TypeScriptIn production

ceu.gg

Minecraft server hosting on self-hosted bare metal, with an orchestration engine built from scratch.

Developer - panel, API, provisioning engine, container images, infrastructure·Dec 2025 - present·Source is private

12.9k+

accounts

who between them created 14.6k+ communities

14.7k+

servers created

70 running at once, daily peak averaged

5

dedicated machines

Rio de Janeiro and Bahia, 4 in use and 1 spare

~160k

lines of TypeScript

Panel and API, both first-party

Platform figures read from the ceu.gg metrics API in August 2026.

~/portfolio/projects/ceu-gg/overview.md

What it is

ceu.gg gives people a Minecraft server for their friends without asking them to understand any of it. Pick a version, press a button, share an address. The free tier stays free, and paid plans exist for anyone who outgrows it.

The panel, the API, the provisioning engine and the container images are all built here. There is no off-the-shelf control panel underneath. Servers run in Kubernetes on self-hosted bare metal, across two Brazilian sites, behind routing I configure. I write the code; a partner works alongside on brainstorming, testing and support. Two things set the shape of everything else: hardware is priced in dollars while the revenue arrives in reais, and there are only so many hours two people have. So simplicity here is a requirement.

  • Everything from the routing table to the panel is first-party
  • Creating a server allocates nothing but a database row
  • Reserved capacity holds because the scheduler refuses to place anything else on it
  • The diagnostic agent reads player-authored text and holds no write tools
  • The chat acts on your account with your permissions, never with any of its own
~/portfolio/projects/ceu-gg/architecture.md

How it is layered

Four layers, each one there to stop a failure inside it from becoming a failure everywhere else.

System diagram of ceu.gg: users reach a cloud gateway through Cloudflare, a site-to-site VPN carries traffic to the self-hosted router, Traefik fronts a Kubernetes cluster holding the servers, core-system and monitoring namespaces, and the bare-metal machines underneath run the VMs, the database and S3 storage.
The whole platform on one page, from the bare metal up to the request path.
  1. 01

    Bare metal

    • Dedicated AMD Ryzen and Intel Xeon servers
    • 3 machines in Rio de Janeiro, 2 in Bahia, one of them still a spare
    • A cold-storage disk passed through to its VM
    • Segmented network
  2. 02

    Proxmox virtualization

    • Control plane and workloads on separate VMs
    • A crash takes down its VM and nothing else
  3. 03

    k3s cluster

    • core-system - the panel and the API
    • servers - one pod per game server: app, watchdog, seeder, backup
    • monitoring - Prometheus and Grafana
    • Per-pod limits, automatic restarts, one deploy shape for everything
  4. 04

    Edge and ingress

    • Cloudflare for DNS and TLS
    • A site-to-site VPN from the cloud gateway to the self-hosted router
    • MikroTik routing into the cluster
    • Traefik with ACME DNS-01 certificates
~/portfolio/projects/ceu-gg/provisioning.md

From a button to a running world

What actually happens between pressing the button and a world being reachable.

  1. 01
    panel

    The wizard asks in player language

    Where do your friends play, plugins or mods. Software, version and template come from the answers, so nobody has to know what Paper, Purpur or Fabric are.

  2. 02
    backend

    Rules are checked, and almost nothing is allocated

    Plan limits, community quota and ABAC permissions resolve first, and all that gets written is a database row, a reserved port and encrypted SFTP credentials. Nothing exists in the cluster yet.

  3. 03
    engine

    The first start goes through a queue

    The entry carries a priority, paid before free, and a CronJob drains the queue every minute against current node capacity.

  4. 04
    cluster

    The seeder fills a fresh volume, then the pod comes up

    The world zip is pulled from MinIO into a new PVC, and the pod starts with its watchdog and backup sidecars.

  5. 05
    watchdog

    Player count is polled over the Minecraft protocol

    A sidecar in every pod asks the game server itself, then reports the number back to the API.

  6. 06
    engine

    An empty server gets stopped

    After ten idle minutes on the free tier the server stops, and its volume sits on the node for another thirty before the archive cron comes for it. The free tier works because most servers spend most of their time stopped.

  7. 07
    storage

    The world is zipped into object storage and the volume is deleted

    Restarting re-seeds from that zip. Nothing is lost, and an idle server holds no cluster capacity.

~/portfolio/projects/ceu-gg/sky.md

The assistant

Sky is the assistant on the platform. It answers in the panel and in Discord, and it can act on the account of whoever is asking - with that person's permissions, never with any of its own.

# the help chat

One engine answers in both places. The widget in the panel and the bot in Discord run the same service, and the Next.js route in front of the widget is a proxy that pipes the reply through as it arrives.

  • Replies stream over Server-Sent Events, and the same channel carries the status line the widget shows while a tool is running.
  • Answers come from a curated set of markdown files about the platform, searched by keyword with heading matches weighted higher. There are no embeddings and no vector database; the corpus is small enough to load whole, so the common question never reaches a search at all.
  • Every Discord channel has a mode - answer everything, answer only when mentioned, or stay out. Ticket channels get a greeting, and Sky keeps quiet when the person talking is staff.
  • The bot runs on exactly one replica at a time, elected through a Postgres advisory lock. Losing the lock disconnects the gateway and another replica takes over.
  • When it cannot solve something, escalate_to_human pings the team in that channel instead of letting the model improvise an answer.

# tool calling

The model can call 18 tools. None of them touch the database: each one is an HTTP call to the platform's own API carrying the JWT of the person in the conversation, so a tool call is authorised the same way a click in the panel is.

  • Permissions come from ABAC, the same matrix the panel reads. Sky never holds a credential of its own.
  • Reading is open to anyone signed in: communities, servers, live status, console output, files, backups, plan and balance.
  • Writing - editing a config file, starting or stopping a server, running a console command, creating a backup - needs an active paid plan, and that check fails closed.
  • Nothing destructive runs on the first call. The tool returns needsConfirmation, Sky has to ask, and only a second call with confirmed: true goes through.
  • A reply gets at most six tool rounds, and the last round is served without tools, so it has to stop investigating and answer.

# diagnostics

When a server will not boot, the console is a wall of English stack traces. Diagnostics is a separate loop that reads the server and writes a report: what is wrong, the evidence for it, and the fixes as buttons.

  • The model gets read-only tools and eight rounds. The prompt pushes it to ask for several files in one round, because rounds are the scarce thing.
  • It finishes by calling submit_diagnosis: a summary in plain Portuguese, findings marked INFO, WARNING or CRITICAL, and proposed actions. If it never calls it, the run returns text and no actions.
  • Every finding has to cite something it actually read - a log line, a config value, a filename. Without that it is a guess, and a guess sends the owner to change the wrong thing.
  • The quota belongs to the server, not to the account clicking. It comes from that server's hosting plan, and a community with several admins would otherwise get a different limit depending on who asked.
  • A limit of zero also means unavailable, which is how the feature gets switched on and off for the free tier from the admin panel without a deploy.
  • A run that passes its cost ceiling stops and returns what it has. A partial report beats an open invoice.
  • Tool names are rewritten into plain language before the report is stored. One real report reached a customer reading get_server retornou: "Não encontrei esse recurso".
~/portfolio/projects/ceu-gg/engineering.md

Problems worth writing down

Four that took real work, plus a few smaller ones I keep coming back to.

cold-storage.md

A server that is not running costs nothing

# the problem

Every idle server pinning a volume to a node is capacity nobody is using. On a free tier, where most servers sit empty most of the time, that is the entire capacity budget.

# what it does

  • Creating a server allocates nothing in the cluster: a row, a reserved port, encrypted SFTP credentials. The first start does all the real work, and a CronJob drains it through WAITING → PROCESSING → SEEDING → STARTING → COMPLETED | FAILED.
  • At rest a world is one object, sv-{id}/.ceugg-world.zip. A cron finds volumes idle past a 30-minute TTL, streams them out and deletes the volume.
  • The write streams straight into a .tmp key, then rclone moveto promotes it. That move is a server-side copy, so it is atomic: a crashed upload leaves a stray .tmp and never touches the live zip.
  • Reads resolve zip-first, which doubles as the migration story: loose objects under the prefix only exist as pre-migration leftovers.

This is what lets 14.7k+ created servers sit on 5 machines. The ones nobody is playing on hold no cluster capacity at all, and come back from a single zip when someone does.

scheduling.md

Capacity you cannot oversell

# the problem

Paid plans promise reserved CPU and RAM. On a cluster that also runs a free tier, "reserved" only means something if the scheduler refuses to hand that capacity to anyone else.

# what it does

  • Nodes carry ceu.gg/tier=high-performance and ceu.gg/storage-gb labels. Disk gating rides on a label because Kubernetes does not expose volume-group size, and the namespaced ServiceAccount cannot list persistentvolumes.
  • High-performance nodes are exclusive. A free or standard server never lands on one, even when it is empty.
  • There is no silent downgrade. A high-performance request that cannot be placed parks in WAITING_RESOURCES and raises an ops alert.
  • Standard nodes hold a hard 20% reserve for paid servers, and free servers never borrow it. Memory is reserved at 100% because it is incompressible; CPU has a tunable request ratio because it is not.

The guarantee holds by refusal, not by observation. A paid server finds its resources free because nothing else was allowed to take them.

sky-tools.md

Letting a model act on a real account

# the problem

In the chat, unlike in diagnostics, Sky can start a server, rewrite a config file and run console commands. The model is the one deciding when to reach for those.

# what it does

  • Tools never touch a repository. Each one is an HTTP call to the platform's own API carrying the user's JWT, so whether an action is allowed gets answered by the same ABAC, specifications and domain events that answer it for a click in the panel.
  • The token is checked for expiry before the call. A dead session returns a session_expired payload that doubles as an instruction: ask the person to sign in again, do not retry.
  • The paid-plan gate fails closed. If the plan lookup itself errors, the write does not happen.
  • Anything destructive returns needsConfirmation instead of running. Sky has to ask, wait for a yes, and call again with confirmed: true.

The model picks what to attempt. It never picks what it is allowed to do. The diagnostic mode answers the same question by deleting the write tools altogether, because there the input is text any player can author.

sky-diagnostics.md

An agent with no write tools

# the problem

Sky reads a server's console, config files and plugin list to explain why it will not boot. Any player on that server can write to all three.

# what it does

  • In diagnostic mode the model gets no write tools at all. So a file named to look like an instruction stays what it is: text in a report.
  • Authorization never comes from the model. The client sends only the id of a proposed action, and the backend re-reads that action from what was persisted on the run and revalidates it from scratch.
  • Actions are a closed allow-list. Each one passes ABAC, plan, quota and cooldown before it runs.

A hostile filename can change what the model says. It cannot change what the system does - that is the whole point of keeping the authority outside the model's output.

$ git log --grep="fix:" --oneline

Heap ceiling against the container limit

--max-old-space-size=768 under a 1Gi limit. Left alone, V8 aims at a ceiling the cgroup will not give it, and the kernel OOM-kills the process before a full GC runs.

libuv threads against cgroup cores

UV_THREADPOOL_SIZE cut from 16 to 8. Node sized its pool from the host's 38 cores, not the 4 it actually had, so the threads oversubscribed by 4× and stalled under archive and SFTP bursts.

A socket leak in the Kubernetes client

Every client construction built a fresh https.Agent, so connections piled up in ESTABLISHED and were never reused. Memoising the agent fixed it.

~/portfolio/projects/ceu-gg/operations.md

Running it

# observability

Prometheus and Grafana on the cluster, OpenTelemetry inside the services: traces, metrics and logs across eight server-side instrumentations. Tracing is currently sampled off in production, because at this size it cost more CPU than the traces were returning.

Grafana dashboard showing cluster CPU and memory utilisation, active pod counts and API gateway throughput.
Without metrics there is no mature technical decision-making, only opinions with a deploy button.
  • Cluster and per-workload CPU
  • Node and pod memory
  • Active pod count and cluster state
  • API gateway latency, error rate and throughput
  • Per-server resource use and availability
  • Node capacity snapshots, feeding the scheduler

# security on a free tier

A free tier is an open invitation. Mass server creation, arbitrary code execution and hostile file uploads all have to be barred.

  • No executable uploads. Plugins, mods and datapacks arrive through the integrated CurseForge and Modrinth catalogue.
  • Everything goes through the panel and the API. There is no shell, and no path from a game server to the host filesystem.
  • Containers run non-root, with CPU and memory limits per pod.
  • Permissions are ABAC throughout: a matrix the frontend reads from the API, never a hardcoded role check.
~/portfolio/projects/ceu-gg/incidents.md

What broke, and what is still open

Two changed how the platform is run.

The cold-storage disk detached

The disk backing cold storage dropped out with no warning.

Redundancy stopped being a later problem, and recovery got a written procedure.

A day without IPv4

IPv4 access was lost for close to a full day.

Access now runs over independent routes, so losing one no longer takes the platform down.

# still open

  • Scheduling places servers by availability. There is no regional strategy yet - fine at this size, and it will not stay fine.
  • Monitoring says network and memory become the first limiters as the platform grows, ahead of CPU.
~/portfolio/projects/ceu-gg/timeline.md

Shipping history

$ git log --tags --oneline

  1. alphaDec 2025

    First servers on own hardware

    A panel, a cluster, and enough of an engine to create a world and connect to it.

  2. v0.1.8 - v0.2.2Feb 2026

    Addressing and control

    Custom subdomains, extra TCP and UDP ports with SRV records for plugins that need their own connections, force stop, and console logs that survive a restart.

  3. v1.0.0Apr 2026

    Public beta

    The startup queue with live progress, the file manager, ABAC permissions with a role hierarchy, Google Drive backups, and audit logs derived from domain events.

  4. v1.5 - v1.6Jul 2026

    Capacity awareness

    Node capacity snapshots, queue admission by available capacity, and the WAITING_RESOURCES state. The memoised https.Agent fix landed here, and Sky reached Discord.

  5. v2.0.0Aug 2026HEAD -> main

    Reserved capacity and a diagnostic agent

    The largest change since beta. Per-server plans with genuinely reserved CPU, RAM and disk; exclusive high-performance node pools bounded by real hardware; a guided creation flow; and Sky's read-only diagnostic mode.

~/portfolio/projects/ceu-gg/stack.md

Full stack

Frontend
Next.js 16React 19TypeScriptTailwind CSSshadcn/uiRadix
Backend
NestJS 11TypeScriptDDDCQRSDomain eventsABACJest
Data
PostgreSQL 16PrismaKafkaMinIOGoogle Cloud StorageSFTP
AI
OpenAITool callingServer-Sent EventsToken and cost accounting
Infrastructure
Kubernetes (k3s)ProxmoxHelmTraefikOpenEBS LVMMikroTikCloudflareDocker
Observability
PrometheusGrafanaSigNozOpenTelemetrypino
Integrations
StripePIX (Efí)Discord.jsResendGoogle DriveCurseForgeModrinth