Orchestrator, dashboard and worker hosts

The orchestrator is the one process that decides how many workers exist and which containers each of them owns. It never simulates anything and it is never on the per-tick path: workers and the gateway talk to each other directly, and the orchestrator only writes leases to the control plane and starts or stops worker instances. Every box in the mesh is the same player build started with a different -nebula-role; the orchestrator is -nebula-role orchestrator.

This page covers NebulaOrchestrator, the dashboard served by OrchestratorHttpServer, and the IWorkerHost abstraction that puts workers on local processes or cloud VMs. For the leases themselves see Control plane; for what happens on a worker when a container moves see Containers and handover.

What the orchestrator does

On startup it:

1. Connects to the control plane and resets it (-nebula-reset, default on), so rows from a previous run never survive.
2. Writes one container_lease row per container in the scene (EnsureContainer for every entry of ContainerRegistry).
3. Seeds mesh settings from -nebula-settings key=value,key=value.
4. Launches the gateway next to itself (-nebula-gateway-id gw1), always through the process host, whatever host the workers use.
5. Starts the dashboard.

Then, every 500 ms, it runs one pass: heartbeat the orchestrator row, reap dead workers, reconcile the desired count, rebalance, finish retirements, relaunch what needs relaunching, and publish a fresh state document for the dashboard. Every dashboard command just schedules the next pass immediately.

Assignment: even and sticky

NebulaOrchestrator.ComputeAssignment is a pure function of (containers, live workers, current leases). It sorts containers by id and workers by index, gives every worker a quota of containers / workers with the remainder going to the lowest indices, keeps every active lease whose owner is alive and still under quota, and deals the rest to the first worker with room. Four workers and four containers means one each; three workers means one of them simulates two; one worker takes everything.

Because existing leases are kept wherever they fit, a rebalance moves as few containers as possible. Each change is applied with AssignContainer, which bumps the container's epoch, and logged as assign <container> -> <worker>. Workers that are retiring are excluded from the assignment set but stay alive to drain.

Worker count is a live setting

DesiredWorkers starts at WorkerCount (-nebula-workers) and is clamped to MaxWorkers (32). Changing it on the dashboard, over the API, or through nebula status-adjacent tooling takes effect on the next pass.

Growing launches a worker on the next free index (NextFreeIndex: the smallest positive index not used by a managed worker or a control-plane row). Indices are reused so worker ids (w1..wN), ports (WorkerBasePort + index, 7101 upwards) and entity-id prefixes stay dense.

Shrinking retires the highest-index worker (PickWorkerToRetire). Retiring means:

1. The worker is dropped from the assignment set; the next pass moves its containers to the survivors.
2. The worker hands its entities over through the normal per-entity handover path. Nothing is special-cased; a drain is just every entity crossing at once.
3. Once it holds no active or assigning lease and reports zero authoritative entities, the orchestrator unregisters and kills it (worker w3 drained; shutting it down).
4. If WorkerDrainTimeoutSeconds (default 10 s) passes first, it is killed anyway and a warning is logged.

Scaling to zero has nobody to hand over to, so leases are released and the retirees drain by timeout.

Dead workers

Liveness is always judged from control-plane heartbeats, whatever the host. A worker whose heartbeat is older than WorkerTimeoutSeconds (default 5 s) is declared dead: the orchestrator calls UnregisterWorker, which orphans its leases, and the same pass reassigns them to the survivors. A replacement is launched on the same index after DeadWorkerReplaceDelaySeconds (default 8 s); containers then flow back to it on a later rebalance.

Two shortcuts exist. If the host reports the instance gone (process exited, VM deleted) the worker is treated as dead at once without waiting for the timeout. And rows the orchestrator never saw alive (leftovers from an earlier run) are never "declared dead", only reset.

The gateway holds nothing authoritative, so a dead worker's entities are dropped and its players are respawned on whichever worker now owns their container.

The dashboard

The orchestrator serves a single-page dashboard from Resources/NebulaDashboard.html on DashboardPort (default 7080, -nebula-dashboard-port; 0 disables it). It binds to localhost unless told otherwise:

The page polls /api/state and shows:

• Mesh totals: live workers, desired, containers, players, bots, server-driven entities, entities (authoritative + ghosts), rebalances.
• Workers: a stepper for the desired count, Add worker and Rebalance now, then one card per worker with its colour, state (starting, ready, draining, relaunching, launching, dead), the containers it owns, players / bots / server-driven counts, authoritative and ghost entity counts, tick time, heartbeat age, and per-card Kill (simulated crash, relaunched) and Remove (graceful drain) buttons.
• Settings: the mesh-wide key/values from game_setting, editable in place (ShooterGame reads npcs here, and damage: all, npcs for invulnerable players while you watch a big NPC fight, or none; the in-game console's damage command sets the same value).
• Containers: container to worker with lease epoch and state.
• Gateways: id, address, heartbeat age.
• Events: the orchestrator's last 60 log lines (it keeps 200), colour-coded by level.

HTTP API

Reads are answered from the listener thread with the last published state. Writes are queued and executed on the orchestrator's main thread; the HTTP response waits up to 3 s for that and returns 503 otherwise. Every successful POST answers {"ok":true,"desired":n}; errors answer {"ok":false,"error":"..."}.

The body parser is deliberately minimal: flat JSON objects with integer or string values. Tools/smoke-test.ps1 -ScaleTo and nebula status use this API; so can you:

Invoke-RestMethod http://localhost:7080/api/state | Select-Object desiredWorkers, totals
Invoke-RestMethod -Method Post http://localhost:7080/api/desired -ContentType application/json -Body '{"desired": 2}'
Invoke-RestMethod -Method Post http://localhost:7080/api/workers/kill -ContentType application/json -Body '{"workerId": "w3"}'

Each worker entry in workers[] carries id, index, color, state, alive, retiring, managed, instance (the host's description, such as pid 1234 or hetzner server 5551 nebula-w1-1 10.0.1.4), address, heartbeatAgeSeconds, relaunchInSeconds, drainRemainingSeconds, tickMs, tickCount, entities, authoritative, ghosts, players, bots, serverDriven, containers[], plus host-specific fields (pid for processes; serverId, serverName, serverStatus, publicIp, privateIp for Hetzner).

Worker hosts

The orchestrator never starts a worker itself. It asks an IWorkerHost to, and the host decides where the worker runs and how to start and stop it. The orchestrator keeps deciding how many and which containers each owns.

public interface IWorkerHost : IDisposable
{
    string Name { get; }                       // "process", "hetzner": shown on the dashboard
    void Initialize(Action<string, string> log); // look up resources, sweep leftovers of a previous run
    bool IsReady { get; }
    string InitializationError { get; }
    IWorkerHandle Launch(WorkerLaunchSpec spec);
    void Kill(IWorkerHandle handle);           // crash semantics: kill the process, delete the machine
    void Tick();                               // main thread, every frame: apply background results
    void WriteHandleJson(IWorkerHandle handle, JsonWriter w);
}

WorkerLaunchSpec is what the orchestrator asks for: WorkerId, Index, Port and CommonArgs (the -nebula-spacetime, -nebula-database, -nebula-gateway and optional -nebula-verbose switches every role inherits). IWorkerHandle is the host's view of one instance: WorkerId, State, Describe, Address (when the host knows it before the worker registers) and Reason. WorkerHandleState is Launching, Running, Exited or Failed.

Hosts must be safe to call every frame. Slow work happens off the main thread and surfaces from Tick(). A host's handle state only lets the orchestrator react sooner when an instance is known to be gone; heartbeats remain the source of truth for liveness.

ProcessWorkerHost

ProcessWorkerHost (-nebula-host process, the default) runs every worker as a child process of the same executable, with -batchmode -nographics -nebula-role worker -nebula-worker-id wN -nebula-worker-index N -nebula-port 710N -nebula-advertise 127.0.0.1 plus the common args, logging to Logs/<id>.log next to the executable. This is the local mesh that nebula start runs. The orchestrator also uses it to start the gateway regardless of the worker host. If the executable is missing it fails the launch with a message pointing at NebulaConfig.WorkerExecutable and -nebula-worker-exe.

HetznerWorkerHost

HetznerWorkerHost (-nebula-host hetzner) creates one Hetzner Cloud VM per worker through the Hetzner API and deletes it on kill or retire. On Initialize it looks up the private network, ssh key and firewall by name and deletes every server labelled nebula-mesh=<mesh>,nebula-role=worker left by a previous run, so a crashed orchestrator never leaves machines billing. It polls the server list every 15 s and marks a handle Exited when its VM is off, deleting or gone. How the VMs boot is described in Deploying to Hetzner.

Its settings are read from provider-neutral switches (CloudHostSettings.FromCommandLine), so a second provider can reuse them:

Adding a provider

Implement IWorkerHost and IWorkerHandle for your provider, then add one case to NebulaOrchestrator.CreateHost:

private IWorkerHost CreateHost(NebulaConfig config)
{
    switch ((config.WorkerHost ?? "process").Trim().ToLowerInvariant())
    {
        case "hetzner":
            return new HetznerWorkerHost(CloudHostSettings.FromCommandLine(OrchestratorId, config.WorkerAdvertiseAddress, config.DashboardPort));
        case "": case "process": case "local":
            return _local;
        default:
            Log("error", $"unknown worker host '{config.WorkerHost}'; using local processes");
            return _local;
    }
}

Keep HetznerWorkerHost open as the reference: do the HTTP work on the thread pool, queue results, and apply them in Tick(); sweep stale machines in Initialize; put your provider's fields in WriteHandleJson so they show up on the dashboard.

Bots vs NPCs

Two different things drive a pawn without a human, and they cost very different amounts:

128 bots on one machine starve the workers; 128 NPCs across four workers cost about 15% of a core per worker. Workers learn whether a client is a bot from the gateway (the client's Hello carries the flag), and both the bot flag and the server-driven flag travel with the entity through ghosting and handover, so the per-worker player / bot / server-driven split on the dashboard stays right after a container moves.

How many NPCs exist is game policy, not Nebula's. ShooterGame's NpcDirector reconciles against the npcs mesh setting on every worker; see Control plane for the setting mechanism and Spawning for SpawnServerDriven.