Manage workers and containers

Use the orchestrator to keep the requested number of workers running and distribute containers between them. The orchestrator does not simulate gameplay. Workers communicate directly with the gateway and with each other during normal simulation.

Run nebula start to launch the standalone .NET orchestrator and gateway alongside Unity workers. nebula deploy uses the same service executables on Linux. Neither service loads the game scene or runs Unity.

Build and start during development

Keep using the same commands from your game project:

nebula build
nebula start --open-ui
nebula stop

nebula build builds the Unity player, exports configuration, container geometry, and persistence field schemas into nebula-services.json, and publishes nebula-orchestrator and nebula-gateway beside the player. Windows executables have an .exe suffix. Building the services requires the .NET 10 SDK; the published executables include the runtime.

After changing only service code, run nebula build services to reuse the last export and skip Unity. Rebuild the full project after changing scenes, containers, settings, or persistence fields. nebula start --build performs the full build before starting. nebula build --linux includes the services and export in the deployment archive.

The CLI supplies the worker executable path to the orchestrator. When launching it directly, pass -nebula-worker-exe <path> with the path to your Unity player. The services read nebula-services.json beside their executables by default; use -nebula-service-manifest <path> to select a different export. Use the same export and protocol version across the mesh.

Understand container assignment

The orchestrator stores each container assignment as a lease in the control plane. A lease names the worker that currently controls the container. Its epoch is a number that increases whenever the assignment changes, which lets workers reject an older assignment.

Every 500 milliseconds, the orchestrator:

1. Reports its own heartbeat.
2. Checks worker heartbeats and removes dead workers.
3. Starts or retires workers until the requested count is reached.
4. Distributes containers as evenly as possible.
5. Publishes a current state snapshot for the dashboard.

Nebula keeps a container on its current live worker when that worker remains within its even share. This reduces container movement during a resize.

For example, four workers and four containers results in one container per worker. With three workers, one worker receives two containers. With one worker, that worker receives every container.

Change the worker count

Set the initial count in NebulaConfig.WorkerCount, with -nebula-workers, or in nebula.json. After the mesh starts, use the dashboard or HTTP API to change it. The count stays between NebulaConfig.MinWorkers (default 1) and NebulaConfig.MaxWorkers (default 32); nebula start --workers N fixes both ends at N, and nebula start --min 1 --max 4 opens the band. nebula deploy takes the same --workers, --min and --max, defaulting to deploy.workers, deploy.minWorkers and deploy.maxWorkers in nebula.json.

Autoscale

NebulaConfig.AutoScale is on by default. Every worker reports the wall-clock time of its simulated tick on each heartbeat; the orchestrator keeps the last ScaleWindowSeconds (20 s) of that per worker and uses the 90th percentile as its utilization: the fraction of the 16.7 ms tick budget the worker is spending. The dashboard draws it as a bar on each worker card.

• Grow when the busiest worker stays above ScaleOutUtilization (0.7) for ScaleHoldSeconds (30 s) and a dry run of the assignment policy with one more worker lowers the predicted peak by at least ScaleMinGain (0.1).
• Shrink when the mesh's mean stays below ScaleInUtilization (0.3) for the same hold and a dry run with one worker fewer keeps everyone under the scale-out line. The worker carrying least is the one that retires. The scaler judges only workers that have joined, so it does not shrink while a launch is still landing (a worker the mesh is driving towards that has not joined, for up to three minutes) or while the workers that have joined are at MinWorkers; it never retires the only live worker to make room for a machine that is still booting.

Only one change happens per hold, and nothing is decided while a launch or a drain is still in flight.

When the mesh is overloaded but an extra worker would not help - one container holds the load and the policy cannot split it - the orchestrator does not grow. It publishes the container in scale.blockedBy and names what that container is actually expensive in, from its cost row:

blocked: cell_3_0_1 carries 0.81 of a tick on its own and cannot be split
  (mostly simulation: 13.40 ms/tick, 0.80 of the tick budget); add a hint or split the cell

scale.blockedComponent is simulation, replication or gateway, and each points at a different fix: split the cell or make its entities cheaper, tighten interest management, or run more gateways.

The reason also says why the planner may not split it, when there is a constraint in the way:

blocked: c0 carries 0.95 of a tick on its own and cannot be split
  (mostly simulation: 15.80 ms/tick, 0.95 of the tick budget)
  - cohesion 7 spans c0, c1, so cutting between them would split it; the item needs 0.95 of a tick
  and is kept whole; add a hint or split the cell

scale.blockedCause is cohesion-group, affinity-group, held, dedicated or no-boundary, and scale.blockedReason is the sentence. no-boundary is the one worth reading twice: it means the hot area is a single container, and Nebula never subdivides one. Bake the area as more cells, register runtime containers inside it, or accept the load.

MinWorkers may be 0; see Scale to zero.

The dashboard's Workers card shows the decision line ("holding: peak 0.74 for 12/30 s", "blocked: ...", "steady: ...") and lets you edit the min/max band, which POST /api/scale/limits with {"min": n, "max": n} also does.

When you add a worker, the orchestrator starts it on the lowest available worker index and rebalances containers.

When you remove a worker, the orchestrator retires the highest-index worker by default. It first assigns the worker's containers to other workers. The retiring worker then transfers its authoritative entities through the normal handover process.

If no other worker can take over the containers and the mesh still wants at least one worker, the orchestrator cancels the retirement instead: the worker stays in service with its containers and the log says so. The orchestrator stops the retiring worker after it has no containers or authoritative entities. It stops the worker after WorkerDrainTimeoutSeconds even if the drain has not completed. If you scale to zero, no worker can receive the entities, so every retiring worker stops after the timeout. Once the orchestrator has stopped a worker, it refuses that worker if its process registers again before the machine is gone, and deals it nothing.

Scale to zero

MinWorkers may be 0 (nebula start --min 0 --max 2, -nebula-min-workers 0, or the min/max band on the dashboard). An idle mesh then retires its last worker: every lease is released and nothing is running but the orchestrator and the gateway. Nothing is lost - entities that were persisted come back with the next worker - and on a cloud host nothing is billed.

The first player after such an idle period pays for it, so Nebula makes the wait explicit instead of failing the join:

1. The client connects to the gateway and is welcomed as usual. There is nowhere to spawn, so the gateway holds the join in JoinState.Starting and tells the client roughly how long a worker takes to boot on this host (IWorkerHost.TypicalBootSeconds: about 3 s for the process host, about 75 s for hetzner and cloud).
2. The gateway reports the waiting client on its control-plane heartbeat. The orchestrator raises the desired worker count to max(1, MinWorkers) at once - no scale-out hold, because somebody is already waiting - and takes a parked worker back from the idle pool if there is one, which turns the boot into a restart.
3. As soon as the worker registers and its lease is active, the gateway places the player. The join finishes on the same connection; the client never reconnects.

While any client is connected or waiting to join, the scaler's floor is one worker whatever MinWorkers says, so a scale-in can never race the wake-up.

The dashboard shows a persistent warning pill while MinWorkers is 0 ("Scale to zero is on: the first player after an idle period waits for a worker to boot (about N s on this host)"), the Gateways card counts clients that are joining, and the decision line reads "world starting: 1 client(s) waiting to join, no worker running" while the mesh boots. nebula status prints the same two lines. In /api/state, hostBootSeconds is the host's estimate and scale.pendingJoins / scale.scaleToZero are the live state.

On the client, NebulaClient.Join, NebulaClient.JoinEstimatedSeconds and the JoinStateChanged event drive a "world starting" screen; see Connecting clients.

Idle pool

Retiring a worker does not always mean throwing its machine away. A host says whether keeping one is worth it, and the orchestrator asks it to park a worker it has drained instead of stopping it:

• The process host does not park. A local process costs nothing to start again, so a retired worker is stopped at once.
• The hetzner host parks. Hetzner bills per started hour, so a VM retired after five minutes is paid for until the hour ends. The machine stays up, the worker keeps heartbeating, and the orchestrator simply deals it no containers. It is deleted a couple of minutes before that paid hour runs out unless something takes it back first.
• The cloud host parks. It asks the hosting platform's resource API to park the machine; the platform keeps it until the moment it stops being free and deletes it itself. Taking it back (unpark) keeps the worker ID; when the platform has already released the machine, the orchestrator launches a fresh worker instead.

A parked worker keeps its worker id and its worker index, so nothing new is ever started on that index while it sits in the pool, and entity ids stay unambiguous. It is never eligible for a lease and is not counted in the desired worker count.

A scale-out takes the most recently parked worker back before it launches anything: that worker is already booted and its build is already downloaded, so it starts holding containers in the next assignment pass instead of a minute later. NebulaConfig.IdlePoolSeconds (or -nebula-idle-pool, or mesh.idlePoolSeconds in nebula.json) shortens how long a parked worker is kept; the default 0 means "as long as it is already paid for", which is the host's own answer.

The dashboard shows an Idle pool card listing each parked worker with how long it has been parked and how much paid time is left, plus two actions: Resume takes it back into service (POST /api/workers/unpark), and Delete drops it now instead of waiting (POST /api/workers/delete-parked). When the host does not park, the card explains that and shows what a cold start costs on it.

Container cost

The utilization band tells you which worker is busy. The Container cost table on the dashboard tells you which container is making it busy, and in which of three ways:

The simulation figure is measured, not derived from the head count: a cell full of static props and a cell full of NPCs chasing paths hold the same number of entities and do not cost the same. Simulation is weighed against the 16.7 ms tick budget, the two byte figures against CostLinkBudgetMbps (100 Mbit/s by default) - a yardstick only, so that milliseconds and bytes can be compared at all; it limits nothing.

Interest management bounds Out and Relay. It does nothing for Sim: five hundred NPCs cost the same to simulate whether anybody is watching them or not. That is why the three are kept apart.

The rest of a tick - physics, the ghost pass, the interest pass, publishing - belongs to no container and is deliberately left out, so a worker's rows add up to less than its utilization. Share is each container's share of the time that could be attributed on that worker.

The same rows are at GET /api/cost:

{"tickPeriodMs":16.67,"linkBytesPerSec":12500000,"containers":[
  {"id":"cell_3_0_1","scope":"","worker":"w1","cost":84.5,"tickMs":13.4,"tickShare":0.91,
   "bytesOut":940000,"gatewayBytes":210000,"ghosts":12,"dominant":"simulation","saturation":0.8}]}

and under cost in GET /api/state. scope is the entity's simulation scope - empty in the public world, the instance key inside an instance - so the rows can be grouped per scope as well as per container.

Capacity and admission

The same rows say how close a container is to what one worker can carry. A container is at capacity when its dominant component reaches CapacitySaturation of that component's budget:

The orchestrator derives the reading once per pass and publishes it on the container's lease row, so every gateway answers from the control-plane document it already mirrors. A scope is reported as full as its worst part, because a scope is one interaction domain and cannot be split past its authored boundaries.

The Container cost table gains a Full column and the Scopes card a Capacity column; GET /api/cost and GET /api/state carry atCapacity per row, capacitySaturation for the mesh, and saturation / dominant / atCapacity per scope.

What happens to an arrival into a full target is the game's decision, not the dashboard's: see When a target is full. By default it is refused with a typed reason; nothing is ever silently split or copied to make room.

Entity cost weights

CostWeights says what a player, a bot, a server-driven entity and anything else cost. That is per category, and in a world with ambient critters and raid bosses in it both are "server-driven". NetworkIdentity.CostWeight (on the prefab, default 1) is a multiplier on the category weight: a boss at 8 costs eight server-driven entities, a critter at 0.25 costs a quarter of one. At 1 everywhere nothing changes.

For something decided at runtime rather than authored, install a callback once at start-up, or set the weight on the entity:

// Once, on a worker. Return a negative number to decline and let the prefab's value stand.
NebulaCost.EntityWeight = id => id.TryGetComponent<Boss>(out _) ? 8f : -1f;

// Or per entity, at any time. This wins over both, and travels with the entity.
identity.SetCostWeight(12f);

Precedence is SetCostWeight (or a weight carried in from another worker) > the callback > the prefab's CostWeight. It is evaluated at spawn, never per tick, and it travels in the entity's handover state, so a boss reports the same cost on whichever worker is simulating it. The weight feeds the planner and the scaler, not just the dashboard: weighting a boss changes where it is placed and when the mesh grows.

Balancing hints

The orchestrator measures how busy each worker is, but it cannot know what a container means. A hint tells it what the game knows. Every field is optional and the default says nothing:

Worked examples:

• A city hub. One cell everybody passes through. Mark it Dedicated and it gets a machine to itself; the rest of the map is dealt across what is left.
• A battle royale. Raise SeamCost on the cells inside the circle at the start of each phase, and the worker boundary moves out of the fight instead of running through it.
• An instanced dungeon. Give the rooms of one instance a shared AffinityGroup so a party never crosses a machine boundary walking through a door.
• A chunked world with nothing special in it. Leave everything default; the measured signal is enough.

Set a hint at bake time (the World window, stored in the manifest), at runtime from game code, or from the dashboard. The runtime value wins over the baked one, and it lives on the container's control-plane row, so an orchestrator restart still sees it.

// From game code on a worker, in the same call that asks for the container.
worker.RequestRuntimeContainer(id, box, new ContainerHint { Dedicated = true });

// Or at any time, for any container.
orchestrator.SetContainerHint("cell_3_0_1", new ContainerHint { CostMultiplier = 2f, SeamCost = 0.5f });

The compact form used by the dashboard and the HTTP API is x2,group=keep,seam=0.5,dedicated; an empty string clears the runtime hint and lets the baked one stand again.

If more containers ask to be Dedicated than the mesh has workers to spare (it needs one more worker than there are dedicated containers), none of them is reserved and they share like anything else. The orchestrator logs it and the dashboard shows the reason above the container list.

Cohesion groups and holds

Two more hints come from the workers rather than from the container rows, and the Cohesion card of the dashboard shows both. A cohesion group is a set of entities that must be simulated by one worker: the planner deals the containers its members occupy as one item, exactly like an AffinityGroup, and a handover of any member takes the others with it. A hold is a worker asking for one container not to be rebalanced for a bounded number of seconds, for a fight or a cutscene that should not be interrupted.

Neither can be honoured unconditionally, and neither fails quietly. A group whose containers need more than one worker's tick budget is dealt whole anyway and reported above the table; a hold never leaves a container unowned, never lasts more than two minutes, and dies with the worker that asked for it.

Assignment plan

The Assignment plan card explains the last re-deal: one row per container that changed hands, with the worker it came from, the worker it went to, and a sentence saying which authored boundary the cut fell on, what the busiest worker was carrying before and after, and which cohesion or affinity group kept containers together.

c4  w1 -> w2   re-deal along the boundary between c3 and c4: c4 joins the run w2 takes
               (2 item(s), 10 of a 15 target); busiest worker 0.81 -> 0.44 of a tick

Above the table, in yellow, is what the planner could not relieve: a container that is carrying its worker on its own and may not be moved away from it, with the reason. Nebula never splits a container and never breaks a group or a hold to relieve a worker, so this is the whole of the answer — it is your cue to author a boundary, not a sign that something failed. The same rows are in assignment.moves and assignment.saturated of /api/state, and the orchestrator's log lines carry the reason in brackets after assign c -> w.

The card hides itself when nothing has moved and nothing is saturated.

Handle a failed worker

Each worker writes a heartbeat to the control plane. When a heartbeat is older than WorkerTimeoutSeconds, the orchestrator treats the worker as dead and assigns its containers to surviving workers.

After DeadWorkerReplaceDelaySeconds, the orchestrator starts a replacement on the same worker index. It can react sooner when the worker process exits or its cloud VM disappears.

Run more than one gateway

One mesh can be served by any number of gateways. The orchestrator starts one (gw1) as a child process next to itself; nebula start and nebula deploy to Hetzner run that one. Whoever runs the gateway fleet, a hosting platform or you, starts the others and puts a load balancer in front of them. The orchestrator does not start, replace, or size gateways beyond its own, and it does not restart gw1 if it exits.

Identity and sessions

Every gateway has a persistent gateway ID (-nebula-gateway-id, default gw1), unique within the mesh, and draws a random incarnation when its process starts. The incarnation is the high half of every session ID the gateway issues, so two gateways, or two starts of the same gateway, never hand out the same client ID. Workers key player state by that session ID.

Each welcome carries a signed session token. A client that reconnects, to any gateway of the mesh, presents it and gets the same session and pawn back. This is what lets a load balancer move clients between gateways without the game noticing more than a pause in input; see Sessions and reconnection.

For this to work, every gateway of a mesh needs:

• The same MeshToken. It authenticates gateways and workers to each other and to the control plane.
• The same AuthSigningKey (NEBULA_AUTH_KEY), which the session tokens and anonymous identities are signed with. With the key derived from the mesh token instead, every deployment changes it; the gateway warns about that.
• A distinct -nebula-gateway-id.
• -nebula-gateway <address:port> set to the address clients use, which is the load balancer's address when there is one. The gateway registers that address in the control plane and the dashboard shows it.

Load reports

Every worker reports its memory on each heartbeat: resident memory (working set), Unity native allocated and reserved, the managed heap and the GC count. GET /api/state publishes them per worker (residentBytes, nativeAllocatedBytes, nativeReservedBytes, managedBytes, gcCount), nebula status shows rss, native and managed columns (local and --cloud), the dashboard shows them on each worker card, and the worker's periodic profile log line ends its counters with mem rss=… native=… managed=…. A worker older than this change reports 0.

Every gateway reports itself on each control-plane heartbeat: welcomed clients with a pawn, clients being authenticated or placed, sessions waiting for a reconnect, packets and bytes per second on the client side, bytes per second on the worker side, CPU as a fraction of one core, memory, the longest loop lag, worker links, and whether it is ready or draining. GET /api/state publishes them per gateway (activeClients, joiningClients, reconnectingClients, packetsIn, packetsOut, bytesIn, bytesOut, workerBytesIn, workerBytesOut, cpu, memoryBytes, loopLagMs, workerConnections, ready, draining), together with incarnation and drainRequested. Size the fleet from these numbers; a connection count alone says nothing about replication cost. The field contract is in the control plane specification.

A gateway whose heartbeat is older than three times WorkerTimeoutSeconds is removed from the list, so /api/state names the gateways that exist rather than every one that ever registered.

Drain a gateway

To take a gateway out of service without dropping its players, ask it to drain:

Invoke-RestMethod -Method Post http://localhost:7080/api/gateways/drain -ContentType application/json -Body '{"id": "gw2", "draining": true}'

The orchestrator sets drainRequested on the gateway's control-plane row. The gateway sees the flag, stops accepting new clients (a new join is refused with gateway is draining, and the client retries after a second), and sends GatewayDraining to every connected client with NebulaConfig.GatewayDrainReconnectSeconds (default 10). Each client reconnects at once with its session token, through the load balancer to another gateway, and continues with its pawn. The gateway reports draining on its heartbeat and answers 503 on its health endpoint. Stop the process once activeClients reaches zero or your own drain timeout passes. Send "draining": false to cancel; the gateway then accepts clients again.

A drain request is bound to the gateway's incarnation: a gateway that restarts registers with a new one, which clears a stale request.

Health checks

The standalone gateway's HTTP server answers GET /healthz with 200 while the gateway is ready (registered, connected to the control plane, not draining) and 503 otherwise. Point the load balancer's health check at it on the web port (WebPort, the gateway port by default). With WebClients off the gateway still serves this one endpoint on that port.

The Gateways table

The dashboard's Gateways table lists every registered gateway with its address, heartbeat age, clients, clients joining, outbound rate, CPU, loop lag, and state (starting, ready, drain requested, draining). Drain asks a gateway to drain and Undrain cancels; the button is disabled while the mesh has only one gateway, because its clients would have nowhere to go.

Open the Nebula Dashboard

To open the local dashboard, visit http://localhost:7080/. Change NebulaConfig.DashboardPort or pass -nebula-dashboard-port to use another port. Set the port to 0 to disable the dashboard.

The dashboard binds to localhost by default. Pass -nebula-dashboard-bind + to listen on every interface, or pass one address to listen only on that interface.

Use the overview to:

• Check live and requested worker counts.
• See which worker controls each container. The Containers table lists them as a tree, each under its parent. A container marked inherited has no lease of its own: the worker that owns its parent simulates it, and the Owner column names the ancestor it inherits from. leased → inherited marks a container authored as leased that sits in a moving carrier without a physics frame, so its carrier's worker simulates it. frame marks a container with its own physics frame.
• Compare authoritative and ghost entity counts.
• Review player, bot, and server-driven entity counts.
• Check tick time and heartbeat age for each worker.
• Add, retire, or stop a worker.
• See every gateway's load and drain one (see Run more than one gateway).
• Edit game-defined mesh settings.
• Pin a dynamic container to a worker or return it to its carrier's worker. A dynamic container authored as leased always has a worker of its own, so it has no pin controls.
• Watch the simulation scopes a mesh has been asked for: their lifecycle state, how many players and entities are in each, how long nothing has wanted one, and how old it is. The card appears once something activates a scope, and is read-only — retiring is decided by ScopeIdleRetireSeconds and ScopeLifecycle.ShouldRetire, not by hand.
• Read recent orchestrator events.

Stopping a worker with Kill simulates a crash. The orchestrator reassigns its containers and starts a replacement. Removing a worker performs a graceful drain and reduces the requested worker count.

Inspect the world map

Open /map from the dashboard to see the mesh in 3D. Every container is drawn as the box it really is, so its height reads as clearly as its footprint. The map shows:

• Static and world-partition cell containers.
• The worker that controls each container.
• Dynamic containers and their carriers, and containers fixed inside a carrier, such as a ship's engine room, drawn where the carrier is now.
• Authoritative entities and ghost copies, at their real height. An entity inside a physics frame is drawn where it is in the world, not at its coordinates in the frame.
• Loaded world cells and container boundaries.

Drag to orbit the camera, right-drag or Shift-drag to pan, and scroll to zoom toward the cursor. Select an item to inspect it. Press F to zoom to the selected item, or the one under the cursor. A player, bot, or carried container stays centred as it moves until you pan or press Esc. With nothing selected, F fits the containers. Press T to look straight down and R to return to the angled view. Use the side controls to switch between worker ownership and entity load or to hide individual layers.

As you zoom in, Focus fade dims whatever lies well away from the point under the cursor, along with boxes much larger than the view, such as the cell around the camera. Players stay on top of other entities, and labels that would overlap are dropped, with player names kept first. Turn off Focus fade to see everything at full strength.

Crowds are thinned to keep the map fast and readable. Each small patch of the screen shows at most a few markers, with at most 2,500 in view. The selected or followed entity is always shown. Players come next, then bots, then everything else, and the same entities stay chosen from frame to frame. The rest appear as faint specks. Zoom in to spread a crowd out and reveal more. A note above the coordinates says how many entities in view are shown.

Workers send summary telemetry once per second. While someone has the map open, workers send detailed entity positions every half second, up to 4,096 authoritative entities per worker. Pass -nebula-telemetry off to a manually started worker to disable this telemetry.

Edit persisted records

Open /persistence from the dashboard to see everything the persistence store holds. The tab is empty when persistence is off. It groups records as a tree: containers first, then the entities saved in each container, with entities carried inside another entity nested under their carrier. Badges mark a record that a client owned, a server-driven entity, a scene object, and a record this build cannot decode.

Select a record to inspect it. The editor shows its identity as read-only text (key, prefab or scene ID, epoch, version, and when and by which worker it was saved), then editable fields for its name, container ID, position, rotation, and velocity, then one row per saved state entry.

The orchestrator uses the persistence field schemas exported during the Unity build to decode saved state. It matches each record's prefab name or scene ID and each NetworkVariable field name to its exported type. Booleans, numbers, strings, enumerations, vectors, quaternions, and colors get a matching control. Custom types, unknown fields, and state written by WritePersistentState appear as hexadecimal bytes that you can replace. Saving rewrites only the entries you changed and keeps the other bytes as they were. Rebuild after changing persistence fields so the dashboard has the current schema.

Use Duplicate as… with a new key to copy a record, for example to place a second object. Use Delete to remove one record, or Clear persistence on the overview to remove every record.

Use the HTTP API

The dashboard and API use the same port. Read /api/state to get the latest orchestrator snapshot.

Invoke-RestMethod http://localhost:7080/api/state | Select-Object desiredWorkers, totals

Send JSON with Content-Type: application/json for operations that change the mesh:

Invoke-RestMethod -Method Post http://localhost:7080/api/desired -ContentType application/json -Body '{"desired": 2}'
Invoke-RestMethod -Method Post http://localhost:7080/api/workers/kill -ContentType application/json -Body '{"workerId": "w3"}'

Each row of containers in GET /api/state names its parent and depth in the container tree, its authority as authored (auto, leased, or inherited), and its mode: leased, inherited, or demoted for a leased container simulated as inherited. worker is the worker that simulates it, and ownerFrom names the ancestor whose lease decides that when the container has none of its own. frame and interest describe its physics frame. Rows under carried carry authority, frame, and interest too.

Successful changes return {"ok":true}. Invalid requests return an error message. An API write can return 503 when it cannot reach the orchestrator's main thread within three seconds.

Choose where workers run

Set NebulaConfig.WorkerHost or pass -nebula-host to select a worker host:

The orchestrator starts one gateway (gw1) as a child process next to itself whatever the host, unless OrchestratorSpawnsGateway is off (-nebula-spawn-gateway false). See Run more than one gateway.

For a local process host, set NebulaConfig.WorkerExecutable or pass -nebula-worker-exe when the orchestrator cannot find the game executable. Worker logs are written to the build's Logs directory.

For Hetzner, let nebula deploy configure the private network, VM image, firewall, SSH key, build URL, and API token. See Deploy to Hetzner Cloud.

The cloud worker host

With -nebula-host cloud, the orchestrator does not call a cloud provider itself. It asks a resource API that is scoped to one deployment for worker machines, and the service behind that API holds the provider credentials and turns each request into a machine. The orchestrator's own machine never sees an account-wide credential, and Nebula's assignment, drain, replacement, and scale-to-zero logic is unchanged; the host only changes where a worker comes from.

The host calls GET /d/self at startup to check the token and read the deployment's region, worker size, and worker ceiling, then GET /d/workers. Workers listed from a previous orchestrator run are deleted, not adopted. It requests a machine with POST /d/workers ({"workerId", "index", "port", "commonArgs"}, where port is the orchestrator's suggestion, WorkerBasePort + index), polls GET /d/workers every five seconds for state and private address, and uses DELETE /d/workers/<id>, POST /d/workers/<id>/park ({"seconds": n}), and POST /d/workers/<id>/unpark. A worker resource is {workerId, resourceId, index, state, privateAddress, size, reason, parkedUntil} with states launching, running, parked, exiting, exited, and failed, wrapped as {"worker": ...} on single answers and {"workers": [...]} on the list; errors come as {"error": {"code", "message"}}. A create that answers 402 means the deployment reached its spend limit and 409 that it is at its worker ceiling; the worker is reported as failed with that reason. The platform decides how each worker is started. One that runs a single worker per machine can set the PORT environment variable and leave out -nebula-port; the worker registers the port it listens on, and the mesh connects to that. The hosting platform documents the API itself.

Test simulation load

Start a bot to test the complete client path:

nebula start --bots 2

A bot is a headless client process. Your game must provide its input behavior. Each bot runs a complete Unity player, so start only as many as you need.

Use NebulaWorker.SpawnServerDriven to test worker simulation without another client process. Your game must provide the entity's behavior and spawning policy.