Control plane contract

Use this specification to integrate with Nebula's control plane or read it from your own tools. The control plane stores process registrations, container leases, and low-frequency game settings for one mesh. It does not carry per-tick entity state or saved entities.

The words MUST, MUST NOT, SHOULD, SHOULD NOT, and MAY describe interoperability requirements. The source of truth is Packages/com.1by3.nebula/Runtime/ControlPlane/: LocalControlPlane.cs (the state machine), ControlPlaneJson.cs (the document and the writes), ControlPlaneHost.cs (the endpoints), and RemoteControlPlane.cs (the reference client).

Compatibility and scope

The orchestrator hosts the control plane on its dashboard HTTP server. The document format has no negotiated version. Run the gateway, orchestrator, and workers from the same Nebula release.

Every write is applied on the orchestrator's main thread, one at a time, in the order received; a request's writes are applied as a group before the next request's. Reads are answered from the last published document without waiting on the main thread.

This contract excludes saved entities. The persistence store the orchestrator serves at /api/store/ is described in Persistence.

Endpoints

Both answer application/json. A reader SHOULD keep one GET open at a time and pass the version of the last document it received, so it is told about a change as soon as it happens and costs the orchestrator one response per ten seconds otherwise.

Scalar types and conventions

Strings are JSON strings, counters and ports are JSON numbers, and times are Unix milliseconds. Vectors are arrays [x, y, z]. Numbers are written at full precision; a reader MUST accept a 64-bit counter and MAY lose precision above 2^53.

The control plane treats process IDs, container IDs, addresses, statuses, states, and setting values as opaque strings. It does not enforce nonempty IDs, address syntax, worker-index uniqueness, worker existence, lease-state membership, or referential integrity.

Nebula's runtime applies these conventions:

• An empty workerId means that a container is unassigned.
• Static container IDs come from the authored or baked manifest.
• A carried dynamic container ID contains # and normally ends with its carrier entity's network ID.
• A runtime container ID is rt_ followed by an invariant-culture u64 decimal value.
• Only leases with hasBounds = true define a runtime-container box. A root box (no parent) is in absolute coordinates; a child box is in its parent container's local space.

The document

document is an identity assigned when the control plane starts empty or is reset. Restoring the document from storage preserves its identity. When the identity changes, workers request missing assignments for containers they still own; removals within the same document remain in effect. A client MUST treat a missing document field as an empty identity. See Restart and restore a mesh.

{
  "version": 42,
  "document": "3f1c0e7a9b6d4c2e8a5f7b1d9c3e6a40",
  "now": 1789430400000,
  "workers": [ { "workerId": "w1", "workerIndex": 1, "address": "10.0.1.3", "port": 7101, "status": "ready", "lastHeartbeat": 1789430399500, "tickCount": 120000, "tickMs": 1.25, "entityCount": 40, "authoritativeCount": 30, "ghostCount": 10, "playerCount": 2, "botCount": 0, "serverDrivenCount": 28 } ],
  "leases": [ { "containerId": "cell-1", "workerId": "w1", "epoch": 3, "state": "active", "updatedAt": 1789430300000, "hasBounds": false },
              { "containerId": "rt_7", "workerId": "w1", "epoch": 1, "state": "active", "updatedAt": 1789430350000, "hasBounds": true, "center": [100, 0, 20], "size": [8, 8, 8] } ],
  "gateways": [ { "gatewayId": "gw1", "incarnation": 2864434397, "address": "203.0.113.9", "port": 7000, "lastHeartbeat": 1789430399800, "drainRequested": false,
                  "pendingJoins": 0, "activeClients": 2, "joiningClients": 0, "reconnectingClients": 0,
                  "packetsIn": 120.5, "packetsOut": 480.2, "bytesIn": 9800, "bytesOut": 61000, "workerBytesIn": 70000, "workerBytesOut": 8000,
                  "cpu": 0.12, "memoryBytes": 91000000, "loopLagMs": 1.5, "workerConnections": 1, "ready": true, "draining": false } ],
  "settings": { "npcs": "12" }
}

workers

leases

The lease epoch is not an entity authority epoch. The wire protocol represents this lease epoch as a u64 in ContainerOwnership, while per-entity messages carry a separate u32 epoch.

gateways

The stats fields are flat on the row. A missing stats field reads as 0 or false.

settings

Settings are a flat object of string keys and string values; the control plane attaches no meaning to either.

Writes

A request body is {"ops":[op, op, ...]}. Each op is an object with an "op" name and the arguments below. Times are set by the host from its own clock; a writer never supplies them.

A missing argument reads as "" or 0. A number MAY be sent as a JSON string. Vectors MUST be arrays of three numbers.

Lease states and transitions

Nebula defines these state strings:

The runtime treats active, draining, and pinned as owning states. The control plane does not enforce the state/worker combinations in this table.

The implemented transition rules are:

AssignContainer and PinContainer are the only epoch-incrementing operations. Consumers MUST therefore consider the state and worker ID as well as the epoch. Release and unregister operations invalidate ownership without burning an epoch.

Worker statuses and liveness

Nebula defines starting, ready, draining, and dead. Registration writes starting; a running worker normally heartbeats ready. draining and dead are reserved in the current runtime. Any status string is accepted.

Liveness is a consumer policy, not a control-plane constraint:

alive = status != "dead" && control_plane_now - lastHeartbeat <= worker_timeout

worker_timeout defaults to 5 seconds. The orchestrator unregisters a worker after this test fails; that write orphans its leases. It unregisters a gateway whose heartbeat is older than three times worker_timeout, so the fleet list names the gateways that exist rather than every one that ever registered. The control plane does not expire entries on its own.

Subscription and failure behavior

RemoteControlPlane keeps one read open with wait=10, parses each answer off the main thread, and raises one Changed callback from the next main-thread Tick() when the version changed. An answer with an unchanged version only refreshes its clock.

IControlPlane.Now on a mirror is the now of the newest document plus local monotonic elapsed time. It is an estimate used for heartbeat ages, not an authoritative simulation clock. On the host it is the orchestrator's clock.

When the orchestrator is unreachable, a mirror keeps its last topology and reports IsConnected = false fifteen seconds after its last successful read. Writes are queued, up to 10000, and posted in order once the orchestrator answers; a write is dropped only when the orchestrator rejects the request with 400 or 401. Workers continue simulating from the last leases they received. During an outage, the orchestrator cannot observe liveness, assign containers, or start replacements.

The host saves the document to its storage a second after a change and restores it at startup when asked (-nebula-reset false). A storage failure is logged and retried; the in-memory document stays authoritative meanwhile.

Operational invariants

An integrating process SHOULD preserve these runtime invariants even though the control plane does not enforce them:

• Give every live worker a unique ID and an index that fits in u16.
• Keep one orchestrator responsible for assignments; it is the only host.
• Assign a lease only to a registered, live worker.
• Use only the defined lease states and pair orphaned with an empty worker ID.
• Treat EnsureRuntimeContainer as first-writer-wins. Use a separate game-level authority when concurrent callers could propose different bounds.
• Keep per-tick entity state on direct service links described by the wire protocol.