Control plane

Use the control plane to share process registrations, container assignments, and low-frequency settings across the mesh. The orchestrator hosts it: the state lives in the orchestrator process, workers and gateways mirror it over HTTP from the orchestrator's dashboard address, and the orchestrator keeps a copy in its database between runs. Nebula processes access it through IControlPlane.

For the exact document format, write operations, state transitions, and failure semantics, see the control-plane contract.

What it is for, and what it is not

Use it for process registration, heartbeats, container assignments, authority epochs, and shared settings. The orchestrator writes assignments. Workers and gateways register themselves and update their heartbeats. Every process reads the same document.

Do not use it for saved entities. Entities that opt into persistence go through IPersistenceStore into the orchestrator's database, next to the control plane but never mixed with it: the control plane starts empty on every run, and a saved world does not.

Do not use the control plane for per-tick state. Workers exchange transforms, input, RPCs, and ghost state over their direct links and through the gateway. Simulation ticks do not wait on it.

If the orchestrator becomes unreachable, workers and gateways continue with the last assignments they received. RemoteControlPlane retries every second and queues its writes until the orchestrator answers again. During the outage nobody reassigns containers, declares workers dead, or starts replacements. Handovers continue between connected workers because they use the last known assignments.

The three implementations

NebulaBootstrap and the standalone services pick the implementation from the role: an orchestrator hosts, a worker or gateway mirrors, and UseLocalControlPlane (or -nebula-local-control-plane) keeps everything in one process. In the local mode the orchestrator skips launching the gateway, reconciling the worker count, and relaunching workers, because no other process could reach its control plane. It is a test harness, not a way to run a mesh.

Reaching the orchestrator

Workers and gateways need the orchestrator's dashboard address. The orchestrator passes it to every process it launches as -nebula-control-plane http://<advertised address>:<dashboard port>/, together with -nebula-token when the mesh has a token. A worker or gateway you start by hand reads NebulaConfig.ControlPlaneUrl (default http://127.0.0.1:7080/) unless you pass the option.

The dashboard port therefore cannot be 0 on an orchestrator that runs a mesh. The standalone orchestrator refuses to start that way.

The mesh token

NebulaConfig.MeshToken (or -nebula-token, or NEBULA_MESH_TOKEN in the standalone services' environment) is a shared secret. When it is set, every control-plane and persistence request must carry it in the X-Nebula-Token header, and the orchestrator answers 401 otherwise. When it is empty the API is open, which is fine on a development machine or a private network. nebula deploy generates a fresh token for every deployment and passes it to the orchestrator through its environment file; the orchestrator hands it to the workers and gateway it launches.

Storage between runs

The host writes the whole document to its storage a second after every change, and reads it back at startup. Where it goes follows NebulaConfig.DatabaseUrl (-nebula-database, or NEBULA_DATABASE_URL for the standalone services), the same setting that places saved entities:

The orchestrator resets the control plane at its first tick regardless (-nebula-reset, default on), because registrations and leases describe processes that are no longer running. Pass -nebula-reset false to bring the stored document back instead. Saved entities are never touched by the reset.

Every SQL statement is written once for both engines. A PostgreSQL URL may carry ?sslmode=require; a Host=...;Username=...; connection string works too. See Persistence for the entity table.

To restart the orchestrator under live workers, survive a database failover, or rehearse a restore from backup, see Restart and restore a mesh.

The document

The control plane is one JSON document with a version number that grows on every change:

The orchestrator assigns each worker a workerIndex. This value becomes the high 16 bits of network IDs created by that worker and selects its diagnostic color.

Each container has an epoch that increases on an accepted assignment or pin operation. A repeated active assignment to the same worker is a no-op; release and unregister operations do not increment it. Workers and the gateway also inspect the lease state and worker ID when deciding who owns a container.

Writes

Every IControlPlane method except the reads is one write. A mirror posts them in order; the host applies them one at a time on its main thread.

Lease states and worker statuses

LeaseState and WorkerStatus in IControlPlane.cs are the string constants the document uses:

Liveness is now - lastHeartbeat <= WorkerTimeoutSeconds (default 5 s), evaluated by the orchestrator against IControlPlane.Now. On the host that is the orchestrator's own clock; a mirror uses the clock the orchestrator sent with the last document plus the time since.

IControlPlane

Every sim process holds one IControlPlane. It exposes mirrored lists (Workers, Leases, Gateways, Settings), a Changed event, and one method per write. All callbacks fire on the main thread from Tick().

public interface IControlPlane : IDisposable
{
    bool IsConnected { get; }
    DateTime Now { get; }
    event Action Changed;

    IReadOnlyList<WorkerInfo> Workers { get; }
    IReadOnlyList<LeaseInfo> Leases { get; }
    IReadOnlyList<GatewayInfo> Gateways { get; }
    IReadOnlyDictionary<string, string> Settings { get; }

    void Connect();
    void Tick();

    void RegisterWorker(string workerId, uint workerIndex, string address, ushort port);
    void HeartbeatWorker(string workerId, string status, in WorkerStats stats);
    void UnregisterWorker(string workerId);
    void RegisterGateway(string gatewayId, string address, ushort port, uint incarnation = 0);
    void HeartbeatGateway(string gatewayId, in GatewayStats stats);
    void UnregisterGateway(string gatewayId);
    void SetGatewayDraining(string gatewayId, bool draining);
    void HeartbeatOrchestrator(string orchestratorId, uint desiredWorkers);
    void SetSetting(string key, string value);
    void EnsureContainer(string containerId);
    void EnsureRuntimeContainer(string containerId, Bounds bounds, string workerId);
    void TouchContainer(string containerId);
    void AssignContainer(string containerId, string workerId);
    void PinContainer(string containerId, string workerId);
    void SetLeaseState(string containerId, string state);
    void ReleaseContainer(string containerId);
    void RemoveContainer(string containerId);
    void ResetControlPlane();
}

ControlPlaneExtensions adds FindWorker, FindGateway, FindLease, GetSetting, GetSettingInt and IsWorkerAlive. GatewayInfo carries the gateway's Incarnation, DrainRequested flag and last Stats; see Run more than one gateway.

How a mirror stays current

RemoteControlPlane keeps one GET /api/control-plane?since=<version>&wait=10 request open. The orchestrator answers as soon as the document's version passes since, or after ten seconds with the unchanged document and a fresh clock. The mirror parses the answer on its reader thread and swaps the lists in from Tick(), where it raises Changed once. IsConnected turns false after fifteen seconds without an answer.

Writes go into a queue; a sender thread posts them as POST /api/control-plane with {"ops":[...]}, up to 64 per request, and retries a request the orchestrator could not take. A request the orchestrator rejects (400, 401) is dropped with a warning, because retrying it would block every later one. Nebula logs control plane: reading ... failed and control plane: writing ... failed once per distinct error.

Mesh settings

Use settings for values that every worker needs but that do not change every tick. Nebula stores each setting as a string key and value. Your game defines what it means.

• Seed settings when the orchestrator starts with -nebula-settings key=value,key=value.
• Edit settings from the dashboard or with POST /api/settings.
• Call ControlPlane.SetSetting from game code on a worker.
• Read settings from IControlPlane.Settings after the Changed event. Use GetSetting or GetSettingInt to supply a fallback value.

Nebula does not provide a singleton gameplay worker. If each worker must respond to a shared value, make the operation idempotent and let each worker reconcile its authoritative entities. See Spawning and mesh settings.

What nebula start and nebula deploy do with it

nebula start launches the standalone orchestrator with -nebula-database sqlite:<project>/Library/Nebula/nebula.db (mesh.database in nebula.json changes it). The orchestrator hosts the control plane on the dashboard port and passes its address to the gateway and every worker. Nothing else needs to run first.

nebula deploy does the same on the orchestrator VM with NEBULA_DATABASE_URL and a fresh NEBULA_MESH_TOKEN in /etc/nebula/env. The database defaults to sqlite:/opt/nebula/data/nebula.db on the VM; run nebula config database to point the deployed mesh at PostgreSQL. See Deploy to Hetzner Cloud.