Nebula
Orchestrator

NebulaOrchestrator

Spins worker processes up and down and authoritatively assigns containers to them through the control plane.

generated from the sources
public sealed class NebulaOrchestrator : MonoBehaviour

Spins worker processes up and down and authoritatively assigns containers to them through the control plane. The CLI runs this loop in a standalone .NET executable. The Unity component remains available for compatibility. The number of workers is a live setting: SetDesiredWorkers (from the web dashboard, see OrchestratorHttpServer) launches new processes or retires existing ones on the fly.

Assignment is a pluggable IAssignmentPolicy (Policy). The baked policy deals containers as evenly as possible across live workers by count, sticky to their current owner so a rebalance moves as few containers as possible. The cost policy deals by the load workers report per container, along a space-filling curve, and is the default once the game registers runtime containers.

Removing a worker is graceful: it is excluded from the assignment set, so the next pass moves its containers to the survivors and the worker hands its entities over through the normal per-entity handover path. Once it holds no leases and reports no authoritative entities (or the drain timeout passes) the process is killed. A worker whose heartbeat stops is declared dead, its containers are reassigned immediately, and a replacement is launched after a short delay.

On a host where an instance costs money to keep and time to recreate (IWorkerHost.SupportsParking), a drained worker is parked into the idle pool instead of killed: it keeps its worker id, its index and its machine, receives no leases, and a later scale-out takes it back before anything new is launched. The host drops a parked instance by itself when keeping it stops being free.

Fields

MaxWorkersLimit

public const int MaxWorkersLimit = ushort.MaxValue;

Hard cap on the worker cap: indices are 16-bit (the high bits of every net id).

Properties

MaxWorkers

public int MaxWorkers { get; }

The most workers this orchestrator will run (NebulaConfig.MaxWorkers).

MinWorkers

public int MinWorkers { get; }

The fewest workers autoscaling will leave running (NebulaConfig.MinWorkers); 0 means the mesh may scale to zero.

ParkedWorkerCount

public int ParkedWorkerCount { get; }

How many workers are in the idle pool. They cost the host something but do no work and are not counted in DesiredWorkers.

Policy

public IAssignmentPolicy Policy { get; set; }

How containers are dealt to workers. Chosen from NebulaConfig.AssignmentPolicy (or -nebula-assignment) at Initialize; game code may replace it at any time. "auto" is the cost policy, for baked and runtime worlds alike; "baked" is the explicit opt-out that deals by count.

TotalCost

public float TotalCost { get; }

Total container cost the mesh carries, per the cost policy, as of the last pass.

Loads

public WorkerLoadTracker Loads { get; } = new WorkerLoadTracker(() => Time.unscaledTime);

The rolling tick-time window behind the scaling signal, sampled from every heartbeat this orchestrator sees.

Config

public NebulaConfig Config { get; }

ControlPlane

public IControlPlane ControlPlane { get; }

OrchestratorId

public string OrchestratorId { get; } = "orch1";

DesiredWorkers

public int DesiredWorkers { get; }

Rebalances

public int Rebalances { get; }

LastAssignmentLog

public IReadOnlyList<string> LastAssignmentLog { get; }

LastMoves

public IReadOnlyList<AssignmentMove> LastMoves { get; }

The last pass's moves and why the planner made them (docs/cohesion-rebalancing.md).

Saturated

public IReadOnlyList<SaturationReport> Saturated { get; }

What the planner reported it could not relieve, and why (SaturationReport).

Events

public IReadOnlyList<OrchestratorEvent> Events { get; }

DashboardUrl

public string DashboardUrl { get; }

Telemetry

public MeshTelemetry Telemetry { get; } = new MeshTelemetry();

The World map's data: static container geometry and the latest telemetry each worker posted (see WorkerTelemetry).

Logs

public LogBuffer Logs { get; } = new LogBuffer();

Recent log lines of every process of the mesh, posted to and read from /api/logs (see LogBuffer).

Persistence

public IPersistenceStore Persistence { get; set; }

The mesh's persistence store, so the dashboard can report how many entities are saved and wipe them (POST /api/persistence/clear). Set by NebulaBootstrap before Initialize; null when persistence is off. The orchestrator never writes records itself.

SharedCache

public ISharedCacheStore SharedCache { get; set; }

The mesh's shared cache store (NebulaSharedCache), served to workers at /cache/{key} (SharedCacheHost) and cleared from the dashboard (POST /api/cache/clear). Set by the bootstrap before Initialize; null when the shared cache is off. The caller owns the store. Separate from Persistence: clearing persistence leaves it alone.

HostName

public string HostName { get; }

AtWorkerCeiling

public bool AtWorkerCeiling { get; }

The mesh is already driving towards as many workers as MaxWorkers allows, so nothing may be added or resumed.

PolicyName

public string PolicyName { get; }

Shown on the dashboard.

PendingJoins

public int PendingJoins { get; }

Clients waiting to be placed: welcomed by a live gateway with nowhere to spawn yet (GatewayInfo.PendingJoins). Non-zero means somebody is looking at a "world starting" screen.

Methods

CostOf

public ContainerCost CostOf(string containerId);

The cost row of one container, or all zeroes when no worker has reported it lately.

CapacityOf

public CapacityInfo CapacityOf(string containerId);

How full a container is, as this orchestrator last derived it. Unknown when no worker has reported it lately.

Initialize

public void Initialize(NebulaConfig config, IControlPlane controlPlane);

SetDesiredWorkers

public void SetDesiredWorkers(int count);

Set how many workers should be running. Extra workers are launched; surplus ones are drained and killed (highest index first).

AddWorker

public void AddWorker();

SetSetting

public bool SetSetting(string key, string value);

Set one mesh-wide setting (dashboard). Game code on the workers decides what it means.

RemoveWorker

public bool RemoveWorker(string workerId = null);

Gracefully remove one worker: a specific one, or the highest-index one when workerId is empty.

KillWorker

public bool KillWorker(string workerId);

Hard-kill a managed worker process (simulates a crash). The reaper reassigns its containers and relaunches it.

RequestRebalance

public void RequestRebalance();

Run an assignment pass now instead of waiting for the next 500 ms tick.

NextFreeIndex

public static uint NextFreeIndex(IEnumerable<uint> used);

Smallest positive index not in used (indices double as ports and entity-id high bits, so they are reused).

EligibleWorkers

public static List<WorkerInfo> EligibleWorkers(IEnumerable<WorkerInfo> live, ICollection<string> retiring, ICollection<string> retired, ICollection<string> parked);

The workers an assignment pass may deal containers to: the live ones minus those draining, just retired, or sitting in the idle pool. A parked worker still heartbeats, so only this filter keeps leases off it. Pure function.

PickWorkerToUnpark

public static string PickWorkerToUnpark(IEnumerable<KeyValuePair<string, float>> parkedAt);

Which parked worker a scale-out takes back first: the most recently parked one. It is the warmest, and on a billed host it is the one with the most paid time left. Pure function; null when the pool is empty.

PickWorkerToRetire

public static string PickWorkerToRetire(IEnumerable<KeyValuePair<string, uint>> candidates);

Which worker leaves first when scaling down: the highest index among those not already retiring.

UnparkWorker

public bool UnparkWorker(string workerId);

Take one worker back out of the idle pool from the dashboard, raising the desired count to match. The ceiling is checked before unparking: raising a desired count that is already at MaxWorkers clamps it back to where it was, and the reconciler would retire the worker that had just been resumed on the very next pass. False when the pool is empty or the mesh is at its ceiling (AtWorkerCeiling tells the two apart for the HTTP answer).

DeleteParkedWorker

public bool DeleteParkedWorker(string workerId);

Drop a parked worker now instead of waiting for the host to let it go (the dashboard's "delete").

ComputeAssignment

public static List<KeyValuePair<string, string>> ComputeAssignment(IList<string> containerIds, IList<WorkerInfo> liveWorkers, IList<LeaseInfo> leases, bool keepOrder = false);

Deal containers across live workers as evenly as possible, keeping existing assignments where they fit. Pure function of (containers, live workers, current leases); returns the changes to apply.

ParameterDescription
keepOrderDeal containerIds in the order given instead of sorting by id. A partitioned world lists its containers in Morton order, so each worker's quota is a contiguous run of spatially neighboring cells - a compact region it can load as a block.

EnsureRuntimeContainer

public string EnsureRuntimeContainer(ulong id, Bounds bounds);
public string EnsureRuntimeContainer(ulong id, ContainerPlacement placement);

Create a runtime container (dashboard, tests, tooling): a lease row carrying bounds (absolute coordinates), unassigned until the next pass deals it. The normal path is a worker asking for one next to its entities (NebulaWorker.RequestRuntimeContainer). Null on success, otherwise the reason.

RemoveRuntimeContainer

public string RemoveRuntimeContainer(ulong id);

Delete a runtime container's lease row, so every process forgets the box. Null on success, otherwise the reason.

ComputeRuntimeAssignment

public static List<KeyValuePair<string, string>> ComputeRuntimeAssignment(IReadOnlyList<LeaseInfo> leases, IList<WorkerInfo> eligible);

Runtime containers (leases carrying a box) stay with the worker that asked for them; the game placed them next to the entities that need them. Only one whose owner is gone or retiring moves, to the eligible worker holding the fewest runtime containers. Pure function; returns the changes to apply.

SetContainerHint

public string SetContainerHint(string containerId, in ContainerHint hint);

Tell the planner something about a container while the mesh runs (ContainerHint): the value goes on its control-plane row, so it outlives this orchestrator and beats whatever was baked. Passing ContainerHint.Default clears it and lets the baked hint stand again. Null on success, otherwise the reason. The next pass acts on it.

PinContainer

public string PinContainer(string containerId, string workerId);

Give a carried container (a ship's interior) a worker of its own: the lease is assigned to workerId and marked LeaseState.Pinned, so it stops following its carrier. The pinned worker receives a permanent ghost of the carrier and simulates whatever is inside. Null on success, otherwise the reason.

UnpinContainer

public string UnpinContainer(string containerId);

Let a pinned carried container follow its carrier again. Null on success, otherwise the reason.

BuildStateJson

public string BuildStateJson();

Everything the dashboard shows, as one JSON document.

On this page