Persistence

Use persistence to keep an entity after the process simulating it stops. A persistent entity is checkpointed into a store while a worker owns it, and brought back when a worker gains the lease of the container it was saved in.

Entities opt in one at a time. Add a PersistentEntity component to the prefab or scene object, then mark the state that must survive. Everything else in the mesh stays transient.

What persists and what does not

A persistent entity is saved with:

• Its prefab (by ID and by asset name) or its scene ID.
• Its container and its pose in that container's local space, when PersistPose is on.
• Its velocity, when PersistVelocity is on.
• Every NetworkVariable field marked [Persist].
• Whatever each NetworkBehaviour writes in WritePersistentState.
• Its authority epoch, whether it was server-driven, whether a client owned it, and the ID of the worker that wrote the record.

Nothing else persists. An entity without a PersistentEntity is lost when its worker stops, and so is any state on a persistent entity that is neither a [Persist] variable nor written by WritePersistentState. Client connections, prediction state, ghosts, and in-flight RPCs are never saved.

Objects your game generates from a chunk's seed, such as rocks and plants, do not need to be entities to keep what players did to them. See Save changes to procedural objects.

The control plane is not affected by any of this. It holds process registrations, container assignments, and shared settings, and it starts empty on every nebula start. Saved entities sit next to it in the orchestrator's database, in a table of their own.

Opt an entity in

Add PersistentEntity to a prefab or a scene object that already has NetworkIdentity. One component per entity.

Keys

The key identifies the entity across runs, workers, and builds. The record in the store is keyed by it.

• A generated key for a spawned entity is <prefab>:<guid>.
• A generated key for a scene entity is scene:<sceneId>.
• Set your own key from the Inspector, or from game code with the Key property, when the game must find the record by name later. EnsureKey() returns the key, generating one if there is none.

A key can be changed until the entity is first saved. After that, PersistentEntity logs a warning and keeps the existing key, because the record already exists under it.

The key travels with the entity through handover, so the next worker updates the same record instead of writing a second one. No wire-protocol change was needed for this: PersistentEntity writes the key in its own handover state.

Scene entities

A scene entity with PersistentEntity saves under scene:<sceneId> by default. Nebula never instantiates a scene object to restore it. Instead, the record waits until the worker spawns the object that is already resident in its scene, applies the saved state to it, and spawns it at the saved epoch plus one.

That means a persistent door keeps its open state through a worker failure, and returns to the state its scene ships with only after you clear its record.

Carried entities

An entity inside a dynamic container is saved against its carrier, not against a static container. Its record stores CarrierKey — the persistence key of the carrier — and its pose in the carrier's local space.

For this to work the carrier itself must carry a PersistentEntity. When it does, the pair comes back together: restoring or spawning the carrier on a worker with authority loads everything saved inside it. When it does not, Nebula logs a debug line and saves only the carried entity's pose, and the container's records are never restored automatically.

Carriers that spawn in the same frame are read together, the same way containers are: one request asks for what up to 256 carriers hold, and a worker has at most two of these requests waiting at a time. A container restore that brings back a hangar full of vehicles therefore costs one request for their cargo rather than one per vehicle. The cargo comes back a frame after its carrier spawns. If the carrier despawns before the answer arrives, its cargo is not restored from that answer.

Where a record says the entity is

A record stores the entity's location as three values: ScopeKey (the empty string for the public world, or the instance key inside an instance), ContainerId, and the local pose. PersistedEntityRecord.Location returns them as one EntityLocation, the same value NetworkIdentity.Location reports on a live entity, and it does not change across a handover or a restart. The entity location contract defines the three values and what stays stable. A record saved before Nebula recorded scope keys reads as the public world.

Find the chunk a record was saved in

You might need to know where a saved entity is without restoring it, for example to respawn a player at a parked vehicle after a worker restart. In a chunk grid that belongs to a scope (any grid but the public world's), a chunk's container ID is a hash of the scope key and the chunk's coordinate. A hash can't be turned back into a coordinate, so RuntimeGrid.TryCoordOf(ulong id, …) only knows the chunks this process has named itself or learned from a lease row. A freshly started process knows none.

So every record also stores PartId, the part ID of its container. It's the same string the container's lease row carries: c/x/y/z for a chunk. Nebula fills it in on every save. To place a record, pass the record itself:

// grid is the RuntimeGrid of the record's scope.
if (grid.TryCoordOf(record, out Vector3Int chunk))
{
    // grid.CenterOf(chunk) is the chunk's center in the grid's frame, and
    // record.LocalPosition is the entity's position inside that chunk.
}

RuntimeGrid.TryCoordOf(record, …) also teaches the grid the chunk's ID, so later TryCoordOf(id, …) calls for it succeed. ChunkKeys.TryCoordOf(record, …) gives the same answer without a grid, so a service outside Unity can use it too. Both check that the part ID hashes to the record's container ID and refuse one that doesn't. Both return false for a record in no container, in a static container, or inside a carrier.

A record saved before Nebula stored part IDs reads with an empty PartId, and a scoped one can't be placed from the record alone. If your game keeps an approximate position for the entity, search the chunks around it:

ContainerRegistry.TryParseRuntimeId(record.ContainerId, out ulong id);
// approximateChunk is a Vector3Int your game kept, for example grid.CoordOf(lastKnownPosition).
if (grid.TryFindCoordNear(id, approximateChunk, 4, out Vector3Int chunk))
{
    // Found. The grid now knows this ID too.
}

TryFindCoordNear hashes every coordinate within the radius of the hint, nearest first, and stops at the first match. That's (2r+1)² hashes on a planar grid and (2r+1)³ on a volumetric one, so keep the radius close to your hint's error. The entity's next save writes its part ID, and you won't need the search for that record again.

Opt variables and state in

Mark a NetworkVariable field with [Persist]:

[RequireComponent(typeof(NetworkIdentity))]
[RequireComponent(typeof(PersistentEntity))]
public sealed class SupplyCrate : NetworkBehaviour
{
    [Persist] public NetworkVariable<int> Ammo = new NetworkVariable<int>(20);
    [Persist] public NetworkVariable<bool> IsOpen = new NetworkVariable<bool>(false);

    public NetworkVariable<float> GlowPulse = new NetworkVariable<float>(0f);   // cosmetic: not saved
}

Each saved variable is stored under the name <BehaviourTypeName>.<FieldName>, here SupplyCrate.Ammo. Assigning a [Persist] variable also marks the entity dirty, so a change is saved at the next opportunity instead of waiting for the full checkpoint interval.

[Persist] on an entity without a PersistentEntity component does nothing.

Write state a variable does not cover

Override WritePersistentState and ReadPersistentState on a behaviour for state that is not in a NetworkVariable. The pair mirrors WriteHandoverState / ReadHandoverState. Both default to writing nothing.

public sealed class CrateContents : NetworkBehaviour
{
    private readonly List<ushort> _itemIds = new List<ushort>();

    public void Add(ushort itemId)
    {
        _itemIds.Add(itemId);
        MarkPersistDirty();                       // checkpoint this entity soon
    }

    public override void WritePersistentState(NetworkWriter writer)
    {
        writer.WriteUShort((ushort)_itemIds.Count);
        for (int i = 0; i < _itemIds.Count; i++) writer.WriteUShort(_itemIds[i]);
    }

    public override void ReadPersistentState(NetworkReader reader)
    {
        _itemIds.Clear();
        int count = reader.ReadUShort();
        for (int i = 0; i < count; i++) _itemIds.Add(reader.ReadUShort());
    }
}

MarkPersistDirty() is a protected NetworkBehaviour method. It is a no-op on an entity with no PersistentEntity. PersistentEntity.MarkDirty() does the same thing from outside a behaviour.

ReadPersistentState runs on a restored entity before it is spawned, so OnNetworkSpawn sees the restored values.

How the blob survives a schema change

PersistentStateCodec writes named, length-prefixed entries rather than a positional layout:

[version:byte=1][count:ushort] { [name:string][length:ushort][bytes] }

Each [Persist] variable is one entry under its name, and each behaviour that writes more than zero bytes adds an entry named <BehaviourTypeName>#state.

On read, entries are matched by name. A variable you removed is skipped, a variable you added keeps its default, and a renamed behaviour starts fresh. Each entry is read through a reader bounded to its own bytes, so a behaviour that reads more than it wrote fails inside its own chunk — the failure is logged and the next entry is still read correctly.

Checkpoint timing

The worker's persistence service runs from the worker's Update, never from the tick loop. An entity is checkpointed when:

• It has never been saved in this process.
• CheckpointSeconds (or NebulaConfig.PersistenceCheckpointSeconds, default 5 s) has elapsed since its last save.
• A [Persist] variable changed or something called MarkPersistDirty().
• It moved more than 0.25 m or turned more than 5 degrees since its last save, and PersistPose is on.
• Authority is about to leave this worker. The final save happens while this worker is still the authority.
• The worker is shutting down. Everything dirty is saved and the store is given a few ticks to flush.

Call worker.Persistence.SaveNow(identity) at a moment the game knows is worth saving, such as a completed objective. It saves immediately, whatever the timer says, and warns if the worker does not own that entity.

Two limits keep the pass cheap: at most one save per entity every 0.5 s (NebulaPersistence.MinSaveIntervalSeconds), and at most 64 saves per frame (NebulaPersistence.MaxSavesPerFrame). Neither the checkpoint nor an unreachable store can stall the simulation. If the store is down, the mesh keeps running and the saves are retried later.

Durability window

Because entities are checkpointed periodically rather than on every change, a worker that crashes can lose whatever changed since an entity's last checkpoint. The worst-case size of that loss — the number to design around — is:

loss window  ≤  PersistenceCheckpointSeconds
              + 0.5 s (NebulaPersistence.MinSaveIntervalSeconds, fixed)
              + ceil(N_dirty / 64) × frame period (NebulaPersistence.MaxSavesPerFrame, fixed)
              + store write latency

• PersistenceCheckpointSeconds is the term to tune: IsDue guarantees an unconditional save at least this often, whatever the dirty flag says. Lowering it lowers the bound by the same amount.
• The minimum save interval (0.5 s, not configurable) is the throttle that keeps one entity from saving more than twice a second: a change can land the instant after a checkpoint and then wait out nearly the whole interval before it is even eligible to save again.
• The per-frame save budget (64 saves per frame, not configurable) spreads out a burst of simultaneously dirty entities across frames; N_dirty is how many other tracked entities become due in the same window, not the worker's total entity count.
• Store write latency is 0 for an in-memory store, up to LocalPersistenceStore.WriteIntervalSeconds (default 1 s) for a local store with a file, and RemotePersistenceStore.FlushIntervalSeconds (0.25 s) plus a network round trip and the orchestrator's own commit for a remote (mesh) store — the worker's own unflushed batch is what a worker crash can still lose after the save left NebulaPersistence.

With the defaults and a lightly loaded worker, the bound is about 5.6 seconds. A wave of 40 NPCs taking damage in the same tick against a mesh's remote store adds roughly another 0.27 s (one frame of budget plus the flush interval and a commit), for about 5.8 seconds total.

This is a bound on loss per change, not a promise that nothing is ever lost — see current limitations. Use SaveNow for a moment that must not wait for the periodic checkpoint at all (a completed objective, a player logging out). The full derivation, with each term tied to its line of code, is in docs/persistence-durability.md in the repository; a conformance test (ConformancePersistenceDurabilityTests) keeps the bound honest by killing a worker mid-checkpoint with 200 dirty entities and asserting the loss stays within it.

The epoch rule

Every record carries the entity's authority epoch: the number that increases when authority or a container assignment changes. The store keeps a save only when its epoch is at least the stored epoch. A worker that lost authority and does not know it yet cannot overwrite the new owner's state with older data.

A restored entity comes back at record.Epoch + 1. Anything still holding the old epoch is stale everywhere in the mesh from that moment on. The restoring worker saves the entity on its next checkpoint pass, instead of waiting for the checkpoint interval. From then on, the store refuses any save the previous owner still had queued.

An entity you give a record with Apply continues that record's epoch in the same way (see Owned records and rejoining players).

When the store refuses a save, the worker or orchestrator that holds the store logs a warning with the key and both epochs, once per key. If the entity it names is the only copy, its saves are being lost: check that its record reached it through Apply or a restore.

Restore after a failure or a restart

When a worker gains the lease of a static container, it does not restore that container's records straight away. Restoring an entity that is still alive somewhere else would create a second copy, so the service is deliberately cautious:

1. Grace period. The worker waits NebulaConfig.PersistenceRestoreGraceSeconds (default 3 s) after the lease appears. A graceful handover from the previous owner arrives inside that window and wins.
2. Load the container. The worker reads every record whose ContainerId is this container and that is not inside a carrier. Containers that are due together are read together: one request asks for up to 256 containers, and a worker has at most two of these requests waiting at a time. A worker that gains hundreds of containers at once, for example after another worker fails, therefore sends a few requests rather than one per container. Containers beyond that wait a frame or two longer.
3. Judge each record. A record is skipped when a client owned it, when an entity with that key is already alive in this process, or when it belongs inside a carrier. A record whose saving worker is still alive on the control plane and that is newer than twice the checkpoint interval waits one more grace period, in case that worker is about to hand the entity over.
4. Restore what is left. The entity is instantiated from its prefab (by ID, falling back to the prefab name if the prefab list was reordered), placed in the container at its saved pose, given its saved state, and spawned with authority at the saved epoch plus one. The worker logs one line per container: persistence: restored N persisted entities into <containerId>.

If a handover arrives for a key that was already restored, the incoming entity wins and the restored copy is despawned without deleting the record.

When the restore of a container finishes — including a container that had nothing saved in it — Nebula raises NebulaLifecycle.OnContainerRestored. It is the moment at which the box is as the store left it, so it is the safe place to spawn anything of your own into it.

After a whole-mesh restart, the same path runs for every container as its first lease lands, so the world comes back container by container rather than all at once.

A worker that is declared dead while it is still running

The orchestrator declares a worker dead when its control-plane heartbeat has not arrived for WorkerTimeoutSeconds (default 5 s). The worker may still be running: a network partition or a long stall can stop its heartbeats without stopping the process. Its containers are assigned to other workers, which restore them from the last checkpoint. Nebula makes sure the old copies do not live on beside the restored ones:

• The worker fences itself. Once its own heartbeat is older than WorkerTimeoutSeconds, NebulaWorker.IsFenced is true. The orchestrator uses the same test to declare a worker dead. While a worker is fenced, it keeps simulating, but it does not save, restore, or hand entities over. Changes stay dirty and are saved when the next heartbeat arrives. A scope part is not retired and ReleaseRuntimeContainer returns false while the worker is fenced.
• Late saves lose. The restored copy's first save carries the new epoch, so the store refuses the old worker's saves (see The epoch rule).
• The old worker drops what it lost. When the old worker reaches the control plane again and sees that it was declared dead, it despawns everything in the containers now assigned to other workers. It does not save those entities first. Then it registers again as an empty worker. Containers that were not reassigned yet are handed over normally later.
• Gateways and workers stop listening to it. A gateway closes its link to a declared-dead worker and forgets that worker's entities, even if the link still works. Clients are never shown the old copy beside the restored one. Players whose pawns were on that worker are placed again, as after any worker failure. Other workers drop their links to it as well.

Owned records and rejoining players

A record saved while a client owned the entity is marked Owned and is never restored automatically — there is no client to own the result. Restore it on demand when the player comes back: spawn the pawn yourself, load the record, and apply it.

private void OnPlayerJoined(NetworkIdentity pawn, string accountId)
{
    var persistence = _worker.Persistence;
    if (persistence == null) return;                       // persistence is off

    var persistent = pawn.GetComponent<PersistentEntity>();
    persistent.Key = $"player:{accountId}";                // settable until the first save

    persistence.Load(persistent.Key, record =>
    {
        if (record == null) return;                        // first time this player joins
        persistence.Apply(record, pawn, applyPose: true);  // false keeps the pawn's spawn point
    });
}

To checkpoint a player's pawn the moment the player drops, rather than on the next checkpoint pass, call worker.Persistence.SaveNow(pawn) from your game mode's OnPlayerDisconnected. The worker keeps the pawn for the reclaim grace after that, and a pawn that is still persistent when the grace runs out keeps its record. See Players who drop and come back.

Load answers on the main thread, with null when there is no record. Apply puts the state and pose back onto an entity you have instantiated but not yet spawned, and also works on an entity this worker already owns.

Apply also gives the entity the record's epoch history, so the store accepts its saves:

• A pawn this worker owns moves to record.Epoch + 1 if its epoch is lower. The pawn's clients and ghosts take the new epoch with its next state update, as they do after a handover.
• An entity you have not spawned yet spawns at record.Epoch + 1, whichever spawn method you call.
• Either way, the worker saves the entity on its next checkpoint pass. From then on, the store refuses saves from any older copy of the player, for example one on a worker that lost it.

Apply the record soon after the spawn, before the game uses the pawn. An AuthorityRpc sent to the pawn before its epoch changes can be rejected as stale (what Nebula promises for a cross-worker call), and a scope transfer prepared with PrepareTransfer before the change can no longer commit.

The rest of the service is on worker.Persistence (a NebulaPersistence, null when persistence is off):

Despawn or keep

NebulaWorker.Despawn(identity, keepPersisted = false) decides whether the entity is also forgotten by the store:

• Despawn(entity) deletes the record. Use it when the entity ceased to exist in the world — it was destroyed, consumed, or picked up.
• Despawn(entity, keepPersisted: true) removes the entity from the mesh but keeps the record. Use it when the world should still contain the entity, such as a player disconnecting.

A despawned carrier sets down whatever rides in it where it stood. To keep a carrier's persistent cargo aboard, for example when a ship is put away in a hangar, pass CargoPolicy.Stow: Despawn(ship, keepPersisted: true, CargoPolicy.Stow) saves each persistent rider aboard the ship before despawning it, and the cargo comes back when the ship is restored. See Stow a ship with its cargo.

Nebula's own paths that merely unload an entity never delete a record: a scene entity leaving with its streamed cell, a ghost being removed, a duplicate restore losing to a handover, and worker shutdown all keep what is saved. A disconnecting client's pawn is despawned with keepPersisted: true when it carries a PersistentEntity.

To forget an entity without despawning it, call worker.Persistence.Delete(key).

Configure it

Set these on NebulaConfig under Persistence and Control plane, or override them on the command line.

An unrecognized mode logs a warning and turns persistence off.

NebulaBootstrap creates the worker's store and the standalone orchestrator creates its own; both pump store callbacks and dispose their stores on shutdown. The orchestrator tells the workers it launches to use remote (or off when it has no store itself), so every worker writes to the orchestrator's database through it.

The orchestrator's database

Saved entities live in the nebula_entity table of the orchestrator's database, next to the control-plane document. DatabaseUrl names the engine: sqlite:<file> (the default for nebula start, at Library/Nebula/nebula.db) or postgres://user:password@host:port/database for a deployed mesh whose world must outlive its VMs. Both use the same statements; the orchestrator creates the table and its indexes (container_id, carrier_key, scope_key) when it starts.

The table's columns are the record fields: entity_key, prefab ID and name, scene ID, container_id and carrier_key, pose and velocity, epoch, server_driven, owned, name, the state blob, version, saved_at, and saved_by.

One writer thread on the orchestrator runs every save, delete, and query in order, so the epoch rule is one conditional upsert and a later save never overtakes an earlier one. If the database is unreachable the queue waits, the connection is retried every two seconds, and the mesh keeps simulating; after three failed attempts at a job, the store gives up on it and logs an error. A read it gives up on is still answered, with an empty result that PersistenceAnswer.Failed marks as a stand-in, and the orchestrator answers the worker's request with HTTP 503, so the worker retries instead of restoring nothing. A save it gives up on fails the next WhenWritten barrier the same way.

A Unity orchestrator (the compatibility role, not the standalone service) carries no database driver: give it file:<folder> or memory.

How workers reach it

A worker's remote store talks to the orchestrator's dashboard address, the same one it mirrors the control plane from, at these endpoints (with the mesh token in X-Nebula-Token when the mesh has one):

Records travel as JSON with the state blob base64-encoded. A batch the orchestrator cannot take is kept and retried, so nothing is lost while it is away; a load is retried until it succeeds.

A worker sends at most two loads at a time, however many it has been asked for; the rest wait in a queue inside the worker. Saves go out one batch at a time. This keeps a burst of loads from delaying the worker's control-plane heartbeat, which goes to the same orchestrator address. A worker whose heartbeat stops is declared dead after WorkerTimeoutSeconds.

Running and resetting it

Saved entities survive every restart of the mesh: nebula start leaves the database alone. To start from an empty world, run:

nebula start --reset-persistence

That passes -nebula-reset-persistence to the orchestrator, which deletes every saved entity as it starts. nebula deploy --reset-persistence does the same on a deployed mesh.

Set the database in nebula.json: mesh.database for local runs (default sqlite:Library/Nebula/nebula.db), deploy.database for a deployed mesh (default SQLite on the orchestrator VM; nebula config database stores a PostgreSQL URL for every project instead).

To see the current state, run:

nebula status

It prints a persistence line with the mode, backend, connection state, and number of saved entities.

From the dashboard

The dashboard shows a persisted tile with the number of records the orchestrator's store holds, and a persistence line in the settings section reading <backend> · connected or disconnected. Both show off when persistence is disabled.

The Clear persistence button deletes every saved entity through POST /api/persistence/clear. It asks for confirmation first.

Offline reads

The records of a world nothing is simulating are ordinary records: you do not have to activate a scope to read what is saved in it.

IPersistenceStore.LoadWhere is the offline read. It answers with every record that matches a predicate, on the main thread, and Load, LoadContainers, and LoadCarried are the same read narrowed to one key, to a list of containers, or to a list of carriers (pass one ID or key to read one). CountRecords(scopeKey, containerId, …) answers the cheap "is there anything saved here at all?" without moving any records — it is a COUNT in the database, and it is what OnScopeActivating's hasRecords is.

store.LoadWhere(r => r.ScopeKey == "settlement/frontier-3", records =>
{
    foreach (var r in records) Debug.Log($"{r.Key} at {r.LocalPosition} in {r.ContainerId}");
});

A tool or a service outside the mesh reaches the same store through RemotePersistenceStore over the endpoints above, with the mesh token, and needs no worker.

Two limits. LoadWhere filters in the caller for the local and remote stores (only CountRecords is pushed down to SQL), so it is meant for tools and game directors, not for a per-frame path. And a record written behind a worker that owns the entity loses to that worker's next checkpoint — the epoch rule doing its job. Write to a world that is not running, or go through the live entity.

Current limitations

• Every load is a round trip to the orchestrator. A worker's remote store keeps no mirror, so restoring a container waits for one request (retried until it succeeds). The orchestrator is the only process that opens the database.
• Live checkpoints overwrite dashboard edits. Records the game is actively simulating are rewritten at the next checkpoint. Editing or deleting a record for an entity that is alive right now changes nothing lasting.
• One orchestrator, one database. There is no orchestrator failover: while the orchestrator is down, workers queue their checkpoints and cannot restore containers. After WorkerTimeoutSeconds without a heartbeat, workers are fenced and stop checkpointing until the orchestrator is back. IPersistenceStore is request/response, with no subscriptions, so another backend fits behind the same interface without changing game code.
• Owned records need game code. Nebula never restores a record saved under a client's ownership. Your game decides when a returning player's entity comes back.
• A record is only as good as its build. A restore needs the prefab in the running build, found by ID or by asset name. A prefab that no longer exists logs a warning and the record stays in the store.