Debugging and observability

A mesh is several processes on several machines, so most debugging is reading the right log. Every Nebula line is prefixed [nebula] so it is greppable in a headless Unity log, the orchestrator keeps an event log on the dashboard, and every role can draw a status overlay. This page lists what each of those shows and what to look for when something is wrong.

The in-game overlay

NebulaDebugOverlay is an IMGUI box in the top-left corner that reads NebulaBootstrap.Instance and prints one block per role present in the process:

The client line is the one to read when input feels wrong. lead is how many ticks ahead of the estimated server tick the client stamps its inputs; adaptive is the part added at runtime because the worker reported inputs arriving late; worker sees is the last lead the worker reported back. corrections and last are how often and how far prediction was corrected; a handover should not change either distribution.

The overlay reads no input of its own, so it works with any input backend. Game code toggles its public Visible field. ShooterGame's ShooterHud finds the overlay in Start, hides it (_overlay.Visible = false) so players see the HUD instead, and flips it on keyboard.f3Key.wasPressedThisFrame. Press F3 in the shooter. Headless roles have the component too but draw nothing.

The in-game console

The backquote key (`) opens the Quantum Console on clients (ShooterConsole instantiates the prefab on clients only). While it is open the cursor is free and the pawn ignores the keyboard and mouse. Commands live in ShooterCommands:

npcs           how many NPCs this client sees, and the mesh-wide total
npcs 200       set the mesh-wide NPC total; workers spawn or despawn to match, no restart

npcs <count> goes through the local pawn as a ServerRpc, so it reaches whichever worker owns the pawn; that worker writes the npcs mesh setting and every worker's NpcDirector reconciles against it. The console prints mesh NPC total set to 200 when the value lands. Any player may run it; it is a load control for the prototype, not an admin command.

NebulaLog and -nebula-verbose

NebulaLog wraps Debug.Log with the [nebula] prefix: Info, Warn, Error, and Debugf, which only prints when NebulaLog.Verbose is true. -nebula-verbose on any role sets it, the orchestrator forwards it to every role it launches, and nebula start --verbose / nebula deploy --verbose pass it to the orchestrator. Verbose adds the launch command lines, dropped control-plane writes while disconnected, and per-message detail; leave it off when measuring tick time.

Warnings and errors are worth grepping for by category: LogWarning, LogError and Exception are what smoke-test.ps1 counts per worker.

The worker tick profile

Every worker logs one line every 5 s with where its tick went:

[nebula] profile 300 ticks/5s 300 frames avg 6.8ms max 29.6ms dup 4 skip 0 gc 1 auth 331 ghosts 64 | npc.brain=0.75 npc.move=3.64 npc.fire=0.65 npc.path=0.08 poll=0.03 ghosts=0.07 physics.sync=0.08 simulate=5.40 poses=0.14 containers=0.56 band=0.26 publish=0.28 gap=9.86

• ticks should be 60 per second and frames close to it. Fewer frames than ticks means the process cannot keep up and Unity is running catch-up FixedUpdates.
• dup is FixedUpdates that found no new tick number to simulate (harmless); skip is tick numbers dropped because the worker fell more than NebulaWorker.MaxCatchUpTicks behind the wall clock. A steady skip is the lag players feel.
• Every section is milliseconds per tick, so the Nebula sections (ghosts, physics.sync, simulate, poses, containers, band, publish) add up to avg. simulate is all of the game's NetworkTick code; gap is everything between two ticks (the physics step, Update, idle time), so avg + gap is about 16.7 ms when the worker is keeping 60 fps.
• Game code adds its own sections with NebulaProfiler.Section("name") and Begin()/End(); ShooterGame's npc.* sections are inside simulate. A call count in brackets appears when a section does not run exactly once per tick.

Read it on a remote mesh with nebula logs --cloud w1 -n 300 | Select-String profile (PowerShell) or | grep profile.

Where logs live

Locally, every process logs next to the build in Builds/<platform>/Logs/ (Win64, MacOS or Linux64; the orchestrator's process host puts them in Logs/ beside the executable):

nebula start deletes nothing; smoke-test.ps1 clears the folder before a run.

In the cloud the paths differ per VM: /var/log/nebula/orchestrator.log and /opt/nebula/bin/Logs/gateway.log on the orchestrator VM, /var/log/nebula-bootstrap.log and /var/log/nebula-worker.log on each worker VM. See Deploying to Hetzner.

nebula status and nebula logs

nebula status
nebula status --json
nebula logs
nebula logs w2 -n 200
nebula logs gateway --follow
nebula logs --cloud w1

status reads /api/state from the dashboard (local, or the deployed one with --cloud) and prints a worker table (state, address, containers, players, bots, NPCs, tick ms, authoritative and ghost counts, heartbeat age) and recent events. logs prints the last 60 lines of a role's log (orchestrator when no role is given) and --follow keeps printing as it grows; it opens the file with sharing because Unity keeps it open for writing. --cloud reads over ssh and cannot follow.

The dashboard event log

The orchestrator keeps its last 200 log lines as events and publishes the newest 60 in /api/state under events[] (time, level, message). The dashboard shows them at the bottom of the page with warnings and errors highlighted. This is the fastest way to answer "what did the orchestrator decide and when": every launch, assignment, retirement, dead-worker declaration and dashboard command lands there.

Reading a handover in the logs

A handover is two lines on two workers. On the old owner:

[nebula] handover OUT Player#281474976710659(e3,quadrant-SE,auth) -> w2 (epoch 4, tick 12345)

On the new owner:

[nebula] handover IN  Player#281474976710659(e3,quadrant-SE,ghost) <- w1 (epoch 4, tick 12346)

The entity prints as name#netId(e<epoch>,<container>,<role>): the epoch and role shown are the ones before the line's transfer applied, and the high 16 bits of the net id are the index of the worker that minted it (281474976710659 is w1, sequence 3).

Across all worker logs the number of handover OUT lines must equal the number of handover IN lines ("out = in"). That is the single invariant smoke-test.ps1 prints in colour: green when they match and are non-zero. A missing IN means the transfer message never arrived or was rejected; look for stale authority transfer for ... ignored (epoch went backwards) or peer worker w2 disconnected on the sender. The same tick number on both sides, plus or minus one, is normal.

A container reassignment is the same thing many times over. On the orchestrator you see assign quadrant-SE -> w1; on the workers a burst of OUT and IN lines as every entity in that container crosses. If a retirement is what triggered it, the orchestrator follows with worker w3 drained; shutting it down, or the warning still reports N authoritative entities after the drain timeout; killing it anyway when something did not hand over within WorkerDrainTimeoutSeconds.

Other lines to know:

Tools/smoke-test.ps1

The smoke test runs the whole mesh headless with bot clients that roam between quadrants and shoot at each other, optionally injects a failure or a scale change, and then reads the logs. It drives the mesh through the CLI (nebula stop, nebula start, nebula stop) and the dashboard API.

pwsh Tools/smoke-test.ps1
pwsh Tools/smoke-test.ps1 -Workers 3 -Bots 3 -Seconds 90
pwsh Tools/smoke-test.ps1 -KillWorker w2 -KillAfter 30
pwsh Tools/smoke-test.ps1 -ScaleTo 2 -ScaleAfter 25 -Seconds 75
pwsh Tools/smoke-test.ps1 -Workers 2 -ScaleTo 4 -ScaleAfter 25

At the end it reports, from the logs:

• orchestrator: assignments, dead-declared, relaunched, scaled-up, retired, drain-timeouts.
• gateway: workers-connected, clients-connected, spawn-requests.
• per worker: registered, peers, spawned, handover-out, handover-in, cross-worker-hits, kills, warnings, errors.
• per bot: welcome, local-player, errors.
• TOTAL handovers: out=N in=N in green when out is non-zero.

A healthy run has out = in, 0 errors, 0 drain-timeouts, and after a -KillWorker exactly one dead-declared and one relaunched.

Tools/typecheck.ps1

The Editor holds an exclusive lock on the project, so a batchmode compile cannot run while it is open. typecheck.ps1 reads every .asmdef under Assets/, generates matching csproj files that reference Unity's managed assemblies directly (resolving package assemblies from Library/ScriptAssemblies or from source), and builds them with dotnet build. It reproduces the asmdef reference graph, so it catches assembly-boundary violations as well as type errors, in seconds, without touching the Editor.

powershell -File Tools/typecheck.ps1
pwsh Tools/typecheck.ps1 -ShowWarnings
pwsh Tools/typecheck.ps1 -ExtraSourceRoots C:\path\to\spacetimedbsdk

-UnityVersion (default 6000.6.0f1) picks the Unity install whose assemblies are referenced. It is a typecheck, not a substitute for compiling inside Unity.

EditMode tests

Packages/com.1by3.nebula/Tests/EditMode holds the tests that run without a mesh: serialization round-trips, the assignment policy and scaling helpers (FourWorkersGetOneContainerEach, DeadWorkersContainerMovesAndNothingElseDoes, NextFreeIndexReusesGapsAndStartsAtOne, HighestIndexWorkerRetiresFirst, and more), container resolution and hysteresis, the local control plane's lease semantics, RPC binding, handover state chunking, NetworkVariable discovery, the interpolator and the sync components. Run them from Window > General > Test Runner in the Editor (EditMode tab), or in batchmode:

Unity -batchmode -projectPath . -runTests -testPlatform EditMode -testResults Temp\editmode.xml

Editor menu items

Everything under Nebula in the menu bar:

The nebula CLI does the same things from a terminal and is the preferred route; see Running locally.

Troubleshooting