Debugging and observability

Start a debugging session by identifying the process that owns the failing operation. Use the client overlay for prediction and connection problems. Use worker logs for simulation and handover problems. Use the gateway log for routing problems. Use the orchestrator log and dashboard for worker lifecycle and container-assignment problems.

Nebula prefixes its log messages with [nebula] so you can filter them in Unity logs.

Use the in-game overlay

Add NebulaDebugOverlay to the same GameObject as NebulaBootstrap. Nebula > Set Up Active Scene adds it for you.

The overlay displays a block for each role in the current process:

When local movement feels wrong, compare these client values:

• lead is the number of ticks the client runs ahead of its estimated worker tick.
• adaptive is the extra lead added because input arrived late.
• worker sees is the latest input lead reported by the worker.
• corrections counts prediction corrections.
• last is the distance of the latest correction.
• max is the largest correction since spawn.
• smoothing offset is how far the pawn is drawn from its simulated position while a correction eases out (see Smooth corrections).

The overlay does not read input. Set its public Visible field from your own input or UI code. Headless processes do not draw it.

Enable detailed logs

Pass -nebula-verbose to a role to set NebulaLog.Verbose. To enable it for a mesh started by the CLI, run nebula start --verbose or nebula deploy --verbose.

Verbose logging includes process launch arguments, discarded control-plane writes during a disconnection, and message details. Disable it when you measure performance because additional logging can affect tick time.

Use NebulaLog.Info, Warn, Error, and Debugf in your own components when you want consistent [nebula] prefixes. Debugf writes only when verbose logging is enabled.

Read the worker profile

Every worker writes a profile line every five seconds:

[nebula] profile 300 ticks/5s 300 frames avg 6.8ms max 29.6ms dup 4 skip 0 budget 0 gc 1 auth 331 ghosts 64 dormant 0 rpcRejected 0 mem rss=812.0MB native=210.3MB managed=64.1MB | poll=0.03 ghosts=0.07 physics.sync=0.08 simulate=5.40 poses=0.14 containers=0.56 band=0.26 publish=0.28 gap=9.86 begin=0.04 frames.sim=0.02 replicate=0.09 untimed=0.03 | frame avg 16.7ms p90 17.4ms max 41.2ms: physics=1.20 fixed=0.10 tick=6.80 update=0.50 late=0.10 idle=8.00

Use these fields to find a performance problem:

• ticks should average the tick rate (TickRate, 60 by default) per second. frames is the number of Unity frames in the window. With the default ServerFrameRate of 0 it stays close to ticks. A higher ServerFrameRate makes it a multiple of ticks, and a lower one makes it a fraction.
• dup counts fixed updates that did not advance to a new network tick. An occasional duplicate is expected.
• skip counts ticks discarded after the worker fell more than NebulaWorker.MaxLagTicks behind. Each one is a tick a predicting client ran and the worker did not. The worker steps a predicted entity through the inputs of skipped ticks when it next ticks, so the entity is normally not corrected, but a value that keeps increasing means the worker cannot maintain the tick rate.
• budget counts the times a frame stopped catching up early because it had spent NebulaConfig.WorkerCatchUpBudgetMs simulating ticks. The ticks left over run in the next frames, or are counted in skip once the worker is more than MaxLagTicks behind. A value that keeps increasing means ticks cost more than the time between them.
• avg and max (before the |) report the average and maximum time spent in a Nebula tick.
• The sections after the first | are milliseconds per tick and add up to the tick. simulate includes your NetworkTick methods. begin, frames.sim (instance scenes and physics frames) and replicate are the passes between the others, and untimed is whatever no section covers. gap is the time between two ticks.
• frame avg, p90 and max are the wall time of a whole Unity frame, from the start of one frame to the start of the next. A worker holding the default rate averages 16.7 ms, one tick period; a p90 or max far above its frame period is a stall.
• The split after the colon is milliseconds per frame and adds up to the frame average. physics is the physics step, fixed the rest of the FixedUpdate phase (script FixedUpdate other than Nebula's tick), tick Nebula's tick, update the Update phase (including Nebula's transport poll and your Update methods), and late the PreLateUpdate phase (your LateUpdate methods). idle is the remainder: waiting for the frame limiter, rendering, the engine's own frame work and garbage collection. A frame that is mostly idle has room to spare; one with little idle and an average above 16.7 ms is falling behind.

The worker sends the same figures on every heartbeat, so GET /api/state shows them per worker as frameAvgMs, frameP90Ms, frameMaxMs, frameSections (the split) and profileSections (every tick section), each a map from name to milliseconds. The values are those of the last completed five-second window. A worker older than this reports 0 and empty maps.

To profile your own game code, create a section with NebulaProfiler.Section("name") and keep it in a static readonly field, then call Begin() and End() around the work. Asking for the same name again returns the same section. The section appears in the worker's profile line and in profileSections, in milliseconds per tick; a section that runs in fewer ticks than the window also shows its call count in parentheses on the profile line. A section nested inside another one (yours inside simulate) is counted in both.

To read profile lines from a deployed worker, run nebula logs --cloud w1 -n 300 and filter for profile.

Read the delivery path

A low worker tick time does not mean players see smooth motion: a snapshot still has to cross the worker's transport, the gateway's loop, and the client's frame. Two more telemetry lines cover that path.

The client writes one line every five seconds (nebula logs bot1, or the player log):

[nebula] client 142 fps entities 1016 in 866 pkt/s 285.0 KB/s 4901 states/s (msgs: state 1788 rpc 279 vars 810) rtt 0ms lead 5 (adj 2, worker saw 4) corrections 2 last 0.10m | full-rate interp: depth 3.8 ticks, starved 3.0% worst +2.0 | frame max 17.8ms fixed/frame max 1 | gaps>3t 1 worst 176t | render delay 4t (stream lag 1t, lateness p95 1.3t, adaptive +1t) | snapshot age ms min 6.5 avg 17.6 max 35.0

• corrections counts reconciliation snaps of the local pawn; steady non-zero counts with a large last distance mean prediction and simulation disagree.
• full-rate interp covers entities the gateway sends every tick. depth is how many ticks of buffer the interpolator has; starved is the share of frames that had to extrapolate past the newest sample. Starvation is what a player sees as remote entities stuttering or rubberbanding.
• render delay is InterpolationDelayTicks plus adaptive, the ticks the measured lateness adds. lateness is the percentile (InterpolationJitterPercentile) of how late world-state packets arrived, for the worst worker; stream lag is the slowest worker's lag behind the newest stream, which is what the delay used with AdaptiveInterpolationDelay off.
• snapshot age is the age of world-state packets on arrival, against the wall clock the workers derive ticks from. On one machine the numbers are exact; across machines read the spread between min and max, which is the jitter the interpolation delay has to cover.
• frame max and fixed/frame max show whether the client itself is stalling.
• carrier hitches appears while the local pawn rides a carrier (a ship): frame-to-frame jumps of the hull that its reported velocity does not explain, with the carrier's container and epoch changes in the same window. A pilot feels every one of these.

The standalone orchestrator and gateway write a loop line every five seconds:

[nebula] loop 240 Hz, period max 6.1 ms, tick avg 0.08 ms max 0.3 ms

period max is the longest wait between two loops. The gateway runs four loops per tick; a period well above its interval means the process is not being scheduled on time (on Windows, the service requests a 1 ms timer and opts out of background timer throttling at startup; the log says so).

Find logs

For a local mesh, find logs in Builds/<platform>/Logs/, where <platform> is Win64, MacOS, or Linux64.

For a Hetzner deployment, use nebula logs --cloud <role>. The CLI reads the appropriate remote path over SSH. See Deploy to Hetzner.

Inspect status and events

Run these commands from the project directory:

nebula status
nebula status --json
nebula logs w2 -n 200
nebula logs gateway --follow
nebula logs --cloud w1

nebula status reads /api/state from the local dashboard. Add --cloud to read the configured deployment. The command shows worker state, addresses, container assignments, entity counts, tick time, heartbeat age, and recent events.

nebula logs prints the last 60 lines from the orchestrator when you omit a role. Pass a role to read another log. Add --follow to keep reading a local file as it grows. Cloud logs do not support --follow.

The orchestrator keeps its 200 most recent event lines. The dashboard displays the newest 60. Check these events when you need to know when a worker launched, stopped, missed heartbeats, or received a new container assignment.

Trace a handover

A cross-worker handover produces one line on the previous worker and one on the new worker:

[nebula] handover OUT Player#281474976710659(e3,quadrant-SE,auth) -> w2 (epoch 4, tick 12345)
[nebula] handover IN  Player#281474976710659(e3,quadrant-SE,ghost) <- w1 (epoch 4, tick 12346)

The entity format is name#networkId(e<epoch>,<container>,<role>). The high 16 bits of the network ID contain the index of the worker that originally spawned the entity. They do not identify the current worker.

Compare the total handover OUT and handover IN lines across worker logs. A missing IN can mean that the peer connection failed or the receiving worker rejected an old epoch. Search for peer worker, stale authority transfer, and not connected; keeping authority near the same timestamp.

The sending and receiving tick numbers normally match or differ by one.

Validate the Unity project

Before you build, run Nebula > Validate Project. The validator checks build scenes, the game mode, container IDs, network prefab registration, world-partition assets, and other setup that could otherwise fail in a build.

Click a validation message in the Unity Console to select the related asset or GameObject when one is available.

Use the Editor menu

Use the Nebula menu for local development:

• Use Build to create a Windows player or Linux dedicated server.
• Use Mesh to start or stop the local mesh, launch another client, or open the dashboard.
• Use World to author and validate a partitioned world.

For repeatable commands and continuous integration, use the nebula CLI.

Troubleshoot common problems