Load testing a gateway fleet

nebula-loadgen is a console program that connects any number of synthetic clients to a gateway address, keeps them playing, and reports what happens to their sessions. Use it to size a gateway, to check that a load balancer keeps each client on one gateway, and to prove that draining or killing a gateway does not lose a player's session or pawn.

It lives in the package's service sources and builds with the .NET SDK:

dotnet run --project Packages/com.1by3.nebula/Services~/Nebula.LoadGen -- --gateway 203.0.113.10:7000 --clients 300 --seconds 600

What a synthetic client does

Each client completes the same handshake a game client does: it connects, sends Hello, receives Welcome with its session id and session token, and is placed on a worker. It then sends an input every tick, pings every half second, and consumes the replication the gateway sends it.

When the gateway tells it to reconnect (GatewayDraining), refuses it with a retry hint, or drops its link, the client reconnects to the same address with its session token. A load balancer in front of the fleet sends that reconnection to another gateway, which reclaims the session from the worker. The client records whether it got the same session id back and whether it kept its pawn.

By default inputs carry no game meaning; behaviours shape them. Workers decode them with your game's input type, so use --bot when your game mode treats bots differently, or point the tool at a mesh whose game ignores malformed input.

Options

Behaviours

By default every client stands still. A behaviour decides what a client does once it has joined: where it moves, which action bits it presses, which RPCs it sends, when it asks for a scope change and when it leaves. Choose them with a weighted mix:

dotnet run --project Packages/com.1by3.nebula/Services~/Nebula.LoadGen -- --gateway 127.0.0.1:7000 --clients 200 --seconds 600 \
  --mix wander:70,travel:20,idle:10 --opt wander.radius=40 --opt travel.every-minutes=2 --opt 'travel.targets=a|b' --seed 7 \
  --behaviour-stats results.jsonl

Counts are exact (200 clients of that mix are 140, 40 and 20) and seeded: the same --seed gives the same client the same behaviour and the same random choices, so a run repeats. --behaviour <name> is a mix of one.

A profile looks like this. Every field is optional, and a relative plugin path is relative to the profile:

{
  "seed": 7, "maxMinutes": 30, "inputHz": 30, "moveSpeed": 5,
  "plugins": ["MyGame.LoadGen.dll"],
  "mix": [
    { "behaviour": "wander", "weight": 70, "options": { "radius": 40 } },
    { "behaviour": "travel", "weight": 20, "options": { "every-minutes": 2, "targets": "a|b" } },
    { "behaviour": "idle", "weight": 10 }
  ]
}

Built-in behaviours

Nebula does not know your input format or how your game changes scope, so the loadgen sends a generic input (a 16 byte payload: tick, client index, sprint flag, move x and z as float32, action bits as uint32) and travel does nothing until a plugin registers a scope-travel handler. A plugin supplies both.

Behaviour stats

--behaviour-stats writes one row per behaviour per second. The columns are t, behaviour, clients, connected, joined, inputsPerSec, actionsPerSec, rpcsPerSec, travelsPerSec, actions, errors, reconnects, rejections, sessionChanges, sessionsLeft, rttP50, rttP95, replicasAvg, bytesInPerSec. Actions are RPCs, scope travels and the actions a behaviour counts itself; inputs are counted apart. errors counts failed sends, behaviours that threw, a missing scope-travel handler and bad packets. The totals (actions, errors, reconnects, ...) are cumulative; the ...PerSec columns are rates over the last interval. Rows from several processes merge by concatenating the files and summing by behaviour and t.

Write a plugin

A plugin is a class library that references Nebula.LoadGen.Sdk (it depends on nothing else from Nebula, so no game type enters Nebula) and has one public class implementing ILoadGenPlugin. Packages/com.1by3.nebula/Services~/Nebula.LoadGen.SamplePlugin is a complete example, with a strafe behaviour:

<ItemGroup>
  <ProjectReference Include="path/to/Nebula.LoadGen.Sdk.csproj" Private="false" ExcludeAssets="runtime" />
</ItemGroup>

Reference the SDK with Private=false: the loadgen already carries it, and two copies would be two sets of types. If your game consumes Nebula as a package, reference the Nebula.LoadGen.Sdk.dll that sits beside the published nebula-loadgen instead. Build the plugin with dotnet build -c Release and load it by path:

nebula-loadgen --gateway ... --plugin path/to/MyGame.LoadGen.dll --mix strafe:50,idle:50 --opt strafe.length=20

The interface (namespace Nebula.LoadGen):

public interface ILoadGenPlugin { void Register(IBehaviourRegistry registry); }

public interface IBehaviourRegistry
{
    void Add(string name, Func<BehaviourOptions, ILoadGenBehaviour> factory); // one instance per client
    void SetInputEncoder(IInputEncoder encoder);       // write the game's own input bytes
    void SetScopeTravel(ScopeTravelHandler handler);   // how a client asks for a scope change
}

public interface ILoadGenBehaviour
{
    void OnJoined(IClientContext client) { }           // session and pawn ready (every rejoin too)
    void Tick(IClientContext client, double dt);       // once per input interval, just before the input is encoded
    void OnLeft(IClientContext client) { }             // link lost, refused or left
}

public interface IInputEncoder { int Encode(in InputSnapshot input, byte[] buffer); }   // buffer >= 256 bytes
public delegate bool ScopeTravelHandler(IClientContext client, string target);

IClientContext is what a behaviour reads and does:

int Index; string Name; string BehaviourName; double Time; Random Rng;   // Rng is seeded: runs repeat
bool Connected; bool Joined; ulong SessionId; ulong PawnNetId;
ReplicaInfo? Pawn; IReadOnlyCollection<ulong> Replicas; bool TryGetReplica(ulong netId, out ReplicaInfo replica);
Vector3 EstimatedPosition; void SetEstimatedPosition(Vector3 position);  // System.Numerics
void SetMove(Vector2 direction, bool sprint = false);     // x is world x, y is world z; stays set
void SetActions(uint bits);  void SetRawInput(byte[]? payload);
void SendServerRpc(ulong netId, byte behaviourIndex, uint methodHash, ReadOnlySpan<byte> args);   // netId 0 = pawn
void SendServerRpc(ulong netId, byte behaviourIndex, string method, ReadOnlySpan<byte> args);
bool RequestScopeTravel(string target);                    // calls your ScopeTravelHandler
void SetFocusHint(double x, double y, double z);  void ClearFocusHint();
void Leave(double rejoinAfterSeconds);                     // goodbye, then a new session; negative never rejoins
void CountAction();  void ReportError(string message);     // per-behaviour stats

ReplicaInfo carries NetId, PrefabId, OwnerClientId, SpawnPosition, and Vars (the newest replicated variables bytes, which your game decodes) with a VarsVersion that rises on each change. BehaviourOptions reads --opt and profile options: GetString, GetDouble, GetInt, GetBool, GetList, Has. InputSnapshot has Tick, ClientIndex, Move, Sprint and Actions.

The loadgen cannot decode your game's pose, so EstimatedPosition is the spawn position plus the loadgen's own dead reckoning of SetMove at --move-speed. If your game's replicated variables hold the real position, decode them in Tick and correct it with SetEstimatedPosition. A behaviour runs on the loadgen's single loop thread, so never block in it, and an exception is caught, counted as an error and does not stop the run.

Reading the report

One line per second:

t=   30s connected=15/15 joined=15 reconnects=15 sessionChanges=0 pawnLosses=0 rejected=135 in=1428 pkt/s 2.19 Mb/s out=510 pkt/s 0.13 Mb/s rtt p50=15.7 p95=16.4 ms replicas avg=42.3 max=88 bytes/client avg=3.1 max=7.4 KB/s gateways[9cd7ccce=15]

The CSV header (with --csv) is t,connected,joined,reconnects,sessionChanges,pawnLosses,rejections,packetsIn,packetsOut,bytesIn,bytesOut,rttP50,rttP95,replicasAvg,replicasMax,bytesPerClientAvg,bytesPerClientMax,gateways.

The program exits with 0 when sessionChanges and pawnLosses are both zero at the end, and 1 otherwise, so it can gate a fleet test.

Two checks worth running

Drain a gateway. Run the tool against a fleet behind a load balancer, then ask the orchestrator to drain one gateway:

curl -X POST http://<orchestrator>:7080/api/gateways/drain -H "X-Nebula-Token: <mesh token>" -d '{"id":"gw2","draining":true}'

Within GatewayDrainReconnectSeconds the gateways field shows that gateway's clients moving to the others, with sessionChanges and pawnLosses still zero.

Kill a gateway. Stop a gateway process under load. Its clients' links drop, they reconnect through the load balancer, and the workers, which keep a pawn for SessionReclaimSeconds after a link drops, hand the same pawns back. pawnLosses stays zero as long as the reconnection lands inside that window.

Soak tests

Nebula.Services.Tests includes a soak suite for interest management: InterestSoakTests, tagged [Category("Soak")] so it does not run with the ordinary unit tests. It runs the real gateway and a real RegionPublisher inside a fake worker, at two world sizes each, for the three shapes from Scaling limits and when to partition:

• a grid of cell containers across four workers,
• a few large zone containers across three workers, and
• one big container on one worker.

Each asserts that a stationary client's replica count and bytes stay bounded independent of world size, that the gateway's cache and worker link count stay bounded by its own clients' interest, and that a client walking a fixed distance sees no gaps, duplicates, or leaked replicas. From the Nebula repository root, run:

dotnet test Packages/com.1by3.nebula/Services~/Nebula.Services.Tests/Nebula.Services.Tests.csproj --filter TestCategory=Soak

It writes its measurements to docs/interest-baseline/soak-results.md in the repository, so a later run can be compared against the recorded baseline.

InterestProbe.Attach(client), attached in a game scene running as a bot or a manual test client, logs the same four properties from inside a running Unity process — see Interest management. It is useful for checking a soak scenario your game defines that the fixed test shapes above do not cover, such as your own world's actual container layout.

To start four headless clients from your Unity project root, run:

nebula start --bots 4 --bot-args "-mygame-bot-line"

Implement the example flag -mygame-bot-line in your game to select a bot behavior. --bot-args appends its value to every bot client's launch arguments. These bots run your actual game code and can exercise gameplay and InterestProbe. Use nebula-loadgen for lightweight synthetic clients that measure gateway connection handling.