Maschinenraum Technical Design
Technical design

Maschinenraum

A multiplayer social room drawn as a neon isometric world. People open a link, pick a nickname, walk around, talk, sit at tables, open doors, and rebuild the room together. A Rust server keeps each room in a single task. A Rust client compiled to WebAssembly draws everything as vector geometry on the GPU.

500concurrent users, the design target
5.8 mschat round-trip p99 at 500 users
0heap allocations per steady-state move
8.2 µsone move fanned out to 64 players

Goals

  • Zero-install: runs in any browser with WebGPU or WebGL2.
  • Up to 50 people per room, around 500 per process, with latency low enough that it feels local.
  • Live editing by anyone let in (or only by holders of the builder key), with every change recorded.
  • A predictable hot path: no locks, no steady-state allocation, bounded queues everywhere.

Out of scope for v1

  • Per-room permissions: access and building are process-wide keys (MR_ACCESS_KEY, MR_BUILDER_KEY). Voice and claimed nicks came later. (Commands are rate limited per connection: a token bucket of 10/s, burst 20, that also charges for search work.)
  • Horizontal scaling across processes (rooms are in-process actors).
  • A rollback command (the event log makes one possible; it isn't built).
01 · Decisions

Decision record

Choices made before any code was written, and the reasoning behind them. Select a row for the alternatives.

02 · System

Crates and runtime

Six crates, and xtask for the checks. protocol is the shared contract (messages, floors, the world tick, walking) and holds every game's rules (the half pipe, transfer, football, light cycles, breakout, Spindizzy, Bomberman, Commando, Kobold, the lexicon's fold, the lift and paternoster), so the client and server cannot disagree within one build; race (ship physics, AI, weapons) and props (deterministic ball and glass physics) are simulation code both sides run. Commando is lockstep: the runner's browser, every onlooker's and the room step the same deterministic game from one input byte a tick. Kobold is integer only, so a run is fixed by its job, its case and the programs; the room scores a submission's 100 cases off its own task. A Kobold duel is two games with the homes swapped, side 1 reading links through the arena's mirror; the room plays it on its own task in well under a millisecond. A relay shift is its line, its seed and its swaps; browsers play it half a second behind the room and replay from the start when a swap comes late, and the room replays it whole at the end to score it. A light show is its program, its seed and its beat period, the period taken from the disco's tempo; only browsers step it, and the room checks a set piece on its own task. The server is split into a library and a binary so tests and benchmarks can drive a room without sockets.

flowchart LR
  protocol["protocol
serde + postcard"] --> server["server
axum · tokio · duckdb"] protocol --> client["client
wasm · wgpu · lyon"] protocol --> loadgen["loadgen
bot swarm"] race["race
ships · AI · weapons"] --> server race --> client props["props
deterministic physics"] --> server props --> client server --> tests["tests/alloc.rs
benches/room.rs"] client --> tbench["benches/tessellate.rs
(native, scene only)"]
ComponentOwnsConcurrency
Room directoryname → room inbox sender, the world mapOne parking_lot mutex, taken on joins, hand-offs, neighbour observation and retirement, never per message
Room taskPlayers, residents, objects, the walk grid, A* scratch, props, the bartender, the games under way (and Commando's copy of each run), the DJ deck, the lexicon's tree, broadcast arenaOne tokio task per room; single owner, no locks; rooms message each other only with try_send
Race sessionOne race at a gateOne task per gate, 60 Hz; inputs bypass the room
Kobold scoringOne submission's 100 casesspawn_blocking; the verdict comes back to the room's inbox as RoomMsg::KoboldScored
Load samplerThe process's CPU and memory, two minutesOne OS thread, a watch rooms read without a lock
CensusEach room's headcountA watch in the directory, written on join and leave
TempoThe disco's tempoA watch in the directory (Directory::tempo), written by the deck, read when a Kobold light show starts
Reader taskNothing; forwards raw framesOne per connection
Writer taskThe socket sinkOne per connection; feeds every queued frame, flushes once per batch
Event log threadThe DuckDB connectionOne OS thread; batched appends. Loads and the hourly archive run on a second thread with a cloned connection
03 · Message lifecycle

From a click to every screen

Pick an action to watch it travel through the server. Each hop is a bounded channel. Watch where frames are decoded, where they are encoded, and what gets logged.

    04 · Room & pathfinding

    The lobby, live

    This is the real rooms/lobby.ron, using the same projection (64 × 32 tiles), the same blocking rules and a breadth-first search (on a flat floor it finds paths as short as the server's A*). Click a tile to walk there, or click a door to toggle it. Hover to read the tile coordinate and the depth value the GPU uses to order it.

    Tile–
    x + y–
    depth_for–
    Object–
    Visited–
    Path–

    The server's A* runs over a per-tile walk grid that also encodes stairs (ramps climb along their rotation and link to the landing a floor up) and checks each step's height: parts of a floor can be raised or sunken, and a step taller than 0.5 is not walked. A search has a 200,000-tile budget. The server keeps the grid up to date as objects change, so a search never rebuilds it. A search marks tiles with a generation number instead of clearing its arrays, and it reuses one queue and one path buffer. Seats and plants are walkable, doors only when open. The start tile is never checked, so a player an edit has boxed in can always walk off.

    05 · Wire protocol

    Postcard, byte by byte

    One binary WebSocket frame per message. Postcard writes the enum variant index, then the fields in order: integers as varints (signed ones zig-zag encoded), strings as a length followed by UTF-8. There are no field names and no version byte. These are real encodings produced by the protocol crate.

    Decoding is zero-copy: C2S<'a> borrows &str straight from the received frame. S2C<'a> uses Cow, so the server encodes from its own state without cloning. Because variant order is the format, new variants are only ever appended.

    06 · Persistence

    Base file plus an append-only log

    A room's layout is always rooms/<name>.ron with its logged edits replayed on top, from the room's latest snapshot (the compacted edits; a new one after 256 more). Chat, join, leave and served are history, archived to Parquet after a day. Game results, Kobold programs and champions, and letters are rows read at load; Kobold duels replay into the ratings, and the lexicon's lines into its tree. Runtime state a crash should not lose (open doors, the DJ deck, matches under way) is written to room_state; seats reset.

    CREATE TABLE events (
      room    VARCHAR NOT NULL,
      seq     BIGINT  NOT NULL,  -- per room, contiguous from 1
      ts_us   BIGINT  NOT NULL,  -- UTC µs since epoch (batch flush time)
      nick    VARCHAR NOT NULL,
      kind    VARCHAR NOT NULL,  -- place | remove | rotate | screen | portal | drop | take | chat | ...
      payload VARCHAR NOT NULL,  -- JSON of eventlog::Event; append-only contract
      PRIMARY KEY (room, seq)
    );

    Replay

    ObjectKindTilerotState after replay

    Two id ranges

    Base-file ids stay below 220. Objects placed at runtime are numbered from 220 upward. Logged remove and rotate events name objects by id, so adding objects to a base file can never shift what an old event points at.

    Write path

    • The room validates an edit, applies it, broadcasts it, then sends LogMsg::Append into a channel bounded at 4096. If the channel is full, the room waits; edits are never dropped.
    • The log thread takes a row, waits 200 ms, drains up to 1024 more rows, and writes them in one Appender flush.
    • Load and Flush requests are processed after the rows queued before them. A room that is loading, or a shutdown in progress, sees every event accepted earlier.
    • SIGTERM flushes before exit. A hard kill can lose up to about 200 ms of events. A DuckDB error exits the process, so memory and disk never disagree.
    07 · Rendering

    Beams, a light volume, six looks

    The client uses no image assets. Every shape is built in code: fills as triangles, edges as analytic glowing beams (tracy's line renderer), avatars as rigged instanced templates. It draws with wgpu (WebGPU, falling back to WebGL2) into an HDR target with a depth buffer: fills write depth with a slope-scaled bias, beams test it and never write it. Text such as names and chat is DOM over the canvas; signs, boards and billboards are stroke-font lines in the scene.

    DataContentsRebuiltPer-frame CPU
    Static sceneFloors (raised parts and risers), walls and their style's dressing, every object, the walls along the seams next door, the city (in photo mode, every room of the world if asked)When an object or a neighbour changesnone
    Light volumeA 3D grid of light from lamps, glowing things and neon, over the room you are inWhen light changes; an edit rebakes only its regionnone
    TemplatesAvatars, props, markersOnce at startupnone
    InstancesPosition, pose, colours, 80 bytes eachEvery frameone buffer write
    Dynamic linesPings, the model's wall, live billboards, games, the lexicon's treeWhen they changerare; every frame while a game moves
    Dynamic fillsA Commando battlefield, painted by the room's painter as it scrolls; its ground stamps the floor stencil so shadows fall on it. Or a Kobold machine floor: the job's hosts, files and kobolds. A room has one or the otherEvery frame while a run or the table's demo shows; every frame in a room with a Kobold floorone buffer write

    Light. The volume holds, per cell, the light's colour and its direction per channel; a surface reads four samples and shades by its own normal, so any number of lights costs the same per pixel. Walls block light by the rule that blocks sight. On WebGPU a compute pass traces bounce light, neon and ambient occlusion over the volume's voxels. Only the room you are in is drawn and lit; next door shows just its walls, doors and windows along the seam.

    Looks. Six shader looks (Diorama, Lounge, Atmosphere, Classic vector, Graphic, Comic) share one scene. Every look in the menu (52) is a profile on one of them that sets every display setting, the vertex, edge and face themes and the line weights, and grades the frame; a Quality setting (Low, Medium, High, Auto, which steps down while frames stay slow) caps what a look may switch on. Photo mode draws lighter while its camera moves and at its best, past Quality, once still.

    flowchart LR
      G["GI compute
    WebGPU only"] --> S S["scene pass
    HDR + depth, MSAA"] --> B["bloom chain
    down and up mips"] S --> C["composite
    AgX or ACES · grade · streak · text mode · film"] B --> C --> O["canvas"]

    The HDR target is Rgba16Float when the adapter can render to, filter and blend it, otherwise Rgba8Unorm on the WebGL2 path. Struct layouts shared with WGSL are mirrored by hand: CameraUniform 304 bytes, Line 64, Instance 80.

    07b · Renderer setup

    Device, targets, bindings, one frame

    One Gfx (client/src/gfx.rs) owns the wgpu device, every target and pipeline, and records each frame into a single command encoder. Three WGSL modules: scene.wgsl (everything in the world), post.wgsl (bloom and the composite) and gi.wgsl (the light tracer, compute). The app hands it a Frame each animation frame: the view, the look and settings, the instances and the dynamic lines.

    Backends

    new_instance_with_webgpu_detection picks WebGPU where the browser has it and WebGL2 otherwise; the status line shows which. The same code runs on both; what differs is what each can do.

    WebGPUWebGL2
    Traced light (compute)Off, On or Ultranone: the baked volume only
    HDR targetsRgba16FloatRgba16Float if it can render to, filter and blend it, else Rgba8Unorm
    Rich extrasdust motes, the nebula (Camera.look.y)left out
    GPU timings (F3)timestamp queries per segmentnone
    Quality on AutoHighLow

    Pipelines only some themes need (ink, hidden lines, vertices) are built the first time a theme asks: WebGL2's shader compiler is slow, and nobody should pay for them at join. client/tests/shaders.rs validates every entry point with naga and translates it to WebGL2 GLSL with each override the client sets (TRAILS, AGX).

    Targets

    TargetFormatLivesPurpose
    SceneHDR, 4× MSAA resolved to single-sampleper frameThe world: fills, beams, rain, reflections
    Depth + stencilDepth24PlusStencil8, multisampledper frameFills write depth with a slope bias, beams only test it; the stencil marks your floor for shadows and reflections
    PhosphorHDR, persistentacross framesAvatar trails: decayed each frame, then deposited into
    Bloom chainHDR, 6 mipsper frameSoft-knee threshold, 6 down, 5 up
    Light volume4 × 3D Rgba16Floatuntil the layout changesColour, and direction per channel, over the room and its neighbours' band
    Voxels, traced light3D Rgba8, 3D Rgba16Float ×3voxels on change; traced light ping-pongsThe tracer's scene and its two result textures
    Tile map2D array Rgba8, a layer per flooron change and on your stepsFog of war and flat lamp light; the race's light strip
    Canvasthe surface's, sRGB or encoded by handper frameThe composite's output; 1.5× when supersampled, 2× for a capture

    Bindings

    scene.wgsl, group 0

    0Camera uniform, 304 bytes (world last: elevation and the line weights)
    2Lights: up to 32 (avatar glows), two vec4 each
    3–4Tile map and its linear sampler
    5–8The light volume: colour, then direction for r, g, b
    9Traced light (the current of two)
    10Neon voxels, for Ultra's floor reflections

    Template lines are vertex data (locations 10–13), never a uniform array: Direct3D took minutes per pipeline over one indexed by vertex_index.

    post.wgsl and gi.wgsl

    0–2Post: the textures of the pass (scene, bloom or phosphor)
    3–4Post: sampler, Params (texel size, threshold, bloom strength, sRGB flag)
    5Post: Photo, two vec4: photo mode, drinks, a look's film
    6Post: Grade, ten vec4: the look's grade, text mode, streak
    0–6GI: Params (four vec4), avatars, voxels, neon, direct light, previous result, next (storage)

    One frame

    Pick a backend and settings to see which passes run. Skipped passes cost nothing: they are not drawn at a lower setting, they are not drawn.

      What crosses to the GPU, and when

      WhenUploadSize
      Every frameCamera uniform304 B
      Every frameInstances: people, props, markers80 B each
      Every frameAvatar glows, the nearest to the camera≤ 1 KB
      Every frame, while it showsPhoto: photo mode, a drink's effect, a look's moving grain32 B
      When they changeDynamic lines: pings, the model's wall, live billboards, games, the lexicon's tree64 B a line
      Every frame, while it showsDynamic fills: a Commando battlefield or a Kobold floor≈ 0.2–0.5 MB
      On your stepFog of war, the rectangle around yousmall
      On an editThe static mesh; the light volume's box around the edit; voxelsregion only
      On joining, or a neighbour's changeThe whole light volume, room and neighbours' band≈ 1–5 MB

      The F3 overlay splits the GPU frame into the same segments as the walk above: fills, avatars, shadows, glass, beams, mirror, avatar beams, phosphor, bloom, composite and gi.

      08 · Performance

      Measured, not assumed

      All numbers come from one desktop machine. Load tests ran the bot swarm on the same host, so the bots and the server competed for CPU.

      Chat round-trip under load

      p50p99max

      Log scale. The bottom row is about 4× the design target.

      Static rebuild per edit

      Native timing. The dashed line is one 60 Hz frame. Large rooms drop a frame on every edit: known hot spot #1.

      One move, decode to fan-out

      Worst-case search

      A serpentine maze where the path visits most tiles. About 217 M tiles/s at every size.

      Capacity estimate

      from benches
      Commands in–per second, all rooms
      Frames out–per second: each action reaches every player in the room
      Busiest room task–
      of one core; excludes socket IO

      Frames out grows with the square of room size. The room-task figure uses the measured fan-out costs (4.1 / 5.8 / 8.2 µs at 8 / 32 / 64 players), interpolated. Socket writes are the larger real cost and aren't included. Measured at 500 users: 49 k frames/s delivered, p99 5.8 ms.

      How it is kept honest

      ToolWhat it proves
      tests/alloc.rsA counting global allocator warms up 50 players, confirms the counter works, then asserts zero allocations across 10,000 moves with fan-out.
      benches/room.rsPathfinding on a worst-case maze, the full move path (decode to try_send), and the codec.
      benches/tessellate.rsThe cost of rebuilding the static mesh after an edit, plus template construction.
      loadgenReal sockets with bots measuring their own chat round-trip through the broadcast path. Reports p50, p90, p99, p99.9 and max.
      --profile profilingA release build with symbols, for Superluminal, perf or samply.
      09 · Failure modes

      What happens when things fill up or fail

      ConditionBehaviourWhy
      Player outbox full (256)kick after the current messageOne stalled client must not stall 63 others
      Room inbox full (1024)wait The reader stops reading and TCP pushes backBackpressure reaches the sender that caused it
      Event log channel full (4096)wait The room pausesEdits are never dropped
      DuckDB write errorexit The process terminatesServing on would let memory and disk diverge
      Corrupt payload on loadpanic in that room onlyNever run a room with silently lost edits
      Malformed frame after joinkickThe client is broken or hostile
      Command bucket emptydrop the commandOne client cannot monopolise a room task (10/s, burst 20, searches charged by size)
      Neighbour room's inbox fullskip that observation or hand-off attemptA blocking send between two full inboxes would deadlock
      SIGTERM / Ctrl-Cflush then exitEverything accepted before the signal is durable
      Client from an older buildbad frame Reload neededNo version handshake; the server always serves a matching client

      Known hot spots

      • A* in the worst case. 275 µs for a corner-to-corner serpentine maze at 128², bounded by the search budget and charged to the client's command bucket. (The static rebuild that was hot spot one is down to 2.1 ms at 128².)
      • Latency checkpoints run in every room, O(n²) relays, whether anyone looks or not; past about 120 people in one room they would starve moves and chat. Gate them on the panel or a #latency board.
      • Text mode and the anamorphic streak are per-pixel heavy (the composite five times per pixel; 20 bloom taps). Both want a low-resolution pre-pass.
      • Chat logging shares the room's backpressure. A DuckDB stall long enough to fill 4096 rows pauses the room.
      10 · Invariants

      Review checklist

      Rules that break silently. Tick them off when reviewing a change; ticks are stored only in this browser.