Lilith Metaverse · Architecture

Multiplayer Netcode, Wire Protocol, and Physics

A focused page within the Lilith Metaverse Architecture documentation. The full map and every sibling page live in the Architecture hub.

9sections14 minread1diagram
← Back to the Architecture hubMarkdown source
On this page

V3 (Lilith's embodied tier) has to put a meditator, a concert performer, an operator, and a low-end browser visitor into the same room and keep their truth from forking. It does this by refusing to let any one engine own room state: a single Rust world server is authoritative, and every client surface — native UE5, the Pixel Streaming worker, the WebGPU fallback, the operator console — speaks one identical Protobuf wire format. That format is defined once in libs/v3/multiplayer-protocol/proto/oshun/v3/multiplayer/v1/multiplayer.proto and then implemented three times over: a Rust crate (libs/v3/multiplayer-protocol/rust/src/lib.rs), a TypeScript codec generated into libs/v3/multiplayer-protocol/src/generated/…/multiplayer.ts and wrapped by src/index.ts, and a hand-rolled Unreal C++ codec (V3/ue/Source/V3Net/Private/V3NetProtocol.cpp). All three are pinned byte-for-byte to the same golden hex fixture, so a snapshot the Rust server emits decodes identically whether Lumen or three.js renders it.

The design choices follow from the product. Because V3 is contemplative and performance, not twitch combat, rollback netcode is deliberately absent — the model is plain server-authoritative client–server with light client-side prediction and one-frame reconciliation. Because the same bytes must survive a WebSocket fallback on a phone, transforms are quantized to integers (millimetres and milli-degrees) and only deltas go on the wire. And because the server is the anti-cheat boundary, physics validation, interaction ownership, and consent gating all live server-side in Rust, with rapier3d as the authoritative physics engine and @dimforge/rapier3d-compat (Rapier WASM) mirroring it for Tier-2 prediction. This page is the prediction/reconciliation, protocol, and physics deep dive behind the "Realtime Backbone" summary in ../V3_ARCHITECTURE.md; the connection lifecycle, sharding, auth, and voice SFU are the companion concern of ./world-server-and-gateway.md.

What ships, honestly#

Three honest layers, verified by reading the code rather than the monolith:

  • Real and test-covered. The Protobuf contract and all three codecs are genuine and asserted byte-identical against a shared golden hex (SNAPSHOT_DELTA_GOLDEN_HEX, 131 bytes, identical in Rust, TS, and UE C++). Delta encode/apply, client prediction (predictSnapshot), the 60 Hz decoder benchmark, version negotiation, rstar R-tree interest management, the bandwidth-budget validator, rapier3d server physics with a teleport-speed anti-cheat, the asana-lock consent state machine, server-validated pickup/place, and the UE OpenXR hand-gesture registry are all real, with unit and integration tests (28 in-file #[test]/#[tokio::test] cases in the world server plus seven integration files under apps/v3/lilith-world-server/tests/, and a UE automation test V3.Net.Protocol.FullPacketSetMatchesRustAndTier2Wire).
  • Library-grade, not a running daemon. The world server's run_service() (world-server lib.rs:4564) binds an axum router that exposes only /healthz, /readyz, and /metrics (build_router, world-server lib.rs:4516). There is no accept loop feeding live client packets into a ticking RoomRegistry yet — the simulation core is exercised by tests, not by a client-serving socket. Treat the netcode as a verified library, not a deployed server.
  • Aspirational in the monolith / corrected here. The monolith says "UE Chaos is the client-side physics engine for Tier 1" and describes a UV3NetDriver/V3NetTransport/UV3WorldSubsystem stack. None of those exist in V3/ue/Source — a grep finds no Chaos, no UV3NetDriver, and V3Net.Build.cs depends only on Core/CoreUObject/Engine/GameplayTags/ V3Core (no Protobuf library, which is why the codec is hand-rolled). What the UE side actually ships is the protocol codec and the hand-gesture registry. The monolith's "zstd-1 on the delta packet" is also unimplemented — there is no zstd anywhere in the V3 Rust tree; compression is delta-encoding plus Protobuf varints only. Finally, the per-mode tick-rate table in the monolith (20 Hz yoga, 10/20 Hz concert audience, …) is a configuration target, not a set of coded presets: TickLoopConfig is a single generic struct whose Default is 50 Hz tick / 20 Hz transform / 60 Hz expression (world-server lib.rs:343).

One wire format, four clients#

The schema is genuinely engine-agnostic. It declares 7 enums and 16 messages (multiplayer.proto). Spatial state is quantized to integers so it survives any renderer: Vector3Mm carries position/velocity as zig-zag sint32 millimetres (proto:67), RotationMilliDegrees carries pitch/yaw/roll as sint32 milli-degrees (proto:73), and AvatarExpression carries an expression_id plus an intensity_basis_points (proto:79) — so the same bytes describe a meditator's micro-smile at 73.00 % intensity whether it is rendered by a MetaHuman or a fallback mesh. The unit of replication is EntityTransform (proto:85): entity_id, position, rotation, velocity, an embedded expression, a lod band, and a last_input_sequence (the field that lets a predicting client know which of its inputs the server has already folded in).

Framing is two oneof envelopes. ClientEnvelope (proto:178) wraps a 6-arm oneof — negotiation_request, presence, voice_control, interaction, gameplay_action, operator_control — each tagged with a session_id, monotonic sequence, and sent_at_ms. ServerEnvelope (proto:193) wraps an 8-arm oneof that adds the two server-only payloads snapshot and snapshot_delta. Capacity is first-class: CapacityTier is CLASS or STADIUM, and it travels in every snapshot so each side knows which budget and interest cap apply.

Hand-rolled codecs, and why the golden hex matters#

The Rust and TypeScript sides use prost / ts-proto, but the Unreal client re-implements the Protobuf wire format from scratch in V3NetProtocol.cpp: WriteVarint (line 22), ZigZagEncode32 (line 11), length-delimited sub-messages (WriteMessage, line 90), and matching decoders that validate the wire type per field and SkipField on unknown tags (line 188). This is the load-bearing risk in a polyglot protocol — two independent encoders can silently disagree on a field's varint or a zig-zag sign — so all three implementations are nailed to one cross-language fixture. The canonical packet is the same in every language (canonical_snapshot_delta_fixture in Rust, canonicalSnapshotDeltaFixture in TS, CanonicalSnapshotDeltaFixture in C++): room lilith-commons-atrium, base sequence 6 → sequence 7, one upserted avatar-lilith-001 transform and one removed avatar-lilith-099. Encoding it must produce exactly:

text
0a0a0803220676332e302e3012156c696c6974682d636f6d6d6f6e732d617472…6c696c6974682d303939

— 262 hex characters (131 bytes), asserted identical by the Rust test protobuf_snapshot_delta_matches_cross_language_golden_wire (protocol crate lib.rs:417), the TS test encodes snapshot deltas to the cross-language protobuf golden bytes (index.spec.ts:50), and the UE automation test, which checks the snapshot, presence, voice, interaction, and both envelope goldens (V3NetProtocolTests.cpp:33–38). The UE codec is honestly scoped: its BuildInEngineProtocolExchangeReport round-trips the four core packet kinds it needs in-engine (PacketKindCount = 4, V3NetProtocol.cpp:967) — snapshot delta, presence, voice control, interaction — and its envelopes carry a single oneof arm each (bHasPresence, bHasSnapshotDelta); it does not implement the gameplay_action/operator_control/full-snapshot arms that the Rust server covers.

Server-authoritative netcode, no rollback#

The server owns state; clients send intent and receive truth. Two server→client shapes carry that truth. A SnapshotPacket is the full room (every EntityTransform); a SnapshotDeltaPacket carries only what changed since a named base_sequence — upserted_transforms plus removed_entity_ids (proto:104). The delta diff is real and shared. In Rust, encode_snapshot_delta (protocol crate lib.rs:85) builds a BTreeMap of the previous frame by entity_id, emits any transform that is new or != its predecessor, sorts the upserts by id for determinism, and lists ids that vanished. apply_snapshot_delta (protocol crate lib.rs:130) replays that onto a base snapshot — remove, then upsert — and the TypeScript applySnapshotDelta (index.ts:163) does the same with a Map, re-sorting by localeCompare. Sequence chaining is the integrity check: a client holds the last full snapshot, and each delta's base_sequence must equal the sequence it last applied, so a dropped or reordered delta is detectable rather than silently corrupting the mirror.

Client prediction and reconciliation#

Prediction is intentionally lightweight — dead reckoning, not rollback. predictSnapshot (index.ts:193) advances every transform by velocity × elapsed, but clamps the extrapolation window to 0–250 ms (Math.max(0, Math.min(250, …)), index.ts:197) so a stale snapshot can never fling an avatar across the room; integration is done in integer millimetres via roundMm to stay bit-stable with the server's quantization. The decode→predict loop is benchmarked: runDecoderBenchmark (index.ts:245) decodes 240 deltas for 256 participants and the test asserts averageFrameMs < 1000/60 and framesPerSecond ≥ 60 (index.spec.ts:115) — i.e. the pure-TypeScript fallback keeps up with a 60 Hz client.

Reconciliation is clearest in the Tier-2 physics model, which simulates a predicted client and the authoritative server side-by-side in two Rapier worlds (physics.ts:57). When a visitor grabs a prop, the predicted world places it at the input frame while the authoritative world only confirms one frame later; the test asserts the visible mismatch is exactly that one frame — frame 8 visibleMispredicted === true, frame 9 false, and maxVisibleMispredictionFrames ≤ 1 (index.spec.ts:239). That is the whole contemplative-netcode bet in one assertion: predict locally for instant responsiveness, accept a single sub-perceptual frame of divergence, then snap to server truth. There is no input-history replay because there is nothing competitive to mis-rank.

sequenceDiagram participant C as Client (UE5 / WebGPU) participant G as Gateway participant W as World Server (Rust, authoritative) Note over C: local grab at input frame 8 C->>C: predict (Rapier) — prop in hand C->>G: ClientEnvelope{ interaction: GRAB } (seq, last_input_sequence) G->>W: route to room shard W->>W: validate (room match, not already held) + Rapier transition W->>W: encode_snapshot_delta (base_seq → seq) W-->>G: ServerEnvelope{ snapshot_delta } G-->>C: snapshot_delta C->>C: apply_snapshot_delta + reconcile (frame 9, mismatch closes)

Interest management and bandwidth budgets#

Stadium rooms can hold far more avatars than any client should receive, so the server culls per viewer. InterestManager::visible_entities (world-server lib.rs:1310) bulk-loads every other participant into an rstar R-tree, takes the nearest-neighbour iterator out to the tier cap, and breaks ties by id for deterministic output. The caps are explicit: 32 visible entities at class tier, 256 at stadium (CLASS_TIER_VISIBLE_ENTITY_CAP / STADIUM_…, world-server lib.rs:506). The EntityTransform.lod field rides along so distant avatars can be sent at coarser detail, and PresencePacket.visibility_band (proto:115) labels the band a subject was sent in.

Bandwidth is a hard, tested budget, not a hope. CLASS_TIER_MAX_BPS = 32_000 and STADIUM_TIER_MAX_BPS = 256_000 (protocol crate lib.rs:10) match the monolith's "≤ 32 kbps in class tier, ≤ 256 kbps at Stadium" line. validate_bandwidth_budget (protocol crate lib.rs:169) computes total_bytes × 8 / duration and reports within_budget, and the test class_tier_snapshot_deltas_stay_under_32_kbps_for_256_participants (protocol crate lib.rs:461) actually simulates 1,200 ticks of a 256-avatar class room, mutating two transforms per tick, summing the encoded delta bytes, and asserting the 60-second average stays under 32 kbps. This is the empirical proof that integer quantization + delta encoding (no zstd) is enough at the class tier.

The fixed-tick loop#

FixedTickLoop (world-server lib.rs:2531) drives broadcasts. TickLoopConfig (world-server lib.rs:336) is a single generic struct — tick_hz, transform_broadcast_hz, expression_interpolation_hz, p99_budget_ms — defaulting to 50/20/60/5.0. Each tick() (world-server lib.rs:2558) advances an accumulator and only emits a SnapshotBroadcastPlan when enough transform-hz have accrued, so a 50 Hz simulation that broadcasts transforms at 20 Hz fires a snapshot every ~2–3 ticks. The per-mode rates in the monolith's table are the intended configuration for each room type; the code is the parameterized engine that would consume them.

Physics: server anti-cheat plus tiered prediction#

Server-authoritative physics is real rapier3d (0.21, in Cargo.toml). PhysicsAuthority (world-server lib.rs:2413) holds a RigidBodySet and ColliderSet; register_participant (world-server lib.rs:2425) spawns a dynamic body at the avatar's millimetre position and parents a capsule collider of half-height 0.9 m, radius 0.25 m (ColliderBuilder::capsule_y(0.9, 0.25), world-server lib.rs:2441) — a human-scale standing capsule. The anti-cheat is validate_state_transition (world-server lib.rs:2483): given the previous and next transform plus elapsed ms, it computes speed in m/s and rejects anything over TELEPORT_SPEED_LIMIT_MPS = 12.0 (world-server lib.rs:1353) with a PhysicsValidationError::TeleportJump carrying the full report. Degenerate inputs are rejected too — NonPositiveDelta for a zero-time step, MissingPosition for a transform without a position. Only after a transition validates does apply_validated_transform (world-server lib.rs:2449) push the new translation and linear velocity into the Rapier body. So a client cannot assert a position the simulation considers impossible.

On the client, Tier 1's physics is whatever the UE project provides in-engine, but V3's own UE modules do not wire a custom Chaos integration (no Chaos references in V3/ue/Source) — the monolith's "UE Chaos for Tier 1" is a description of the engine default, not V3-authored code. Tier 2's prediction physics is genuinely shipped: simulateRapierPropPickupPrediction (physics.ts:57) builds a real Rapier WASM world with gravity { x: 0, y: -9.81, z: 0 } (physics.ts:132), a fixed floor collider, and a kinematic prop, and steps it frame-by-frame to produce the prediction/reconcile trace described above. Its capability descriptor advertises a rapier-step:16ms-budget operational metric (lilith-engine-web-fallback).

Interaction primitives — validated, owned, consented#

The monolith lists pickup/place, asana lock, and a hand-gesture set as "server-validated, same across tiers." In code these are three Rust authorities plus a UE registry.

Pickup / place is InteractionAuthority (world-server lib.rs:1539). pickup (world-server lib.rs:1573) requires the actor to be a participant in the room, the object to exist, the object's room to match the request (cross-room pickup is rejected with ObjectRoomMismatch, world-server lib.rs:1476), and the object to be unheld (else ObjectAlreadyHeld); on success it sets held_by_session_id, bumps server_revision, and emits an interaction.pickup.accepted event. place (world-server lib.rs:1602) additionally requires the placing session to be the current holder (ObjectNotHeldBySession) and the placed transform's entity_id to match the object. The integration test server_validated_pickup_and_place_mutate_authoritative_object_state (tests/pickup_place.rs) walks a full grab→place cycle and checks the event topics and sequence numbers.

Asana lock is a consent state machine, AsanaLockAuthority (world-server lib.rs:1795), with status Requested → Consented → Locked → Released (AsanaLockStatus, world-server lib.rs:1663). An instructor requests a lock targeting a student; only the student may consent — the actor must equal student_session_id or it fails with ConsentActorMismatch (world-server lib.rs:1863) — after which the instructor may activate_lock, and either party may release. Every transition appends to an audit log (asana_lock.requested/consented/locked/released). The integration test asana_lock_covers_request_consent_lock_release_and_audit_log (tests/asana_lock.rs) asserts each status, the consented_at_ms stamp, and the ordered audit trail — this is the literal encoding of the monolith's "instructor → student-consent → server-validated → audit-logged" chain. A parallel tara-physical-adjustment-explicit-consent authority gates body adjustments the same way (both appear in SERVICE_DESCRIPTOR.capabilities, world-server lib.rs:42).

Hand gestures live on the UE client in FV3HandGestureRegistry (V3/ue/Source/V3Input/Private/V3HandGestureRegistry.cpp). BuildLaunchRegistry (line 187) ships exactly seven: namaste, the four-mudra set (mudra-jnana/-chin/-varuna/-prana), applause, and snap. Each gesture is sourced from OpenXR hand tracking (InputSignature must start with OpenXR.Hands.), names required semantic poses and ≥5 tracked joints with per-joint confidence thresholds, and is ClientPredictedSequenced: it triggers client-side within a ≤ 25 ms budget and must not wait for server acknowledgement, then broadcasts within a ≤ 50 ms p95 budget. Validate (line 88) enforces all of this, and ValidateLaunchRegistry (line 332) asserts the set is complete, ids are unique, all are client-triggered, and worst-case broadcast p95 stays ≤ 50 ms — the same client-prediction posture as prop pickup, applied to expression. (These gestures map to the protocol's GameplayActionPacket/InteractionPacket arms when sent.)

Version negotiation and failure modes#

The handshake is content-based, not positional. negotiate_protocol_version (protocol crate lib.rs:189, mirrored by TS negotiateProtocolVersion, index.ts:206) takes the client's and server's supported ProtocolVersion lists and selects the highest mutually supported (major, minor, patch); with no overlap it returns NEGOTIATION_STATUS_UNSUPPORTED_VERSION and an unsupported_version error code carrying the server's supported set so the client can report a precise mismatch. The test exercises both the accept path (picks 3.0.0 over a shared 2.9.x) and the reject path (a 4.0.0-only client). The shipped version is v3.0.0 (PROTOCOL_V3_0_0, index.ts:82; protocol_v3_0_0(), protocol crate lib.rs:29; FV3NetProtocolVersion defaults to 3/0/0, V3Net.h:52).

Decode is fail-closed at every layer. The UE decoders return false on a wrong wire type, a truncated length-delimited field, or a varint that overruns the buffer (ReadVarint/ReadLengthDelimited, V3NetProtocol.cpp:151); Rust returns a prost::DecodeError; and physics/interaction validators return typed errors rather than guessing. Unknown fields are skipped, not fatal (SkipField), which is what lets a v3.0.x client tolerate additive fields from a slightly newer server. The @oshun/multiplayer-protocol package also publishes a capability descriptor with per-stage latency budgets — presence-packet:8ms-budget, snapshot-delta:16ms-budget, gateway-pin:24ms-budget (index.ts:31) — and a gatewayPin is the contract a session carries once the edge has bound it to a shard (LilithGatewayPinSchema, libs/contracts/src/v3/lilith.ts:33).

How this connects#

The protocol is the seam between every other Realtime Backbone page. The gateway terminates connections, authenticates them, and pins each session to a world shard before any of these packets flow — see ./world-server-and-gateway.md for the connection lifecycle, JWT rotation, QUIC/WebTransport→WebSocket fallback ladder, and voice SFU. Which surface a visitor lands on (and therefore which codec runs) is decided upstream by ./tier-routing-and-pixel-streaming.md. The two protocol peers are ./tier1-ue5-client.md (the UE C++ codec and hand-gesture registry described here) and ./tier2-fallback-web-client.md (the TypeScript decoder, predictor, and Rapier WASM physics). Expression intensity and voice positioning ride the same transforms covered in ./avatar-animation-and-audio.md, and the ≤ 25 ms concert music-sync constraint that shares this tick budget is detailed in ./saraswati-stage-pipeline.md. For where these numbers are watched in production, see ./observability-performance-security-and-launch.md.