# Multiplayer Netcode, Wire Protocol, and Physics

V3 (Lilith's embodied tier) has to put a meditator, a concert performer, an
operator, and a low-end browser visitor into the _same_ room and keep their
truth from forking. It does this by refusing to let any one engine own room
state: a single Rust world server is authoritative, and every client surface —
native UE5, the Pixel Streaming worker, the WebGPU fallback, the operator
console — speaks one identical Protobuf wire format. That format is defined once
in
`libs/v3/multiplayer-protocol/proto/oshun/v3/multiplayer/v1/multiplayer.proto`
and then implemented three times over: a Rust crate
(`libs/v3/multiplayer-protocol/rust/src/lib.rs`), a TypeScript codec generated
into `libs/v3/multiplayer-protocol/src/generated/…/multiplayer.ts` and wrapped
by `src/index.ts`, and a _hand-rolled_ Unreal C++ codec
(`V3/ue/Source/V3Net/Private/V3NetProtocol.cpp`). All three are pinned
byte-for-byte to the same golden hex fixture, so a snapshot the Rust server
emits decodes identically whether Lumen or three.js renders it.

The design choices follow from the product. Because V3 is contemplative and
performance, **not twitch combat, rollback netcode is deliberately absent** —
the model is plain server-authoritative client–server with light client-side
prediction and one-frame reconciliation. Because the same bytes must survive a
WebSocket fallback on a phone, transforms are quantized to integers (millimetres
and milli-degrees) and only _deltas_ go on the wire. And because the server is
the anti-cheat boundary, physics validation, interaction ownership, and consent
gating all live server-side in Rust, with `rapier3d` as the authoritative
physics engine and `@dimforge/rapier3d-compat` (Rapier WASM) mirroring it for
Tier-2 prediction. This page is the prediction/reconciliation, protocol, and
physics deep dive behind the "Realtime Backbone" summary in
[../V3_ARCHITECTURE.md](../V3_ARCHITECTURE.md); the connection lifecycle,
sharding, auth, and voice SFU are the companion concern of
[./world-server-and-gateway.md](./world-server-and-gateway.md).

## What ships, honestly

Three honest layers, verified by reading the code rather than the monolith:

- **Real and test-covered.** The Protobuf contract and all three codecs are
  genuine and asserted _byte-identical_ against a shared golden hex
  (`SNAPSHOT_DELTA_GOLDEN_HEX`, 131 bytes, identical in Rust, TS, and UE C++).
  Delta encode/apply, client prediction (`predictSnapshot`), the 60 Hz decoder
  benchmark, version negotiation, `rstar` R-tree interest management, the
  bandwidth-budget validator, `rapier3d` server physics with a teleport-speed
  anti-cheat, the asana-lock consent state machine, server-validated
  pickup/place, and the UE OpenXR hand-gesture registry are all real, with unit
  and integration tests (28 in-file `#[test]`/`#[tokio::test]` cases in the
  world server plus seven integration files under
  `apps/v3/lilith-world-server/tests/`, and a UE automation test
  `V3.Net.Protocol.FullPacketSetMatchesRustAndTier2Wire`).
- **Library-grade, not a running daemon.** The world server's `run_service()`
  (`world-server lib.rs:4564`) binds an axum router that exposes only
  `/healthz`, `/readyz`, and `/metrics` (`build_router`,
  `world-server lib.rs:4516`). There is **no accept loop feeding live client
  packets into a ticking `RoomRegistry`** yet — the simulation core is exercised
  by tests, not by a client-serving socket. Treat the netcode as a verified
  library, not a deployed server.
- **Aspirational in the monolith / corrected here.** The monolith says "UE Chaos
  is the client-side physics engine for Tier 1" and describes a
  `UV3NetDriver`/`V3NetTransport`/`UV3WorldSubsystem` stack. **None of those
  exist in `V3/ue/Source`** — a grep finds no `Chaos`, no `UV3NetDriver`, and
  `V3Net.Build.cs` depends only on `Core`/`CoreUObject`/`Engine`/`GameplayTags`/
  `V3Core` (no Protobuf library, which is why the codec is hand-rolled). What
  the UE side actually ships is the protocol _codec_ and the hand-gesture
  registry. The monolith's "zstd-1 on the delta packet" is also unimplemented —
  there is no `zstd` anywhere in the V3 Rust tree; compression is delta-encoding
  plus Protobuf varints only. Finally, the per-mode tick-rate table in the
  monolith (20 Hz yoga, 10/20 Hz concert audience, …) is a **configuration
  target, not a set of coded presets**: `TickLoopConfig` is a single generic
  struct whose `Default` is 50 Hz tick / 20 Hz transform / 60 Hz expression
  (`world-server lib.rs:343`).

## One wire format, four clients

The schema is genuinely engine-agnostic. It declares **7 enums and 16 messages**
(`multiplayer.proto`). Spatial state is quantized to integers so it survives any
renderer: `Vector3Mm` carries position/velocity as zig-zag `sint32` millimetres
(proto:67), `RotationMilliDegrees` carries pitch/yaw/roll as `sint32`
milli-degrees (proto:73), and `AvatarExpression` carries an `expression_id` plus
an `intensity_basis_points` (proto:79) — so the same bytes describe a
meditator's micro-smile at 73.00 % intensity whether it is rendered by a
MetaHuman or a fallback mesh. The unit of replication is `EntityTransform`
(proto:85): `entity_id`, `position`, `rotation`, `velocity`, an embedded
`expression`, a `lod` band, and a `last_input_sequence` (the field that lets a
predicting client know which of its inputs the server has already folded in).

Framing is two `oneof` envelopes. **`ClientEnvelope`** (proto:178) wraps a 6-arm
`oneof` — `negotiation_request`, `presence`, `voice_control`, `interaction`,
`gameplay_action`, `operator_control` — each tagged with a `session_id`,
monotonic `sequence`, and `sent_at_ms`. **`ServerEnvelope`** (proto:193) wraps
an 8-arm `oneof` that adds the two server-only payloads `snapshot` and
`snapshot_delta`. Capacity is first-class: `CapacityTier` is `CLASS` or
`STADIUM`, and it travels in every snapshot so each side knows which budget and
interest cap apply.

### Hand-rolled codecs, and why the golden hex matters

The Rust and TypeScript sides use `prost` / `ts-proto`, but the Unreal client
**re-implements the Protobuf wire format from scratch** in `V3NetProtocol.cpp`:
`WriteVarint` (line 22), `ZigZagEncode32` (line 11), length-delimited
sub-messages (`WriteMessage`, line 90), and matching decoders that validate the
wire type per field and `SkipField` on unknown tags (line 188). This is the
load-bearing risk in a polyglot protocol — two independent encoders can silently
disagree on a field's varint or a zig-zag sign — so all three implementations
are nailed to one cross-language fixture. The canonical packet is the same in
every language (`canonical_snapshot_delta_fixture` in Rust,
`canonicalSnapshotDeltaFixture` in TS, `CanonicalSnapshotDeltaFixture` in C++):
room `lilith-commons-atrium`, base sequence 6 → sequence 7, one upserted
`avatar-lilith-001` transform and one removed `avatar-lilith-099`. Encoding it
must produce exactly:

```
0a0a0803220676332e302e3012156c696c6974682d636f6d6d6f6e732d617472…6c696c6974682d303939
```

— 262 hex characters (131 bytes), asserted identical by the Rust test
`protobuf_snapshot_delta_matches_cross_language_golden_wire`
(`protocol crate lib.rs:417`), the TS test
`encodes snapshot deltas to the cross-language protobuf golden bytes`
(`index.spec.ts:50`), and the UE automation test, which checks the snapshot,
presence, voice, interaction, and _both_ envelope goldens
(`V3NetProtocolTests.cpp:33–38`). The UE codec is honestly scoped: its
`BuildInEngineProtocolExchangeReport` round-trips the **four core packet kinds**
it needs in-engine (`PacketKindCount = 4`, `V3NetProtocol.cpp:967`) — snapshot
delta, presence, voice control, interaction — and its envelopes carry a single
`oneof` arm each (`bHasPresence`, `bHasSnapshotDelta`); it does not implement
the `gameplay_action`/`operator_control`/full-`snapshot` arms that the Rust
server covers.

## Server-authoritative netcode, no rollback

The server owns state; clients send intent and receive truth. Two server→client
shapes carry that truth. A **`SnapshotPacket`** is the full room (every
`EntityTransform`); a **`SnapshotDeltaPacket`** carries only what changed since
a named `base_sequence` — `upserted_transforms` plus `removed_entity_ids`
(proto:104). The delta diff is real and shared. In Rust, `encode_snapshot_delta`
(`protocol crate lib.rs:85`) builds a `BTreeMap` of the previous frame by
`entity_id`, emits any transform that is new or `!=` its predecessor, sorts the
upserts by id for determinism, and lists ids that vanished.
`apply_snapshot_delta` (`protocol crate lib.rs:130`) replays that onto a base
snapshot — remove, then upsert — and the TypeScript `applySnapshotDelta`
(`index.ts:163`) does the same with a `Map`, re-sorting by `localeCompare`.
Sequence chaining is the integrity check: a client holds the last full snapshot,
and each delta's `base_sequence` must equal the sequence it last applied, so a
dropped or reordered delta is detectable rather than silently corrupting the
mirror.

### Client prediction and reconciliation

Prediction is intentionally lightweight — dead reckoning, not rollback.
`predictSnapshot` (`index.ts:193`) advances every transform by
`velocity × elapsed`, but clamps the extrapolation window to **0–250 ms**
(`Math.max(0, Math.min(250, …))`, `index.ts:197`) so a stale snapshot can never
fling an avatar across the room; integration is done in integer millimetres via
`roundMm` to stay bit-stable with the server's quantization. The decode→predict
loop is benchmarked: `runDecoderBenchmark` (`index.ts:245`) decodes 240 deltas
for 256 participants and the test asserts `averageFrameMs < 1000/60` and
`framesPerSecond ≥ 60` (`index.spec.ts:115`) — i.e. the pure-TypeScript fallback
keeps up with a 60 Hz client.

Reconciliation is clearest in the Tier-2 physics model, which simulates a
predicted client _and_ the authoritative server side-by-side in two Rapier
worlds (`physics.ts:57`). When a visitor grabs a prop, the predicted world
places it at the input frame while the authoritative world only confirms one
frame later; the test asserts the visible mismatch is exactly that one frame —
frame 8 `visibleMispredicted === true`, frame 9 `false`, and
`maxVisibleMispredictionFrames ≤ 1` (`index.spec.ts:239`). That is the whole
contemplative-netcode bet in one assertion: predict locally for instant
responsiveness, accept a single sub-perceptual frame of divergence, then snap to
server truth. There is no input-history replay because there is nothing
competitive to mis-rank.

```mermaid
sequenceDiagram
    participant C as Client (UE5 / WebGPU)
    participant G as Gateway
    participant W as World Server (Rust, authoritative)
    Note over C: local grab at input frame 8
    C->>C: predict (Rapier) — prop in hand
    C->>G: ClientEnvelope{ interaction: GRAB } (seq, last_input_sequence)
    G->>W: route to room shard
    W->>W: validate (room match, not already held) + Rapier transition
    W->>W: encode_snapshot_delta (base_seq → seq)
    W-->>G: ServerEnvelope{ snapshot_delta }
    G-->>C: snapshot_delta
    C->>C: apply_snapshot_delta + reconcile (frame 9, mismatch closes)
```

## Interest management and bandwidth budgets

Stadium rooms can hold far more avatars than any client should receive, so the
server culls per viewer. `InterestManager::visible_entities`
(`world-server lib.rs:1310`) bulk-loads every other participant into an `rstar`
R-tree, takes the nearest-neighbour iterator out to the tier cap, and breaks
ties by id for deterministic output. The caps are explicit: **32** visible
entities at class tier, **256** at stadium (`CLASS_TIER_VISIBLE_ENTITY_CAP` /
`STADIUM_…`, `world-server lib.rs:506`). The `EntityTransform.lod` field rides
along so distant avatars can be sent at coarser detail, and
`PresencePacket.visibility_band` (proto:115) labels the band a subject was sent
in.

Bandwidth is a hard, tested budget, not a hope. `CLASS_TIER_MAX_BPS = 32_000`
and `STADIUM_TIER_MAX_BPS = 256_000` (`protocol crate lib.rs:10`) match the
monolith's "≤ 32 kbps in class tier, ≤ 256 kbps at Stadium" line.
`validate_bandwidth_budget` (`protocol crate lib.rs:169`) computes
`total_bytes × 8 / duration` and reports `within_budget`, and the test
`class_tier_snapshot_deltas_stay_under_32_kbps_for_256_participants`
(`protocol crate lib.rs:461`) actually simulates **1,200 ticks of a 256-avatar
class room**, mutating two transforms per tick, summing the encoded delta bytes,
and asserting the 60-second average stays under 32 kbps. This is the empirical
proof that integer quantization + delta encoding (no `zstd`) is enough at the
class tier.

### The fixed-tick loop

`FixedTickLoop` (`world-server lib.rs:2531`) drives broadcasts. `TickLoopConfig`
(`world-server lib.rs:336`) is a single generic struct — `tick_hz`,
`transform_broadcast_hz`, `expression_interpolation_hz`, `p99_budget_ms` —
defaulting to 50/20/60/5.0. Each `tick()` (`world-server lib.rs:2558`) advances
an accumulator and only emits a `SnapshotBroadcastPlan` when enough transform-hz
have accrued, so a 50 Hz simulation that broadcasts transforms at 20 Hz fires a
snapshot every ~2–3 ticks. The per-mode rates in the monolith's table are the
_intended configuration_ for each room type; the code is the parameterized
engine that would consume them.

## Physics: server anti-cheat plus tiered prediction

Server-authoritative physics is real `rapier3d` (0.21, in `Cargo.toml`).
`PhysicsAuthority` (`world-server lib.rs:2413`) holds a `RigidBodySet` and
`ColliderSet`; `register_participant` (`world-server lib.rs:2425`) spawns a
dynamic body at the avatar's millimetre position and parents a **capsule
collider of half-height 0.9 m, radius 0.25 m**
(`ColliderBuilder::capsule_y(0.9, 0.25)`, `world-server lib.rs:2441`) — a
human-scale standing capsule. The anti-cheat is `validate_state_transition`
(`world-server lib.rs:2483`): given the previous and next transform plus elapsed
ms, it computes speed in m/s and rejects anything over
`TELEPORT_SPEED_LIMIT_MPS = 12.0` (`world-server lib.rs:1353`) with a
`PhysicsValidationError::TeleportJump` carrying the full report. Degenerate
inputs are rejected too — `NonPositiveDelta` for a zero-time step,
`MissingPosition` for a transform without a position. Only after a transition
validates does `apply_validated_transform` (`world-server lib.rs:2449`) push the
new translation and linear velocity into the Rapier body. So a client cannot
assert a position the simulation considers impossible.

On the client, Tier 1's physics is whatever the UE project provides in-engine,
but **V3's own UE modules do not wire a custom Chaos integration** (no `Chaos`
references in `V3/ue/Source`) — the monolith's "UE Chaos for Tier 1" is a
description of the engine default, not V3-authored code. Tier 2's prediction
physics is genuinely shipped: `simulateRapierPropPickupPrediction`
(`physics.ts:57`) builds a real Rapier WASM world with gravity
`{ x: 0, y: -9.81, z: 0 }` (`physics.ts:132`), a fixed floor collider, and a
kinematic prop, and steps it frame-by-frame to produce the prediction/reconcile
trace described above. Its capability descriptor advertises a
`rapier-step:16ms-budget` operational metric (`lilith-engine-web-fallback`).

## Interaction primitives — validated, owned, consented

The monolith lists pickup/place, asana lock, and a hand-gesture set as
"server-validated, same across tiers." In code these are three Rust authorities
plus a UE registry.

**Pickup / place** is `InteractionAuthority` (`world-server lib.rs:1539`).
`pickup` (`world-server lib.rs:1573`) requires the actor to be a participant in
the room, the object to exist, the object's room to match the request
(**cross-room pickup is rejected** with `ObjectRoomMismatch`,
`world-server lib.rs:1476`), and the object to be unheld (else
`ObjectAlreadyHeld`); on success it sets `held_by_session_id`, bumps
`server_revision`, and emits an `interaction.pickup.accepted` event. `place`
(`world-server lib.rs:1602`) additionally requires the placing session to be the
current holder (`ObjectNotHeldBySession`) and the placed transform's `entity_id`
to match the object. The integration test
`server_validated_pickup_and_place_mutate_authoritative_object_state`
(`tests/pickup_place.rs`) walks a full grab→place cycle and checks the event
topics and sequence numbers.

**Asana lock** is a consent state machine, `AsanaLockAuthority`
(`world-server lib.rs:1795`), with status
`Requested → Consented → Locked → Released` (`AsanaLockStatus`,
`world-server lib.rs:1663`). An instructor _requests_ a lock targeting a
student; only the **student** may `consent` — the actor must equal
`student_session_id` or it fails with `ConsentActorMismatch`
(`world-server lib.rs:1863`) — after which the instructor may `activate_lock`,
and either party may `release`. Every transition appends to an audit log
(`asana_lock.requested/consented/locked/released`). The integration test
`asana_lock_covers_request_consent_lock_release_and_audit_log`
(`tests/asana_lock.rs`) asserts each status, the `consented_at_ms` stamp, and
the ordered audit trail — this is the literal encoding of the monolith's
"instructor → student-consent → server-validated → audit-logged" chain. A
parallel `tara-physical-adjustment-explicit-consent` authority gates body
adjustments the same way (both appear in `SERVICE_DESCRIPTOR.capabilities`,
`world-server lib.rs:42`).

**Hand gestures** live on the UE client in `FV3HandGestureRegistry`
(`V3/ue/Source/V3Input/Private/V3HandGestureRegistry.cpp`).
`BuildLaunchRegistry` (line 187) ships exactly seven: `namaste`, the four-mudra
set (`mudra-jnana`/`-chin`/`-varuna`/`-prana`), `applause`, and `snap`. Each
gesture is sourced from OpenXR hand tracking (`InputSignature` must start with
`OpenXR.Hands.`), names required semantic poses and ≥5 tracked joints with
per-joint confidence thresholds, and is `ClientPredictedSequenced`: it triggers
client-side within a **≤ 25 ms** budget and must _not_ wait for server
acknowledgement, then broadcasts within a **≤ 50 ms** p95 budget. `Validate`
(line 88) enforces all of this, and `ValidateLaunchRegistry` (line 332) asserts
the set is complete, ids are unique, all are client-triggered, and worst-case
broadcast p95 stays ≤ 50 ms — the same client-prediction posture as prop pickup,
applied to expression. (These gestures map to the protocol's
`GameplayActionPacket`/`InteractionPacket` arms when sent.)

## Version negotiation and failure modes

The handshake is content-based, not positional. `negotiate_protocol_version`
(`protocol crate lib.rs:189`, mirrored by TS `negotiateProtocolVersion`,
`index.ts:206`) takes the client's and server's supported `ProtocolVersion`
lists and selects the **highest mutually supported** `(major, minor, patch)`;
with no overlap it returns `NEGOTIATION_STATUS_UNSUPPORTED_VERSION` and an
`unsupported_version` error code carrying the server's supported set so the
client can report a precise mismatch. The test exercises both the accept path
(picks 3.0.0 over a shared 2.9.x) and the reject path (a 4.0.0-only client). The
shipped version is `v3.0.0` (`PROTOCOL_V3_0_0`, `index.ts:82`;
`protocol_v3_0_0()`, `protocol crate lib.rs:29`; `FV3NetProtocolVersion`
defaults to 3/0/0, `V3Net.h:52`).

Decode is fail-closed at every layer. The UE decoders return `false` on a wrong
wire type, a truncated length-delimited field, or a varint that overruns the
buffer (`ReadVarint`/`ReadLengthDelimited`, `V3NetProtocol.cpp:151`); Rust
returns a `prost::DecodeError`; and physics/interaction validators return typed
errors rather than guessing. Unknown fields are _skipped_, not fatal
(`SkipField`), which is what lets a v3.0.x client tolerate additive fields from
a slightly newer server. The `@oshun/multiplayer-protocol` package also
publishes a capability descriptor with per-stage latency budgets —
`presence-packet:8ms-budget`, `snapshot-delta:16ms-budget`,
`gateway-pin:24ms-budget` (`index.ts:31`) — and a `gatewayPin` is the contract a
session carries once the edge has bound it to a shard (`LilithGatewayPinSchema`,
`libs/contracts/src/v3/lilith.ts:33`).

## How this connects

The protocol is the seam between every other Realtime Backbone page. The gateway
terminates connections, authenticates them, and pins each session to a world
shard before any of these packets flow — see
[./world-server-and-gateway.md](./world-server-and-gateway.md) for the
connection lifecycle, JWT rotation, QUIC/WebTransport→WebSocket fallback ladder,
and voice SFU. Which surface a visitor lands on (and therefore which codec runs)
is decided upstream by
[./tier-routing-and-pixel-streaming.md](./tier-routing-and-pixel-streaming.md).
The two protocol _peers_ are [./tier1-ue5-client.md](./tier1-ue5-client.md) (the
UE C++ codec and hand-gesture registry described here) and
[./tier2-fallback-web-client.md](./tier2-fallback-web-client.md) (the TypeScript
decoder, predictor, and Rapier WASM physics). Expression intensity and voice
positioning ride the same transforms covered in
[./avatar-animation-and-audio.md](./avatar-animation-and-audio.md), and the ≤ 25
ms concert music-sync constraint that shares this tick budget is detailed in
[./saraswati-stage-pipeline.md](./saraswati-stage-pipeline.md). For where these
numbers are watched in production, see
[./observability-performance-security-and-launch.md](./observability-performance-security-and-launch.md).

## Related

- [World Server and Realtime Gateway](./world-server-and-gateway.md) — the
  connection lifecycle, sharding, auth, and voice SFU behind this protocol
- [Tier 1 UE5 Client](./tier1-ue5-client.md) and
  [Tier 2 Fallback Web Client](./tier2-fallback-web-client.md) — the two codec
  peers
- [Tier Routing and Pixel Streaming](./tier-routing-and-pixel-streaming.md) —
  which surface (and codec) a visitor receives
- [Subsystem Glossary and Layout](./subsystem-glossary-and-layout.md) and
  [Product Promise and Architecture](./product-promise-and-architecture.md) —
  where the Realtime Backbone sits in the whole
- [Avatar, Animation, and Audio](./avatar-animation-and-audio.md) and
  [Saraswati Stage Pipeline](./saraswati-stage-pipeline.md) — what rides the
  transforms and the music-sync budget
- The orientation hub [../V3_ARCHITECTURE.md](../V3_ARCHITECTURE.md) </content>
  </invoke>