Lilith World Server and Realtime Gateway

The world server and the realtime gateway are the two halves of V3's authoritative multiplayer backbone: the server that owns the truth of every room (who is where, what they touched, who may hear whom) and the edge that terminates client connections and routes their packets to the right server shard. They exist because V3 deliberately refuses to let Unreal Engine's own replication stack own room state — a single Rust authority has to speak to four very different client surfaces (native UE5, a Pixel Streaming worker, a fallback WebGPU browser client, and an operator console) over one identical wire format, so the truth can never fork per-tier. The world server lives at apps/v3/lilith-world-server/src/lib.rs (an ~215 KB axum/Tokio crate with 28 #[test]/#[tokio::test] cases plus a live-infra integration test); the gateway lives at apps/v3/lilith-realtime-gateway/src/lib.rs (~123 KB, 24 tests, real quinn QUIC). Both consume the same Protobuf contract published as @oshun/multiplayer-protocol (libs/v3/multiplayer-protocol/), and the Unreal client re-implements that exact wire format byte-for-byte in C++.

This page is the deep dive behind the "Realtime Backbone" summary in the orientation hub ../V3_ARCHITECTURE.md. It is the networking companion to Netcode Protocol and Physics (which goes deeper on prediction and reconciliation), and it sits directly downstream of Tier Routing and Pixel Streaming — once the router has decided which surface a visitor gets, the gateway is the door they actually knock on. Read it alongside Tier 1 UE5 Client and Tier 2 Fallback Web Client, which are the two protocol peers the backbone serves.

What ships, honestly#

The wire protocol, the world-server simulation library, and the gateway's transport/auth/voice library are real and tested — but the daemons that wrap them are not yet the full real-time servers the monolith describes, and the Unreal integration is not what the monolith says it is. Three honest layers:

• Real and exercised by tests. The three-language Protobuf wire format (Rust, TypeScript, and hand-rolled UE C++) is asserted byte-identical against a shared golden hex fixture. The world server's authoritative core — the fixed-tick snapshot loop, rstar R-tree interest management, rapier3d physics anti-cheat, asana-lock and physical-adjustment consent gating, consistent-hash sharding, 60 s reconnect retention, and real Postgres + Redis durable persistence with a live crash-recovery round-trip — are all genuine, deterministic, and covered. The gateway's real QUIC echo round-trip (quinn), HS256 JWT verification with one-time refresh rotation, the WebTransport→WebRTC→WebSocket fallback ladder, and the LiveKit-equivalent SFU fan-out/HRTF model are all real code with real tests.
• Scaffolded, not wired into a running server. Critically, neither binary actually runs the simulation in production form. run_service() for the world server (lib.rs:4564) binds an axum router that exposes only /healthz, /readyz, and /metrics (build_router, lib.rs:4516); the gateway's run_service() (lib.rs:2277) serves only a health body. There is no accept loop feeding live client packets into a ticking RoomRegistry, no QUIC listener bound to the session router, and no live world-server↔gateway socket path. The rich logic is library-grade and test-driven; the assembled, client-serving daemon is not here yet.
• Aspirational in the monolith / corrected here. The monolith's V3_ARCHITECTURE.md describes a UV3NetDriver (a UNetDriver subclass), a cxx-bridge V3NetTransport, and a UV3WorldSubsystem owning a local mirror table. None of those exist in V3/ue/Source. What exists is a real protobuf codec (FV3NetMultiplayerProtocolCodec) and a deterministic voice probe — described honestly below. The monolith's "zstd-1 on the delta packet" is also not implemented (there is no zstd anywhere in the V3 Rust tree); compression is delta-encoding plus Protobuf varints only. And "cross-shard chat via the V1 event bus" is, in code, an in-process Vec.

One packet shape for every tier#

Everything rests on a single Protobuf schema: libs/v3/multiplayer-protocol/proto/oshun/v3/multiplayer/v1/multiplayer.proto. It is genuinely engine-agnostic. Positions are integer millimetres (Vector3Mm with zig-zag sint32 fields), rotations are integer milli-degrees (RotationMilliDegrees), and a per-avatar AvatarExpression carries an expression id plus an intensity in basis points — so the same bytes describe a meditator's micro-smile whether they are rendered by Lumen or by three.js. Client→server traffic is a ClientEnvelope oneof (negotiation_request, presence, voice_control, interaction, gameplay_action, operator_control); server→client is a ServerEnvelope oneof that adds snapshot and snapshot_delta. Capacity is a first-class enum (CAPACITY_TIER_CLASS, CAPACITY_TIER_STADIUM).

Delta encoding is real and shared. encode_snapshot_delta (libs/v3/multiplayer-protocol/rust/src/lib.rs:85) diffs two snapshots through a BTreeMap/BTreeSet, emitting only upserted_transforms (entities that are new or whose transform changed) and removed_entity_ids; apply_snapshot_delta (:130) reconstructs the full snapshot on the receiver. Bandwidth is then gated against a per-tier budget: validate_bandwidth_budget (:169) measures average bits-per-second against CLASS_TIER_MAX_BPS = 32_000 or STADIUM_TIER_MAX_BPS = 256_000 (:10–:11). The crate's own test class_tier_snapshot_deltas_stay_under_32_kbps_for_256_participants (:461) proves 256 moving avatars fit the class budget on deltas alone — which is why zstd was never needed in practice.

The contract has three coherent implementations, and a cross-language golden proves they agree:

The Rust test protobuf_snapshot_delta_matches_cross_language_golden_wire (rust/src/lib.rs:417) and the UE automation test V3NetProtocolTests.cpp both assert equality against the same hex string (0a0a0803220676332e302e30…). The UE test is explicit about its intent — its case is titled "Snapshot delta matches Rust and Tier-2 golden wire." That shared fixture is the contract's real teeth: a drift in any one language's encoder breaks a committed test.

Lilith World Server — the authority#

The authoritative tick#

The simulation clock is FixedTickLoop (lib.rs:2532). Its tick() (:2558) runs an accumulator: each tick it adds transform_broadcast_hz and, when the accumulator crosses tick_hz, increments the snapshot sequence and produces a SnapshotBroadcastPlan. With the canonical config (50 Hz internal tick, 20 Hz transform broadcast) the test fixed_tick_loop_emits_twenty_transform_snapshots_per_second (:5607) confirms exactly twenty snapshots per simulated second. snapshot_broadcast_plan (:2577) walks every active room, builds a SnapshotPacket from each participant's EntityTransform, and accounts encoded bytes through the same encode_wire_packet the wire crate uses — so the bandwidth figure the server plans against is the literal on-wire size, not an estimate. Expression is carried separately at a higher interpolation rate (expression_interpolation_hz) for lip-sync continuity between transform frames.

Interest management with an R-tree#

A viewer never receives every entity. InterestManager::visible_entities (lib.rs:1310) bulk-loads all other participants' world-space points into an rstar::RTree, then takes the cap nearest neighbours of the viewer, where cap comes from visible_entity_cap (:2523): CLASS_TIER_VISIBLE_ENTITY_CAP = 32 or STADIUM_TIER_VISIBLE_ENTITY_CAP = 256 (:506–:507). The result is distance-sorted with a stable id tiebreak. Two load tests pin the caps: interest_manager_caps_class_visibility_at_32_nearest_entities (:5734) and stadium_interest_load_test_confirms_256_entity_cap (:5783). This is a genuine spatial index, not a "return the first N" stub — the nearest-N guarantee is what keeps a stadium-tier viewer's snapshot bounded while the room holds thousands.

Physics authority and anti-cheat#

PhysicsAuthority (lib.rs:2414) holds a real rapier3d RigidBodySet and ColliderSet; each participant is registered as a dynamic body with a humanoid capsule collider (capsule_y(0.9, 0.25)). The anti-cheat gate is validate_state_transition (:2483): it computes metres-per-second from the positional delta over elapsed_ms and rejects anything above TELEPORT_SPEED_LIMIT_MPS = 12.0 (:1353) with a PhysicsValidationError::TeleportJump carrying the offending speed. Two further authorities enforce V3's bodily-safety promises rather than mere movement: AsanaLockAuthority (:1795) refuses to lock an avatar into a guided posture without consent, and PhysicalAdjustmentConsentAuthority (:2161) gates teacher-initiated adjustments behind an explicit, audited consent dialog (PhysicalAdjustmentConsentDialog, with an audit-event trail). These are the contractual backbone behind Tara live classes — see Tara, Classes, Aja, and the Commons.

Sharding, reconnect, and cross-shard chat#

Rooms above a tier are distributed by a ConsistentHashRing (lib.rs:1078) over ShardNodes with virtual replicas; ShardedRoomDirectory (:1160) routes a room to a shard and spawn_routed_room places it. Reconnect tolerance is real: retain_session_for_reconnect (:3819) parks a departing participant for RECONNECT_SESSION_RETENTION_MS = 60_000 (:508), resume_retained_session (:3839) rejoins them if they return inside the window, and purge_expired_retained_sessions (:3877) reaps the rest. One honest correction: CrossShardEventBus (:1121) is an in-process structure (a Vec<CrossShardChatEvent> filtered by target room), not an integration with V1's event bus as the monolith implies — see V1 Integration and Identity Bridge for what the real V1 bridge does and does not cover.

Durable persistence#

This is the strongest "really runs against infrastructure" piece. The hot path stays in memory (RoomRegistry), but apps/v3/lilith-world-server/src/durable_persistence.rs adds two real backends. PostgresDurableSessionStore (durable_persistence.rs:164) uses sqlx to create and write the v3_session_boundary_writes table, persisting a JSON checkpoint envelope at each session boundary (SessionBoundaryKind::Start/Checkpoint/End). RedisHotStateStream (:281) appends each hot-state checkpoint to a real Redis Stream via XADD under the v3:hot-state prefix and reads the latest back with XREVRANGE COUNT 1. Crucially, the checkpoint's participant transforms are prost-encoded with the canonical wire bytes before being wrapped in JSON (CheckpointRecord, :83–:144), so the durable representation can never drift from the on-wire one. On crash, recover_room_from_redis_hot_state (:362) replays the latest stream entry into a fresh RoomRegistry::restore_checkpoint. The integration test durable_persistence_recovers_room_from_real_postgres_and_redis exercises a real Postgres + Redis round trip — this is not a mock.

Hot-reload of scenes and assets#

HotReloadPoller (lib.rs:787) polls a published scene-graph and asset manifest at HOT_RELOAD_POLL_INTERVAL_MS = 30_000 (:637), tracking SceneRevisions through a SceneRevisionCatalog and applying new revisions on instance-restart boundaries; FileAssetManifestSource (:760) reads manifests from disk with typed AssetManifestErrors. This is how a re-published Lilith Commons scene reaches a live shard without a redeploy — the authoring side is Authoring and Content Pipeline.

Lilith Realtime Gateway — the edge#

The transport-selection ladder#

web_client_echo_with_transport_fallback (lib.rs:1814) encodes the monolith's preference order as real branching against a WebClientNetworkProfile: if UDP is allowed it returns WebTransport over QUIC (route_note: "webtransport-quic"); else if TURN/TCP is allowed it returns a WebRTC data channel relayed over TURN/TCP; else if WebSockets are allowed it falls to a WebSocket last resort (/v3/realtime/ws, subprotocol oshun.v3.realtime+json); else it errors with "no realtime transport available." The QUIC path is not a placeholder: spawn_quinn_webtransport_echo_server (:1873) and quinn_webtransport_echo_round_trip (:1898) open a real quinn endpoint with a self-signed cert and complete a bidirectional-stream echo, decoding a real WebTransport CONNECT/draft-02 frame.

Auth handshake#

authenticate_gateway_handshake (lib.rs:434) parses a Bearer token and calls verify_auth_primitives_hs256_jwt (:2476), which does a genuine HS256 HMAC verification: it base64url-decodes header/payload/signature, rejects any header whose alg/typ is not HS256/JWT, recomputes the signature over the signing input, then validates exp/nbf, requires token_type == "access", and enforces an optional required scope. The gateway also implements full one-time refresh-token rotation (JwtRefreshRotationStore, :166), so a rotated refresh token is single-use and an explicitly revoked family is rejected (tests jwt_refresh_rotation_issues_one_time_pair_and_authenticates_rotated_access_token and jwt_refresh_rotation_rejects_explicitly_revoked_refresh_token). These JWTs are the V1 identity service's auth-primitive tokens — the bridge is documented in V1 Integration and Identity Bridge.

Session pinning and reconnect#

A session is pinned to a shard by GatewayConsistentHashRing::route_session (lib.rs:662) and held in a SessionPinDirectory (:686) for SESSION_PIN_TTL_SECONDS = 60 (:675). Transient drops survive via GatewayReconnectDirectory (:806) for GATEWAY_RECONNECT_RETENTION_SECONDS = 60 (:676), with a RECONNECT_RESUME_BUDGET_MS = 3_000 (:1068) resume target — the same 60 s window the world server honours, so a reconnecting client lands back on the shard that still holds its retained participant state.

Voice SFU and the HRTF model#

Voice routes through VoiceSfuModule (lib.rs:1474), whose compatibility mode is LiveKitServerSdkEquivalent. route_gateway_packet (:1504) validates an Opus 24 kbps mono payload, refuses muted senders, fans the packet out to every other peer in the room, and attaches per-recipient HRTF metadata. The mouth-to-ear latency is modelled explicitly:

latency_ms = OPUS_FRAME_DURATION_MS(20) + gateway_to_sfu_ms
           + sfu_fanout_ms + (recipient_regional_rtt_ms / 2)

checked against SFU_MOUTH_TO_EAR_LATENCY_BUDGET_MS = 80 and a per-region REGIONAL_VOICE_LATENCY_BUDGET_MS = 50 (:1061–:1062), with a packet-loss budget of VOICE_LOSS_RATE_BUDGET = 0.005. Operator mute/kick commands route through apply_operator_command (:1569) within an OPERATOR_SFU_COMMAND_BUDGET_MS = 200 deadline. Per-tenant voice policy (who may speak, push-to-talk vs open mic) is resolved through VoiceChatPolicyDirectory (:1346). The audio rendering itself is on the client — see Avatar, Animation, and Audio.

The Unreal side: a codec and a probe, not a NetDriver#

The monolith's network-integration section is the largest gap between description and code, so this section states what is actually in V3/ue/Source.

V3Net is a wire codec, not a UNetDriver. V3NetProtocol.cpp implements FV3NetMultiplayerProtocolCodec — a hand-written Protobuf encoder/decoder (zig-zag WriteSInt32, length-delimited WriteMessage, a ReadVarint state machine) producing bytes identical to the Rust/TS encoders. BuildInEngineProtocolExchangeReport (V3NetProtocol.cpp:958) round-trips the full packet set (snapshot-delta, presence, voice-control, interaction, plus client/server envelopes) entirely in-engine and reports bDecodedFullPacketSet. The module's V3Net.Build.cs depends only on Core/CoreUObject/Engine/ GameplayTags/V3Core — there is no NetworkCore dependency, no cxx bridge, and no quinn. The C++ structs are plain FV3Net* value types (V3Net.h), not UObject actors with replication flags. This is a real, valuable, test-covered protocol bridge — it just is not the engine-integrated net driver the monolith claims.

V3Voice is a deterministic spatialization probe. FV3VoiceRealtimeGatewayClient (V3VoiceRealtimeGateway.cpp) contains genuine positional audio math: BuildPositionalMetadata computes distance, azimuth, and elevation from millimetre relative positions, and ApplyResonanceAudioSpatialization applies a real equal-power pan law plus distance attenuation. Notably its mouth-to-ear formula (ReceiveGatewayFrame, :523) is the same as the Rust gateway's — frame + gateway_to_sfu + sfu_fanout + rtt/2 — so the two stacks agree on the latency model by construction. But ConnectToRoom (:397) validates config and participants and checks IsResonanceAudioRuntimeAvailable() via FModuleManager; it does not open a WebRTC socket. RunSixteenParticipantLatencyProbe and RunRegionalLatencyValidationProbe are deterministic in-engine probes over test fixtures, not live SFU sessions.

V3World is a scene/launch validator, not a UV3WorldSubsystem. FV3AtriumPlaceholderSceneBuilder (V3World.cpp) builds and validates a placeholder Atrium scene spec, and BuildColdJoinValidationReport produces a join-latency percentile report from deterministic phase fixtures with seeded jitter — a synthetic budget check (96 samples against a 5 s budget), not a measurement of real joins. ConnectLocalWorldServerStack validates a service name and port (lilith-world-server:43101); it is configuration validation, not a live connection. These modules are covered by UE automation (V3NetProtocolTests.cpp, V3VoiceRealtimeGatewayTests.cpp, V3AtriumPlaceholderSceneTests.cpp), and the codec test's golden-hex assertion is genuinely strong — but treat the "connects to the world server" language as a fixture, per Tier 1 UE5 Client.

A join, end to end#

The intended flow — assembled from the real functions above, even though the serving daemon does not yet wire them together — looks like this:

Capacity math: Pixel Streaming workers are clients#

The capacity model that makes V3 affordable is that a Pixel Streaming worker is itself a full multiplayer client of the world server, so fan-out scales with native_ue + px_stream_workers + tier2_fallback, not with unique humans. A 4,096-attendee stadium concert produces only ~1,025 connected world-server clients, because the crowd band consumes one multicast video stream (a single master capture worker is the client) rather than 3,072 per-attendee subscriptions. For a Tara class-tier room (≤ 64 attendees) the math is flat — every attendee is a client. This is an architectural model in the monolith and the relay's autoscaler rather than something the (health-only) world-server daemon measures today; the relay that schedules those workers is detailed in Tier Routing and Pixel Streaming.

Related#

• Netcode Protocol and Physics — prediction, reconciliation, and the physics contract in depth
• Tier Routing and Pixel Streaming — the router and GPU edge that sit in front of the gateway
• Tier 1 UE5 Client and Tier 2 Fallback Web Client — the two protocol peers the backbone serves
• Avatar, Animation, and Audio — client-side voice rendering and expression interpolation
• Tara, Classes, Aja, and the Commons — asana-lock and physical-adjustment consent in practice
• V1 Integration and Identity Bridge — the JWTs the gateway verifies and the V1 services behind them
• Saraswati Stage Pipeline, Authoring and Content Pipeline — what the hot-reload poller consumes
• Subsystem Glossary and Layout and Product Promise and Architecture — the orientation pages, and the hub ../V3_ARCHITECTURE.md
• Data Tenancy and Residency, Persona, Policy, Provenance, and Rights, Commerce and Royalties, and Observability, Performance, Security, and Launch — the cross-cutting concerns the backbone participates in