V3 — Architecture (Lilith Metaverse, Tara Studio, Saraswati Stage)

Status: V3 launch architecture only.
Source: derived from V3/V3_features.md (product surface scope) and V3/V3_TODOS.md (implementation source of truth).
Audience: engineering, platform, infrastructure, art, audio, and live-service leads building V3.
Date: 2026-05-12

This architecture reference has been decomposed into focused pages under architecture/. This file remains the canonical hub (every section heading is preserved, so existing arch§"…" anchors keep resolving); the in-depth, code-grounded companion pages live under architecture/. Start at the architecture page index. The product feature map is in V3_features.md.

Purpose#

This document describes the runtime, package, and integration architecture that V3 must ship behind. It is grounded in the V3 feature map (V3_features.md) and the V3 backlog (V3_TODOS.md), and is anchored to the Unreal Engine 5 project at V3/ue/ plus the supporting Rust services, TypeScript libraries, and V1-monorepo extensions V3 requires. Companion docs: V3/V3_features.md (product surface scope), V3/V3_TODOS.md (implementation source of truth), and V3/V3_DEPENDENCIES.md (external dependency registry).

V3 ships as a tiered client stack with Unreal Engine 5 as the canonical runtime:

• Tier 1 — UE5 canonical client. A single UE5 project drives every premium V3 surface: native desktop (Windows / macOS / Linux), native VR (Quest 3, Vision Pro, PSVR 2, SteamVR), native console (PS5 / XSX where applicable), native mobile (iOS / Android via UE Mobile), and the web via UE Pixel Streaming (server-rendered UE frames streamed to the browser over WebRTC). Tier 1 owns the full V3 quality envelope: Lumen GI, Nanite geometry, MetaHumans, Niagara VFX, MetaSounds, Sequencer-driven concerts, Chaos physics.
• Tier 2 — Lightweight web fallback (three.js). A reduced-fidelity WebGPU client at apps/v3/lilith-web-fallback/ for users on hardware too constrained for UE Pixel Streaming, on bandwidth too constrained for server-streamed video, or in regions where Pixel Streaming POPs aren't available yet. Same multiplayer wire protocol, same VRM avatars, same V1 BFF — but rendered locally with a simplified scene graph (no Lumen, no Nanite, lower-poly avatars, fewer particle systems). Tenant features degrade gracefully: Tara classes and Lilith Commons work; Saraswati Stadium-tier concerts surface a degraded preview with audio-only or pre-rendered video paths.

V3 layers on top of the Oshun V1 monorepo without forking it: the V1 BFF, contracts, identity, audit-platform, residency, isis-generation-control, sophia-evidence, lilith-persona-policy, aja-embodiment, and aje commerce packages are reused as-is or extended in-place. New V3-specific code lives under V3/ue/ (the Unreal project), apps/v3/* (supporting services and the fallback client), and libs/v3/* (shared libraries), and integrates with V1 through the existing event-bus and BFF route conventions.

Cross-Reference Convention#

Throughout the V3 docs, bare §N and §N.M references resolve to V3_TODOS.md (the only V3 doc with section numbering used as hyperlinks). Explicit doc prefixes are used where ambiguity matters:

• §40 → V3_TODOS.md §40.
• deps§14 → V3_DEPENDENCIES.md §14 (e.g., generative-music providers).
• features§"<anchor>" → V3_features.md by section anchor.
• arch§"<anchor>" → this document, by section anchor.
• v1arch§"<anchor>" → V1/ARCHITECTURE.md by section anchor.
• v1todos§N → V1/TODOS.md §N.
• v2arch§"<anchor>" → V2/V2_ARCHITECTURE.md by section anchor (UE5 patterns shared with the V2 fighting game).

The features and architecture docs are referenced by named section anchors; the backlog (V3_TODOS.md) and dependency registry (V3_DEPENDENCIES.md) are referenced by numbered section ids.

Table of Contents#

• V3 Product Promise
• Subsystem Glossary
• High-Level Architecture
• Tier Selection and Client Routing
• Project Layout
• UE5 Module Split
• Game Feature Plugins (Tenants)
• Lilith World Server
  • World-Server ↔ UE Network Integration
  • Capacity Math With Pixel Streaming Workers as Clients
  • Snapshot & delta encoding
  • Voice SFU
• Lilith Realtime Gateway
• Pixel Streaming Infrastructure
  • Per-POP Capacity, GPU Quota, and Multi-Cloud Posture
• Tier-1 UE5 Client Architecture
• Tier-2 Fallback Web Client
  • Asset baking pipeline
• Avatar Pipeline (MetaHumans + VRM)
  • Dual-Authoring Requirement for Premium Personas
  • Provenance and Identity
• Animation Pipeline
• Spatial Audio
• Multiplayer Netcode
• Physics and Interaction
• Aja Body-Aware Coaching Pipeline
• Saraswati Music Generation Pipeline
• Saraswati Live Performance Pipeline (Sequencer + MetaSounds)
  • Concert Authoring Pipeline
  • Between-Song Speech
  • Performance Flow
  • Tier-2 During a Concert
• Tara Live Class Pipeline
• Lilith Commons Pipeline
• Lilith Studio Authoring Split (Web vs. UE Editor)
• Content Pipeline and Source Control
• Data Architecture and Tenancy
  • Datastores
  • Hot / durable split, by data class
  • Tenancy and isolation
  • Residency
  • DSAR
• V1 Integration (BFF, Contracts, Auth, Audit)
• Identity and Account Bridge
• Persona Policy at the Body
• Provenance and Rights
• Commerce and Royalties
• Observability, Eval, Release Gates
• Performance Budgets
  • Cross-Tier Co-Existence
  • Stadium-Tier Scaling Strategy
  • Stadium Concerts in Tight-Quota POPs
• Build, Cook, Patch (UE5)
• Security, Privacy, Compliance
  • UE Anti-Mod and Anti-Cheat Posture
• Testing, QA, Accessibility
• Launch Readiness
• Cross-References

V3 Product Promise#

V3 ships as one embodied, multi-user surface for Oshun. Three tenants ride one substrate; the substrate carries V1's identity, memory, grounding, safety, provenance, and rights guarantees into a 3D space. UE5 is the canonical engine; the three.js fallback exists to make sure no one is locked out at the door. See V3_features.md § "V3 Product Promise" for the full list.

Subsystem Glossary#

Names below are fixed vocabulary for V3; module names, namespaces, asset paths, and game feature plugins inherit from them.

Platform substrates (V3 metaverse)

UE5 engine modules (live under V3/ue/Source/)

Game Feature Plugins (modular, hot-swap; live under V3/ue/Plugins/)

Tenant adapter packages (TypeScript-side glue)

Cross-domain extensions (V1 → V3)

Surfaces

High-Level Architecture#

Tier Selection and Client Routing#

When a user clicks "enter Lilith" from the V1 Oshun shell or an external landing page, the tier-router service (a thin route inside the V1 BFF at /api/v3/lilith/launch) decides where to send them:

1. Native UE5 client installed. Deep-link launches the local UE5 binary (Steam / Epic / App Store / Quest Store / PSN / etc.). Lowest latency, best fidelity, no streaming cost.
2. Premium browser path (UE Pixel Streaming). When no native client is installed but device + bandwidth + region capacity allow:
   • Device must report WebRTC + H.264 / AV1 decode.
   • Sustained bandwidth ≥ 8 Mbps measured during a 2-second probe.
   • A POP with available UE workers is within ≤ 60 ms. The user is matched to a Pixel Streaming worker; UE renders server-side and streams frames over WebRTC. Input is forwarded via the WebRTC data channel.
3. Fallback web client (three.js). When (1) and (2) are not available:
   • Device fails the WebRTC / decode / bandwidth probe; or
   • No Pixel Streaming POP is within the latency budget; or
   • User explicitly selects "Lite mode" from the launcher; or
   • User is in a Pixel-Streaming-restricted region (where Aje / GDPR / residency rules require local rendering). The fallback client renders locally on WebGPU (with WebGL2 secondary fallback). Same world server, same identity, same content catalog; reduced visual fidelity per features§"Tier-2 Fidelity Envelope (Known Gaps from Canonical)".
4. Static launch landing. If the browser cannot run WebRTC, WebGPU, or WebGL2, the router returns a static launch landing target with the local-rendering-unsupported reason code instead of claiming a runnable WebGL2 fallback.

The router emits a LilithLaunchDecision event with reason codes for analytics and capacity planning. Users can override the choice via a launcher dropdown.

Capacity planning notes:

• Pixel Streaming workers are sized for one UE process per concurrent browser session. A 24-vCPU GPU node hosts ~6 concurrent sessions at Tara class-tier fidelity, ~3 at Stadium-tier concert fidelity. Fleet is auto-scaled by predicted programming load + observed queue depth.
• Native clients carry no per-session GPU cost on the backend.
• Fallback clients carry no per-session GPU cost on the backend.

Project Layout#

oshun/
├── apps/
│   ├── oshun/                                  # V1 (reused, extended)
│   │   ├── web/                                # adds /v3/* routes + /lilith-studio
│   │   ├── mobile/                             # adds /v3 stack (companion + native deep-link)
│   │   └── admin/                              # adds /lilith operator console
│   └── v3/                                     # NEW
│       ├── lilith-web/                         # browser shell + Pixel Streaming player
│       ├── lilith-web-fallback/                # three.js + WebGPU lightweight fallback
│       ├── lilith-world-server/                # Rust authoritative state
│       ├── lilith-realtime-gateway/            # Rust QUIC/WebTransport + WebRTC + voice SFU
│       ├── lilith-pxstream-relay/              # Rust Pixel Streaming relay + signaller
│       ├── saraswati-service/                  # Rust music gen orchestrator + worker pool
│       └── lilith-commerce-service/            # Rust commerce, tickets, royalty waterfall
├── libs/
│   ├── v3/                                     # NEW
│   │   ├── lilith-web-pxstream/                # Pixel Streaming player wrapper (TS)
│   │   ├── lilith-engine-web-fallback/         # three.js + WebGPU + Rapier WASM wrapper
│   │   ├── avatar-pipeline/                    # VRM importer, costume slots, provenance (TS)
│   │   ├── spatial-audio/                      # HRTF / mix-bus runtime for the fallback path
│   │   ├── multiplayer-protocol/               # shared Rust + TS + UE-C++ bindings
│   │   ├── tara-studio/                        # tenant adapter (TS-side glue)
│   │   ├── saraswati-stage/                    # tenant adapter (TS-side glue)
│   │   ├── lilith-commons/                     # tenant adapter (TS-side glue)
│   │   ├── aja-pose/                           # on-device pose estimation + cue generator
│   │   ├── memory-iris-spatial/                # scene/pose memory scope (extends V1 Iris)
│   │   ├── isis-music/                         # Saraswati music gen workflow
│   │   ├── isis-motion/                        # avatar motion gen workflow
│   │   ├── isis-world-asset/                   # world-asset gen workflow
│   │   ├── psyche-3d/                          # 3D embodiment runtime extensions
│   │   ├── sophia-saraswati-grounding/         # artist-backstory grounding adapter
│   │   └── lilith-identity-bridge/             # V1 account ↔ UE session bridge
│   ├── contracts/                              # V1 (extended with v3 contracts)
│   │   └── src/v3/                             # NEW: V3-specific contracts
│   ├── oshun/                                  # V1 (reused)
│   ├── shared/                                 # V1 (reused)
│   └── ...
└── V3/
    ├── V3_features.md
    ├── V3_ARCHITECTURE.md
    ├── V3_DEPENDENCIES.md
    ├── V3_TODOS.md
    └── ue/                                     # UE5.5 LTS project (canonical V3 client)
        ├── V3.uproject
        ├── Config/
        │   ├── DefaultEngine.ini
        │   ├── DefaultGame.ini
        │   ├── DefaultInput.ini
        │   ├── DefaultEditor.ini
        │   └── PixelStreaming.ini
        ├── Content/
        │   ├── Avatars/                        # MetaHuman + VRM canonical
        │   ├── Venues/                         # Commons + Tara + Saraswati venues
        │   ├── Animations/                     # Asana, gesture, stage motion catalogs
        │   ├── Audio/                          # MetaSounds graphs, music buses
        │   ├── VFX/                            # Niagara stage / ritual effects
        │   ├── UI/                             # UMG widgets, CommonUI styles
        │   ├── Sequences/                      # Sequencer concert cinematics
        │   └── Localization/                   # All locales
        ├── Source/
        │   ├── V3Core/
        │   ├── V3Gameplay/
        │   ├── V3World/
        │   ├── V3Avatar/
        │   ├── V3Animation/
        │   ├── V3Input/
        │   ├── V3Net/
        │   ├── V3Voice/
        │   ├── V3UI/
        │   ├── V3Audio/
        │   ├── V3VFX/
        │   ├── V3Cinematics/
        │   ├── V3Persistence/
        │   ├── V3OnlineServices/
        │   ├── V3Telemetry/
        │   ├── V3Editor/
        │   └── V3Tests/
        ├── Plugins/                            # Game Feature plugins (tenants + modes)
        │   ├── V3Tenant_TaraStudio/
        │   ├── V3Tenant_SaraswatiStage/
        │   ├── V3Tenant_LilithCommons/
        │   ├── V3Mode_TaraLiveClass/
        │   ├── V3Mode_SaraswatiConcert/
        │   └── ... (one plugin per mode listed in arch§"Subsystem Glossary")
        └── Build/                              # Per-platform build outputs

UE5 Module Split#

Each module owns a clear concern and is built with *.Build.cs. C++ is the primary gameplay language; Blueprint is for designers, data assets, state graphs, and cinematic glue only. Frame-deterministic combat is not a V3 concern (V3 is a contemplative metaverse, not a twitch game), but authoritative-state code paths must be C++.

Disabled in DefaultEngine.ini to keep editor cold-start ≤ 25 s: any plugin not referenced by V3 modules.

Game Feature Plugins (Tenants)#

Tenants and modes are shipped as Game Feature Plugins so each can be hot-loaded / unloaded at runtime and so the build pipeline can include or exclude them per platform / region without engine rebuilds.

Each plugin has its own *.uplugin with EnabledByDefault: false, a GameFeatureData asset that registers gameplay tags, ability sets, world- template scenes, and Niagara/MetaSound assets specific to that mode.

Lilith World Server#

The world server (apps/v3/lilith-world-server/) is an axum-based Rust service that owns authoritative state for every active room. Responsibilities:

• Room lifecycle — spawn / join / leave / shutdown; persistence sync.
• Tick loop — fixed 50 Hz internal tick; broadcast at 20 Hz transform / 60 Hz interpolated expression.
• Physics authority — Rapier (Rust); client predicts in UE via Chaos, in fallback via Rapier WASM. Anti-cheat rejects teleports > 12 m/s, asana-lock without consent, cross-room interactions.
• Interest management — spatial AABB tree caps per-client visible-entity count; cross-shard chat via V1 event bus.
• Persistence sync — Postgres durable writes at session boundaries + 60 s checkpoints; Redis Streams hot state.
• Sharding — rooms above class tier sharded by (venue_id, instance_id); the Realtime Gateway routes clients to shards.
• Hot-reload — polls UE-published scene-graph + asset manifest at 30 s; applies new revisions on instance-restart boundaries.

The world server is engine-agnostic on the wire: UE Tier-1 clients and Tier-2 fallback clients send the same Protobuf packets defined in @oshun/multiplayer-protocol.

World-Server ↔ UE Network Integration#

UE5 ships its own replication stack (NetDriver + UE Iris). V3 does not use that stack for room state — the Rust world server is the single authority. Concretely:

• V3Net is a UNetDriver subclass UV3NetDriver registered as the default network driver for the V3 game type. It bypasses UE's built-in reliable channel system for V3World/V3Avatar actor state.
• On connect, UV3NetDriver opens a QUIC stream to the Realtime Gateway using the multiplayer-protocol Rust bindings exposed through a V3NetTransport C++ module compiled from cxx-bridge-generated headers.
• Replicated room state from the world server is applied to UE actors via a UV3WorldSubsystem that owns the local-mirror entity table and dispatches transform / expression / interaction events to the right AActor / ACharacter / UMG widget. UE's bReplicates flag is set to false on room actors; their state is driven externally.
• UE Iris is used for two narrow channels: (1) transient client-only UI / cinematic state inside the local UE process (e.g., Pixel Streaming worker driving an attendee browser through a Sequencer-side cutscene), and (2) V3OnlineServices calls into EOS (friends / parties / presence). Neither carries authoritative room state.
• Replication target rate is 20 Hz transform / 60 Hz expression interpolated client-side, matching the gateway. The UE tick budget for applying remote-entity updates is ≤ 1.5 ms per tick under class-tier capacity, ≤ 4 ms under stadium-tier capacity.

This design lets one Rust world server speak to every tier (native UE, Pixel Streaming worker, Tier-2 fallback) through a single protocol and keeps the authority surface uniform.

Capacity Math With Pixel Streaming Workers as Clients#

Each Pixel Streaming worker is itself a full multiplayer client of the world server. World-server fan-out scales with:

world_server_clients = native_ue_clients + px_stream_workers + tier2_fallback_clients

not with the count of unique users. For a Stadium-tier concert under the seating plan in arch§"Stadium-Tier Scaling Strategy":

The same 4 096-attendee concert produces only ~1 025 connected world-server clients because the crowd band consumes a multicast video stream, not a per-attendee multiplayer subscription.

For a Tara class-tier room (≤ 64 attendees) the math is simpler: every attendee is a connected client regardless of tier, so world-server client count equals attendee count.

Per-shard capacity targets follow. A single world-server shard can co-host several rooms, so per-shard client counts are aggregate across the rooms a shard holds — not the per-room capacity tiers in features§"Rooms, Sharding, Capacity":

Snapshot & delta encoding#

Same as the engine-agnostic protocol (see v1arch§"Communication Patterns" for V1's analogous design). Transform snapshots delta-encoded against the previous snapshot; per-avatar fields packed to ≤ 32 bytes; zstd-1 on the delta packet. Snapshot rate 20 Hz; expression interpolated 60 Hz client-side via 60 Hz lipsync sub-channel.

Voice SFU#

Voice routes through the Realtime Gateway's SFU module (LiveKit-compatible Rust stack). UE clients receive voice streams via the Pixel Streaming data channel (when streaming) or directly via WebRTC (native UE). UE positions voice in 3D using Resonance Audio or Steam Audio; the fallback uses Web Audio + WASM HRTF.

Lilith Realtime Gateway#

The gateway (apps/v3/lilith-realtime-gateway/) terminates client connections and forwards traffic to the correct world-server shard. It also serves the Pixel Streaming relay fleet for browser-tier sessions:

• Transport selection — WebTransport (QUIC) preferred; WebRTC data channel fallback; WebSocket last resort.
• Auth handshake — JWT from V1 identity service.
• Session pinning — consistent-hash shard pin; survives transient reconnects ≤ 60 s.
• Voice routing — voice packets to SFU module.
• Pixel Streaming bridge — when a session is served via Pixel Streaming, the gateway forwards multiplayer packets between the UE Pixel Streaming worker (acting as a server-side client) and the world server. Voice is mixed in the gateway and streamed to the UE worker; the worker spatializes via Resonance / Steam Audio and the browser receives mixed HRTF audio in the H.264 / AV1 stream.

Pixel Streaming Infrastructure#

UE Pixel Streaming makes the canonical UE5 client reachable from any WebRTC-capable browser without an install. Architecture:

Components:

• Matchmaker (apps/v3/lilith-pxstream-relay/) — Rust axum service. Decides which POP / signaller handles each browser session based on geo, load, programming priority (e.g., concert attendees in-window get priority). Issues short-lived session JWTs.
• UE Pixel Streaming Signaller — Epic's signalling server (Node.js) deployed per POP behind the matchmaker. Manages SDP exchange.
• UE Worker Process — headless UE5 client launched per browser session. One process per session at GA. Renders frames; encodes H.264 (broad compat) or AV1 (modern); streams over WebRTC; receives input over the WebRTC data channel.
• Voice path — voice mixed in the gateway and forwarded to the UE worker via an inbound audio stream; UE spatializes; outbound audio rides the same WebRTC video stream.
• Auto-scaling — fleet of GPU nodes (NVIDIA L40S / L4 / RTX 6000 Ada or AMD MI300) auto-scaled by queue depth and predicted concert / festival load. Pre-warming for scheduled concerts ≥ 30 min before start.
• POPs at GA — us-east-1, us-west-2, eu-west-1, ap-northeast-1, ap-south-1, ap-southeast-2, sa-east-1; latency budget ≤ 60 ms browser ↔ POP. See arch§"Per-POP Capacity, GPU Quota, and Multi-Cloud Posture" for the full per-POP coverage matrix.

Cost model: a 24-vCPU GPU node hosts ~6 concurrent class-tier sessions or ~3 stadium-tier sessions. Native clients carry no per-session backend GPU cost.

Quality: Pixel Streaming targets 60 fps at 1080p / H.264 4 Mbps for class-tier; 60 fps at 1440p / AV1 8 Mbps for concerts where browser decode supports AV1 (Chrome, Edge — Safari fallbacks to H.264).

Per-POP Capacity, GPU Quota, and Multi-Cloud Posture#

POP capacity is gated by GPU instance availability, not by control-plane software. The ap-south-1 and ap-northeast-1 regions have historically tight GPU quota and require active capacity management; the posture per POP:

Capacity management commitments:

• Quarterly quota review with AWS / Azure / GCP TAMs covering each POP and the next 6 months of scheduled concert / festival load.
• Pre-reservation for scheduled events ≥ 90 days in advance in tight POPs (ap-south-1, ap-northeast-1); ≥ 30 days in standard POPs.
• Spot-burst budget for Saraswati festival events that exceed steady POP capacity; spot interruption gracefully promotes affected sessions to a reserved on-demand pool with operator notification.
• Cross-region routing is honored only when the user's residency rules permit. EU users do not failover to non-EU POPs. India users prefer ap-south-1 but may route to eu-west-1 with consent during capacity shortfalls.
• Region-capacity backpressure (see features§"Pixel Streaming Abuse Posture") routes free-tier walk-ups to the Tier-2 fallback when a POP exceeds 85% utilization, preserving paid-session headroom.
• Wave-1 regions without a primary POP at GA — none. The seven GA POPs cover every wave-1 region within the ≤ 60 ms browser ↔ POP budget. Specifically: AU + NZ via ap-southeast-2; BR + AR + CL via sa-east-1; MX via us-west-2 with cross-border consent; CA via us-east-1 / us-west-2; IN via ap-south-1; JP via ap-northeast-1; EU wave-1 via eu-west-1.

This posture is documented per POP at apps/v3/lilith-pxstream-relay/config/pops/ and audit-logged on change.

Tier-1 UE5 Client Architecture#

Per-platform deployment:

Pixel Streaming uses the same UE binary that runs natively on Windows; the build pipeline produces a Win-Server cook with headless flags for the worker fleet and a normal Win64 cook for desktop users.

Tier-2 Fallback Web Client#

The fallback (apps/v3/lilith-web-fallback/) is a Next.js app rendering through libs/v3/lilith-engine-web-fallback/:

• Engine. three.js (≥ r170) on WebGPU with WebGL2 secondary fallback.
• Physics. Rapier WASM client-side prediction only; server-authoritative.
• Audio. Web Audio + Resonance Audio JS for HRTF; ambisonic fallback for low-end CPU.
• Avatars. Same VRM 1.0 source as Tier-1; rendered at a reduced LOD (≤ 80 K triangles vs Tier-1's full MetaHuman) and a simplified PBR material.
• Scenes. Same authoring source as Tier-1, automatically baked to a fallback variant (no Nanite, no Lumen — pre-baked lightmaps + simpler shaders).
• Tenant feature parity:
  • Tara live classes: full parity (lower-fi visuals).
  • Tara on-demand: full parity.
  • Lilith Commons venues: full parity at reduced fidelity.
  • Saraswati class- and hall-tier concerts: full parity with reduced particle / lighting complexity.
  • Saraswati Stadium-tier concerts: degraded preview — audio-only path or pre-rendered concert recording streamed back from Yemaya, with a note encouraging the user to upgrade to Pixel Streaming or install the native client.
• Wire protocol. Same @oshun/multiplayer-protocol Protobuf packets; full presence / voice / interaction parity.

The fallback is intentionally not a replacement for the canonical experience — it's the "no one gets locked out" path.

Asset baking pipeline#

A nightly job at tools/v3/fallback-bake/ consumes the UE5 cooked assets from staging and produces fallback variants:

• glTF 2.0 + KTX2 + Draco for meshes.
• Pre-baked lightmaps from UE5 lightmaps.
• Simplified PBR materials (no virtual textures, no parallax occlusion, no MetaHuman material expressions).
• Lower-LOD versions of every Nanite-enabled mesh.
• Opus 96 kbps stereo music streams (matching Tier-1).

Bake outputs land in s3://oshun-v3-fallback-assets/ with the same versioning scheme as Tier-1.

Avatar Pipeline (MetaHumans + VRM)#

The avatar pipeline supports two sources, both normalized to a single runtime representation:

• VRM 1.0 — canonical for community avatars, Ready-Player-Me imports, VRoid imports, and Oshun gallery avatars. Imported via a UE plugin (UE-VRM4U or an in-house port) on Tier 1 and a TS importer on Tier 2.
• MetaHumans — canonical for high-fidelity verified personas (verified Tara instructors who opt into a custom MetaHuman; all Saraswati artist primary avatars). Authored via MetaHuman Creator; MetaHuman Animator handles facial capture.

Both source types are bound to a common Oshun avatar skeleton (60 bones) with shared blendshape semantics (viseme, emotion, gaze, brow).

Dual-Authoring Requirement for Premium Personas#

The Tier-2 fallback rasterizes only the VRM 1.0 path. A MetaHuman avatar cannot render on three.js as-is — MetaHuman uses a UE-specific skeletal mesh, MetaHuman-specific facial blend-shape rig, strand-based Groom hair, and Substrate material expressions, none of which port to glTF cleanly.

Every premium persona (every Saraswati artist + any verified Tara instructor who opts into MetaHuman) therefore ships as two parallel avatar assets:

The VRM proxy is not an automated bake of the MetaHuman — automated MetaHuman → VRM conversion produces unacceptable likeness drift in practice. The proxy is authored by hand from the same reference art the MetaHuman was built from, targeting a similar silhouette / palette / proportions at ≤ 80 K triangles. Hair is rendered as card geometry, not strands; facial expressions use a reduced viseme + emotion blend-shape set; costume slots reuse the same material-slot identifiers as the MetaHuman so per-realm wardrobe rules and provenance attribution apply uniformly.

Authoring budget at GA: ~5 working days per persona for the VRM proxy once the MetaHuman master is signed off. Six Saraswati personas + a post-GA queue of Tara instructor opt-ins.

Animation retargeting: a single canonical animation library (motion-matched clips, asana poses, gesture catalog) is authored against the Oshun 60-bone skeleton. UE plays it on MetaHuman skeletons via UE's IK Retargeter; the fallback plays it on VRM proxies via a parallel retarget table at libs/v3/avatar-pipeline/src/retarget/. The retarget table is generated from the same source rig data and validated on a regression set of canonical poses.

Likeness-drift release gate: VRM proxies are reviewed by the persona's editorial owner against the MetaHuman master from three canonical camera angles (front, three-quarter, profile) before GA; drift score on a labeled gallery must be ≤ the threshold defined in §34 (Saraswati Artist Persona Authoring).

Provenance and Identity#

Every avatar carries an Isis provenance bundle (model card, prompt hash for generated avatars, importer user-id for imported); UE attaches the bundle as an asset metadata key, the fallback attaches it to the asset manifest entry. Both variants of a premium persona share the same provenance bundle id.

Identity binding (V1 user-id ↔ avatar-id) is enforced by Lilith-Identity-Bridge; UE asks the V1 BFF for the current binding on session start. Swap cooldown 24 h.

Animation Pipeline#

Per v2arch§"Animation Pipeline" patterns:

• AnimBP per archetype — neutral, kneeling, sitting, standing, lying, performing (stage), conducting (instructor pointing / demonstrating).
• Motion Matching for Lilith Commons free-roam locomotion (Mover 2.0 replaces CMC for exploration).
• Asana retargeting — 300+ canonical asana poses retargeted to the Oshun skeleton; ChooserTable selects per-asana variant by lineage tag.
• Mocap pipeline — Vicon / OptiTrack body capture (editorial); Move.AI / Plask markerless capture (iteration); Cascadeur for non-mocap stage motion; MetaHuman Animator for facial capture; Live Link for performance capture sessions.
• Hand IK — driven by OpenXR hand-tracking on VR clients; fingertip IK targets used for mudra and prop interaction.

The fallback runs the same canonical animation source at a lower frame rate (interpolated) and without micro-cloth simulation.

Spatial Audio#

Tier 1 (UE5). MetaSounds is the canonical audio runtime. Spatialization uses Resonance Audio (default) or Steam Audio (VR opt-in for higher fidelity occlusion). Music streams sync via NTP-style server timestamps with ≤ 25 ms drift target across all Stadium-tier attendees.

Tier 2 (Fallback). Web Audio API + Resonance Audio JS for HRTF; ambisonic-decoded-to-stereo fallback for low-end CPU. Identical mix-bus semantics (voice / music / effects / ambience / accessibility).

Voice chat is opus 24 kbps mono per speaker; UE clients receive directly via WebRTC; Pixel Streaming clients receive the pre-mixed audio in the H.264 / AV1 video stream (spatialization done in the UE worker).

Multiplayer Netcode#

Client-server with server-authoritative state. Rollback is not used (V3 is contemplative / performance, not twitch combat).

Per-mode tick targets:

Bandwidth budget ≤ 256 kbps per client at Stadium tier; ≤ 32 kbps in class tier.

UE-side: V3Net translates UE FNetSerialize-style state to the wire protocol Protobuf packets. UE's built-in replication is bypassed for room state (server is non-UE) but used internally for transient UI / cinematic state across Pixel Streaming clients.

Physics and Interaction#

UE Chaos is the client-side physics engine for Tier 1; Rapier WASM for Tier 2. Server-authoritative physics is Rapier on the world server.

Interaction primitives (server-validated, same across tiers):

• Pickup / place (cross-room rejected).
• Sit / lie / kneel / stand IK auto-align via Mover + AnimBP.
• Asana lock (instructor → student-consent → server-validated → audit-logged).
• Hand-gesture registry (namaste, mudra set, applause, snap).
• In-world camera with provenance-tagged output.

Aja Body-Aware Coaching Pipeline#

Aja extends the V1 embodied-instruction substrate into 3D. A student opts in to camera-based pose estimation; the camera frames are processed entirely on-device, and only derived pose features and aggregate metrics ever leave the client.

Tier-1 UE clients receive cue events over the gateway and render world-space widgets near the student avatar. Tier-2 clients receive identical cue events and render HUD text + Web Audio TTS.

Camera frames never leave the device.

Saraswati Music Generation Pipeline#

The catalog-generation pipeline is engine-agnostic — it runs entirely in the apps/v3/saraswati-service/ Rust orchestrator and the V1 substrates, with no UE dependency. Tracks are produced ahead of release, reviewed by a human curator, and stamped with an Isis provenance bundle before they appear in any client.

Saraswati Live Performance Pipeline (Sequencer + MetaSounds)#

UE Sequencer drives the canonical concert experience. A concert is a Sequencer asset (/Game/Saraswati/Concerts/<artist>/<concert>.uasset) referencing:

• Persona avatar (MetaHuman + costume binding).
• Setlist (data asset listing tracks + cue tracks).
• Stage scene (venue level + Niagara stage effects).
• Music streams (MetaSound subgraphs per song).
• Cue tracks (lighting, particles, audience-camera, persona motion preset, between-song speech triggers).
• Audience choreography (light-emoji parsing, applause beds).
• Pre-generated speech bank (see "Between-Song Speech" below).

Concert Authoring Pipeline#

A 60-minute concert with 12–15 songs and per-song cue tracks is heavy editorial work. The end-to-end pipeline:

GA-cadence target: 2 concerts per month per artist (features.md § "Scheduled Concerts and Drops") with 6 artists at GA = 12 concerts / month. Editorial staffing model in features§"Operator Staffing Model" below.

Between-Song Speech#

Between-song speech is pre-generated and queued, not live-generated during the concert. Live generation introduces latency that the concert's beat-sync planning can't tolerate and would expose policy violations to a live audience without an operator review gate.

Pipeline:

1. Speech prompts. Editorial team writes ~3–5 prompts per between-song slot (dedicate a song, acknowledge a recurring fan persona by handle, talk about an inspiration). Slots are determined by the setlist.
2. Generation. Each prompt routes through Sophia (factuality check), the persona-policy classifier (tone), and the persona's TTS voice signature. Output is mixed against the persona's voice profile and stored as a MetaSound stream in the concert's Sequencer asset.
3. Review gate. Every speech artifact is reviewed by editorial before the concert; operators can swap in alternates during the dress rehearsal.
4. Live selection. At showtime, the Sequencer plays from the pre-rendered bank. Concert director (or AI-director rule) chooses which of the slot's 3–5 alternates to play based on the live moment (e.g., pick the version that acknowledges the named fan who is actually in attendance).
5. Cold-fallback. Each slot ships with a generic neutral take in case the AI-director can't make a context-appropriate choice.

Live unscripted speech (where a fan asks the persona a question during the concert) is not supported at GA. Q&A sessions happen in a post-show green-room mode (V3Mode_SaraswatiClub) with a smaller audience and operator oversight, where pre-generation + retry budget is viable. Post-GA: streaming TTS with a 5-second policy-check buffer is on the roadmap.

Performance Flow#

1. Performance plan loader. V3Mode_SaraswatiConcert activates; Sequencer asset loaded; per-song cue tracks bound; speech bank pre-warmed.
2. Pre-show warm-up. Audience admit gate; MetaSound ambient loop; countdown UMG widget. Pixel Streaming workers for the front + mezzanine bands pre-spawn ≥ 30 min before start.
3. Stage transition. Sequencer plays entrance cinematic; performer MetaHuman + costume + lighting cue sweep.
4. Per-song execution. MetaSound graph streams the song; cue tracks trigger Niagara FX, light cues, camera angles, audience-cam sweeps; server emits NTP-style sync timestamps for the music bed. Tier-2 crowd-band attendees consume the LL-HLS multicast from the master capture worker (see arch§"Stadium-Tier Scaling Strategy").
5. Between-song speech. Sequencer plays the AI-director-selected pre-rendered speech artifact; persona-policy lock enforced; operator override available.
6. Audience interaction. Fans send light-emoji / song-request / tip / Q&A-for-green-room over the gateway; server validates and forwards as gameplay tags / Niagara cues.
7. Recording. Sequencer's MoviePipeline captures the master record on the master-capture worker; Isis provenance attached + C2PA manifest written.
8. Royalty distribution. Performance-end event triggers Aje settlement.

Tier-2 During a Concert#

For Tier-2 fallback attendees during a concert: the Sequencer-driven cue track is forwarded as a series of cue events; the fallback renders simplified particle / light effects on cue. Tier-2 attendees in the crowd band consume the LL-HLS multicast and a thin spatial seat. Tier-2 attendees in the mezzanine band (rare — typically those who chose Lite mode explicitly despite having Pixel Streaming capability) get a degraded but locally-rendered version of the stage with audio in sync with the crowd-band stream.

Tara Live Class Pipeline#

Lilith Commons Pipeline#

Commons venues re-render V1 domain experiences inside UE5 (Tier 1) with fallback variants for Tier 2. Each venue's UE level + Game Feature plugin exposes the V1 domain surface through world-space UMG widgets:

• Tara Ritual Rooms. Tara session schema rendered as Sequencer-driven ritual; multi-user presence; group meditation timer (V3UI widget).
• Nyx Observatory. V1 sky-event data → planetarium dome shader; group observation sessions; ambient cues.
• The Stacks (Nisaba). Bookshelf actors; pull-book interaction reveals passage UMG overlay with V1 Nisaba content surface.
• Veritas Debate Hall. Podium-state machine in V3Gameplay; central + radial podiums; Sophia inline citations as world-space widgets.
• Metis Lecture Hall. Tiered seating; slide projection on planar decals; cohort break-out cells (sub-levels).
• Arete Atrium of Practice. Journal walls (UMG widget arrays); cohort review circles.

Tier-2 fallback renders each venue from the baked glTF scene with the same world-space widget HTML overlays.

Lilith Studio Authoring Split (Web vs. UE Editor)#

Most V3 creators (Tara instructors, Saraswati editorial staff contracting from outside, partner studios) do not have UE5 installed and shouldn't need to. Lilith Studio therefore exposes authoring in two surfaces with a clear split, and a synchronization layer that keeps both consistent:

Implications:

• Tara instructors author entirely from the web. Asana sequences, schedules, recordings opt-in, profiles all live in the V1 web Studio subroute (apps/oshun/web/src/app/lilith-studio/tara/*). The UE5 client renders the published AsanaSequence manifest at runtime; no UE installation needed by the instructor.
• Saraswati editorial staff author proposals from the web and delegate UE-side cue track authoring to the V3 in-house concert production team. Cross-team handoff happens through the Sequencer assembly task in the authoring pipeline (see arch§"Concert Authoring Pipeline").
• Venue and Niagara authoring requires UE. Environment artists and VFX leads work in UE Editor; their output is published to MinIO and pulled by every UE client + the fallback bake job.
• MetaHumans require UE + MetaHuman Creator. VRM proxies for premium personas can be authored without UE (in VRoid / Blender + web upload to Lilith Studio).
• Web ↔ UE sync. Lilith Studio publishes everything through V1's environment-promotion gates. The UE Editor opens content through the Lilith Studio publish queue, not directly from MinIO; this keeps editorial gates uniform.

This split is the source of truth for "who edits what where" — if a new authoring surface is proposed, it's added here with its primary surface designated.

Content Pipeline and Source Control#

UE assets (uasset, umap) live under V3/ue/Content/ and are version-controlled via Perforce following standard UE practice (binary asset support; per-file locking). Code modules (V3/ue/Source/) and game feature plugins (V3/ue/Plugins/) live in git alongside the rest of the monorepo and are mirrored to Perforce on commit for asset coexistence.

• Asset publish. Lilith Studio publishes from the UE Editor through the V1 Studio editorial workflow; staging → preview → prod via V1's environment-promotion gates.
• Hot-reload. UE clients hot-reload Game Feature plugins on a release cadence; world server polls the asset manifest at 30 s for fallback-bake invalidation.
• Fallback bake. Nightly job consumes UE cooked assets and produces glTF + KTX2 + lightmap variants for Tier-2.

Source control rules from v2arch§"Content Pipeline & Source Control" carry over: Perforce ignore lists, asset-naming conventions, automated asset linting, derived data cache.

Data Architecture and Tenancy#

V3 stores no state of its own outside the V1 data substrate. Every V3 datastore is a database, stream, or bucket already operated for V1 — V3 adds schemas, stream topics, and bucket prefixes rather than new infrastructure. State is split along a hot / durable boundary: low-latency mutable session state in Redis, the authoritative record of truth in PostgreSQL, and large binary assets in object storage behind the CDN. Two PostgreSQL capabilities carry specialized load — the pgvector extension for discovery embeddings and TimescaleDB hypertables for eval telemetry — but they are columns and tables inside the same Postgres cluster, not separate stores.

Datastores#

The world server (arch§"Lilith World Server") writes durable Postgres rows at session boundaries plus 60-second checkpoints, and keeps the live tick state in Redis. A room's checkpoint is owned by the room, not by any one attendee, so it is written to the data plane of the room's tenant realm (the realm that owns the venue); attendees from other residency zones interact with the room through the gateway without their durable per-user rows leaving their own zone. If a world-server shard restarts, it rehydrates room state from the last Postgres checkpoint and the asset manifest; no hot state is the record of truth.

Hot / durable split, by data class#

Transient avatar motion is never persisted; "avatar history" in a DSAR export means the avatar-binding, costume, and swap history rows, not transform data. Live voice and text exist only in a rolling 60-second buffer except where a segment is flagged by moderation or captured into a recording — those segments become durable Postgres rows linked to the moderation case or the recording's provenance bundle.

Tenancy and isolation#

V3's three tenants — Tara Studio, Saraswati Stage, Lilith Commons — are modeled as tenants in the V1 tenancy graph (Tenant, TenantHierarchyEdge, TenantPolicyInheritance). V3 does not invent a parallel tenancy model:

• RBAC. Operator, editorial, instructor, and artist roles are V1 roles scoped to a tenant. A Tara editorial operator cannot act on Saraswati persona dossiers; a Saraswati editorial operator cannot approve Tara instructor verifications. Lilith-Safety is a shared pool with cross-tenant scope for in-the-moment moderation, but the tone-policy rule sets it applies are owned per tenant (see features§"Operator Staffing Model").
• Row-level scoping. Every v3-schema row carries a tenant_id. Catalog, recommendation, and feature-flag queries filter on it; the V1 tenant-isolation test suite is extended with V3 fixtures and is launch-gating.
• Room-boundary segregation. A room belongs to exactly one tenant realm. Shards may co-host rooms from different realms for packing efficiency, but interest management, presence, replication, and chat are all scoped to a single room — no event crosses a room boundary — so realms stay logically segregated even on a shared shard. This is logical segregation by room scope, not physical per-realm shard partitioning.
• Generation isolation. Isis generation jobs (music, motion, world asset) carry the requesting tenant; provenance bundles record it; royalty waterfalls settle within the tenant that owns the persona or instructor.

Residency#

V3 inherits V1's residency model end to end. Every request carries a residency context derived from the user's primary data plane; the BFF, world server, Pixel Streaming matchmaker, generation pipelines, and provider calls all honor it:

• Storage residency. A user's durable rows (attendance, ledgers, consent, recordings they appear in) live in their residency zone's data plane. Postgres and S3 are partitioned by zone.
• Compute residency. The tier router pins EU users to EU Pixel Streaming POPs even at higher latency; users in Pixel-Streaming-restricted regions are routed to the locally-rendered Tier-2 fallback (see arch§"Tier Selection and Client Routing"). Cross-region routing is offered only where the user's rules permit and only with explicit consent, recorded as a V1 ConsentRecord.
• Voice / video residency. Voice transcripts and recording media stay in the residency zone; the Pixel Streaming worker for an EU session runs on an EU POP, so server-side rendering never moves a user's frames out of zone.

DSAR#

V3 session data flows into V1's existing DSAR pipeline. A DSAR export for a V3 user includes avatar history, in-world transcripts, attendance, signed-edition holdings, recordings the user appears in, Pixel Streaming session telemetry, consent decisions, and the ticket / refund / credit ledger. Deletion cascades across Postgres rows, S3 media (recording re-render with the user redacted), and the Redis hot state, and is recorded in the audit platform.

V1 Integration (BFF, Contracts, Auth, Audit)#

V3 rides on the V1 monorepo without forking it. The integration points:

BFF routes. V3 reuses V1's apps/oshun/bff and adds a /api/v3/lilith/* route family for sessions, classes, concerts, artists, venues, commerce, studio authoring, and the operator console. Two routes are specific to the tiered client stack:

• /api/v3/lilith/launch — the tier-router decision endpoint (see arch§"Tier Selection and Client Routing").
• /api/v3/pxstream/match — the Pixel Streaming session matchmaker (proxies to the lilith-pxstream-relay service).

The world server and tenant services are not exposed directly to clients; all catalog, identity, billing, and scheduling calls go through the BFF, and all real-time traffic goes through the Lilith Realtime Gateway.

Contracts. V3-specific Zod contracts live under libs/contracts/src/v3/, alongside the V1 contracts and sharing the V1 OpenAPI codegen. V3 adds (among others) LilithLaunchDecision, PxStreamSession, PxStreamWorker, AsanaSequence, Setlist, Track, ArtistPersona, AvatarBinding, and ProgramSlot. Contracts that V3 shares with V1 (identity, billing, residency, consent) are reused unchanged.

Auth. Auth uses the V1 JWT primitive. The world server validates the JWT on every connect; the Realtime Gateway re-validates on transport handshake; UE V3OnlineServices bridges the same JWT through the platform OAuth bridge for native console / headset sign-in flows.

Audit. Every operator action, persona-policy decision, generation job, ticket transaction, consent state change, and royalty settlement is written to V1 @oshun/audit-platform with V3-scoped event types, so V3 inherits V1's audit-query, DSAR, and retention tooling rather than duplicating it.

Event bus. V3 services publish and subscribe on V1's @oshun/event-bus (Redis Streams), so cross-shard presence, takedown cascades, and continuity writes into V1 domains (Arete practice logs, Iris memory) flow over the existing bus rather than a V3-private channel.

Identity and Account Bridge#

libs/v3/lilith-identity-bridge/ (Lilith-Identity-Bridge) is the single translation layer between a V1 Oshun account and an embodied V3 session. There is no V3-private account store — a V3 user is a V1 user, and the bridge resolves the embodied identity on top of that account.

Bridge responsibilities:

• Account resolution. On session start, the UE client (or Tier-2 fallback) presents a V1 JWT; the bridge resolves it to the V1 user-id and the user's residency zone. Native console / headset sign-in (Quest, Vision Pro, PSVR 2) comes through the platform OAuth bridge: the headset issuer's account is mapped to the V1 account, the external-account key is persisted, and a v3.account_avatar.linked audit event is written.
• Avatar binding. Exactly one active avatar per session. The active avatar-id is persisted as v3.activeAvatarId on the immutable V1 profile. Avatar swap is gated by a 24-hour cooldown enforced server-side (an anti-harassment measure — see features§"Identity and Account Bridge").
• Display name. Defaults to the V1 Oshun display name. A per-realm override is allowed (anti-stalking); the override is sanitized, written to the V1 audit log, and surfaced as a row in the Lilith Operator Console.
• Reputation. Derived from instructor ratings, audience reactions, and operator-visible review counts. Never exposed as a numeric score — the bridge emits graded bands (new / known / trusted / flagged) only.
• Presence privacy. The bridge carries the user's "online in Lilith" visibility setting (V1 default plus per-realm override) into the presence digest the world server broadcasts.

The bridge is server-authoritative for every decision; UE and the fallback only render the resolved identity. Bind / unbind / override / swap events all carry into V1 audit, and DSAR export of a V3 user includes UE session data and Pixel Streaming session telemetry alongside the V1 account record.

Persona Policy at the Body#

Persona and tone policy at the body is enforced by @oshun/lilith-body-policy, a server-side adapter over V1's @oshun/persona-policy-lilith. The adapter is server-authoritative: UE clients run only client-side previews (a visual costume-rule check before a costume change commits), and every binding decision is made on the world server or the relevant tenant service.

Invocation points:

AI-persona enforcement. Each of the six GA Saraswati AI artists has a policy profile generated from its signed avatar-roster dossier. The profile enforces a stay-in-persona lock for safe prompts, gracefully no-ops role-swap and hidden-instruction attempts, and hands medical / legal / financial / public-figure-voice-copy / political-persuasion requests to V1 Lilith operator queues. Supervised Tara AI instructor personas carry the same enforcement plus their human sponsor's editorial lock. The enforcement path is verified by an adversarial prompt suite (36 persona × prompt cases at GA) that must produce zero policy breaches as a release gate.

Crisis-language detection runs on the same voice-transcript and text-chat stream; a confirmed signal triggers the in-world soft-prompt and hands off to the V1 crisis-routing pipeline with an operator notification.

Provenance and Rights#

Lilith-Rights is the in-world arm of V1's provenance and takedown machinery. Every generated artifact created inside Lilith carries an Isis provenance bundle, and exports carry C2PA manifests.

Provenance bundle. Attached to every generated asset — music tracks, avatar motion, costumes, scenes, avatars, and in-world camera captures. The bundle records the model card, prompt, seed, reference set, watermark hash, consent id, performing identity, room / venue id, timestamp, and geo-residency tag. On Tier 1 the bundle is stored as a UE asset-metadata key; on Tier 2 it is stored in the asset-manifest entry. Both variants of a premium persona (MetaHuman master and VRM proxy) share one provenance-bundle id, and the bundle index is a Postgres table in the v3 schema.

C2PA. Video and audio exports — concert recordings, on-demand class recordings, signed-edition media, in-world screenshots and clips — are written with a C2PA manifest so the asset is verifiable off-platform.

Takedown cascade. A provenance withdrawal at the track level cascades within a 24-hour SLA across every surface the artifact touches: archived live performances are flagged historical, in-world replays are paused, signed editions are marked historical and resale-royalty routing is paused, and off-platform reshares trigger takedown notices. The cascade is driven over the V1 event bus, so a withdrawal initiated from the V1 takedown pipeline fans out into V3 automatically. Confirmed unlicensed-use claims (adjudicated by Themis) trigger a track unrelease plus royalty restitution.

Consent and rights records. Voice-clone contributor agreements, instructor verification dossiers, and recording consent ledgers are stored encrypted in Lilith-Rights with V1 audit coverage; access is gated by the Lilith-Rights operator scope and every access event is audit-logged.

Commerce and Royalties#

apps/v3/lilith-commerce-service/ (Rust) is the commerce engine for all three tenants. It is built on V1 Aje (crypto substrate) and the V1 payment substrate, and writes its ledgers to the v3 Postgres schema with on-chain mirroring via Aje.

Payment rails:

• Stripe Connect — fiat payouts to instructors and artists in supported regions; fiat ticket / subscription purchase.
• Aje — crypto payment, royalty waterfalls, and signed-edition mint; on-chain ownership and audit.
• Apple IAP / Google Play Billing / PSN Wallet bridges — used where a platform's store policy requires its own billing flow; entitlements are delivered through the same commerce ledger so a purchase reconciles identically regardless of rail.

Capabilities:

• Tickets. Named on issue, capped per fan, transferable only through a signed-return-only resale flow with no-markup anti-scalp checks.
• Tips. Per-instructor / per-artist / per-session, routed in fiat or crypto with payer, recipient, and platform receipts.
• Subscriptions. Tara passes, Saraswati Club passes, Commons memberships; recurring billing on Stripe or Aje.
• Cancellation / refund. The three-tier refund schedule from features§"Cancellation, Refund, and No-Show Policy" — full / partial+credit / no-refund — with a 90-day Lilith-credit ledger; platform-fault cancellations add goodwill credit and pause the affected concert's royalty waterfall.
• Royalty waterfall. Per-track royalty splits are encoded at generation time; per-stream and per-performance payouts settle per quarter on-chain via Aje, routing a contributor share to the human creators (voice contributors, training-data rights-holders) whose work the persona depends on.
• Signed editions. Limited-supply minting via the Aje signed-edition primitive; resale enforces secondary-sale royalty routing back to creators; a track takedown flags outstanding editions as historical.

Every commerce action — purchase, refund, credit, chargeback, payout, royalty settlement — is audit-logged into V1 @oshun/audit-platform, and the full ticket / refund / credit ledger is covered by DSAR.

Observability, Eval, Release Gates#

OTel end-to-end (UE clients via OpenTelemetry C++; Pixel Streaming workers; fallback clients; gateway; world-server; tenant services; V1 substrates; data stores). Trace IDs span the full lifecycle, so a single trace covers a browser session from the tier-router decision through the Pixel Streaming worker, the gateway, the world-server shard, and any tenant-service call it triggers.

Beyond the standard per-service RED metrics, V3 emits these tier-stack-specific metrics:

• v3_pxstream_session_count per POP.
• v3_pxstream_session_p99_latency browser ↔ POP.
• v3_pxstream_worker_utilization per POP.
• v3_pxstream_decode_failure_rate per browser type.
• v3_tier_router_decision_share (native / pxstream / fallback / lite).
• v3_fallback_session_count.
• v3_fallback_session_quality (FPS distribution).

Release gates (defined in §75, Launch Readiness):

• Tier-1 UE5 cert approvals across every platform.
• Pixel Streaming p99 ≤ 60 ms RTT and ≥ 99.5% session-start success at 5× expected concurrency for 7 consecutive days.
• Fallback path passes WCAG 2.2 AA, performance budgets, feature parity checks for class- and hall-tier rooms.
• Cross-tier wire-protocol parity tests green.

Performance Budgets#

Latency and sync targets are per-tier. Cross-tier co-existence in the same room is bounded separately and is structurally looser — see "Cross-Tier Co-Existence" at the end of this section.

Cross-Tier Co-Existence#

Different tiers in the same room experience different end-to-end latencies and cannot be held to the same instantaneous-sync bound. Concrete implications and design accommodations:

• Music. Each tier hears the music with its own latency offset. Drift within a tier is held to ≤ 25 ms (the sync target). Drift between a Pixel Streaming attendee and a native-UE attendee is structurally larger — video encode + decode adds ≥ 100 ms. We treat music as ambient-shared rather than precisely-shared across tiers: clapping, cheering, light-emoji are not beat-synchronized across tier boundaries.
• Voice chat. Voice between two native-UE clients clears the 80 ms bound. Voice that traverses a Pixel Streaming worker adds the video-stream codec round-trip on the listener side. Stage performers (instructors, Saraswati personas) always use native-UE clients or workstation-class Pixel Streaming with reserved bandwidth, so performer-to-audience voice stays under the conversational threshold. Audience-to-audience voice between tiers is tolerated but not optimized.
• Interaction. Asana lock, prop pickup, ticket gate, podium turn — all authoritative on the world server and therefore see only the gateway RTT for authority. Video-stream latency on the Pixel Streaming path adds to the user's perceived response time but not to the authority decision.
• Audience choreography (light-emoji bursts). Server batches emoji events into a 200 ms window before broadcasting, smoothing the visible band across tiers.

Stadium-Tier Scaling Strategy#

A 4 096-attendee Stadium-tier concert cannot put every attendee on a dedicated Pixel Streaming worker — that would require ~1 366 GPU nodes (at ~3 sessions per 24-vCPU node) and is unaffordable. The actual seating plan is:

Native UE5 attendees with installed binaries render locally and stay in the front-256 or mezzanine band based on ticket tier. Tier-2 fallback attendees default to the crowd band; they receive the LL-HLS stream alongside a fallback-rendered spatial seat (avatar + thin Niagara-equivalent particle approximation).

The multicast LL-HLS stream is produced once per concert by a single "master capture" UE worker running the canonical Sequencer at full fidelity; Yemaya transcodes that worker's output into the HLS ladder and fans it out to a CDN. This collapses crowd-band GPU cost from O(N) to O(1).

Front-256 and mezzanine-band seats are sold through Lilith Commerce (see features§"Tickets, Tips, and Subscriptions"); crowd-band seats default to free or low-cost; upgrade-to-mezzanine is a ticket-resale path.

Approximate GPU cost at GA for one Stadium-tier concert. The two Pixel Streaming bands run at different fidelity presets, so sessions per 24-vCPU node differ:

Native-UE-installed attendees consume zero backend GPU cost and reduce the front + mezzanine worker count proportionally. At a typical native-UE install rate of 40% across the front + mezzanine bands, the worst-case ~$1 075/hr drops to ~$650/hr. Front-band tickets are priced to absorb at least the worst-case GPU cost ($427 / 256 ≈ $1.70/attendee/hr); mezzanine-band tickets absorb the mezzanine cost proportionally; crowd-band tickets carry near-zero GPU cost so they can be free or low-cost.

The crowd-band design degrades interactivity (spectator with chat / emoji only) but is honest about what 4 000+ concurrent interactive UE workers actually cost and routes the overflow into a workable production model.

Stadium Concerts in Tight-Quota POPs#

A concert hosted in ap-south-1 (capacity 250 PS sessions) or ap-northeast-1 (300 PS sessions) cannot serve a Stadium-tier worst case in-region (1 025 PS sessions). Three concrete accommodations:

1. Per-POP stadium capacity cap. Concerts hosted in tight-quota POPs ship with a smaller front + mezzanine band size at GA: front ≤ 96, mezzanine ≤ 192 (≈ 288 PS sessions in worst case, fitting within ap-northeast-1 capacity with room for non-concert traffic). The crowd band is unaffected (LL-HLS multicast scales independently). The concert remains a "stadium-tier event" but with a smaller premium-tier seating area.
2. Native-UE assumption. If the region's tracked native-UE install rate is ≥ 60%, the operator may opt to keep the full 256 + 768 band sizes since the PS worker demand falls below capacity. The decision is recorded in the concert's pre-show capacity plan.
3. Cross-region burst with consent. EU / NA users do not failover across residency boundaries. APAC users without strict residency rules may be offered a cross-region routing consent prompt to reach a higher-capacity POP (e.g., a Tokyo-based concert routes Korean attendees to Tokyo as expected, but Indian attendees may be offered routing to eu-west-1 with consent during shortfalls).

The concert scheduling editorial flow surfaces the chosen accommodation during the concert's pre-show capacity-plan step; users see the applicable front-band capacity in the ticket UI before purchase.

Build, Cook, Patch (UE5)#

V3's UE5 build and cook pipeline follows the same shape as the V2 fighting game's (v2arch§"Build, Cook, Patch"), adapted for V3's Pixel Streaming worker target and Tier-2 fallback bake:

• Engine version. UE 5.5 LTS, locked across the launch window.
• Build farm. Distributed compile via UE Build Tool + IncrediBuild (Windows) or FASTBuild (cross-platform).
• Cook targets at GA. Win64, Mac (Apple Silicon), Linux, Win-Server (Pixel Streaming workers), iOS, Android, Quest, visionOS, PS5, XSX (post-GA if appetite warrants).
• Pak files. Chunked by Game Feature plugin so the tenant for the user's destination loads first.
• Patch pipeline. UE patcher + Iostore; delta patches per platform.
• Pixel Streaming worker image. Headless cook of the Win-Server target baked into a container with the GPU runtime; pushed nightly.
• Fallback bake. Nightly job at tools/v3/fallback-bake/.

Security, Privacy, Compliance#

V3 inherits V1's security, privacy, and compliance posture — JWT auth, residency enforcement, encrypted-at-rest storage, the audit platform, and the DSAR pipeline all carry over unchanged. The additions below are specific to the tiered client stack, principally Pixel Streaming:

• WebRTC DTLS for Pixel Streaming sessions; per-session short-lived credentials.
• Per-region routing rules (e.g., GDPR-region browser sessions route only to EU POPs even at higher latency).
• Pixel Streaming worker images carry no persistent data — session state is fully ephemeral; logs redact user content (no avatar pose / voice transcript retention beyond the rolling 60 s buffer).

UE Anti-Mod and Anti-Cheat Posture#

V3 is a contemplative metaverse, not a competitive twitch game, so the threat model is abuse and impersonation, not skill cheats. The posture is light-touch but explicit:

• Authority is server-side, always. Every room state mutation — prop pickup, asana lock, attendance, ticket gate, podium turn, signed-edition transfer — is decided on the Rust world server. A modified client can only show different things locally; it cannot affect other users, ticket inventory, or commerce. This is the primary defense and it does not depend on client integrity.
• Persona / costume integrity. Avatar swap, costume slot changes, and persona-impersonation attempts route through Lilith-Rights at the server. A modded client cannot present a Saraswati persona avatar to other users without the server endorsing it.
• Voice-similarity check on persona impersonation attempts (see features§"Voice Cloning Boundaries").
• No client-side integrity attestation at GA. EasyAntiCheat / BattlEye are not enabled by default — they're invasive, add user friction, and are mismatched with V3's threat model. Native UE builds do enable platform-level integrity (App Sandbox on iOS / Vision Pro, Quest Store signing, Sony cert) which is the appropriate bar.
• Pixel Streaming sessions are inherently server-integrity-bounded — the UE process runs on Oshun infrastructure, not the user's machine, so client-side modding is not possible.
• Tier-2 fallback runs untrusted client code, same as V1's web surfaces — no special defense beyond V1's existing posture, since authority is always server-side.
• Reverse engineering of UE binaries is tolerated for inspection and accessibility tooling (mods that improve a user's own experience without affecting others) and prohibited for abuse / impersonation / commerce evasion (handled through Lilith-Safety reports and account-level action).
• Anti-grief defenses sit in Lilith-Safety: rate limits on interactions, abuse-signal classifier on Pixel Streaming worker behavior, per-user / per-IP caps (see features§"Pixel Streaming Abuse Posture").
• Post-GA, if Saraswati Stage develops a competitive ranking layer (e.g., remix-chart competitions, fan-curation leaderboards), the posture is revisited and client-side attestation may be added for those specific surfaces.

This is set deliberately at the lower-friction end of the spectrum. V3 does not chase the V2 fighting-game anti-cheat profile (v2arch§"Anti-Cheat") because the contemplative-tenant threat model doesn't require it.

Testing, QA, Accessibility#

UE-specific test layer:

• Automation Spec runner for gameplay logic.
• Gauntlet end-to-end (concert run-throughs, class run-throughs, cross-platform smoke).
• Functional Test Map suites per game feature plugin.
• Golden replay harness for concerts and asana sequences.

Plus the engine-agnostic harness (Rust load-gen, music-sync drift harness, Aja pose accuracy harness, persona-policy adversarial suite, axe-core, screen-reader audit) covering both Tier 1 and Tier 2 surfaces.

Launch Readiness#

See §75 (Launch Readiness) for the gating checklist. High-level for the tiered stack:

• UE5 cert approvals across Win / Mac / Linux / iOS / Android / Quest / Vision Pro / PSVR 2 / SteamVR.
• Pixel Streaming fleet at 5× expected concurrency for 7 consecutive days with p99 ≤ 60 ms RTT and ≥ 99.5% session-start success.
• Fallback path passes WCAG 2.2 AA + performance budgets + cross-tier feature parity tests for Tara / Commons / Saraswati class- and hall-tier rooms.
• ≥ 12 verified Tara instructors live; ≥ 6 Saraswati artists with full debut catalogs; ≥ 8 Commons venues open.
• Operator console fully staffed for GA week with documented runbooks.
• Generation provider failover drilled; provenance inspector signed off.
• Eval-set release gates green; adversarial scan green across UE source, Rust services, and TS packages.

Cross-References#

• V3 features: V3/V3_features.md (named anchors).
• V3 backlog: V3/V3_TODOS.md (§N).
• V3 dependencies: V3/V3_DEPENDENCIES.md (deps§N).
• V1 architecture: V1/ARCHITECTURE.md (v1arch§"<anchor>").
• V2 architecture: V2/V2_ARCHITECTURE.md (v2arch§"<anchor>") for UE5 patterns shared with the V2 fighting game.
• V1 backlog: V1/TODOS.md (v1todos§N).
• V1 features: V1/features.md (v1features§"<anchor>").