Lilith Metaverse · Reference & analysis

V3 — Dependency Registry (Lilith Metaverse, Tara Studio, Saraswati Stage)

This file is the V3-scoped dependency contract.

46sections30 minread44tables

On this page

Scope: Dependencies that V3 launch depends on — Unreal Engine 5.5 LTS and its plugins, Pixel Streaming infrastructure, middleware, generative-AI providers (music, voice, motion, world-assets), XR SDKs, hardware peripherals, cloud / infrastructure services, language runtimes, build tooling, and the V1 internal packages on the V3 critical path. The Tier-2 three.js fallback stack is included here as a constrained sub-tree. Out-of-V3 dependencies are catalogued in the whole-monorepo /DEPENDENCIES.md and in V1/DEPENDENCIES.md. Companion docs: V3/V3_features.md, V3/V3_TODOS.md, V3/V3_ARCHITECTURE.md. Date: 2026-05-12

Purpose#

This file is the V3-scoped dependency contract. For each dependency it answers three questions a reviewer needs before launch:

  1. What is it — one sentence on the dependency itself.
  2. Where does V3 use it — the V3 features or TODOS sections that need it.
  3. How is it wired — env vars, package paths, ports, version source.

Pinned versions live in V3/ue/V3.uproject, each plugin's *.uplugin, each *.Build.cs, the per-platform configs in V3/ue/Config/, pnpm-workspace.yaml (TypeScript), each Rust service's Cargo.toml, and the per-service Docker compose files. This document references those sources of truth rather than duplicating them.

Cross-Reference Convention#

Throughout the V3 docs, bare §N and §N.M references resolve to V3_TODOS.md (the canonical backlog). Within this file, bare numeric references (e.g., §3, §14) point to this document's own sections. Explicit doc prefixes are used where ambiguity matters:

  • todos§40V3_TODOS.md §40.
  • arch§"<anchor>"V3_ARCHITECTURE.md by section anchor.
  • features§"<anchor>"V3_features.md by section anchor.
  • v1deps§NV1/DEPENDENCIES.md §N (where V3 reuses V1 dependencies as-is).
  • v2deps§NV2/V2_DEPENDENCIES.md §N (UE plugins shared with V2).

Table of Contents#

  1. Internal Packages — V3 Modules
  2. Internal Packages — V1 Foundations Reused
  3. UE5 Engine Version
  4. UE5 First-Party Plugins
  5. UE5 Third-Party Plugins
  6. Pixel Streaming Infrastructure
  7. Audio Middleware (UE5)
  8. Application Runtimes
  9. Backend Frameworks (Services)
  10. Web Shell + Pixel Streaming Player
  11. Tier-2 Fallback Web Client Stack
  12. Mobile (Companion + UE Mobile)
  13. VR / AR / XR Stack
  14. Generative AI — Music
  15. Generative AI — Voice and Speech
  16. Generative AI — Motion and Avatar
  17. Generative AI — World Assets
  18. Generative AI — Text and Lyrics
  19. Pose Estimation and Embodied Coaching
  20. Multiplayer Networking and SFU
  21. Physics
  22. Spatial Audio and DSP
  23. Animation & Mocap Pipeline
  24. Music Licensing and Rights Infrastructure
  25. Avatar Source Pipelines
  26. Data Stores
  27. Object Storage and CDN
  28. Identity, Auth, and Provisioning
  29. Payment, Billing, Royalties
  30. Provenance and C2PA
  31. Observability and Telemetry
  32. Search and Vector Stores
  33. Inbound and Outbound Channels
  34. Cloud Providers and Infrastructure
  35. Build, Cook, Patch (UE5)
  36. CI/CD and Build Tooling
  37. Source Control
  38. Testing Tooling
  39. Security and Secrets
  40. Containerization and Local Development
  41. Compliance, Age Gating, Rating Boards
  42. Hardware Peripherals
  43. Esports / Tournament / Streaming Toolkit (post-GA)

1. Internal Packages — V3 Modules#

Package Path What V3 uses it for
UE5 project V3/ue/ The canonical V3 client; one project, every premium surface.
@oshun/lilith-web-pxstream libs/v3/lilith-web-pxstream/ Browser-side Pixel Streaming player wrapper (TS).
@oshun/lilith-engine-web-fallback libs/v3/lilith-engine-web-fallback/ Tier-2 web fallback engine: three.js + WebGPU + Rapier WASM + audio runtime.
@oshun/avatar-pipeline libs/v3/avatar-pipeline/ VRM 1.0 importer + Oshun rig binding + costume slots + provenance (shared TS surface).
@oshun/spatial-audio libs/v3/spatial-audio/ Tier-2 HRTF rendering, ambisonic fallback, mix buses, voice chat client (Web Audio + WASM).
@oshun/multiplayer-protocol libs/v3/multiplayer-protocol/ Shared wire format: Rust crate + TS bindings + UE C++ bindings.
@oshun/tenant-tara-studio libs/v3/tara-studio/ Tara Studio TS-side glue (booking, marketplace, V1 BFF adapters).
@oshun/tenant-saraswati-stage libs/v3/saraswati-stage/ Saraswati Stage TS-side glue (persona dossier, performance plan driver, fan-economy adapters).
@oshun/tenant-lilith-commons libs/v3/lilith-commons/ Lilith Commons TS-side glue (venue catalog, programming calendar bindings).
@oshun/aja-pose libs/v3/aja-pose/ On-device pose estimation, asana classifier, alignment cue generator.
@oshun/memory-iris-spatial libs/v3/memory-iris-spatial/ Scene/pose memory scope; extends V1 Iris.
@oshun/isis-music libs/v3/isis-music/ Music generation workflow registered with V1 Isis control plane.
@oshun/isis-motion libs/v3/isis-motion/ Avatar motion generation workflow.
@oshun/isis-world-asset libs/v3/isis-world-asset/ World-asset generation workflow.
@oshun/psyche-3d libs/v3/psyche-3d/ 3D embodiment runtime extensions: lipsync, expression coherence, gesture (TS glue).
@oshun/sophia-saraswati-grounding libs/v3/sophia-saraswati-grounding/ Sophia adapter for grounding Saraswati artist backstories.
@oshun/lilith-identity-bridge libs/v3/lilith-identity-bridge/ Translates V1 account ↔ UE session ↔ Tier-2 session; reputation bands.

Rust crates under apps/v3/*/Cargo.toml and shared crates under libs/v3/multiplayer-protocol/rust/:

Crate Path What V3 uses it for
lilith-world-server apps/v3/lilith-world-server/ Authoritative state, physics, room sharding, presence, persistence sync.
lilith-realtime-gateway apps/v3/lilith-realtime-gateway/ QUIC/WebTransport termination; WebRTC fallback; voice SFU.
lilith-pxstream-relay apps/v3/lilith-pxstream-relay/ Pixel Streaming matchmaker + signaller integration; UE worker pool manager.
saraswati-service apps/v3/saraswati-service/ Music generation orchestration + performance engine + concert calendar.
lilith-commerce-service apps/v3/lilith-commerce-service/ Tickets, tips, subs, royalty waterfall, signed-edition mint via Aje.
multiplayer-protocol (Rust) libs/v3/multiplayer-protocol/rust/ Wire format, packet schema, snapshot delta encoding; codegen for TS + UE.

2. Internal Packages — V1 Foundations Reused#

V3 deliberately reuses V1's foundational packages without duplication:

Package Path V3 role
@oshun/contracts libs/contracts/ Zod contracts; V3 contracts live under libs/contracts/src/v3/.
libs/openapi libs/openapi/ OpenAPI 3.1 specs and clients for V3 BFF routes.
libs/proto libs/proto/ gRPC for service-to-service traffic.
@oshun/persistence libs/oshun/persistence/ Prisma renderer; V3 schemas under v3.* namespace.
@oshun/event-bus libs/shared/event-bus/ Cross-tenant events.
@oshun/queue libs/shared/queue/ Saraswati gen queue, royalty settlement queue, recording transcode queue.
@oshun/http-client libs/shared/http-client/ Idempotent calls into Isis, Sophia, generative providers.
@oshun/data-residency libs/shared/data-residency/ V3 sessions honor V1 residency primitives.
@oshun/inbound-integrations libs/shared/inbound-integrations/ Yoga Alliance API, Stripe Connect, Aje, music-licensing API connectors.
@oshun/audit-platform libs/shared/audit-platform/ All operator actions, persona decisions, generation jobs audited here.
@oshun/identity libs/shared/identity/ RBAC; V3 adds tenant Tara / Saraswati / Commons scopes.
@oshun/auth-primitives libs/shared/auth-primitives/ JWT / session primitives extended with realm, avatar_id claims.
@oshun/design-tokens libs/oshun/design-tokens/ Lilith canonical tokens carried into UE UMG + Tier-2 HUD surfaces.
@oshun/ui libs/oshun/ui/ Shared 2D UI for HUDs, modals, calendars.
@oshun/navigation libs/oshun/navigation/ Deep links into V3 routes, command-surface registration.
@oshun/analytics libs/oshun/analytics/ Route-level analytics and experiment scoping for V3.
@oshun/testing libs/shared/testing/ Test utilities shared with V1.
@oshun/offline libs/oshun/offline/ Offline shell, retry, recent-content fallback (limited utility in real-time V3).
@oshun/evidence-sophia libs/oshun/evidence-sophia/ Grounding for artist backstories and instructor lineage claims.
@oshun/persona-policy-lilith libs/oshun/persona-policy-lilith/ Persona policy applied to embodied avatars.
@oshun/generation-control-isis libs/oshun/generation-control-isis/ Generation control plane; V3 adds music/motion/world-asset workflows.
@oshun/embodiment-aja libs/oshun/embodiment-aja/ Aja substrate; V3 extends with in-world pose estimation.
@oshun/embodiment-psyche libs/oshun/embodiment-psyche/ Psyche; V3 extends with 3D-aware lipsync and gesture.
@oshun/memory-iris libs/oshun/memory-iris/ Iris memory; V3 adds scene/pose scope.
libs/aje libs/aje/* Crypto substrate for tickets, signed editions, royalty waterfall.
libs/yemaya libs/yemaya/* Off-line media render for recap reels, signed-edition video, Stadium-tier fallback proxy.
libs/themis libs/themis/* Adjudication for music rights, instructor credential disputes, persona impersonation.

3. UE5 Engine Version#

Dependency Version Notes
Unreal Engine 5.5 LTS Lock to 5.5.x; no engine upgrade across the launch window without explicit signoff
Engine source path Engine/ (Epic) Built from source if engine modifications required; binary otherwise
Visual Studio 2022 17.10+ Windows toolchain
Xcode 16.x macOS + iOS + visionOS
clang (Linux) 16+ Linux client + Pixel Streaming Win-Server cook on Linux container
Android NDK r25+ (UE-pinned) Android + Quest builds
PS5 / XSX toolchains platform-issued Console builds; behind platform-cert gates

V3 shares the UE5.5 LTS choice with V2 (the fighting game). Sharing the engine version simplifies toolchain, plugin compatibility, and shared patch / hotfix processes across the two products. See v2deps§1 for the parallel V2 pinning.

4. UE5 First-Party Plugins#

Enabled in V3/ue/V3.uproject Plugins list.

Plugin Used by V3 for
PixelStreaming Browser delivery of the canonical UE5 client (see §6).
PixelStreamingPlayer Browser-side WebRTC player components (for embedding in the V3 web shell).
EnhancedInput Input mapping across KBM, touch, gamepad, OpenXR controllers, hand-tracking.
CommonUI Front-end shell, input-style routing, focus graph (V3UI).
ModelViewViewModel HUD attribute binding (V3UI).
GameplayAbilities (GAS) Light use: interaction, asana lock, gesture, applause (V3Gameplay).
MotionWarping Throw / asana entry alignment to props or asana targets.
AnimationWarping Foot lock, stride warping for traversal in Commons free-roam.
MotionMatching World-free-roam traversal in Lilith Commons venues.
Mover (2.0) Locomotion replacement for CMC in exploration modes.
ChooserTable Asana variant selection by lineage tag; gesture variant selection.
Niagara Stage particles, ritual effects, asana cues, light shafts, weather (V3VFX).
GeometryScripting Editorial-only stage builds.
ChaosCloth Costume cloth, robes, scarves.
MetaSound Native audio runtime; all music and ambient (V3Audio).
AudioModulation Dynamic mix, ducking, music stack (V3Audio).
AudioGameplayVolume Venue acoustic zones (e.g., reverb in stone halls, dry in studio).
ReplicationGraph Interest management for stadium-tier concert replication priority bands.
NetworkPrediction Used in non-rollback channels for asymmetric flows.
GameFeatures Modular tenants + modes shipped as Game Feature Plugins.
ModularGameplay Per-mode actor extension.
Iris UE5 networking layer.
OnlineSubsystem Cross-platform online abstraction; presence, friends.
TraceInsights Perf profiling.
Gauntlet End-to-end automation runner.
FunctionalTesting Per-feature in-engine automation.
AutomationController CI test orchestration.
MoviePipeline Concert master recording, cinematic capture for trailers.
Replays Demo recording for replay system.
Mover2 Newer locomotion path used for Commons + World Tour-style Lilith spaces.
GameplayCameras (UE5.5) Camera-rig framework for in-world camera, audience cameras.
ContentBrowserAliasIDs Editorial workflow for content versioning.
MetaHuman High-fidelity MetaHuman avatars (Saraswati artists; verified premium Tara instructors).
MetaHuman Animator Facial capture pipeline for premium personas.
LiveLinkFace Performance capture for cinematic sequences.

5. UE5 Third-Party Plugins#

Plugin / Vendor Used by V3 for
VRM4U (or in-house port) VRM 1.0 avatar import in UE5; required for community / Ready-Player-Me / VRoid.
Resonance Audio (Google) Default HRTF / spatial audio renderer on UE for non-VR clients.
Steam Audio (Valve) Higher-fidelity spatial audio with occlusion + reflections (VR opt-in).
OpenXR Plugin (Epic) OpenXR runtime for cross-headset VR support.
OnlineServicesEOS Default OnlineServices provider (cross-platform).
EOSShared / EOSVoiceChat Friends, presence, party, voice chat backbone.
Wwise (optional, see §7) Bank-style audio authoring (decision recorded in §35 launch readiness).
AssetLinter (V3) Internal plugin enforcing V3 naming conventions and asset budgets.
MetaHuman SDK (separate from MetaHuman plugin) third-party tools for MH workflow.
WebRTC Importer (Epic) Used by Pixel Streaming for ICE / DTLS / SRTP.
OmniCloth / Nvidia Flow (optional, premium concerts only) cloth + volumetric sim.
Adobe Substance Plugin Editorial-only material runtime import from Substance Painter.

6. Pixel Streaming Infrastructure#

Pixel Streaming makes the canonical UE5 client reachable from any WebRTC-capable browser without an install. See arch§"Pixel Streaming Infrastructure" for the architecture diagram.

Component Version target V3 use
UE PixelStreaming plugin UE 5.5 LTS bundled Native UE pipeline for server-side rendering + WebRTC encode + stream.
UE Signalling Server Epic OSS (Node.js) One signaller per POP; SDP exchange between browser and UE worker.
Stun / Turn servers coturn (OSS) NAT traversal for WebRTC sessions.
lilith-pxstream-relay (internal) Rust Matchmaker; selects POP by geo / load / programming priority; issues session JWTs.
NVIDIA L40S / L4 / RTX 6000 Ada GPU silicon Per-POP worker GPU. ~6 class-tier sessions per 24-vCPU node; ~3 stadium-tier.
AMD MI300 (alt) GPU silicon Alternate GPU silicon for cost-optimized POPs.
WebRTC stacks browser-native Browser-side stream + input data channel.
AV1 encode NVIDIA NVENC / AMD VCN High-fidelity concert streams to Chrome / Edge browsers that decode AV1.
H.264 encode NVIDIA NVENC / AMD VCN Default encode for broad browser compat (including Safari).
Opus (audio in stream) libopus Audio in the WebRTC stream.
Cloud GPU providers see §34 AWS g6e / p4 / Azure NVadsA10 v5 / Lambda / Hetzner GPU.
Auto-scaler Karpenter / KEDA Auto-scale UE worker fleet by queue depth + scheduled concert load.
Worker container image Docker (Ubuntu + UE Win-Server cook + GPU runtime) Built nightly; pushed to ECR / GHCR.

POPs at GA: us-east-1, us-west-2, eu-west-1, ap-northeast-1, ap-south-1, ap-southeast-2, sa-east-1. Latency budget ≤ 60 ms browser ↔ POP. Per-region routing honors GDPR / residency rules. See arch§"Per-POP Capacity, GPU Quota, and Multi-Cloud Posture" for the full coverage matrix.

7. Audio Middleware (UE5)#

Dependency Version V3 use
MetaSounds UE 5.5 native Default audio runtime; all music graphs, ambient, UI SFX.
AudioModulation UE 5.5 native Mix-bus runtime, ducking, music-bed transitions.
AudioGameplayVolume UE 5.5 native Venue acoustic zones.
Resonance Audio (Google) OSS plugin Default HRTF renderer for non-VR clients.
Steam Audio (Valve) OSS plugin Higher-fidelity HRTF + occlusion + reflections (VR opt-in; concert stadium tier).
Wwise (Audiokinetic) optional Bank-style audio if editorial requires it; per-platform license.
Opus (libopus) ^1.5 Voice codec; HRTF mixed client-side.
FFmpeg ^7.0 Server-side transcode (recordings → on-demand library) and Yemaya fallback proxy.

Default decision: MetaSounds + Resonance Audio for non-VR, MetaSounds + Steam Audio for VR. Wwise stays as an optional path if Saraswati audio team needs bank-style authoring.

8. Application Runtimes#

Runtime Version source V3 use
C++ UE5 toolchain per-platform UE5 client modules (V3/ue/Source/*) and plugins.
Node.js engines.node in app package.json; target node22 V1 BFF extension; Pixel Streaming UE Signalling Server (Node.js).
TypeScript typescript in pnpm-workspace.yaml Tenant adapter packages, Tier-2 fallback, browser shell.
Rust per-crate Cargo.toml; toolchain pinned in rust-toolchain.toml World server, gateway, pxstream-relay, saraswati-service, commerce-service, shared protocol.
Python per-service pyproject.toml Pose-estimation training jobs, music-licensing offline jobs.
Swift Xcode 16+ UE Native iOS / Vision Pro shim modules where needed.
Kotlin Android Studio UE Native Android shim modules where needed.
WASM rustc target wasm32-unknown-unknown Rapier physics + audio DSP + protocol encoder for Tier-2.

9. Backend Frameworks (Services)#

Framework Version source V3 use
Fastify fastify: ^4.28.1 V1 Oshun BFF, extended with V3 routes.
axum axum = "0.7" (pinned per Rust service) World server, gateway, pxstream-relay, saraswati-service, commerce-service.
Hono hono: ^4.0.0 + @hono/* Lightweight service routers (Lilith Studio internal API).
Prisma prisma: ^5.20.0 Per-tenant Postgres schemas in V3 namespace.
Zod zod (pinned through @hono/zod-openapi) All V3 contracts; OpenAPI generation; runtime guards.
tokio tokio = "1" (pinned per Rust service) Async runtime for all Rust services.
quinn quinn: ^0.10 QUIC implementation in the realtime gateway.
livekit livekit-server-sdk: ^1.x Voice SFU module embedded in the gateway.

10. Web Shell + Pixel Streaming Player#

The browser-side surface that hosts the Pixel Streaming player and the launcher / tier-router landing page.

Library Version source V3 use
Next.js next: ^14.2.21 apps/v3/lilith-web browser shell; V1 web extension routes.
React react: ^18.3.1 All web shell surfaces.
@tanstack/react-query ^5.90.16 Catalog / billing / programming data fetch from V1 BFF.
@epicgames-ps/lib-pixelstreamingfrontend-ue5.5 latest Epic's Pixel Streaming browser library (WebRTC player, input).
@oshun/lilith-web-pxstream libs/v3/lilith-web-pxstream/ Internal wrapper around the Epic library with V3-specific UX + auth.
@oshun/design-tokens (V1 reuse) Lilith canonical design language carried into the browser shell.

11. Tier-2 Fallback Web Client Stack#

The no-one-locked-out path. Activated by the tier router when Pixel Streaming isn't available or the user explicitly chooses Lite mode.

Library Version source V3 use
Next.js next: ^14.2.21 apps/v3/lilith-web-fallback.
React react: ^18.3.1 UI shell.
three ^0.170.0 Core 3D render engine (Tier-2).
@react-three/fiber ^8.17.0 React bindings for three.js.
@react-three/drei ^9.105.0 High-level three.js helpers (controls, post-processing).
@react-three/postprocessing ^2.16.0 Post-processing chain.
@rapier3d/core (WASM) ^0.14.0 Tier-2 client-side physics prediction.
webxr-emulator dev-only Emulating XR sessions during CI for the fallback.
wasm-bindgen consumer pkgs per-package Rust-WASM bridges for protocol encoding and audio DSP.
@livekit/components-react ^2.5.0 WebRTC voice chat client.

12. Mobile (Companion + UE Mobile)#

Library Version (package.json) V3 use
Expo ~51.0.0 V1 mobile shell; V3 companion stack inside apps/oshun/mobile.
React Native 0.74.5 V1 mobile + V3 companion UI runtime.
expo-router ~3.5.0 V3 stack routing in the companion shell.
expo-haptics ~13.0.0 Hand-gesture haptic feedback in the companion surfaces.
expo-camera ~15.0.0 Aja pose estimation source (opt-in).
expo-av ~14.0.0 Audio playback fallback path.
expo-secure-store ~13.0.2 Credential storage shared with V1 mobile.
react-native-mmkv ^2.12.0 Fast key-value cache.
Deep link to UE5 native app expo-linking Launches the canonical UE5 mobile client from companion screens.
UE Mobile (iOS / Android) UE5.5 LTS The canonical mobile V3 client; built from V3/ue/.

The split: the React Native companion app inside apps/oshun/mobile provides V1 continuity surfaces (booking, Arete log, ticket inventory) and a "launch Lilith" deep-link. The actual 3D embodied experience runs in the UE5 mobile client.

13. VR / AR / XR Stack#

Platform / SDK Version target V3 use
OpenXR OpenXR 1.0+ UE plugin; cross-headset input, render, hand-tracking, anchors.
Meta Quest SDK (Quest 3, Pro) v68+ Quest-native UE binary; hand-tracking, passthrough, body tracking.
Apple visionOS SDK visionOS 2.x UE visionOS pipeline.
PSVR 2 SDK (Sony) latest UE PS5 build; submitted through Sony approval.
SteamVR / OpenVR latest Valve Index, HTC Vive Focus support via OpenXR.
WebXR Device API browser-native Tier-2 entry path on Quest browser; reduced fidelity.
Khronos glTF / VRM 1.0 spec versions Avatar and scene asset format for community avatars (UE imports via VRM4U).
Steam Deck (Proton) latest Linux UE5 cook + Proton verification badge ("Verified" target).

Hand-tracking: OpenXR hand-tracking extension; controller-only fallback. Body tracking: Quest 3 inside-out + OpenXR body-tracking when adopted.

Steam Deck specifics: V3 ships as a "Steam Deck Verified" or "Playable" candidate. The build uses the standard Linux UE5 cook with Steam Deck detection (gamepad / touch hybrid input mode + 720p / 40 fps default preset). Tested per Valve's hardware compatibility guidelines pre-launch and gated by the cert task in V3_TODOS.md § 69.

14. Generative AI — Music#

Music generation underpins Saraswati Stage. V3 uses a multi-provider strategy with explicit licensing constraints.

Provider / Model Version target Use case Licensing posture
Suno API v4+ Vocal + accompaniment for indie folk, hip-hop, ambient. Commercial license at Pro/Premier tier; full output ownership confirmed pre-launch.
Udio API v2+ Vocal + accompaniment for electronic, classical-crossover. Commercial license confirmed.
Stable Audio Open / 2.0 API + on-prem Ambient stems, sound-baths, devotional cells. OSS or commercial Stability AI license; on-prem option for compliance.
MusicGen (Meta) API + on-prem Backing-track gen; open-weight on-prem option. Open-weight; commercial use validated.
AudioCraft (Meta) API + on-prem Backing + sound-effect gen. Open-weight; commercial use validated.
Mubert API Royalty-free ambient stems for low-stakes Commons ambient. Commercial subscription.
Custom in-house model self-hosted (post-GA) Per-Saraswati-persona fine-tunes for voice + style consistency. Trained on licensed datasets only; per-contributor royalty waterfall.

Failover policy: Saraswati performances must not go silent. Real-time generation provider failure falls back to pre-rendered catalog tracks. Concert sync drift target ≤ 25 ms maintained across providers.

Routing through V1 Isis: @oshun/isis-music registers a workflow with V1 Isis (v1deps§9). UE's MetaSound layer consumes the resulting streams via the world-server-issued sync timestamps.

15. Generative AI — Voice and Speech#

Provider / Model Version target Use case
ElevenLabs API v2+ Voice cloning, TTS for instructors (with consent), Saraswati persona speech.
Cartesia Sonic API v2+ Low-latency real-time voice; Tara live-class TTS where latency matters.
PlayHT API v3+ Fallback TTS provider; multi-locale.
Coqui TTS (open) on-prem On-prem TTS for residency-restricted regions.
WhisperX OSS Server-side transcript for crisis / harassment detection (60 s buffer).
Whisper (OpenAI) API Multi-locale transcription for accessibility captions.

Voice-cloning consent flow + public-figure registry sit in front of any voice signature build (see features§"Voice Cloning Boundaries").

16. Generative AI — Motion and Avatar#

Provider / Model Version target Use case
DeepMotion Animate 3D API v3+ Video-to-motion-capture for instructor demo recordings.
Rokoko hardware + SDK Pro-grade mocap suit for editorial team motion authoring.
Move.AI API Markerless mocap from video.
Cascadeur desktop app Keyframe + AI-assist for non-mocap motion (stage performance moves).
MetaHuman Animator UE-bundled Facial capture for high-fidelity recording exports + Saraswati personas.
Ready Player Me API v2+ Avatar import bridge for community avatars (via VRM4U).
VRoid Hub API Community avatar import (verified before publish).
Stable Diffusion + LoRA Isis pipeline Text-to-costume texture generation; per-creator LoRA fine-tunes (verified).
Sora / Veo (post-GA) API when available Text-to-video for editorial trailers; not real-time in-world.

Motion provenance: every motion clip carries an Isis provenance bundle.

17. Generative AI — World Assets#

Provider / Model Version target Use case
Stable Diffusion XL / 3.x Isis pipeline Texture generation for stage props, ritual implements, costume textures.
Adobe Substance 3D Sampler desktop Material capture from reference photos.
Meshy.ai API Text-to-3D for low-stakes props (cushions, stools, candles).
Tripo3D API Text-to-3D alternative.
Luma AI Genie API Image-to-3D for editorial bootstrapping.
Polycam / Reality Capture desktop Photogrammetry for high-fidelity stage props and venue scans.

Asset publishing gate: every generated asset passes through Lilith Studio editorial review and Isis provenance verification before production.

18. Generative AI — Text and Lyrics#

Provider / Model Version target Use case
Claude (Anthropic) claude-opus-4-7, claude-sonnet-4-6 Lyric gen, artist-persona dialog, instructor copy, Sophia grounding.
GPT-4o / o3 / o4 (OpenAI) latest Alternate provider for diversity and cost-quality choice.
Gemini 3 Pro latest Cost-mode alternative for non-critical paths.
Llama 3.x / 4.x (Meta) on-prem option Residency-restricted regions, on-prem deployments.
Mistral Large / Codestral API + on-prem Multi-locale lyric gen; on-prem option.

Lyric tone runs through Lilith persona-policy; factuality runs through Sophia for any claim touching real culture, persons, or events.

19. Pose Estimation and Embodied Coaching#

Library / Model Version target Use case
MediaPipe Pose (Google) tasks-vision 0.10+ On-device 2D pose estimation from camera (both UE and Tier-2 clients).
MediaPipe Holistic tasks-vision 0.10+ Combined body / face / hands estimation.
MoveNet (TF) tflite + tfjs Lightweight on-device pose model.
BlazePose GHUM 3D MediaPipe 3D pose estimation where camera quality allows.
OpenPose (server-side QA) OSS Validation set evaluation only.
AlphaPose (server-side QA) OSS Validation set alternative.
Custom asana classifier trained in-house Per-asana classification head over pose features.
OpenXR hand-tracking OpenXR ext (UE plugin) VR pose estimation source on Tier-1 VR clients.
Apple Vision Body Tracking iOS 17+, visionOS 2+ iOS / Vision Pro body tracking through UE Native.

Pose data privacy: on-device by default; aggregate metrics only to Iris; camera frames never leave device.

20. Multiplayer Networking and SFU#

Library / Service Version target Use case
WebTransport (W3C) browser-native Tier-2 client ↔ gateway transport (QUIC).
WebRTC data channel browser-native Tier-2 fallback transport; Pixel Streaming input channel.
WebSocket browser-native Last-resort fallback for restrictive networks.
quinn (Rust) ^0.10 QUIC server in the realtime gateway.
str0m / webrtc-rs latest Native Rust WebRTC stacks for SFU.
LiveKit Server SDK ^1.x Voice SFU module; Apache 2.0 OSS.
Janus / MediaSoup (alt) latest Alternative SFUs; evaluated, not default.
UE OnlineSubsystem (EOS) UE 5.5 Presence, friends, parties on UE clients.
UE Iris networking UE 5.5 Non-rollback channels inside UE.

V3 implements the authoritative world server outside UE in Rust so that UE clients and Tier-2 fallback clients can share a single state authority.

Voice chat: Opus codec (libopus); 24 kbps mono per speaker; HRTF mixed client-side (UE: Resonance / Steam Audio; Tier-2: Web Audio + WASM DSP).

21. Physics#

Library Version target Use case
Chaos (UE5) UE 5.5 native UE client-side physics prediction.
Rapier 3D (Rust) ^0.21 Authoritative physics on world server.
Rapier WASM @dimforge/rapier3d Tier-2 client-side prediction physics on web.
PhysX (alt) bundled Considered as Chaos alternative; rejected (Chaos parity is enough).

Physics scope at GA: prop interaction, posture IK, asana lock, camera collisions. No active ragdolls; no destructible meshes (out-of-V3 scope).

22. Spatial Audio and DSP#

Library Version target Use case
MetaSounds (UE) UE 5.5 native UE audio runtime; music graphs, voice routing, ambient.
Resonance Audio (Google) OSS UE plugin HRTF + ambisonic on UE non-VR clients.
Steam Audio (Valve) OSS UE plugin Higher-fidelity HRTF + occlusion (VR opt-in; stadium tier).
Web Audio API browser-native Tier-2 audio graph.
Resonance Audio JS OSS Tier-2 HRTF via WASM DSP.
AudioWorklet browser-native Low-latency audio DSP on Tier-2 web.
libopus ^1.5 Voice codec.
FFmpeg ^7.0 Server-side transcode (recordings → on-demand library).

Music sync drift target: ≤ 25 ms across all attendees in Stadium-tier concerts. Achieved via server-issued sync timestamps and NTP-style client clock adjustment.

23. Animation & Mocap Pipeline#

Same pattern as v2deps§5; the Tara + Saraswati motion catalogs ride this pipeline.

Dependency Used by V3 for
Vicon / OptiTrack Body mocap recording (editorial studio shoots).
MetaHuman Animator Facial capture pipeline (default).
Faceware Performance face-capture (optional).
Move.AI / Plask Markerless mocap for pickup sessions.
Cascadeur Keyframe + AI-assist animation.
Autodesk Maya Animation cleanup, retargeting.
MotionBuilder Mocap retargeting and cleanup.
Blender Alt DCC for indie passes; community asset prep.
Houdini Procedural venue authoring for high-end Commons venues.
ZBrush Character sculpt for MetaHuman + premium VRM.
Substance Painter Material / decal painting.
Marvelous Designer Cloth simulation source.

24. Music Licensing and Rights Infrastructure#

Service / Org Use case
ASCAP / BMI / SESAC (US) Performing-rights registration for off-platform distribution.
PRS for Music (UK) Performing-rights in the UK.
GEMA (DE) Performing-rights in DE.
JASRAC (JP) Performing-rights in JP.
MCPS / MLC (mechanical) Mechanical rights registration where applicable.
SoundExchange (US digital) Digital performance royalties for non-interactive streaming.
The MLC US mechanical-licensing collective.
HFA / Songfile (mechanical) Mechanical license clearance.
Auddly / Sound Credit Credit and split-sheet tooling.
Pex (rights identification) Identify uploaded material against rights registry; takedown automation.
YouTube Content ID (post-GA) Off-platform monetization for Saraswati tracks on YouTube.
Rights ID (Audible Magic) Pre-publish fingerprint check against major-label catalogs.

Editorial process: every Saraswati track passes a fingerprint check (Audible Magic / Pex) before release; matches trigger editorial review. Voice-similarity registry checks against a public-figure voice corpus.

25. Avatar Source Pipelines#

Source Use case
Oshun Avatar Gallery Canonical avatar set authored by Oshun art team.
MetaHumans High-fidelity premium personas (Saraswati artists; verified Tara instructors who opt in).
Ready Player Me Community avatars (imported via VRM4U); verified before publish.
VRoid Hub Community avatars; verified before publish.
Polycam / RealityCapture Photogrammetry-sourced avatars (post-GA).

All avatars are normalized to a runtime that supports both MetaHuman and VRM 1.0; the Tier-2 fallback rasterizes only the VRM 1.0 path (MetaHumans render to their VRM proxy on Tier-2).

26. Data Stores#

Store Version source V3 use
PostgreSQL + pgvector docker/docker-compose.dev.yml Durable state: tenant catalogs, attendance, royalty records, vector search for assets.
Redis Streams docker/docker-compose.dev.yml Hot room state, session token cache, voice transcript rolling buffer.
MinIO / S3 docker/docker-compose.dev.yml Object store: avatars, scenes, music, recordings, signed editions.
Qdrant (optional) vectors profile Vector search for asana / track / artist / scene similarity.
Neo4j (optional) graph profile Relationship graph: artist → lineage → track → contributor → fan.
Elasticsearch (optional) search profile Full-text search across instructor / artist / venue / track catalogs.
Kafka (optional) streaming profile Cross-tenant streaming for analytics, royalty settlement, Aje events.

27. Object Storage and CDN#

Service Use case
MinIO (dev) Local development.
AWS S3 (prod) Asset object store with regional buckets for residency.
Cloudflare R2 (alt) Egress-optimized object store for music streaming.
Cloudflare CDN Global asset distribution for scenes, music, avatar textures.
Bunny.net (alt) Alternative low-cost CDN.
AWS CloudFront Alternative CDN with AWS-native auth.

All CDN assets signed-URL-only; signed by the V1 identity service.

28. Identity, Auth, and Provisioning#

V3 reuses V1's identity / auth stack with UE-specific bridges:

Service / Tool V3 use
@oshun/identity RBAC; V3 adds tenant Tara / Saraswati / Commons scopes.
@oshun/auth-primitives JWT with realm, avatar_id claims.
@oshun/auth-client V3 client SDK for session bootstrap.
UE OnlineServicesEOS UE-side OAuth bridge to V1 account.
OAuth bridges Quest / Vision Pro / PSVR2 OAuth linking to V1 account.
SCIM / SSO Tenant-administered SCIM/SSO for partner organizations (carried from V1).

29. Payment, Billing, Royalties#

Service Use case
Stripe Connect Fiat payouts to verified instructors (Tara) and rights-holders (Saraswati).
Stripe Billing Subscriptions.
Apple In-App Purchase iOS / Vision Pro in-app payments where required by Apple policy.
Google Play Billing Android in-app payments where required by Google policy.
Sony PSN Wallet PSVR2 payment routing.
Meta Horizon Store Quest in-app payments.
Aje (V1) Crypto-native tickets, signed-edition mint, royalty waterfall on-chain.
Plaid Bank-link for fiat payouts where applicable.
TaxJar / Avalara Sales-tax on tickets / subscriptions per region.
Vertex (alt) Alternative tax engine.

30. Provenance and C2PA#

Service / Lib Use case
c2pa-rs / c2pa-node C2PA manifest writes on video / audio exports.
Truepic / Truera (alt) Alternative C2PA provider.
Steg.AI Watermark embedding in generated audio / video / images.
OriginTrail (alt) Knowledge-graph provenance for content lineage.
Adobe CAI tools C2PA authoring and verification.
V1 @oshun/provenance In-house provenance substrate carried through V1 Isis.

UE exports through MoviePipeline carry C2PA manifests written via c2pa-rs invoked from a post-export hook.

31. Observability and Telemetry#

V3 reuses V1 observability stack with UE-specific OTel exporters:

Tool Use case
OpenTelemetry End-to-end tracing: UE clients (OTel C++), Pixel Streaming workers, gateway, world-server, tenant services, V1.
Jaeger Trace UI in dev and staging.
Prometheus Metrics scrape.
Grafana Dashboards under grafana://v3/*.
Loki Structured-log storage with redaction.
Tempo (alt) Alternative trace store.
Sentry Client error reporting (UE + Tier-2 + companion mobile).
Datadog (alt-cloud) Alternative observability stack.

V3-specific eval dashboards:

  • v3/world-server/perf, v3/realtime-gateway/lat.
  • v3/pxstream/session-count, v3/pxstream/p99-lat, v3/pxstream/worker-utilization.
  • v3/voice/lat, v3/music/sync-drift.
  • v3/saraswati/gen-queue.
  • v3/tara/aja-accuracy.
  • v3/lilith-safety/queue-sla.
  • v3/tier-router/decision-share.

32. Search and Vector Stores#

Tool Use case
Postgres + pgvector Default vector store for asset / artist / track / sequence similarity.
Qdrant (optional) Higher-scale vector search if needed.
Elasticsearch Full-text search for catalogs.
Typesense / Meilisearch (alt) Alternative search.

33. Inbound and Outbound Channels#

Channel Use case
V1 event bus Cross-tenant events between V3 and V1 domains.
V1 webhook simulator Inbound webhooks (Yoga Alliance, Stripe, Aje, music-rights services).
Telegram / WhatsApp / push / email / SMS / Discord / Slack V1 messaging-channel substrate; V3 sends class / concert reminders, drop notifications, festival announcements.
C2PA verifier Inbound verification when receiving externally-signed assets.

34. Cloud Providers and Infrastructure#

Provider V3 use
AWS (primary) World-server fleet, S3, CloudFront, Route 53; g6e / p4 GPU for UE Pixel Streaming.
GCP (alt) Alternate region for residency requirements; A100 / L4 GPU options.
Azure (alt) NVadsA10 v5 GPU option for Pixel Streaming.
Cloudflare CDN, Workers (edge auth / signed URLs), R2.
Lambda Cloud / CoreWeave (alt) Spot GPU for Pixel Streaming bursts.
Hetzner (cost-optimized) Static asset CDN backup; dev / staging environments.
Fly.io (edge) Realtime gateway edge POPs in low-latency regions.
Vercel (web only) Browser shell + Tier-2 fallback hosting (Next.js).
EKS / GKE Container orchestration for world-server / gateway / tenant services / Pixel Streaming.
HashiCorp Vault Secret management.
Terraform (>= 1.7) Infrastructure-as-code.

GPU POPs at GA: us-east-1, us-west-2, eu-west-1, ap-northeast-1, ap-south-1, ap-southeast-2, sa-east-1.

35. Build, Cook, Patch (UE5)#

Tool V3 use
UE Build Tool (UBT) UE5 client builds across all platforms.
UnrealAutomationTool (UAT) Cook + package + stage automation.
IncrediBuild Distributed compile on Windows build farm.
FASTBuild Distributed compile cross-platform alternative.
UE Patcher + Iostore Delta patches per platform.
Pak files Chunked by Game Feature plugin for staged loading.
Pixel Streaming worker image Headless Win-Server cook + GPU runtime; pushed nightly to ECR / GHCR.
Fallback bake (tools/v3/fallback-bake/) Nightly job; consumes UE cooked assets and emits glTF + KTX2 + lightmaps for Tier-2.
MoviePipeline Concert master recording / trailer capture.

36. CI/CD and Build Tooling#

Tool V3 use
Nx Monorepo build orchestration (existing V1 setup; V3 projects added).
pnpm TypeScript dep management.
cargo + cargo-nextest Rust build and test.
UE BuildGraph UE5 CI pipelines.
EAS Build (Expo) Companion mobile build pipeline.
Xcode Cloud / fastlane iOS / Vision Pro CI (UE Native).
Gradle Android CI.
GitHub Actions CI orchestration.
Docker + Buildx Multi-arch container build for services + UE worker images.
Helm + Kustomize Kubernetes deployments for services.
Argo CD GitOps deployment.

37. Source Control#

Tool V3 use
Perforce UE assets (uasset, umap) under V3/ue/Content/; binary asset support; per-file locking.
Git UE code modules (V3/ue/Source/), plugins (V3/ue/Plugins/), all TS / Rust packages, V3 docs.
git-p4 / git-LFS bridges Mirror between git and Perforce on commit boundaries.
UEFN-style asset linting Pre-commit hook for asset naming and budget compliance.

Asset / code split mirrors V2's pattern (v2deps§13). Perforce ignore lists, asset-naming conventions, and derived-data-cache configuration are shared with V2 where possible.

38. Testing Tooling#

Tool V3 use
Vitest TypeScript unit tests (V1 standard).
Playwright Tier-2 fallback E2E + browser shell E2E.
Detox Companion mobile E2E.
cargo test / nextest Rust unit + integration tests.
UE Automation Spec UE gameplay logic.
Gauntlet UE end-to-end (concert run-throughs, class run-throughs, smoke).
FunctionalTesting UE per-feature in-engine.
Locust / k6 Realtime load testing of gateway + world-server.
Custom Rust load-gen V3-specific simulator for class / stadium tier concurrency.
Pixel Streaming load-gen Headless browser fleet simulating browser-tier sessions.
WebXR emulator Headless VR session simulation in CI.
Axe-core Web accessibility audits.
Lighthouse Performance budgets on V1 web extensions + Tier-2.
WireMock / Polly.JS External provider stubbing.
Audio sync harness V3-specific harness for music-sync drift measurement.
Pose accuracy harness V3-specific harness for Aja validation.
Adversarial prompt suite Saraswati persona-policy and Lilith-safety adversarial test set.

39. Security and Secrets#

Tool V3 use
HashiCorp Vault Secret store.
AWS Secrets Manager (alt) Alternative secret store.
OIDC / OAuth (V1) Auth.
Cloudflare WAF Web application firewall.
OWASP ZAP (CI) Security scanning.
Snyk Dependency vulnerability scanning.
trivy Container vulnerability scanning.
gitleaks Secret leak detection in repo.
UE code-signing toolchain Per-platform signing for UE binaries.

40. Containerization and Local Development#

Tool V3 use
Docker Compose Local infrastructure stack (extends docker/docker-compose.dev.yml).
k3d / kind Local Kubernetes for end-to-end testing.
mkcert Local TLS certs for WebTransport development.
tilt / skaffold Local dev loop for Rust services.
UE Editor Local UE development; binary-installed Epic build of UE5.5 LTS.

Local dev: pnpm dev:v3 brings up world-server, gateway, pxstream-relay (in single-node mode), saraswati-service, commerce-service, Postgres, Redis, MinIO. UE development uses the locally-installed Editor opening V3/ue/V3.uproject. Pixel Streaming requires a local NVIDIA / AMD GPU for the worker process; on dev machines without one, the browser launcher routes to the Tier-2 fallback for development.

41. Compliance, Age Gating, Rating Boards#

Authority V3 use
GDPR / CCPA / DSA / DSAR / CalOPPA Privacy compliance (carried from V1).
COPPA Minors under 13 not admitted; minor-mode 13+.
ESRB / PEGI / CERO / USK Content rating for native UE5 builds.
Apple App Review iOS + Vision Pro distribution of UE5 mobile.
Google Play Review Android distribution of UE5 mobile.
Sony / Meta / Valve cert Quest / PSVR2 / SteamVR distribution of UE5 binaries.
MUSO / IFPI Music-rights compliance liaison.
PCI DSS Payment compliance (Stripe / Aje).
Yoga Alliance Instructor credential verification API.

Age gating: V3 admits users 13+; Saraswati Stage 16+ at GA. Per-region age gates per local law. Minor-mode safety defaults enforced by Lilith-Safety.

42. Hardware Peripherals#

Peripheral V3 use
Meta Quest 3 / Pro Primary VR headset target.
Apple Vision Pro Premium AR/VR headset target.
PSVR 2 Console VR target.
Valve Index PC VR via SteamVR.
HTC Vive Focus 3 Standalone enterprise VR.
Bluetooth heart-rate sensors Optional Tara biofeedback (Polar H10, Apple Watch).
Camera (webcam, phone) Aja pose estimation source (opt-in).
Voice headset (any) Voice chat I/O.
Haptic vests (post-GA) Optional concert haptic feedback (bHaptics Tactsuit X40).
MIDI controllers Saraswati Studio music authoring (editorial team).

43. Esports / Tournament / Streaming Toolkit (post-GA)#

Tool V3 use (post-GA)
OBS / Streamlabs Concert / class broadcast tooling.
Twitch / YouTube Live External live-stream targets for Saraswati concerts.
Restream / Castr Multi-platform live-stream relay.
Discord stage channels Cross-platform community gathering during concerts.
Vimeo OTT On-demand catalog distribution (optional).

(Competitive yoga is not a V3 launch concept; tournament tooling is reserved for post-GA evaluation.)