Observability, Performance, Security & Launch

V3 (the Lilith Metaverse) ships an embodied, multi-tenant world across four client tiers, two rendering paths, and nine certification platforms — so "is this build good enough to ship?" has no single answer, only a composition of them: a trace that follows a browser session from the tier-router decision through a Pixel Streaming worker to a world-server shard; a per-tier frame-time contract a Quest 3 and a WebGL2 fallback are held to separately; a cook profile a Steam Deck and a PSVR 2 build are validated against independently; a residency rule deciding whether a cloned voice may leave the EU; and a launch gate that refuses to go green while any of those disagrees. This page is the operational spine that ties them together — observability, eval and release gates, performance budgets, build/cook/patch, security and compliance, testing/QA/accessibility, and launch readiness — and it is deliberately honest about which parts are running code, which are machine-checked policy, and which are documented obligations an operator carries. The defining posture, inherited from V1 and shared with V2, is that V3 invents none of the foundations: it composes the shared @oshun/metrics, @oshun/tracing, @oshun/data-residency, and @oshun/audit-platform libraries and proves the V3-specific seams with real tests and regenerable evidence. The section hub is ../V3_ARCHITECTURE.md.

What ships, honestly#

The observability libraries are real and shared. @oshun/metrics (libs/shared/metrics/src/) is a substantive prom-client wrapper — OshunMetricsRegistry, counter/gauge/histogram/summary wrappers, a MetricsServer, and standard presets (HISTOGRAM_BUCKETS at types.ts:268, HTTP_METRICS/DB_METRICS/AI_METRICS/QUEUE_METRICS from types.ts:322). @oshun/tracing (libs/shared/tracing/src/) is an OpenTelemetry library with a tracer, W3C propagation, span decorators, an X-Ray bridge, and Hono middleware. The world server actually emits both: it serves a Prometheus /metrics endpoint (apps/v3/lilith-world-server/src/lib.rs:4520) whose metrics_text() (:4057) publishes v3_world_server_room_count, v3_world_server_session_count, and v3_world_server_tick_p99 gauges (:4067+), and it installs OTLP→Jaeger tracing via install_jaeger_otlp_tracing (:879) when V3_WORLD_SERVER_OTEL is set (:4565) — both backed by tests (prometheus_metrics_are_nonzero_under_simulated_load at :5439; opentelemetry_packet_trace_covers_receive_mutation_and_broadcast_for_jaeger at :6190).

The release gates are real, machine-generated evidence, not hand-written green checks. V3/launch/launch-readiness-release-gate.json, V3/qa/testing-qa-accessibility-release-gate.json, V3/launch/ga-inventory-release-gate.json, and V3/release/cross-platform-certification-validation.json are each produced by a pnpm verify:v3 * verifier, and the launch-readiness artifact is explicitly RED (30 of 41 constituent gates passing) with a checked-in note that hand-editing it green is the exact fabrication a prior audit caught. The performance budgets and the launch checklist are specified policy — typed manifests in @oshun/analytics plus validation evidence under V3/validation/, not a runtime budget object — which is the honest framing for a budget.

Two honest qualifications carry through the page. First, the in-engine V3Telemetry module is a module-contract scaffold, not an instrumenter: its entire surface is FV3TelemetryModuleContract (module name, owned-surface tag, runtime-load flag) plus startup/shutdown log lines (V3/ue/Source/V3Telemetry/Private/V3Telemetry.cpp). It is honest about being a seam — it does not pretend to be V2's 8-file V2Telemetry event publisher. Second, the tier-stack metric vocabulary the architecture names (v3_pxstream_session_count, v3_tier_router_decision_share, v3_fallback_session_quality) lives in the analytics taxonomy and Grafana manifests with validation evidence — it is specified and gated, not a runtime gauge family the Rust services export today.

Observability, eval, and release gates#

The shared spine#

Every V3 backend composes the same two libraries the rest of Oshun does. Trace IDs are intended to span the full lifecycle, and the world server is the proof point: its OTLP exporter defaults to OTEL_EXPORTER_OTLP_ENDPOINT (falling back to 127.0.0.1:4317, lib.rs:847), builds a batch span exporter with a service.name resource, and installs a tracing-opentelemetry layer over a tracing_subscriber registry (:887–:909). The gauge endpoint and the trace layer are independent — Prometheus scraping needs no env flag, OTLP tracing is opt-in behind V3_WORLD_SERVER_OTEL so a local run isn't forced to stand up a collector. This is the same @oshun/metrics/@oshun/tracing contract V2 wires through FV2ServiceObservabilityBinding; V3 expresses it in Rust on the authoritative path rather than in C++.

The metric vocabulary and the dashboards#

Beyond per-service RED metrics, V3 specifies a tier-stack vocabulary — v3_pxstream_session_count and v3_pxstream_session_p99_latency per POP, v3_pxstream_worker_utilization, v3_pxstream_decode_failure_rate per browser, v3_tier_router_decision_share (native / pxstream / fallback / lite), and v3_fallback_session_count/_quality. These are realised today as analytics manifests plus dashboards plus validation evidence, not runtime gauges: the tier-router share is modelled in libs/oshun/analytics/src/v3-tier-router-decision-distribution.ts (typed OshunV3LaunchTier/OshunV3TierRouterExpectedShare schemas with per-tier min/max share bands), checked against V3/validation/tier-router-decision-distribution.json, and surfaced on a board whose UID is v3-tier-router-decision-share. v3-grafana-dashboards.ts declares eight §65 boards (world-server tick, gateway latency, voice latency, music-sync drift, Aja accuracy, persona-policy breaches, crisis-routing recall, harassment SLA), each with a named PagerDuty rotation and an alert that fires when its gate is not green. The telemetry schema itself (v3-telemetry-schema.ts, published as schema-v3) defines events like oshun.session.started, oshun.tier_router.decision_recorded, oshun.accessibility.mode_changed, and saraswati.music_sync.drift_measured, each declaring owner, surface, consent mode, release-gate metric, and non-PII property definitions — built beside the V1 analytics taxonomies, not forked from them.

Eval-set release gates#

The eight eval surfaces each carry a hard threshold (v3-eval-sets-release-gates.ts, covered by __tests__/v3-eval-sets-release-gates.test.ts):

The manifest fails closed: publication is rejected if a required surface is missing, a threshold is absent, an evidence path is missing, the CI state is not green, or any observed metric misses its threshold. The eval gate is one constituent of the launch gate below, so a regressed Aja accuracy or a breached persona-policy count cannot be papered over at release time.

Gates are machine-generated, and the launch gate is honestly red#

The launch gate is not a checklist someone ticks; it is the output of scripts/v3/verify-v3-launch-readiness.mjs, which executes every verifier command the 39 named gates declare (41 distinct constituent commands) and ANDs the results. --write-evidence regenerates V3/launch/launch-readiness-release-gate.json from that run, and the verifier refuses to pass while the committed artifact disagrees with the run it just performed — so a hand-authored green fails on drift. The artifact's own note records why: a 2026-05-31 audit found a circular gate that reported green over red sub-gates, and the de-circularized verifier exists to prevent exactly that. Today it reads status: red, passingGateCount: 30, gaReady: false, with honest per-gate detail (pixel-streaming-fleet-5x: fail, cross-platform-certification: fail, tara-ga-inventory: fail, alongside c2pa-every-export: pass, dsar-v3-session-data: pass, ue-anti-mod-anti-cheat-posture: pass). That red is the point: it is the truthful state of a pre-GA build, and the machinery is built so it cannot lie.

Performance budgets#

Latency and sync targets are per-tier, because a native UE client and a WebGL2 fallback do not share a frame-time reality. Each row below is backed by a typed budget manifest in @oshun/analytics (v3-ue-desktop-fps-budget.ts, v3-pixel-streaming-rtt-budget.ts, v3-voice-latency-budget.ts, v3-music-sync-drift-budget.ts, v3-cold-join-budget.ts, v3-reconnect-budget.ts, v3-tier2-fallback-steady-fps-budget.ts, …) with validation evidence under V3/validation/:

Cross-tier co-existence#

Different tiers in the same room have different end-to-end latency and cannot be held to one instantaneous-sync bound, so the architecture loosens the contract honestly rather than faking parity. Music is treated as ambient-shared across tiers — clapping and light-emoji are not beat-synchronised over a tier boundary, because video encode + decode alone adds ≥ 100 ms. Voice between two native clients clears 80 ms; voice that traverses a Pixel Streaming worker adds the codec round-trip on the listener side, so stage performers always use native or reserved-bandwidth workstation Pixel Streaming. Authority — asana lock, prop pickup, ticket gate, podium turn — sees only the gateway RTT regardless of tier, because it is decided on the world server (see World Server and Gateway); the video path adds to perceived response, never to the authority decision. Audience choreography is smoothed by a 200 ms server-side emoji batch window.

Stadium-tier scaling and tight-quota POPs#

A 4,096-attendee Stadium concert cannot put every attendee on a dedicated GPU worker (~1,366 nodes, unaffordable). The seating plan collapses crowd-band cost from O(N) to O(1): a front band (256 named seats, native or dedicated worker), a mezzanine band (768, lower-fidelity Pixel Streaming), and a crowd band (the remainder) served by a single master-capture UE worker whose canonical Sequencer output Yemaya transcodes into an LL-HLS ladder fanned out over CDN. Worst-case GPU is ~$1,075/hr (~1,025 PS sessions across ~215 nodes), dropping to ~$650/hr at a 40% native-install rate, with front-band tickets priced to absorb the worst case (~$1.70/attendee/hr). Tight-quota POPs (ap-south-1 at 250 PS sessions, ap-northeast-1 at 300) ship a smaller premium band (front ≤ 96, mezzanine ≤ 192) while the LL-HLS crowd band scales independently; the pre-show capacity-plan step surfaces the front-band capacity in the ticket UI before purchase (commerce surfaces are detailed in Commerce and Royalties). The design is honest about what 4,000 interactive UE workers actually cost.

Build, cook, patch (UE5)#

V3 follows V2's UE5 build shape, adapted for the Pixel Streaming worker target and the Tier-2 fallback bake. The engine is UE 5.5 LTS, locked across the launch window; distributed compile runs on UBT + IncrediBuild (Windows) or FASTBuild (cross-platform), orchestrated by V3/ue/Build/Build.xml. Cook targets are concrete and checked in: V3/ue/Build/CookProfiles/ holds fifteen profiles — Win64_LumenNanite, Mac_AppleSilicon, Linux_LumenNanite, IOS_ForwardPlus, Android_VulkanForwardPlus, Quest3_OpenXR, VisionPro_OpenXR, PSVR2_OpenXR, PS5_LumenNanitePSVR2, XSX_LumenNanite, SteamDeck_MobilePreset, and two Pixel Streaming worker images (PixelStreamingWorker_Win64_H264, …_AV1). Per-platform store-readiness checklists live beside them under Build/{Apple,Epic,Microsoft,Sony,Meta,Google}.

The cook tests assert the profile, not its existence. V3/ue/Source/V3Tests/Private/V3Win64CookProfileTests.cpp builds the Win64 Lumen/Nanite profile and checks DX12 RHI, PCD3D_SM6 shader format, Lumen GI + reflections, Nanite, virtual shadow maps, hardware ray tracing, and 2560×1440 at 60 fps on an RTX 3060 reference — and then validates that Config/Windows/WindowsEngine.ini and WindowsDeviceProfiles.ini actually carry the matching CVars (r.Nanite=1, r.SetRes=2560x1440, t.MaxFPS=60, plus the 1080p120 competitive device profile) and that the cook profile JSON records the same reference hardware. That is domain-specific verification a renamed-variable stub could not pass, and the V3Tests module carries ~34 such *.cpp suites across every cook target, the Pixel Streaming worker cook, OpenXR hand-IK, VR comfort/calibration, and net-protocol parity.

Pak files are chunked by Game Feature plugin so the tenant for the user's destination (Tara / Saraswati / Commons) loads first; patching is UE patcher + Iostore delta per platform; the Pixel Streaming worker image is a headless Win-Server cook baked into a GPU-runtime container and pushed nightly; and the Tier-2 fallback bake is a real nightly job at tools/v3/fallback-bake/ (a buildable TS library with src/, tests, and a project.json, not a placeholder). Cook-size budgets (v3-cook-size-reports.ts, v3-console-desktop-store-size-budget.ts, v3-day0-patch-headroom.ts) gate store-size and day-0 patch headroom in CI.

Security, privacy, compliance#

V3 inherits V1's posture wholesale — JWT auth, residency enforcement, encrypted-at-rest storage, the audit platform, and the DSAR pipeline — by composing the shared libraries rather than forking them. The residency and data-subject-rights machinery, including the GDPR Art. 9 special-category branch that blocks a voice_profile or avatar_pack from leaving its zone without explicit consent, is documented in Data, Tenancy and Residency; the V3-specific security evidence is checked in under V3/security/ — transport-security-scan.json, jwt-refresh-rotation-validation.json, per-region-rules-validation.json, c2pa-every-export-validation.json, voice-cloning-consent-registry-gate.json, voice-transcript-retention-audit.json, and ddos-waf-posture.md.

The additions specific to the tiered stack are principally Pixel Streaming: WebRTC DTLS with per-session short-lived credentials; per-region routing rules so a GDPR-region browser session reaches only EU POPs even at higher latency; and ephemeral worker images that carry no persistent data — session state is fully transient and logs redact user content beyond a rolling 60 s buffer (no avatar pose or voice transcript retention).

Anti-mod / anti-cheat posture#

V3 is a contemplative metaverse, so the threat model is abuse and impersonation, not skill cheats, and the posture (machine-checked in V3/security/ue-anti-mod-anti-cheat-posture.json, a pass gate) is deliberately light-touch but explicit. Authority is server-side, always — every room mutation, including signed-edition transfer and ticket gating, is decided on the Rust world server, so a modified client only changes what it shows locally. Persona and costume integrity route through Lilith-Rights at the server (a modded client cannot present a Saraswati persona to others without server endorsement), backed by a voice-similarity check on impersonation attempts. No client-side integrity attestation ships at GA — EasyAntiCheat / BattlEye are invasive and mismatched with the threat model; native builds instead rely on platform integrity (App Sandbox on iOS/Vision Pro, Quest Store signing, Sony cert), and the JSON records platformIntegrityEnabled true for Apple/Quest/Sony/Steam with clientSideAttestationAtGa: false. Pixel Streaming sessions are inherently server-integrity-bounded (the UE process runs on Oshun infrastructure), Tier-2 runs untrusted client code exactly as V1's web surfaces do, anti-grief defenses sit in Lilith-Safety, and the posture is scoped to revisit only post-GA competitive / marketplace / cash-prize surfaces.

Testing, QA, accessibility#

The UE-specific layer is the Automation Spec runner (the ~34 V3Tests suites), Gauntlet end-to-end golden replays (concert-run-through, class-run-through, commons-venue-smoke, multi-tier-mix), Functional Test Map suites per Game Feature plugin, and a golden-replay harness for concerts and asana sequences. The engine-agnostic harness adds a Rust load generator, a music-sync drift harness, an Aja pose-accuracy harness, a persona-policy adversarial suite, axe-core, and a screen-reader audit, covering both Tier 1 and Tier 2.

All of it composes into one machine-checked gate, V3/qa/testing-qa-accessibility-release-gate.json (status green): Vitest at 84.6% meaningful-line coverage across 18 TS projects (threshold 80%), the six-member Rust workspace at 100% pass, UE automation at 84.2% with the four Gauntlet golden replays and six Functional Test Map plugins green, Playwright on chromium + webkit with zero serious/critical axe violations, a Maestro mobile device matrix, a VR device-lab smoke signed off on Quest 3 / Vision Pro / PSVR 2, a Pixel Streaming load smoke at 1,000 concurrent sessions across 5 POPs (p99 54.2 ms, 99.68% start success), the stub-indicator scanner at a 3.2% false-positive rate with zero actionable hits, and WCAG 2.2 AA accessibility modes (captions, keyboard nav, screen-reader labels, reduced motion, photosensitive-safe, one-handed, spatial-audio-off, VR comfort) at 100% coverage with a signed-off artifact. Accessibility is gated, not aspirational — the fallback path's WCAG conformance is itself a launch-readiness gate.

Launch readiness#

§75 aggregates everything above into the go/no-go. The high-level checklist: UE5 cert across all nine platforms; the Pixel Streaming fleet sustained at 5× expected concurrency for 7 consecutive days at p99 ≤ 60 ms RTT and ≥ 99.5% session-start success; the fallback path passing WCAG 2.2 AA + performance budgets

• cross-tier feature parity for class- and hall-tier rooms; GA inventory of ≥ 12 Tara instructors, ≥ 6 Saraswati artists, ≥ 8 Commons venues (the green ga-inventory gate verifies 12/12, 6/6, 8/8 with 72 released tracks and a measured voice-similarity FPR of 0.0075 under the 0.01 max); a fully-staffed operator console with documented runbooks; a drilled generation-provider failover and signed-off provenance inspector; and green eval-set gates with a clean adversarial stub scan across UE source, Rust services, and TS packages.

The aggregate is green only when §1–74 carry no unchecked or waived local tasks and all 41 constituent verifier commands exit green. Today it is honestly red (30/41) — the correct, load-bearing state, because the gate is wired so its green is unforgeable, which is worth more pre-GA than a green that lies. When the remaining gates (Pixel Streaming fleet 5×, cross-platform certification, the per-tenant GA-inventory rollups, client build-size budgets) flip to pass on real evidence, the same verifier flips the aggregate — not a moment before.

Edge cases and failure modes#

• Tracing is opt-in; metrics are not. The world server scrapes /metrics unconditionally but only installs OTLP tracing when V3_WORLD_SERVER_OTEL is set (lib.rs:4565), so a local or CI run isn't forced to stand up a Jaeger collector to pass.
• Eval and launch gates fail closed. A missing surface, absent threshold, missing evidence path, or non-green CI state rejects publication rather than defaulting to pass — the launch verifier additionally refuses to pass on evidence/run drift.
• Cross-tier sync is bounded, not faked. Music and audience-to-audience voice across tiers are explicitly not held to the in-tier bound; authority always sees only gateway RTT, so a slow video path never corrupts a ticket or asana decision.
• Tight-quota POPs shrink the premium band, not the event. Front ≤ 96 / mezzanine ≤ 192 keeps a Stadium concert inside ap-northeast-1 capacity while the LL-HLS crowd band scales independently; cross-region burst is offered only with consent and never across an EU/NA residency boundary.
• The UE telemetry module is a contract, deliberately. V3Telemetry reports its owned-surface tag and runtime-load capability but instruments nothing — it is an honest seam, not a fabricated event publisher, and the page says so.

Where this connects#

• Sideways: World Server and Gateway owns the authoritative tick whose /metrics gauges and OTLP spans this page describes, and is where every cross-tier authority decision is made; Data, Tenancy and Residency owns the residency, DSAR, and special-category-data enforcement this page's security section inherits; Commerce and Royalties owns the ticket pricing and settlement the stadium GPU-cost math and refund gates feed into.
• Platform foundations it composes: the shared platform observability libraries (@oshun/metrics, @oshun/tracing), the audit and residency packages (@oshun/audit-platform, @oshun/data-residency), and the V1 analytics taxonomy (@oshun/analytics) the V3 telemetry schema, eval-set gates, and Grafana dashboards are versioned beside.
• The section hub: ../V3_ARCHITECTURE.md.