Observability, Performance, Testing & the Content Pipeline

V5 is one Unreal Engine 5 open-world narrative universe split into five playable ruleset cells — a 1947-noir Urban cell, a Prohibition-era Period cell, an 1899 Frontier, a Witcher-flavoured Hunter cell, and a hard-SF Sci-Fi cell — stitched together by a cross-cell Mind Palace, and targeting nine launch platforms from a PS5 down to an iPad. So the question this page answers — how do you know the build is good enough to ship? — has no single answer, only a composition: a telemetry event that survives batching and schema-validation at the client edge before it ever reaches ClickHouse; a per-platform frame-time budget a Switch 2 and a Windows-ultra rig are held to separately; a feel test that proves an Urban two-protagonist chase swap still feels like the GTA scene it was modelled on; and a launch gate that refuses to go green while any of those disagrees. This page is the quality spine that ties them together — observability, performance budgets, testing and the eval gates, the content pipeline, and the launch-readiness gate — and it is deliberately honest about which parts are running code, which are machine-checked policy, and which are documented obligations that no build yet satisfies. The defining posture, shared with V2 and V3, is that V5 invents none of the foundations where it can compose them, and that a gate's red is worth more than a green that lies. The section hub is ../V5_ARCHITECTURE.md.

What ships, honestly#

The in-engine telemetry is real and substantive, not a contract scaffold. V5Telemetry (V5/ue/Source/V5Telemetry/, V5TelemetrySystems.cpp ~299 lines, V5TelemetryTypes.h ~220 lines, a 140-line test) implements the actual event batcher, a TypeBox schema validator, the 32-entry critical-event taxonomy, the opt-out filter, and the DSAR export as deterministic Blueprint function libraries. The testing surface is real: V5/ue/Source/V5Tests/ carries a 511-line C++ harness (V5TestSystems.cpp) plus a 5,804-line suite file with 59 IMPLEMENT_SIMPLE_AUTOMATION_TEST cases under the V5.Tests.* namespace, and the performance budgets are a cross-validating gate — a checked-in profile catalog bound to per-platform .ini config and proven by a 223-line Python validator and a UE automation test, not a runtime budget object.

Two honest qualifications carry through the page. First, V5 does not wire the shared observability spine into its services the way V2 does: a grep for @oshun/metrics or @oshun/tracing across apps/v5/ returns nothing. Those libraries are real and substantive in the monorepo (libs/shared/metrics/src/registry.ts is a 65-symbol prom-client wrapper), but V5's 16 backend services compose @v5/service-shared and express observability through ClickHouse/Grafana telemetry plus contract-declared regional failover, not a per-service RED-metrics endpoint. Second, V5 ships zero binary .uasset files (the content is logic and 29 .uplugin manifests); the "content pipeline" below is therefore a real process and validator surface whose teeth today are 59 manifest tests and 68 Python validators, not authored art. The launch-readiness gate is honestly red — 17 of 17 blocking gates pending, releaseBlocked: true — and the machinery is built so that red cannot be forged.

Observability and telemetry#

The in-engine batcher#

V5's client telemetry is event-batched, and the batching is a real algorithm. UV5_Telemetry_EventBatcher::BuildBatches (V5/ue/Source/V5Telemetry/Private/V5TelemetrySystems.cpp:110) walks a stream of FV5TelemetryEvents and flushes a batch when any of three bounds would be crossed — MaxBatchBytes (2048), MaxEvents (500), or MaxBatchAgeSeconds (5) — tagging each finalized batch with the flush reason it tripped (max_bytes / max_events / max_age, or drain for the trailing remainder) and recording whether sequence ordering was preserved. These are the exact 2 KB / 5 s / 500 events limits the architecture's Telemetry section names, implemented rather than asserted.

Validation happens at the edge before transport. UV5_Telemetry_Schema_TypeBox::ValidateEvent (:151) resolves each event to a typebox.v5.telemetry.<event>.v1 schema and rejects it on a missing schema, a schema-name mismatch, a payload over MaxPayloadBytes, or any missing required field (event, account, region, sequence) — returning a typed EV5TelemetryValidationStatus rather than a bare bool. The critical-event catalog (BuildCriticalEventHandlers, :191) enumerates 32 handlers across session, mission, combat, Mind Palace deduction/accusation, dialogue paragon/renegade, vehicle, frontier horse-bond, ship burn/flip/dock, alchemy and sign-cast, workshop publish/subscribe, online matchmaking/leaderboard/replay, ops crash/anti-cheat, and currency-balance — each carrying a privacy class (AccountOpaque by default; privacy_consent_change is Anonymous). Privacy is enforced, not promised: ShouldCollectEvent (:264) drops everything for an opted-out account except the essential privacy_consent_change event, and BuildExport (:273) assembles a redacted, companion-app-verified, 30-day-SLA DSAR export filtered to the requesting account's own events. The module is pure client-side logic — it builds, validates, and redacts; the network transport to ClickHouse lives on the services side.

The ingest contract and the shared spine#

The server end is the telemetry service (apps/v5/telemetry/), a NestJS service whose contract.json declares one endpoint — POST /v5/telemetry/batch, requiresJwt, supportsOfflineQueue, regional — with ClickHouse primary storage, Kafka cache, Grafana analytics, capabilities batch-limits / schema-validation / privacy-opt-out / clickhouse-export, and a regional-active-active failover plan (iad/fra/sin → pdx/dublin/syd, dataResidencyPinned: true). service.ts composes @v5/service-shared's handleRealRequest rather than a bespoke handler. This is the honest shape of V5 observability today: a schema-validating, residency-pinned ingest contract feeding per-cell Grafana dashboards — not the per-service Prometheus /metrics family that V3's Rust world server exports or the @oshun/metrics/@oshun/tracing binding V2 mandates in every service. The shared spine exists in the platform; V5 has not yet adopted it in-service, and the architecture's reference to a apps/v5/telemetry/schema/ directory is a specification — that path is not populated in-repo.

Performance budgets#

The platform profile catalog#

The per-frame budget is per-platform and explicit, because a Switch 2 and a Windows-ultra rig do not share a frame-time reality. The architecture's budget table (PS5 33.3/16.6 ms, PS5 Pro down to 8.3 ms, Switch 2 with a fatter 28 ms GPU slice, Steam Deck at 33.3 ms, eleven rows in all) is bound to a launch-locked catalog at V5/ue/Content/V5Performance/Data/performance_platform_profiles.json (schemaVersion: 1, contentLock: "launch-v5"): 11 budget profiles across 9 platform groups, with 11 release-blocking regression suites. Each profile carries its upscaler (TSR/MetalFX), GI and Nanite strategy, per-mode targets (targetFps, frameBudgetMs, gpuBudgetMs, cpuBudgetMs, a dynamic-resolution range, and a profiledFrameP95Ms), scalability knobs, a profilingEvidence block (a .utrace artifact, ≥10 capture-minutes, ≥5 scenarios), and a release-blocking regression entry.

The cross-validating gate#

V5/tools/performance/validate-performance-budgets.py is the gate, and it does domain-specific verification a renamed-variable stub could not pass. It checks the profile inventory against an explicit EXPECTED_PROFILE_IDS set and the exact (targetFps, frameBudgetMs, gpuBudgetMs, cpuBudgetMs) tuple for every profile/mode against an EXPECTED_BUDGETS table; it asserts the special per-platform flags (switch-2 reduced-Lumen + SDF GI and Hard-Vacuum cook excluded, macos-apple-silicon MetalFX, steam-deck Proton-verified, ipad-pro-m3 touch-only Mind Palace); it requires profiledFrameP95Ms ≤ frameBudgetMs; and then it re-parses each platform's .ini (e.g. V5/ue/Platforms/PS5/Config/PS5Engine.ini) and confirms the config section's profileId, memory budget, upscaler, GI/Nanite strategy, regression-suite id, and every per-mode FPS/budget/resolution match the JSON. The matching UE test, FV5PerformanceBudgetCatalogTest (V5.Tests.PerformanceBudgets, V5TestsAutomation.cpp:324), independently loads the catalog and asserts the 9/11/11 counts, that the profiled p95 stays within budget, that every profile blocks release through its regression suite, and that the four special-platform flags are set. Two independent readers — Python in CI and C++ in-engine — must agree with the data, which is how the budget gets enforced rather than merely documented.

Arcade cabinet#

V5 also cooks a distinct location-based SKU. V5ArcadeCabinet (V5/ue/Source/V5ArcadeCabinet.Target.cs) is a four-player local-co-op cabinet target with its own compile definitions (V5_ARCADE_CABINET, V5_CABINET_ONLINE_SERVICES=0), packaged through a dedicated ArcadeCabinetEngine.ini, profiled at 4K60 with a 16.6 ms budget, and validated by its own manifest test (FV5ArcadeCabinetBuildTargetManifestTest). It cooks only a curated roster (Spec Ops Co-op, Horde Defense, Fleet Co-op, Hunter Card Game, Training Range) and hides open online/PvP rulesets behind the build-target gate.

Testing and the eval gates#

The automation surface#

The UE automation layer is V5Tests (V5/ue/Source/V5Tests/, depending only on Core/CoreUObject/Engine/Json/V5Core). The bulk of its 59 suites are content-as-data manifest validators — they load a plugin's JSON, assert its schema version, content counts, and stable ids, and check release-gate booleans (FV5CrossCellReplayEditorManifestTest, for instance, verifies 7 supported cells, 14 source clips, 13 cross-cell splices, and that every clip is hash-verified, privacy-scrubbed, and rights-cleared). This is honest about what V5 is today: a logic-and-manifest skeleton whose tests gate the data, since there is no binary content to gate. Three suites, though, exercise the quality machinery itself.

Feel tests#

The feel-test catalog (UV5_Test_FeelTestCatalog::BuildFeelTestScenarios, V5TestSystems.cpp:108) defines nine reviewer-facing scenarios, each pinned to its genre inspiration: Urban heist-chase-swap (GTA) and fish-market brawl (Sleeping Dogs), Period made-man escort (Mafia) and crime-scene interrogation (LA Noire), Frontier outlaw-trail ride (Red Dead), monster-hunter contract (The Witcher), Sci-Fi squad interrupt (Mass Effect) and zero-G duel (The Expanse), and the cross-cell Mind Palace case file (Sherlock). Every scenario declares a runtime path, required gameplay systems, and ≥3 blocking success criteria plus ≥3 failure modes. FV5FeelTestCatalogTest (V5.Tests.Feel.Catalog) asserts exactly nine, unique ids, and that precisely one — Mind Palace — is the cross-cell meta-layer. Each scenario is mirrored as a source-controlled *.feeltest.json record under V5Tests/FeelTests/ so the nightly driver can assemble its plan without loading content assets, and the C++ catalog and the nine JSON files are kept in lockstep by the suite.

Golden-replay determinism#

UV5_Test_GoldenReplayHarness is the determinism dividend. BuildInputStream (:331) emits a deterministic stream — six input events, four expected outputs, and a StableFingerprint computed by HashCombineFast over the scenario id, seed, and every input frame/payload (BuildFingerprint, :71). ValidateReplay (:362) matches expected outputs to actuals within a ±2-frame tolerance and an optional numeric tolerance, and returns Passed only when no issue is logged. FV5GoldenReplayHarnessTest (V5.Tests.GoldenReplay) proves both directions: a faithful replay passes with all four outputs matched, and mutating a single actual value to "false" flips the result to Failed with a reported issue — a test that would fail against a stub that always returned success.

The Gauntlet driver#

UV5_Test_GauntletDriver::BuildNightlyRun (:410) assembles the nightly plan: one parallel worker (conservative for small hosts), the golden-replay harness plus all nine feel scenarios for ten entries total, each with a real UE automation command and a per-scenario timeout. BuildCommandLine (:439) emits the unattended invocation (-NullRHI -NoSound -Unattended -NoSplash -TestExit=... -ExecCmds=... -ReportExportPath=...), and ValidateNightlyRun (:447) refuses a plan that drops a feel scenario, omits the golden-replay entry, runs more than one worker, or leaves any entry non-blocking. The architecture's release-gate list (nightly Gauntlet pass, each cell's feel suite, adversarial stub-scan, >99.5% PS5 crash-free, memory-within-budget on a 10-hour run, packet-loss tolerance at 5%/20%, localization and accessibility) composes on top of this driver.

The content pipeline#

Source control, CI, and where the content actually is#

V5 splits its repository in two: the Oshun monorepo (Git) holds all logic, manifests, and tooling; binary assets live in Perforce (Helix Core) under V5/ue/Content/, outside Git's scope, with per-asset checkout locking to avoid binary merge conflicts. Branching is trunk-based with short-lived feature branches; CI is Buildkite for tests, Horde for distributed cook, and Perforce asset CI. Build/cook/patch mirror V4: UnrealBuildTool + Horde for distributed compile, per-platform parallel and iterative cook, and chunked UnrealPak iterative patches sized per-cell for partial-download installs. The deeper cook, packaging, and content-as-data accounting are owned by the sibling telemetry, build & data page.

Validators are the teeth#

Because V5 ships 0 .uasset and 29 .uplugin, the content pipeline's enforcement today is data validation, and it is extensive. V5/tools/ carries 68 test_validate_*.py suites plus validate-v5-docs.py, which performs referential integrity across the planning docs and content manifests (every mission's required NPC exists, every vehicle's required mesh is declared). These Python validators and the 59 UE manifest tests are two independent checks over the same JSON, which is what keeps content-as-data honest while the binary content is unauthored.

Creator suite#

The creator surface is a set of UE-Editor + Slate tools — Mission Editor, Heist Author, Case Author, Contract Author, Ship Loadout Author, Bestiary Author, Cinematic Director Mode, the Cross-Cell Replay Editor, and a one-click Workshop Publisher — each backed by a manifest and a V5Tests manifest validator (the replay editor's, above, is representative).

The launch-readiness gate (honestly red)#

§"Launch Readiness" aggregates everything above into a single go/no-go, and it is the page's load-bearing example of fail-loud honesty. V5/release/launch-readiness-manifest.json (schemaVersion: 2) tracks 17 blocking gates, and today every one reads pending — passedGates: 0, cellsComplete: 0, languagePacksLocked: 0, platformsCertPassed: 0, releaseBlocked: true, greenBuildId: null. Its correctionNote records why: a prior revision fabricated a shippable game — 17/17 gates passed, a 99.72% crash-free rate over 125,000 PS5 sessions, nine platform cert passes, a completed external accessibility audit — with no packaged build, hardware, sessions, or external engagement behind any of it. V5/REMEDIATION_2026-06-12.md reset it (and moved the backlog from 905 [x] to 875 [x], promoting zero marks to done). The note is blunt about the single real signal: "the only verified quality signal to date is the one-off 2026-05-31 on-box automation run (298/298 after remediation); no green nightly pipeline exists."

The gate cannot be hand-edited green. V5/tools/release/validate-launch-readiness.py (399 lines) fails closed: a gate may claim passed only with backing evidence that exists on disk; releaseBlocked: false requires every gate passed and a real greenBuildId; a cert-passed platform needs memoryWithinBudget plus an existing certEvidencePath; a "complete" accessibility audit needs a real named consultant ("TBD" is rejected); the crash-free gate needs a cohort ≥100,000 with rate above threshold; and it runs its own regex stub-scan over the runtime tree for coming soon|not implemented|todo|fixme|tbd. It composes the cell, performance-budget, and live-service validators, so a regression in any of them blocks launch. The UE test FV5LaunchReadinessManifestTest (V5.Tests.LaunchReadiness) mirrors the fail-closed logic in C++ — its own comment records that it once hardcoded the fabricated 17/17 expectations and now validates internal consistency instead.

Edge cases and failure modes#

• Telemetry validates and redacts before it leaves the client. A malformed or oversized event is rejected at the edge by ValidateEvent; an opted-out account emits nothing but the essential consent event; a DSAR export is redacted and scoped to the requester. The module fabricates no transport — it is honest about being client-side logic feeding the services contract.
• The performance budget is enforced by two independent readers. Python in CI and C++ in-engine both check the same catalog and .ini bindings, so a config that drifts from the budget fails the build rather than shipping a stutter.
• The launch gate fails closed, and its red is the point. Zero of 17 gates pass today; the validator refuses a green without on-disk evidence and a real build id, and the correctionNote documents the exact fabrication the fail-closed rewrite exists to prevent.
• Observability is not yet on the shared spine. No V5 service declares @oshun/metrics/@oshun/tracing; the honest framing is a ClickHouse/Grafana ingest pipeline with residency-pinned failover, not a per-service RED-metrics endpoint — a real gap relative to V2/V3, named rather than papered over.

Where this connects#

• Sideways: telemetry, build & data owns the cook, arcade-cabinet packaging, and the content-as-data accounting (0 .uasset, 29 .uplugin) this page's pipeline and budgets assume; accessibility, security & compliance owns the WCAG audit, DSAR pipeline, residency, and ratings reviews that several launch gates above depend on.
• Platform foundations it composes: the shared platform observability libraries (@oshun/metrics, @oshun/tracing) — real in the monorepo, and the spine V5's services have yet to adopt — beside which the telemetry contract and Grafana dashboards are versioned.
• The section hub: ../V5_ARCHITECTURE.md.