# Telemetry, Performance, Testing & Release Gates

This page is V2's answer to a single question: _how do you know the build is
good enough to ship?_ For a cross-platform competitive game — the fighting
roster on one side, the racing rulesets in `V2/balance/racing` on the other,
both feeding a ranked ladder — that question has no single answer, only a
composition of four interlocking systems. **Telemetry** is what the live game
tells you; **performance budgets** are the per-platform frame-time contract that
keeps 60 fps a competitive-integrity guarantee rather than a setting;
**testing** is the automation that proves the deterministic simulation still
behaves frame-for-frame; and the **release gate** composes all three (plus cert,
canary, and crash-rate) into a go/no-go. The determinism that powers rollback is
what makes this tractable: because a match is a pure function of its inputs, you
can re-run it from a recorded input stream and assert the result hash exactly —
a testing affordance most games simply do not have. This page is the quality
spine, and it is deliberate about which parts are running C++ and CI, which are
machine-checked policy, and which are documented obligations no build yet
satisfies. The section hub is [../V2_ARCHITECTURE.md](../V2_ARCHITECTURE.md).

## What ships, honestly

The **in-engine telemetry is real and substantial**, not a contract scaffold:
`V2Telemetry` (`V2/ue/Source/V2Telemetry/`, nine files, ~2,900 lines) implements
the event taxonomy, the batcher, HMAC event signing, integrity heartbeats, and
the PII/privacy/sampling policies as deterministic Blueprint libraries. The
**testing surface is real and large**: `V2Tests` holds **407 `*.spec.cpp`
suites** across 442 files, including a literal **200-replay golden corpus**
re-simulated frame-by-frame on every run. The **frame-budget and launch gates
are real CI tooling** — `check-frame-budget.py` and
`check-v2-launch-readiness.py` both run in `.github/workflows/v2-build.yml`, not
as prose.

Two honest qualifications carry through the page. **First**, the architecture
claims "all Node packages under `apps/v2/` declare `@oshun/metrics` and
`@oshun/tracing` from day one"; the ground truth is **50 of 91** service
package.json files declare them directly — the in-engine
`FV2ServiceObservabilityBinding` is the contract that names the surface, but the
per-package adoption is partial, and that gap is named here rather than papered
over. **Second**, the **launch-readiness gate is honestly fail-closed**: its
validator forbids the §138 checklist from marking anything `[x]` because no
build, beta, cert pass, or drill exists, and the machinery is written so that
red cannot be forged.

## Telemetry & analytics

### The in-engine event spine

V2's client telemetry is a registered, schema-validated taxonomy.
`UV2TelemetryBlueprintLibrary::BuildDefaultEventSchemas`
(`V2/ue/Source/V2Telemetry/Private/V2TelemetryBlueprintLibrary.cpp:130`) defines
**39 event schemas**, each an `FV2TelemetryEventSchema` carrying a category, a
reliability class, its required fields, a `bStripPII` flag, and — critically —
`bAllowedDuringRollback = false` (`V2TelemetryTypes.h:95`). The taxonomy spans
the standard lifecycle (`v2.player.session.started/ended`,
`v2.match.level.loaded`, `v2.player.achievement.unlocked`,
`v2.cosmetic.purchased`, `v2.player.ui.interacted`, `v2.player.error.reported`)
and the fighting-game-specific events that make balance legible:
`v2.match.combat.move.used`, `…damage.applied`, `…combo.ended`,
`…resource.used`, `…finisher.triggered`, `…attack.blocked`, and the
release-gated `v2.match.combat.frame-data.drift`. Reliability is a real
three-valued enum — `BestEffort`, `Reliable`, `ReleaseGate` — so a crash, a
frame-budget breach, a cheat-detection signal, or a frame-data drift is tagged
as a gate-quality event at the schema, not by convention.

The defining architectural choice is the destination. `FV2TelemetrySenderConfig`
(`V2TelemetryTypes.h:297`) pins `Endpoint = "event-bus://@oshun/event-bus"` and
`EventBusPackageName = "@oshun/event-bus"`: V2 publishes to canonical
`v2.match.*`, `v2.cosmetic.*`, and `v2.player.*` topics on the shared bus,
**not** to a V2-private ingest. `BuildEventBusPublishRecord` (`:689`) stamps
each event with its topic, source domain, schema version, and
`bMayInfluenceRollback = false`. That single decision is what lets the analytics
warehouse, the Maat balance team, and the Iris AI-commentary system all consume
one stream without V2 reaching into anyone's database.

The batcher is a real algorithm, not a buffer that flushes on a timer.
`FV2TelemetryBatch::CanAddEvent` (`V2TelemetryTypes.h:347`) admits an event only
while the batch stays under both `MaxEventsPerBatch` (64) and `MaxBatchBytes`
(64 KB), and `UV2TelemetrySubsystem` (`V2TelemetrySubsystem.h:48`) — a real
`UGameInstanceSubsystem` — drives `EnqueueEvent` → `Flush(reason)` →
`CaptureRuntimeSnapshot`, where the flush reason is the typed
`EV2TelemetryFlushReason` (`Interval` / `Size` / `Shutdown` / `Manual`). A
server-side variant, `FV2ServerLogAggregationPolicy`, rejects raw log lines
(`bRejectRawLogLines`), requires structured game events, and mirrors them into
the same event-bus batch — so dedicated-server logs and client telemetry land in
one schema-validated pipe.

### Signed events and integrity heartbeats

Telemetry that feeds anti-cheat must be tamper-evident, and V2 implements that
with real cryptography rather than a flag. `SignTelemetryEvent` (`:494`) builds
a canonical, attribute-sorted payload and signs it with `ComputeHmacSha1Hex`
(`:90`), which calls `FSHA1::HMACBuffer` — genuine HMAC-SHA1 over the UTF-8
payload, not a hash placeholder. `VerifySignedTelemetryEvent` (`:512`)
recomputes the MAC, checks the session/key id, and — when a
`v2.match.replay.hash` event fails verification — sets `bReplayEventTampered`,
the signal the integrity layer escalates on. `BuildIntegrityHeartbeat` /
`VerifyIntegrityHeartbeat` (`:554` / `:578`) extend this to a per-session
heartbeat that binds the build hash, content version, and replay-seed checksum;
a mismatch sets `bIntegrityHeartbeatDesync`, catching a client whose simulation
has diverged from the authoritative build. Every one of these structs carries
`bMayInfluenceRollback = false` — the integrity-telemetry path is observable but
can never perturb the deterministic sim it observes.

### Ingest, PII, sampling, and the balance dashboards

`FV2TelemetryIngestPipelineConfig` declares the warehouse path as an explicit
stage list — `Firehose → PiiStrip → Warehouse` — with `bPiiStripAtIngest` and
`bRejectRawAccountIdentifiers` set, so raw account ids never reach analytics;
the full PII-allowlist contract lives in
[Security & Compliance](./security-compliance-and-sister-monorepo-integration.md).
`FV2TelemetryPrivacyPolicy` makes `bOptOutPersistsAcrossSaveReset` a structural
property (a save wipe cannot silently re-enable collection), and
`FV2GhostDataSharingPolicy` keeps ghost capture `bOptInRequired`. Sampling is
policy-bounded but safe: `FV2TelemetrySamplingPolicy` randomizes a per-session
sample percentage yet forces `bCrashEventsAlwaysSampled` and
`bCheatDetectionEventsAlwaysSampled`, so the gate-quality events are never
sampled away.

The payoff is the **balance dashboard set**: `BuildDefaultBalanceMetrics`
(`:193`) defines 12 `FV2BalanceDashboardMetric`s — `move_pick_rate`,
`win_rate_on_hit`, `win_rate_on_block`, `frame_data_drift`, the
`hot_cold_fighter_heatmap`, `finisher_trigger_rate`, `rage_quit_rate`,
`frame_budget_p99`, and the player funnels (`match_acceptance_funnel`,
`per_fighter_retention`, `churn_risk_weekly`) — each wired to a Grafana URI and
a source event. Because `frame-data.drift` events compare authoring frames to
runtime frames, the same dataset the
[data pipeline](./build-cook-assets-data-and-production.md) exports as
`frame-data.csv` is what the dashboard argues from, and an A/B framework
(`FV2BalanceExperimentPolicy`, `bRollbackRequired = true`) lets a balance change
be canaried with a rollback runbook before it goes wide.

## Performance budgets

### The per-platform frame-time contract

The frame-time contract is per-platform and explicit, because frame pacing is
input timing in a competitive game:

| Target                       | Resolution   | FPS     | Notes                           |
| ---------------------------- | ------------ | ------- | ------------------------------- |
| PS5 / Xbox Series X          | 4K           | 60      | RT off (Pro: RT shadows)        |
| PS5 Pro (perf)               | 1080p        | 120     |                                 |
| Xbox Series S                | 1440p        | 60      | reduced VFX density             |
| Switch 2 (docked / handheld) | 1080p / 720p | 60 / 30 | quality-band drop               |
| PC baseline (3060/4060)      | 1440p        | 60      | medium                          |
| Steam Deck                   | 800p         | 60      | Verified rating launch-blocking |

The per-frame split is a hard budget, and it exists as data, not prose:
`BuildDefaultQAReleaseGateFeatureSet`
(`V2/ue/Source/V2Tests/Private/V2AutomationHarness.cpp:404`) populates
`SystemFrameBudgets` as gameplay **4 ms**, animation **3 ms**, render **8 ms**,
audio **1 ms**, network **0.5 ms**, and incremental GC **1 ms**;
`LoadingBudgets` bound cold-boot to 30 s on PS5 and a mid-session match load to
8 s; and a `LatencyBudget` block pins input-to-render at **35 ms @ 60 fps / 25
ms @ 120 fps** with a 16.6 ms sim tick and a 2 ms controller poll, flagged
`bMeasuredWithControllerCameraRig` so the number is an end-to-end measurement,
not a synthetic estimate. `FV2CrashPerfPolicy` mirrors the 16.667 ms frame
budget against the per-platform budget ids (`PC.60Hz`, `PS5.60Hz`, `XSX.60Hz`,
`Switch2.60Hz`) and emits `v2.match.performance.frame-budget-exceeded` when a
frame blows it.

### The frame-budget gate is real CI

The budget is _enforced_, not merely documented.
`V2/ue/Tools/check-frame-budget.py` (324 lines) parses Unreal JSON/CSV profiling
exports, identifies combat frames by marker (`phase: Combat`,
`combatFrame: true`, a `scope`/`stat` containing `V2Combat`) or by being nested
under a combat-labeled collection, reads the worst millisecond field from each
sample, and **fails the build on any combat frame over 16.6 ms**. It ships a
`--self-test` mode that proves a good fixture passes and an over-budget fixture
fails — so the gate's own correctness is regression-tested. Both run in CI:
`.github/workflows/v2-build.yml` invokes `check-frame-budget.py --self-test`
(line 2312) and the real profiling run (line 2360) after automation, on the
Win64 reference runner, per `V2/docs/ci/frame-budget-gate.md`. A regression past
budget fails the PR rather than shipping a stutter.

## Testing, QA & cert

### The automation surface

The UE automation layer is `V2Tests` — **407 `*.spec.cpp` suites** (30 in
`Automation/`, the rest under `Private/`) across 442 files — covering GAS, the
motion parser, the hitbox engine, rollback determinism, save round-trip, and
data-table integrity: precisely the subsystems where a silent regression is a
balance or fairness bug rather than a cosmetic one. `Services.Module.spec.cpp`
asserts the service surface directly (`:45`): 62 required Oshun domain adapters
and exactly 2 observability bindings, each verified to name `@oshun/metrics` and
`@oshun/tracing`, to be `bRequiredFromDayOne`, and to stay off rollback.

### Golden-replay regression — the determinism dividend

`GoldenReplayCorpus.spec.cpp` (`V2/ue/Source/V2Tests/Private/Netcode/`) is the
load-bearing test, and the one a non-deterministic game cannot write. The corpus
is **200 deterministic input-stream replays**
(`constexpr int32 CorpusSize = 200`) persisted as JSON manifests under
`Content/V2/Tests/GoldenReplays/Corpus/`, eight gameplay archetypes (neutral
walk, eight-way run, stance cycling, heat/ki charge, power-crush guard, pressure
strings, hit exchange, juggle carry), each storing its exact per-frame input
stream plus state hashes sampled every 10 frames and a final hash. The
regression re-runs every manifest through `FV2SimWorld` and **fails on the first
divergent frame** — so a rendering or logic change that perturbs the sim by even
one frame is caught before it ships. The generator runs only under the explicit
`-V2GenerateGoldenCorpus` flag, self-verifies each manifest by an immediate
re-simulation before writing, and never overwrites silently; cross-process
stability of `FV2SimWorld::ComputeStateHash` is itself part of what the
regression asserts. The release-gate feature set fixes the contract at
`RequiredReplayCount = 200`, `FramesPerReplaySample = 600`,
`bFrameByFrameDivergenceDetection = true`, `bRunsPerBuild = true`, and
`bNightlyAllCookedBuilds = true`.

### Gauntlet, cert & dogfood

Above the unit/replay layers, `BuildDefaultGauntletDrivers`
(`V2AutomationHarness.cpp:223`) assembles full-game smoke suites (launch,
tutorial, arcade ladder, online-headless 1v1) that run on every PR. The cert
surface is enumerated as data in the same feature set:
`CertCompliance.PlatformCertIds` lists `PlayStation.TRC`, `Xbox.XR`,
`Nintendo.Lotcheck`, `SteamDeck.Verified`, and `MicrosoftStore.Cert`, with
`bAntiCheatCompatibilityCertPerPlatform` and a trophy/achievement plan that
`AvoidsForcedGrind`. `BetaDogfood` declares a 30-day internal closed beta, a
pre-launch network-test weekend, and a press-playable feature freeze. These are
specifications the launch gate below composes — real structured policy, but
policy whose teeth are the validators, not a runtime that has executed them.

## Observability & the release gate

### The shared spine, and an honest coverage gap

V2's intended posture is that every backend surface emits through the shared
platform libraries. The in-engine contract is real:
`FV2ServiceObservabilityBinding`
(`V2/ue/Source/V2Services/Public/V2ServiceTypes.h:83`) and
`BuildRequiredObservabilityBindings` (`V2OshunDomainAdapters.cpp:121`) produce
two bindings — `v2-services-node` over `apps/v2/*` and `v2-services-unreal` over
`V2/ue/Source/V2Services` — each pinning `@oshun/metrics` / `@oshun/tracing`,
`bRequiredFromDayOne = true`, and `bMayInfluenceRollback = false`, and the
Services automation spec enforces all of it. The shared libraries are themselves
substantial: `@oshun/metrics` (`libs/shared/metrics/src/registry.ts`, 416 lines)
is a real prom-client registry, and `@oshun/tracing` ships a tracer,
propagation, decorators, and middleware.

The honest qualification is coverage. The architecture asserts _all_ Node
packages under `apps/v2/` declare the spine "from day one"; in fact **50 of 91**
package.json files declare `@oshun/metrics` and `@oshun/tracing` directly — the
data-pipeline and analysis libs (e.g. `@v2/ab-test-analysis-pipeline`) compose
domain packages instead. So the accurate framing is: the binding is the day-one
_contract_ and the in-engine test enforces it, while per-package adoption across
the 91 services is partial — a real gap, named rather than implied away. Beyond
the spine, observability includes a symbolicated crash reporter, in-the-loop dev
fault injection (`bDevFaultInjectionEnabled`), per-platform perf dashboards, the
network-quality dashboard (matchmaking p99, rollback-frame distribution, packet
loss, rage-quit rate), an anti-cheat false-positive dashboard, and
canary-analysis automation for the online services.

### The launch-readiness gate (honestly fail-closed)

The release gate is the composition — regression suite + golden replays +
frame-budget + crash rate + cert pass + RTC pass + canary pass — and §138 Launch
Readiness aggregates it into one go/no-go. Its data lives at
`V2/balance/launch/launch-readiness.json` (schema
`v2.launch.readiness-data.v1`): 60-day internal dogfood across
PC/PS5/XSX/Switch2, a 5× network-stress multiplier, a five-dashboard monitoring
set, a canary/rollback plan, **27 critical journeys**, **33 feature gates**, and
**21 exit criteria** (60 fighters, 30 stages, 100 ms playable RTT, a 30 ms
local-feel reference, and a `noComingSoonPanelsVisible` proof).

What makes this honest is the validator.
`V2/ue/Tools/check-v2-launch-readiness.py` (404 lines, run in CI at
`v2-build.yml:2125`) cross-checks the contract, the data, the C++ catalog
source, and the docs — and then **fails closed on the TODO**. Its embedded rule
(lines 327–339) is blunt: because no build, deployment, beta, cert, or drill
exists, the §138 checklist rows must stay `[ ]`/`[~]`, and **any `[x]` is
treated as a fabricated completion** unless it is on an explicit evidence
allowlist with committed proof on disk. Today that allowlist holds exactly
**one** row — the §6.7 per-move verification, backed by the on-box Linux-editor
automation report
(`V2/docs/qa/automation-runs/2026-06-12-linux-editor-report.json`, requiring a
`Success` run of `V2.Combat.MoveData.AssetContract`). Every other launch row is
held red by construction. The gate is therefore not green and the tooling
refuses to let it be hand-edited green — the correct posture for a launch gate
over a game that has not yet been built, beta'd, or certified.

```mermaid
flowchart TB
  subgraph pr["Every PR / nightly"]
    A["V2Tests · 407 spec suites"]
    G["Gauntlet smoke<br/>(launch · tutorial · ladder · online 1v1)"]
    R["200 golden replays<br/>FV2SimWorld · frame-exact"]
    C["check-frame-budget.py<br/>≤ 16.6 ms or fail"]
  end
  subgraph obs["Live observability"]
    T["V2Telemetry → @oshun/event-bus"]
    M["@oshun/metrics · @oshun/tracing<br/>(50/91 services adopt)"]
    D["crash · netquality · anti-cheat dashboards"]
  end
  A & G & R & C --> Gate{"check-v2-launch-readiness.py<br/>fail-closed"}
  T --> D --> Gate
  M --> Gate
  Gate -->|"§138 rows held [ ]/[~]<br/>1 evidence-backed [x]"| Red["releaseBlocked (today)"]
  Gate -.->|"only on real build · beta · cert · canary"| Ship["green = ship"]
```

## Edge cases and failure modes

- **Telemetry can never perturb the sim.** Every schema sets
  `bAllowedDuringRollback = false` and every publish/binding/policy struct sets
  `bMayInfluenceRollback = false`; observability rides strictly _off_ the
  deterministic path it observes.
- **Tampered replay-hash telemetry is flagged, not trusted.** A
  `v2.match.replay.hash` event with a bad HMAC trips `bReplayEventTampered`, and
  a heartbeat whose build hash / content version / replay-seed checksum diverges
  trips `bIntegrityHeartbeatDesync` — real HMAC-SHA1 verification, escalated to
  the integrity layer in
  [Online Backbone & Competitive Integrity](./online-backbone-and-competitive-integrity.md).
- **Opt-out survives a save wipe; crash/cheat events survive sampling.**
  `bOptOutPersistsAcrossSaveReset` and the always-sample overrides are
  structural, so privacy and gate-quality signals are not silently lost.
- **One divergent frame fails the build.** The golden corpus re-simulates
  through `FV2SimWorld` and stops on the first hash mismatch — the determinism
  dividend the [combat](./combat-system-gas-frame-data-and-determinism.md) and
  [rollback](./rollback-netcode-and-tag-team.md) layers earn.
- **The launch gate fails closed, and its red is the point.** The validator
  rejects any §138 `[x]` without committed automation evidence, so a build
  cannot be declared launch-ready by editing a checklist.

## Where this connects

- **Up to what it measures:**
  [Combat: GAS, Frame Data & Determinism](./combat-system-gas-frame-data-and-determinism.md)
  and [Rollback Netcode & Tag-Team](./rollback-netcode-and-tag-team.md) — the
  determinism the golden-replay regression and frame-data drift metric exploit.
- **Sideways:**
  [Build, Cook, Assets, Data & Production](./build-cook-assets-data-and-production.md)
  (the cook frame-budget gate and the `frame-data.csv` export the dashboards
  read) and
  [Security, Compliance & Sister-Monorepo Integration](./security-compliance-and-sister-monorepo-integration.md)
  (the PII allowlist applied at telemetry ingest).
- **Integrity:**
  [Online Backbone & Competitive Integrity](./online-backbone-and-competitive-integrity.md)
  — where signed telemetry and integrity heartbeats feed anti-cheat and the
  replay-hash validator.
- **Platform:** the [shared platform](../../platform/overview.html)
  observability libraries (`@oshun/metrics`, `@oshun/tracing`) every V2 service
  composes.