# Esports, Companion & AI Services A fighting game becomes a sport, and then a show, in three steps that all live _outside_ the match. The match itself is the deterministic, integer-frame simulation described in [Rollback Netcode & Tag-Team](./rollback-netcode-and-tag-team.md); everything on this page wraps that simulation without ever being allowed to perturb it. The **esports backend** turns a verified result into a bracket, a Crown Points standing, and a broadcast; the **companion app** gives a player a phone-and-watch surface for frame data, replays, store, and coach notes that is deliberately kept off the competitive path; and the **AI services** — an AI Director that paces single-player encounters and a real-time AI commentary stack that casts a match with synthesized voice, lip-synced faces, ducked music, and live translation — make a V2 broadcast feel _produced_. The unifying design rule is that V2 builds none of the underlying intelligence: it composes the shared Oshun substrates (Iris, Isis, Psyche, Euterpe, Uzume, Veritas, Shakti) through thin, rollback-aware `apps/v2/` bridges, supplies the fighting-game context, and consumes a result it can audit. The section hub is [../V2_ARCHITECTURE.md](../V2_ARCHITECTURE.md). ## What ships, honestly The split here is the same one the sibling [Online Backbone & Competitive Integrity](./online-backbone-and-competitive-integrity.md) page draws, and it is worth stating plainly. The **bridges and tooling are real code.** `apps/v2/` ships the AI-services integration packages — `iris-commentary-orchestration`, `psyche-ai-director-hints`, `isis-commentary-lipsync-pipeline`, `psyche-commentary-face-animation`, `euterpe-commentary-ducking`, `iris-realtime-translation`, and `iris-accessibility` — each a substantive TypeScript module that composes a shared substrate and enforces a rollback boundary. `apps/v2/esports-tools/` ships `@v2/esports-tools` with three real engines (broadcast, arena LED/projection, fact-checking). `apps/oshun/mobile/v2/` ships the companion shell model. The in-engine seams are real Unreal C++: `V2/ue/Plugins/V2AICommentary/` and `V2/ue/Plugins/V2AdaptiveAI/` carry typed contracts and Blueprint libraries, and the AI Director is a genuine C++ state machine, not a config struct. The gRPC wire contract for this page's three services **now ships**: real proto3 contracts live at `libs/proto/src/oshun/v2/esports/esports.proto`, `.../director/ai_director.proto`, and `.../ai_commentary/commentary.proto` (registered in `@oshun/proto`'s loader and service registry), and `@v2/grpc-services` binds all three — `oshun.v2.esports.Esports`, `oshun.v2.director.Director`, `oshun.v2.ai_commentary.Commentary` — onto one `@grpc/grpc-js` server backed by the real bridges (`@v2/esports-tools` + `@maat/strategy` Crown Points, `@v2/psyche-ai-director-hints`, `@v2/iris-commentary-orchestration`), with in-process round-trip tests covering streaming, fallback, and rollback rejection. The **other** V2 proto packages the architecture names (online, live, progression, community, racing, heist, world-boss) remain the _designed_ contract, not files in the tree. And the **generation itself is provider-gated**: the bridges orchestrate, validate, budget, and route, but the live commentary text, the synthesized voice, and the generated face animation come from model-backed substrates (and ultimately providers) those bridges call. The orchestration ships; the generation is a capability that fails loud when its provider is absent, exactly the seam the platform uses everywhere. ## The esports backend V2 treats competitive play as a first-class surface. The backend's job is to take a result the [online backbone](./online-backbone-and-competitive-integrity.md) has already verified against the replay hash, and turn it into a standing and a show — while keeping every one of those operations provably off the deterministic path. ```mermaid flowchart TB Event[Tournament event] --> TO[TO badge tools] TO --> Rule[Per-event rule file · JSON] Rule --> Lock[Client Tournament Edition lock · stamped in replay header] Event --> Bracket[Bracket · Start.gg / Challonge / Battlefy] Bracket --> Ingest[Result ingest · V2 pulls, never client-pushed] Ingest --> Verify{Replay-hash verified?} Verify -->|yes| Crown[Crown Points pool] Verify -->|no| Hold[Held for review] Crown --> Finals[World Final qualification] Event --> Tools["@v2/esports-tools"] Tools --> Cast["@uzume/broadcast · switcher · multiviewer · graphics · replay · SRT"] Tools --> Arena["@uzume/lumina + @uzume/prism · LED / projection"] Tools --> Fact["@veritas/fact-checking · post-match reporting"] ``` ### Broadcast, arena, and fact-checking tooling `@v2/esports-tools` is the real operator surface. Its broadcast engine, `buildV2EsportsBroadcastPipeline` (`apps/v2/esports-tools/src/esports-broadcast-pipeline.ts:193`), composes seven `@uzume/broadcast` engines into one plan: a multi-platform streaming engine that upserts per-destination encodings (1080p60 / 8 Mbps default), a stream-health dashboard with adaptive bitrate bands (floor 65%, ceiling 140%, 500 kbps steps), an ATEM-default switcher with `CLEAN`/`PGM`/`REPLAY` aux buses, a director multiviewer layout, three CasparCG graphics templates (scorebug, bracket, next-match countdown), an EVS-LSM replay controller, and SRT contribution feeds. The plan is explicitly inert on the match: it returns `offRollback: true`, `mayInfluenceRollback: false`, and `deterministicImpact: 'none'` (`:341`). Its `reasonCodes` builder _refuses_ to let standalone OBS, NDI, or vMix be a primary integration (`:542`); OBS WebSocket is retained for **one** purpose only — streamer-mode notification suppression (`V2_ESPORTS_BROADCAST_OBS_WEBSOCKET_PURPOSE`, `:31`). Sibling engines in the same package compose `@uzume/lumina` + `@uzume/prism` for arena LED walls and projection mapping (presentation/show-control state only, `mayInfluenceRollback: false`) and `@veritas/fact-checking` for post-match reporting and bracket-result verification, where a winner-record conflict holds publication without ever changing match authority. ### Tournament integrity and Crown Points The single load-bearing integrity feature lives one page over but is worth restating: **Tournament Mode freezes the balance hotfix at the TO's snapshot and stamps that Tournament Edition into the replay header** (see [Security, Compliance & Sister-Monorepo Integration](./security-compliance-and-sister-monorepo-integration.md)), so a recorded set is reproducible against the exact ruleset it was played under. Results are **server-authoritative by ingestion shape**: V2 _pulls_ results from Start.gg / Challonge / Battlefy via `@oshun/inbound-integrations` rather than accepting client-pushed scores, and every match is cryptographically verified against its golden-replay hash before it counts. Only then does the Crown Points formula apply — `base_points[round] × region_factor × participant_factor × season_decay`, with round bases stepping 50 (R32) → 100 (R16) → 200 (QF) → 400 (SF) → 800 (GF) — feeding World Final qualification through the `@maat/strategy`-owned Pro Circuit calendar. SLOs are concrete: result ingest p99 ≤ 5 s, live bracket fetch p99 ≤ 200 ms, broadcast overlay update p99 ≤ 100 ms. ### The results archive and the gRPC seam The public-facing spectator surface is real and static: `apps/v2/web/esports/results/` ships `index.html`, `results.js`, `results.css`, and a `results.json` (`schema: v2.esports.resultsArchive.v1`) that is a searchable tournament archive — searchable by fighter, player, date, and event, with explicit ranking weights (championship 120, major 90, regional 55, local 25, plus win and replay bonuses) and per-event brackets, match scores, and replay URIs. The gRPC service backing live operation — `Esports.RegisterEvent(rule_file) → event_id`, `SubmitResult(event_id, match_result, replay_hash) → ack`, `GetBracket(event_id)`, `QualifyForFinals(season_id, region)` — **ships** at `libs/proto/src/oshun/v2/esports/esports.proto`, served by `@v2/grpc-services`: `SubmitResult` fact-checks every result through `@veritas/fact-checking` (winner agreement across bracket, report, and replay records plus the sha256 replay hash) before crediting `calculateMaatV2CrownPoints`, and a conflict is `RESULT_HELD_FOR_REVIEW` with zero credit. The integrity claim rides exactly where the backbone page puts it: on the replay-hash verification the engine's determinism already guarantees. ## The companion app The companion is real and deliberately scoped. `apps/oshun/mobile/v2/` (`companionAppModel.ts`, `V2CompanionApp.tsx`, `V2CompanionLaunchTile.tsx`) builds its shell via `buildV2CompanionModel()` (`companionAppModel.ts:394`), which composes the shared Oshun client stack — `@oshun/shell-core`, `@oshun/auth-client`, `@oshun/concordia-integration`, `@oshun/trust-safety`, `@iris/agents`, and `@sophia/client` (`:31`). It is a **read surface**: move lists and a signature-clip library that are offline-ready, plus replay viewer, profile/match-history, friends-and-party, store-and-wishlist, and Battle Pass surfaces gated behind auth. ### Why it sits outside the competitive path The companion is, by explicit design, **not a participant in a ranked match** — an integrity decision as much as a product one, because a phone is not a deterministic match client. Its second-screen mode during a live match is modeled with `gameplayAuthority: false`, `cosmeticOnly: true`, `optInRequired: true`, and a 200 ms latency budget (`:265`): it receives a low-bandwidth opponent move list and frame-data panels for spectating, and can produce no gameplay effect. The one thing the companion _does_ let you play is a suite of daily reward micro-games (Combo Tap Trial, Drift Steer, Trivia Battle, Daily Puzzle Kombat, Mini Pit Crew), and even those are kept honest: every one carries `hmacRequired: true` and `serverValidated: true` (`:315`), so a reward is signed and server-checked and can never touch a ranked ladder. Push runs over APNS / FCM / web-push with per-category topics; a wearable glance (Apple Watch / Wear OS) surfaces daily stats, friend match starts, and Battle Pass tier progress. ### Assistants, RAG, and trust-and-safety The companion is where V2 wires the shared assistant plane. `buildV2CompanionModel` resolves `@iris/agents` launch readiness (`resolveIrisAgentsLaunchReadiness`) into a `ready`/`blocked` status — a fail-loud seam, not an assumed-available one — and selects a frame-data-query agent and a support/appeal agent, each carrying a `requiresHumanReview` flag. A local `@sophia/client` (port 3737) backs the frame-data / codex / wiki-citation RAG surfaces. `@oshun/trust-safety` classifies assistant-behavior policy hits (the model carries a sample prompt-injection hit) into a severity class with triage/action SLA budgets, and `@oshun/concordia-integration` registers the appeals nav item, a PII-redacting OTLP telemetry sink, and case-update subscriptions rate-limited to 30/min. The public API the companion and third-party tools share is OAuth 2.1 + PKCE with per-app scopes, an OpenAPI 3.1 document at `libs/openapi/v2/companion.yaml`, and rate limits of 100/min unauthenticated, 1000/min per registered app, and 30/min on personal data — API p99 ≤ 400 ms, push delivery p99 ≤ 5 s. ## AI services: a produced broadcast This is where V2's integration ambition shows, and where the rollback boundary is most aggressively defended in code. Two services live here — the AI Director (single-player pacing) and AI commentary (the produced broadcast) — and both are built so that nothing they emit can ever enter the deterministic simulation except as an explicitly serialized, hashed, match-start input. ```mermaid flowchart LR Match[Deterministic match · v2.match.* events] --> Dir["psyche-ai-director-hints
@psyche/behavior-prediction"] Dir -->|match-start snapshot only| Net[V2Netcode · deterministic input vector] Match --> Cue["iris-commentary-orchestration
@iris/conversation-orchestration"] Cue --> Voice[Synthesized caster VO · persona TTS] Voice --> Duck["euterpe-commentary-ducking
@euterpe/master sidechain"] Cue --> Face["psyche-commentary-face-animation
@psyche/avatar-lipsync + expressions"] Face --> Lip["isis-commentary-lipsync-pipeline
@isis/visual-dubbing"] Cue --> Tr["iris-realtime-translation
@iris/voice"] Cue --> A11y["iris-accessibility
@psyche/caption-streaming"] Voice -.audio/subtitle only.-> Audio[V2Audio · gameplay-inert] ``` ### The AI Director `@v2/psyche-ai-director-hints` is real domain code. It composes `@psyche/behavior-prediction` — `createBehaviorPredictionEngine` with a first-order Markov model, a 5-action horizon, and 30 s prediction validity (`psyche-ai-director-hints.ts:309`) — and maps each predicted action into a fighting-specific tendency vector (`pressure`, `defense`, `spacing`, `throwGame`, `resourceUse`) by term-matching the action against move families (`deriveTendencyVector`, `:428`). The architecturally critical part is the rollback discipline, and it is _enforced_, not described: live tendency hints are available only off-rollback (offline/training), and `assertValidLiveRequest` throws if rollback is enabled (`:240`). For online rollback-with-CPU, the only legal path is `buildV2AdaptiveAIMatchStartSnapshotPlan`, which samples Psyche at match load and bakes a `serializedTendencyVector` plus a `sha256` `vectorHash` into a `V2AdaptiveAIDeterministicMatchStartInputVector` that `v2NetcodeTreatsAsOrdinaryDeterministicInput` (`:395`). Mid-match updates are queued for the next match start; every hint carries `mayInfluenceRollback: false`. The in-engine `V2AdaptiveAI` plugin mirrors this: a real C++ state machine `EV2AIDirectorState { Calm, Build, Climax, Recovery }` (`V2AdaptiveAITypes.h:33`) whose `EvaluateAIDirectorDecision` hard-disables the Director — `bDirectorActive = false`, `bUsesStaticRuleOfDay = true` — whenever the player opts out, the session is not single-player, **or** the mode is ranked or tournament (`V2AdaptiveAIBlueprintLibrary.cpp:234`). Style matchup hints come from `@shakti/sota-critical` (boxer / striker / grappler / submission-specialist) over an off-rollback `shakti.player.style.updated` stream. Under the EU AI Act the Director is **Limited risk**: opt-out switches to a static CPU profile before any classifier use, and a DSR delete cascades to the Director profile, the Psyche tendency vector, and the Shakti classification. ### AI commentary and the post-match experience `@v2/iris-commentary-orchestration` composes `createMatchCommentaryStream` from `@iris/conversation-orchestration` (`v2-ai-commentary.ts:113`) into a cue plan whose every field declares the boundary: `rollbackPolicy: 'off-rollback-audio-subtitle-only'`, `allowedOutputChannels: ['audio', 'subtitle']`, `rejectsRollbackFrameOrigin: true`. Validation _rejects_ `fromRollbackFrame` outright (`:244`) and requires a `fallbackSeed`, because the real-time budget is a hard contract: the 800 ms KO-trigger budget decomposes into 420 ms LLM round trip + 240 ms persona TTS + 100 ms lipsync + 40 ms safety margin (`:23`), and on a miss the system falls back to a pre-recorded bank keyed by `(cueKind, ruleset, commentatorId, seed)`. The mirror C++ contract `FV2AICommentaryCueContract` carries the same budgets and the same off-rollback flags (`bOffRollbackBubble`, `bGameplayInertOutput`, `bFeedsSimulationState = false`, `V2AICommentaryTypes.h:53`). The post-match path `buildV2AIPostMatchExperiencePlan` (`:445`) is real selection logic: it sorts highlight candidates by score and takes the top three to five, requires the canonical moment kinds (longest-combo, counter-hit-punish, power-play, dramatic-comeback, KO), and emits an analysis screen, an auto-captioned clip library, and a compliance block whose disclosure text — "AI commentary is machine-generated presentation content." — is mandatory under EU AI Act cross-ref §80.4. Per `deps§67`, the generation providers are named (Anthropic Claude Sonnet 4 — the `claude-sonnet-4` registry default, with the exact version a configuration choice — for play-by-play text, ElevenLabs Multilingual v2 for TTS, Isis `ai-video` for the highlight auto-edit), with on-cap routing to `@iris/conversation-providers-local`; the Calliope commentator roster lives at `V2/ue/Content/V2/Audio/Commentary/commentator-personas.json` with cleared voice-clone metadata and talent signoff. ### The supporting render bridges The remaining bridges turn commentary text into a finished broadcast, each a real package composing a real substrate: - **`isis-commentary-lipsync-pipeline`** (`@isis/visual-dubbing`) extracts phonemes and viseme timing, drives visual dubbing, validates ≤ 45 ms lip-sync drift, and exports UE curves to `/Game/V2/Audio/Commentary/Lipsync` — a real replacement for Audio2Face. - **`psyche-commentary-face-animation`** reads `@iris/voice` output metadata and drives `@psyche/avatar-lipsync` viseme frames plus `@psyche/avatar-expressions` ARKit expression curves for the caster face. - **`euterpe-commentary-ducking`** does genuine audio-engineering math: `computeArrangementDensity` from `@euterpe/accompany` and the `@euterpe/master` `masteringChain.dbToLinear` to sidechain-duck `V2.DynamicMusic` under `V2.Commentary` to a −18 LUFS commentary target with a −1 dB true-peak ceiling (`commentary-ducking.ts:166`). - **`iris-realtime-translation`** (`@iris/voice`) localizes spectator chat, commentary subtitles, broadcast overlays, and the companion second-screen feed; **`iris-accessibility`** composes `@iris/accessibility` + `@psyche/caption-streaming`. Every one of these declares `mayInfluenceRollback: false` and routes only audio/visual/subtitle output back to `V2Audio` — review-friendly, off the deterministic path, and degrading to silence or the fallback bank rather than fabricating a caster when a provider is absent. ### Edge cases and failure modes - **The rollback envelope is sealed in code, not prose.** Psyche, Iris, Shakti, and Calliope can be called freely off-rollback, but inside the rollback loop the only Psyche-derived input is a hashed, match-start-sampled vector; live RPCs are rejected (`assertValidLiveRequest` throws; the cue validator rejects `fromRollbackFrame`). - **Latency is a contract with a fallback, not a hope.** If the 420/240/100/40 ms commentary budget is exceeded, the cue resolves to the pre-recorded bank keyed by `(cueKind, ruleset, commentatorId, seed)` — deterministic and provider-free. - **The Director is off in ranked and tournament, by construction.** `EvaluateAIDirectorDecision` falls to the static rule-of-the-day the instant a session is ranked, tournament, multiplayer, or opted-out — no profile is even consulted. - **Results are pulled and hash-verified, never pushed.** A bracket integration cannot inject a result; ingestion verifies the golden-replay hash before Crown Points credit, and Veritas holds publication on a winner-record conflict without touching match authority. - **The companion cannot reach the ladder.** Second-screen is `cosmeticOnly` with `gameplayAuthority: false`; the only playable surface is HMAC-signed, server-validated daily reward micro-games. ## Where this connects - **Sideways:** [Online Backbone & Competitive Integrity](./online-backbone-and-competitive-integrity.md) (the backbone the companion reads, the ranked-pro circuit, and the replay-hash determinism the esports backend depends on), [Live-Ops, Store, Progression & Community](./live-ops-store-progression-and-community.md) (the `@maat/strategy` calendar, Crown Points windows, and store the companion surfaces), and [Telemetry, Performance, Testing & Release Gates](./telemetry-performance-testing-and-release-gates.md) (the `v2.match.*` stream the AI Director and commentary consume). - **Integrity:** [Security, Compliance & Sister-Monorepo Integration](./security-compliance-and-sister-monorepo-integration.md) (Tournament Mode hotfix freeze, ranked-pro MFA, and the EU AI Act conformity export for the Director and commentary). - **Platform:** the [domain-orchestration](../../platform/oshun-domain-libraries.html) layer that owns the Iris / Isis / Psyche / Euterpe / Uzume / Veritas / Shakti substrates these `apps/v2/` bridges compose, and the [contracts](../../platform/contracts.html) tier that defines the `@oshun/event-bus` topics they publish and subscribe. - The section hub: [../V2_ARCHITECTURE.md](../V2_ARCHITECTURE.md).