A fighting game becomes a sport, and then a show, in three steps that all live
outside the match. The match itself is the deterministic, integer-frame
simulation described in
Rollback Netcode & Tag-Team; everything on
this page wraps that simulation without ever being allowed to perturb it. The
esports backend turns a verified result into a bracket, a Crown Points
standing, and a broadcast; the companion app gives a player a
phone-and-watch surface for frame data, replays, store, and coach notes that is
deliberately kept off the competitive path; and the AI services — an AI
Director that paces single-player encounters and a real-time AI commentary stack
that casts a match with synthesized voice, lip-synced faces, ducked music, and
live translation — make a V2 broadcast feel produced. The unifying design rule
is that V2 builds none of the underlying intelligence: it composes the shared
Oshun substrates (Iris, Isis, Psyche, Euterpe, Uzume, Veritas, Shakti) through
thin, rollback-aware apps/v2/ bridges, supplies the fighting-game context, and
consumes a result it can audit. The section hub is
../V2_ARCHITECTURE.md.
What ships, honestly#
The split here is the same one the sibling
Online Backbone & Competitive Integrity
page draws, and it is worth stating plainly. The bridges and tooling are real
code. apps/v2/ ships the AI-services integration packages —
iris-commentary-orchestration, psyche-ai-director-hints,
isis-commentary-lipsync-pipeline, psyche-commentary-face-animation,
euterpe-commentary-ducking, iris-realtime-translation, and
iris-accessibility — each a substantive TypeScript module that composes a
shared substrate and enforces a rollback boundary. apps/v2/esports-tools/
ships @v2/esports-tools with three real engines (broadcast, arena
LED/projection, fact-checking). apps/oshun/mobile/v2/ ships the companion
shell model. The in-engine seams are real Unreal C++:
V2/ue/Plugins/V2AICommentary/ and V2/ue/Plugins/V2AdaptiveAI/ carry typed
contracts and Blueprint libraries, and the AI Director is a genuine C++ state
machine, not a config struct.
The gRPC wire contract for this page's three services now ships: real proto3
contracts live at libs/proto/src/oshun/v2/esports/esports.proto,
.../director/ai_director.proto, and .../ai_commentary/commentary.proto
(registered in @oshun/proto's loader and service registry), and
@v2/grpc-services binds all three — oshun.v2.esports.Esports,
oshun.v2.director.Director, oshun.v2.ai_commentary.Commentary — onto one
@grpc/grpc-js server backed by the real bridges (@v2/esports-tools +
@maat/strategy Crown Points, @v2/psyche-ai-director-hints,
@v2/iris-commentary-orchestration), with in-process round-trip tests covering
streaming, fallback, and rollback rejection. The other V2 proto packages the
architecture names (online, live, progression, community, racing, heist,
world-boss) remain the designed contract, not files in the tree. And the
generation itself is provider-gated: the bridges orchestrate, validate,
budget, and route, but the live commentary text, the synthesized voice, and the
generated face animation come from model-backed substrates (and ultimately
providers) those bridges call. The orchestration ships; the generation is a
capability that fails loud when its provider is absent, exactly the seam the
platform uses everywhere.
The esports backend#
V2 treats competitive play as a first-class surface. The backend's job is to take a result the online backbone has already verified against the replay hash, and turn it into a standing and a show — while keeping every one of those operations provably off the deterministic path.
This picture is wider than it can be shown here. Move across it, or read it as text.
Drawn from its Mermaid source.
As text
Parts: - Tournament event - TO badge tools - Per-event rule file · JSON - Client Tournament Edition lock · stamped in replay header - Bracket · Start.gg / Challonge / Battlefy - Result ingest · V2 pulls, never client-pushed - Replay-hash verified? (decision) - Crown Points pool - Held for review - World Final qualification - @v2/esports-tools - @uzume/broadcast · switcher · multiviewer · graphics · replay · SRT - @uzume/lumina + @uzume/prism · LED / projection - @veritas/fact-checking · post-match reporting Connections: - Tournament event to TO badge tools - TO badge tools to Per-event rule file · JSON - Per-event rule file · JSON to Client Tournament Edition lock · stamped in replay header - Tournament event to Bracket · Start.gg / Challonge / Battlefy - Bracket · Start.gg / Challonge / Battlefy to Result ingest · V2 pulls, never client-pushed - Result ingest · V2 pulls, never client-pushed to Replay-hash verified? - Replay-hash verified? to Crown Points pool: yes - Replay-hash verified? to Held for review: no - Crown Points pool to World Final qualification - Tournament event to @v2/esports-tools - @v2/esports-tools to @uzume/broadcast · switcher · multiviewer · graphics · replay · SRT - @v2/esports-tools to @uzume/lumina + @uzume/prism · LED / projection - @v2/esports-tools to @veritas/fact-checking · post-match reporting
Show source
flowchart TB
Event[Tournament event] --> TO[TO badge tools]
TO --> Rule[Per-event rule file · JSON]
Rule --> Lock[Client Tournament Edition lock · stamped in replay header]
Event --> Bracket[Bracket · Start.gg / Challonge / Battlefy]
Bracket --> Ingest[Result ingest · V2 pulls, never client-pushed]
Ingest --> Verify{Replay-hash verified?}
Verify -->|yes| Crown[Crown Points pool]
Verify -->|no| Hold[Held for review]
Crown --> Finals[World Final qualification]
Event --> Tools["@v2/esports-tools"]
Tools --> Cast["@uzume/broadcast · switcher · multiviewer · graphics · replay · SRT"]
Tools --> Arena["@uzume/lumina + @uzume/prism · LED / projection"]
Tools --> Fact["@veritas/fact-checking · post-match reporting"]
Broadcast, arena, and fact-checking tooling#
@v2/esports-tools is the real operator surface. Its broadcast engine,
buildV2EsportsBroadcastPipeline
(apps/v2/esports-tools/src/esports-broadcast-pipeline.ts:193), composes seven
@uzume/broadcast engines into one plan: a multi-platform streaming engine that
upserts per-destination encodings (1080p60 / 8 Mbps default), a stream-health
dashboard with adaptive bitrate bands (floor 65%, ceiling 140%, 500 kbps steps),
an ATEM-default switcher with CLEAN/PGM/REPLAY aux buses, a director
multiviewer layout, three CasparCG graphics templates (scorebug, bracket,
next-match countdown), an EVS-LSM replay controller, and SRT contribution feeds.
The plan is explicitly inert on the match: it returns offRollback: true,
mayInfluenceRollback: false, and deterministicImpact: 'none' (:341). Its
reasonCodes builder refuses to let standalone OBS, NDI, or vMix be a primary
integration (:542); OBS WebSocket is retained for one purpose only —
streamer-mode notification suppression
(V2_ESPORTS_BROADCAST_OBS_WEBSOCKET_PURPOSE, :31). Sibling engines in the
same package compose @uzume/lumina + @uzume/prism for arena LED walls and
projection mapping (presentation/show-control state only,
mayInfluenceRollback: false) and @veritas/fact-checking for post-match
reporting and bracket-result verification, where a winner-record conflict holds
publication without ever changing match authority.
Tournament integrity and Crown Points#
The single load-bearing integrity feature lives one page over but is worth
restating: Tournament Mode freezes the balance hotfix at the TO's snapshot and
stamps that Tournament Edition into the replay header (see
Security, Compliance & Sister-Monorepo Integration),
so a recorded set is reproducible against the exact ruleset it was played under.
Results are server-authoritative by ingestion shape: V2 pulls results from
Start.gg / Challonge / Battlefy via @oshun/inbound-integrations rather than
accepting client-pushed scores, and every match is cryptographically verified
against its golden-replay hash before it counts. Only then does the Crown Points
formula apply —
base_points[round] × region_factor × participant_factor × season_decay, with
round bases stepping 50 (R32) → 100 (R16) → 200 (QF) → 400 (SF) → 800 (GF) —
feeding World Final qualification through the @maat/strategy-owned Pro Circuit
calendar. SLOs are concrete: result ingest p99 ≤ 5 s, live bracket fetch p99 ≤
200 ms, broadcast overlay update p99 ≤ 100 ms.
The results archive and the gRPC seam#
The public-facing spectator surface is real and static:
apps/v2/web/esports/results/ ships index.html, results.js, results.css,
and a results.json (schema: v2.esports.resultsArchive.v1) that is a
searchable tournament archive — searchable by fighter, player, date, and event,
with explicit ranking weights (championship 120, major 90, regional 55, local
25, plus win and replay bonuses) and per-event brackets, match scores, and
replay URIs. The gRPC service backing live operation —
Esports.RegisterEvent(rule_file) → event_id,
SubmitResult(event_id, match_result, replay_hash) → ack,
GetBracket(event_id), QualifyForFinals(season_id, region) — ships at
libs/proto/src/oshun/v2/esports/esports.proto, served by @v2/grpc-services:
SubmitResult fact-checks every result through @veritas/fact-checking (winner
agreement across bracket, report, and replay records plus the sha256 replay
hash) before crediting calculateMaatV2CrownPoints, and a conflict is
RESULT_HELD_FOR_REVIEW with zero credit. The integrity claim rides exactly
where the backbone page puts it: on the replay-hash verification the engine's
determinism already guarantees.
The companion app#
The companion is real and deliberately scoped. apps/oshun/mobile/v2/
(companionAppModel.ts, V2CompanionApp.tsx, V2CompanionLaunchTile.tsx)
builds its shell via buildV2CompanionModel() (companionAppModel.ts:394),
which composes the shared Oshun client stack — @oshun/shell-core,
@oshun/auth-client, @oshun/concordia-integration, @oshun/trust-safety,
@iris/agents, and @sophia/client (:31). It is a read surface: move
lists and a signature-clip library that are offline-ready, plus replay viewer,
profile/match-history, friends-and-party, store-and-wishlist, and Battle Pass
surfaces gated behind auth.
Why it sits outside the competitive path#
The companion is, by explicit design, not a participant in a ranked match —
an integrity decision as much as a product one, because a phone is not a
deterministic match client. Its second-screen mode during a live match is
modeled with gameplayAuthority: false, cosmeticOnly: true,
optInRequired: true, and a 200 ms latency budget (:265): it receives a
low-bandwidth opponent move list and frame-data panels for spectating, and can
produce no gameplay effect. The one thing the companion does let you play is a
suite of daily reward micro-games (Combo Tap Trial, Drift Steer, Trivia Battle,
Daily Puzzle Kombat, Mini Pit Crew), and even those are kept honest: every one
carries hmacRequired: true and serverValidated: true (:315), so a reward
is signed and server-checked and can never touch a ranked ladder. Push runs over
APNS / FCM / web-push with per-category topics; a wearable glance (Apple Watch /
Wear OS) surfaces daily stats, friend match starts, and Battle Pass tier
progress.
Assistants, RAG, and trust-and-safety#
The companion is where V2 wires the shared assistant plane.
buildV2CompanionModel resolves @iris/agents launch readiness
(resolveIrisAgentsLaunchReadiness) into a ready/blocked status — a
fail-loud seam, not an assumed-available one — and selects a frame-data-query
agent and a support/appeal agent, each carrying a requiresHumanReview flag. A
local @sophia/client (port 3737) backs the frame-data / codex / wiki-citation
RAG surfaces. @oshun/trust-safety classifies assistant-behavior policy hits
(the model carries a sample prompt-injection hit) into a severity class with
triage/action SLA budgets, and @oshun/concordia-integration registers the
appeals nav item, a PII-redacting OTLP telemetry sink, and case-update
subscriptions rate-limited to 30/min. The public API the companion and
third-party tools share is OAuth 2.1 + PKCE with per-app scopes, an OpenAPI 3.1
document at libs/openapi/v2/companion.yaml, and rate limits of 100/min
unauthenticated, 1000/min per registered app, and 30/min on personal data — API
p99 ≤ 400 ms, push delivery p99 ≤ 5 s.
AI services: a produced broadcast#
This is where V2's integration ambition shows, and where the rollback boundary is most aggressively defended in code. Two services live here — the AI Director (single-player pacing) and AI commentary (the produced broadcast) — and both are built so that nothing they emit can ever enter the deterministic simulation except as an explicitly serialized, hashed, match-start input.
This picture is wider than it can be shown here. Move across it, or read it as text.
Drawn from its Mermaid source.
As text
Parts: - Deterministic match · v2.match.* events - psyche-ai-director-hints @psyche/behavior-prediction - V2Netcode · deterministic input vector - iris-commentary-orchestration @iris/conversation-orchestration - Synthesized caster VO · persona TTS - euterpe-commentary-ducking @euterpe/master sidechain - psyche-commentary-face-animation @psyche/avatar-lipsync + expressions - isis-commentary-lipsync-pipeline @isis/visual-dubbing - iris-realtime-translation @iris/voice - iris-accessibility @psyche/caption-streaming - V2Audio · gameplay-inert Connections: - Deterministic match · v2.match.* events to psyche-ai-director-hints @psyche/behavior-prediction - psyche-ai-director-hints @psyche/behavior-prediction to V2Netcode · deterministic input vector: match-start snapshot only - Deterministic match · v2.match.* events to iris-commentary-orchestration @iris/conversation-orchestration - iris-commentary-orchestration @iris/conversation-orchestration to Synthesized caster VO · persona TTS - Synthesized caster VO · persona TTS to euterpe-commentary-ducking @euterpe/master sidechain - iris-commentary-orchestration @iris/conversation-orchestration to psyche-commentary-face-animation @psyche/avatar-lipsync + expressions - psyche-commentary-face-animation @psyche/avatar-lipsync + expressions to isis-commentary-lipsync-pipeline @isis/visual-dubbing - iris-commentary-orchestration @iris/conversation-orchestration to iris-realtime-translation @iris/voice - iris-commentary-orchestration @iris/conversation-orchestration to iris-accessibility @psyche/caption-streaming - Synthesized caster VO · persona TTS to V2Audio · gameplay-inert (dotted): audio/subtitle only
Show source
flowchart LR
Match[Deterministic match · v2.match.* events] --> Dir["psyche-ai-director-hints<br/>@psyche/behavior-prediction"]
Dir -->|match-start snapshot only| Net[V2Netcode · deterministic input vector]
Match --> Cue["iris-commentary-orchestration<br/>@iris/conversation-orchestration"]
Cue --> Voice[Synthesized caster VO · persona TTS]
Voice --> Duck["euterpe-commentary-ducking<br/>@euterpe/master sidechain"]
Cue --> Face["psyche-commentary-face-animation<br/>@psyche/avatar-lipsync + expressions"]
Face --> Lip["isis-commentary-lipsync-pipeline<br/>@isis/visual-dubbing"]
Cue --> Tr["iris-realtime-translation<br/>@iris/voice"]
Cue --> A11y["iris-accessibility<br/>@psyche/caption-streaming"]
Voice -.audio/subtitle only.-> Audio[V2Audio · gameplay-inert]
The AI Director#
@v2/psyche-ai-director-hints is real domain code. It composes
@psyche/behavior-prediction — createBehaviorPredictionEngine with a
first-order Markov model, a 5-action horizon, and 30 s prediction validity
(psyche-ai-director-hints.ts:309) — and maps each predicted action into a
fighting-specific tendency vector (pressure, defense, spacing,
throwGame, resourceUse) by term-matching the action against move families
(deriveTendencyVector, :428). The architecturally critical part is the
rollback discipline, and it is enforced, not described: live tendency hints
are available only off-rollback (offline/training), and assertValidLiveRequest
throws if rollback is enabled (:240). For online rollback-with-CPU, the only
legal path is buildV2AdaptiveAIMatchStartSnapshotPlan, which samples Psyche at
match load and bakes a serializedTendencyVector plus a sha256 vectorHash
into a V2AdaptiveAIDeterministicMatchStartInputVector that
v2NetcodeTreatsAsOrdinaryDeterministicInput (:395). Mid-match updates are
queued for the next match start; every hint carries
mayInfluenceRollback: false. The in-engine V2AdaptiveAI plugin mirrors this:
a real C++ state machine EV2AIDirectorState { Calm, Build, Climax, Recovery }
(V2AdaptiveAITypes.h:33) whose EvaluateAIDirectorDecision hard-disables the
Director — bDirectorActive = false, bUsesStaticRuleOfDay = true — whenever
the player opts out, the session is not single-player, or the mode is ranked
or tournament (V2AdaptiveAIBlueprintLibrary.cpp:234). Style matchup hints come
from @shakti/sota-critical (boxer / striker / grappler /
submission-specialist) over an off-rollback shakti.player.style.updated
stream. Under the EU AI Act the Director is Limited risk: opt-out switches
to a static CPU profile before any classifier use, and a DSR delete cascades to
the Director profile, the Psyche tendency vector, and the Shakti classification.
AI commentary and the post-match experience#
@v2/iris-commentary-orchestration composes createMatchCommentaryStream from
@iris/conversation-orchestration (v2-ai-commentary.ts:113) into a cue plan
whose every field declares the boundary:
rollbackPolicy: 'off-rollback-audio-subtitle-only',
allowedOutputChannels: ['audio', 'subtitle'],
rejectsRollbackFrameOrigin: true. Validation rejects fromRollbackFrame
outright (:244) and requires a fallbackSeed, because the real-time budget is
a hard contract: the 800 ms KO-trigger budget decomposes into 420 ms LLM round
trip + 240 ms persona TTS + 100 ms lipsync + 40 ms safety margin (:23), and on
a miss the system falls back to a pre-recorded bank keyed by
(cueKind, ruleset, commentatorId, seed). The mirror C++ contract
FV2AICommentaryCueContract carries the same budgets and the same off-rollback
flags (bOffRollbackBubble, bGameplayInertOutput,
bFeedsSimulationState = false, V2AICommentaryTypes.h:53). The post-match
path buildV2AIPostMatchExperiencePlan (:445) is real selection logic: it
sorts highlight candidates by score and takes the top three to five, requires
the canonical moment kinds (longest-combo, counter-hit-punish, power-play,
dramatic-comeback, KO), and emits an analysis screen, an auto-captioned clip
library, and a compliance block whose disclosure text — "AI commentary is
machine-generated presentation content." — is mandatory under EU AI Act
cross-ref §80.4. Per deps§67, the generation providers are named (Anthropic
Claude Sonnet 4 — the claude-sonnet-4 registry default, with the exact version
a configuration choice — for play-by-play text, ElevenLabs Multilingual v2 for
TTS, Isis ai-video for the highlight auto-edit), with on-cap routing to
@iris/conversation-providers-local; the Calliope commentator roster lives at
V2/ue/Content/V2/Audio/Commentary/commentator-personas.json with cleared
voice-clone metadata and talent signoff.
The supporting render bridges#
The remaining bridges turn commentary text into a finished broadcast, each a real package composing a real substrate:
isis-commentary-lipsync-pipeline(@isis/visual-dubbing) extracts phonemes and viseme timing, drives visual dubbing, validates ≤ 45 ms lip-sync drift, and exports UE curves to/Game/V2/Audio/Commentary/Lipsync— a real replacement for Audio2Face.psyche-commentary-face-animationreads@iris/voiceoutput metadata and drives@psyche/avatar-lipsyncviseme frames plus@psyche/avatar-expressionsARKit expression curves for the caster face.euterpe-commentary-duckingdoes genuine audio-engineering math:computeArrangementDensityfrom@euterpe/accompanyand the@euterpe/mastermasteringChain.dbToLinearto sidechain-duckV2.DynamicMusicunderV2.Commentaryto a −18 LUFS commentary target with a −1 dB true-peak ceiling (commentary-ducking.ts:166).iris-realtime-translation(@iris/voice) localizes spectator chat, commentary subtitles, broadcast overlays, and the companion second-screen feed;iris-accessibilitycomposes@iris/accessibility+@psyche/caption-streaming.
Every one of these declares mayInfluenceRollback: false and routes only
audio/visual/subtitle output back to V2Audio — review-friendly, off the
deterministic path, and degrading to silence or the fallback bank rather than
fabricating a caster when a provider is absent.
Edge cases and failure modes#
- The rollback envelope is sealed in code, not prose. Psyche, Iris, Shakti,
and Calliope can be called freely off-rollback, but inside the rollback loop
the only Psyche-derived input is a hashed, match-start-sampled vector; live
RPCs are rejected (
assertValidLiveRequestthrows; the cue validator rejectsfromRollbackFrame). - Latency is a contract with a fallback, not a hope. If the 420/240/100/40
ms commentary budget is exceeded, the cue resolves to the pre-recorded bank
keyed by
(cueKind, ruleset, commentatorId, seed)— deterministic and provider-free. - The Director is off in ranked and tournament, by construction.
EvaluateAIDirectorDecisionfalls to the static rule-of-the-day the instant a session is ranked, tournament, multiplayer, or opted-out — no profile is even consulted. - Results are pulled and hash-verified, never pushed. A bracket integration cannot inject a result; ingestion verifies the golden-replay hash before Crown Points credit, and Veritas holds publication on a winner-record conflict without touching match authority.
- The companion cannot reach the ladder. Second-screen is
cosmeticOnlywithgameplayAuthority: false; the only playable surface is HMAC-signed, server-validated daily reward micro-games.
Where this connects#
- Sideways:
Online Backbone & Competitive Integrity
(the backbone the companion reads, the ranked-pro circuit, and the replay-hash
determinism the esports backend depends on),
Live-Ops, Store, Progression & Community
(the
@maat/strategycalendar, Crown Points windows, and store the companion surfaces), and Telemetry, Performance, Testing & Release Gates (thev2.match.*stream the AI Director and commentary consume). - Integrity: Security, Compliance & Sister-Monorepo Integration (Tournament Mode hotfix freeze, ranked-pro MFA, and the EU AI Act conformity export for the Director and commentary).
- Platform: the
domain-orchestration layer that
owns the Iris / Isis / Psyche / Euterpe / Uzume / Veritas / Shakti substrates
these
apps/v2/bridges compose, and the contracts tier that defines the@oshun/event-bustopics they publish and subscribe. - The section hub: ../V2_ARCHITECTURE.md.