# Online Services & Infrastructure

A heist crew forming up in 1947 Urban, a posse queuing for a Frontier showdown,
a 5v5 Capital Raid in the Sci-Fi cell, and a lone detective syncing a Mind
Palace clue from the companion app on a train all touch the _same_ backbone.
They sign in through one account, carry one Bureau XP ledger, route through one
skill-rated matchmaker, and land their scores, replays, and cosmetics in one set
of domain-isolated stores. That backbone is not a diagram on a slide: it is
sixteen deployable services under `apps/v5`, each a TypeScript/NestJS package
with a real Postgres/Redis domain behind it, fronted by a sixteen-endpoint
Unreal C++ client catalog in `V5/ue/Source/V5OnlineServices` that speaks to them
over genuine HTTP. This page is the product-and-code tour of that layer: the
services, what each really _does_, the request seam that connects client to
cloud, and the backend infrastructure — containers, clusters, regions, health,
and graceful degradation — that keeps it standing. The connection models, PvP,
ranked, and esports surfaces that _ride_ this backbone live in
[Modes & Multiplayer](./modes-and-multiplayer.md); the seasons, calendar,
workshop marketplace, and community programs are in
[Live Service & Community](./live-service-and-community.md); the engine-side
save, persistence, and cross-play machinery is the architecture companion,
[Persistence, Online Services & Cross-Play](../architecture/persistence-online-and-crossplay.md).
For the full feature scope this slots into, start at the hub:
[../V5_features.md](../V5_features.md).

## What ships, honestly

The split is clean, and stating it up front keeps the rest of the page honest.
**Two tiers are real code**, and a small set of seams are labelled as plan or
fixture rather than wired integration.

The **client tier is real Unreal C++.** `V5/ue/Source/V5OnlineServices` ships a
sixteen-endpoint service catalog
(`UV5_Online_ServiceCatalog::BuildEndpointCatalog`) fronted by a family of
`UBlueprintFunctionLibrary` request builders — one per service
(`UV5_Online_AuthService`, `…MatchmakingService`, `…AntiCheatClient`,
`…WorkshopClient`, and the rest) plus a cross-cell-world builder — over a typed
`V5OnlineServicesTypes.h` surface. Crucially the transport underneath is
**genuine network I/O**: `V5OnlineServices.Build.cs` declares the `HTTP` and
`Json` modules (the build comment marks this "the first `FHttpModule` usage in
V5 UE"), and `FV5OnlineHttp::BuildHttpRequest` turns a request value into a real
`IHttpRequest` from `FHttpModule::Get()`. Authentication is real cryptography:
`FV5JwtCrypto` is a self-contained FIPS 180-4 SHA-256, RFC 2104 HMAC-SHA256, RFC
7519 HS256 JWT, and RFC 7636 PKCE S256 implementation, unit-tested against
published vectors.

The **service tier is real TypeScript + NestJS.** `apps/v5` holds sixteen
service packages — each with a `contract.json`, a `Dockerfile`, and `k8s/`
manifests — and every one delegates its `handle()` to `handleRealRequest` in
`@v5/service-shared`, which resolves **live `pg.Pool` + `ioredis` clients** and
dispatches to a domain that is not CRUD: a Glicko-2 rating pipeline, an OAuth
refresh-rotation flow with reuse detection, a hash-chained balance ledger, a
version-vector CRDT Mind Palace merge. Each carries Vitest cases that assert
computed answers, not shapes.

Three honest labels travel with the rest of the page:

- **There are two backend code paths, and the live one is real.** The pure
  `evaluateServiceSpecificRule` in `runtime.ts` still returns the old
  `jwt.v5.<account>.oauth` fixture string for golden contract tests — but the
  deployed `service.handle()` calls `handleRealRequest` →
  `executeRealServiceRequest`, the Postgres/Redis-backed path. The fixture path
  validates contract shape; the real path is what runs in the container.
- **Kernel anti-cheat is a planned client tier.** The client builds EAC signal
  requests and `platformPolicy` names the signed Windows driver
  (`EasyAntiCheat_EOS.sys`), but that kernel driver is named, not in-tree — and
  the server-side strike journal that _does_ ship is real.
- **The design's service table is a subset, and "Voice" is design-named.** The
  feature brief lists ten representative rows (including a `Login` and a `Voice`
  service); the shipped backbone is sixteen packages where `Login` is the `auth`
  service and there is **no** standalone `voice` package in `apps/v5` — in-game
  voice is routed through party/presence, not a separate in-tree service.

## The service catalog — sixteen services on one backbone

Gameplay code never hand-rolls a URL. `BuildEndpointCatalog` enumerates all
sixteen services, each as an `FV5OnlineEndpointDefinition` carrying its verb,
path, `bRequiresJwt`, and `bSupportsOfflineQueue` flags; a `UV5_Online_*`
builder fills an `FV5OnlineServiceRequest`, and the catalog is the single source
of truth for ports and routing. The service catalog (`service-catalog.json`)
declares the same sixteen on the server, with ports assigned as
`4210 + service index` (`auth = 4210` … `faction-rep = 4225`, identical in
`FV5OnlineHttp::ServicePort` and the catalog).

| Service                   | Client builder                         | Real domain behind it                                                           |
| ------------------------- | -------------------------------------- | ------------------------------------------------------------------------------- |
| **auth**                  | `UV5_Online_AuthService`               | `AuthDomain` — OAuth exchange, HS256 JWT, refresh rotation + reuse detection    |
| **friends**               | `UV5_Online_FriendsService`            | `FriendsDomain` — cross-platform graph + Redis presence heartbeats              |
| **parties**               | `UV5_Online_PartiesService`            | `PartiesDomain` — group/ready-check, cross-cell Bureau-HQ sessions              |
| **matchmaking**           | `UV5_Online_MatchmakingService`        | `MatchmakingDomain` — Glicko-2 + Redis ZSET ticket queue                        |
| **leaderboards**          | `UV5_Online_LeaderboardsService`       | `LeaderboardDomain` — per-cell/per-season Redis ranked boards                   |
| **replays**               | `UV5_Online_ReplaysService`            | `ReplaysDomain` — chunked upload, signed download grants, AI highlight reels    |
| **telemetry**             | `UV5_Online_TelemetryClient`           | `TelemetryDomain` — batched event ingest, ClickHouse routing                    |
| **anti-cheat**            | `UV5_Online_AntiCheatClient`           | `AntiCheatStore` — eight detectors + persisted strike journal                   |
| **crash-reporting**       | `UV5_Online_CrashReportingClient`      | `CrashReportingDomain` — dedupe + symbolication queue                           |
| **companion-app-bridge**  | `UV5_Online_CompanionAppBridge`        | `CompanionAppBridgeDomain` — read-only home view, save export/import, AR/VTuber |
| **mindpalace-cloud-sync** | `UV5_Online_MindPalaceCloudSyncClient` | `MindPalaceCloudSyncDomain` — version-vector CRDT graph merge                   |
| **workshop**              | `UV5_Online_WorkshopClient`            | `WorkshopDomain` — publish/moderate, paid-mod marketplace, refunds              |
| **compliance-dsar**       | `UV5_Online_ComplianceDSARClient`      | `ComplianceDsarStore` — GDPR/CCPA/DSA/COPPA + data residency                    |
| **live-service-calendar** | `UV5_Online_LiveServiceCalendarClient` | `LiveServiceCalendarDomain` — seasonal feeds with ETags                         |
| **balance-ledger**        | `UV5_Online_BalanceLedgerClient`       | `BalanceLedgerDomain` — hash-chained append-only ledger                         |
| **faction-rep**           | `UV5_Online_FactionRepClient`          | `FactionRepDomain` — reputation writes + weekly decay                           |

### The HTTP seam and fail-loud transport

`FV5OnlineHttp::BuildHttpRequest` constructs the real request: it sets the verb,
`Content-Type`/`Accept: application/json`, an `Authorization: Bearer` header
when a JWT is present, the routing headers `X-V5-Account` and `X-V5-Region`, and
the JSON body for non-GET methods. Base-URL resolution is fail-soft and never
fabricated — `V5_ONLINE_BASE_URL` is treated as an API gateway, else
`V5_ONLINE_HOST` plus the service's catalog port.

The latent action node `UV5_Online_HttpRequest::Activate` is where the fail-loud
discipline shows. A request whose endpoint `bRequiresJwt` but whose token is
empty **short-circuits to a 401 before touching the network**
(`auth.jwt.required`). A genuine transport failure resolves to
`TransportFailureResponse` (`service.unreachable`, `HttpStatus = 0`) and sets
`bQueuedOffline` from the endpoint's offline-queue flag. A `503` likewise flips
`bQueuedOffline` and parses the backend's structured `errorCode`/`errorMessage`.
No path fabricates a success it did not receive — the response booleans reflect
the wire, not a hope.

## Identity: JWT, PKCE, and the single account

Login is the gateway every other service trusts, so it is real on both sides of
the wire. On the client, `UV5_Online_AuthService::IssueOAuthJwt` mints a genuine
HS256 token: it assembles RFC 7519 registered claims (`iss = v5.bureau.auth`,
`sub`, `aud = v5.client`, `provider`, `iat`, `nbf`, `exp` with a 3,600-second
TTL, `jti`), derives the `jti` from `SHA-256(account|provider|iat)` so each
issuance is unique yet deterministic, signs with `FV5JwtCrypto::IssueHs256`, and
derives an **opaque refresh token** as
`v5rt_ + base64url(HMAC(SHA-256(key.refresh), SHA-256(accessJwt)))` — revocable
server-side and not user-forgeable. `VerifyAccessJwt` does the real inverse: a
constant-time signature comparison plus `exp`/`nbf` enforcement, and
`VerifyPkceChallenge` checks `BASE64URL(SHA256(verifier)) == challenge`.

Server-side, `AuthDomain` (`domain/auth.ts`) is Postgres-backed and stricter
still. `resolveCanonicalAccount` enforces **oldest-verified-credential-wins**:
the first verified `(platform, externalSubject)` pair binds a canonical
`v5acct.…` id, and later exchanges for the same pair return that id — one human
cannot fork into multiple V5 accounts per platform, which is exactly what
cross-progression needs to stay coherent across all nine platforms.
`refreshAccessToken` implements RFC 6749 §6 rotation with **RFC 6819 reuse
detection**: presenting an already-rotated token sets every row in the family to
`revoked = TRUE` and returns `reuse_detected`. Refresh tokens are stored only as
SHA-256 hashes, never cleartext. The canonical account this issues is the spine
the [cross-play policy](../architecture/persistence-online-and-crossplay.md)
builds on, and the client mirror `BuildPlatformCredentialLinkRequest` binds an
external subject to it.

## Matchmaking and the skill model

The client builders are deliberately thin — `BuildQueueRequest` and
`BuildCrossplayQueueRequest` carry mode, region, an integer `SkillRating`, ping,
and (for cross-play) `Platform`, `ModeId`, and `bCrossplayOptOut` to a per-cell
queue path. The authority is server-side and genuine. `MatchmakingDomain`
(`domain/matchmaking.ts`) runs a real **Glicko-2** update (Glickman 2013):
`glicko2Update` computes the `g(φ)` and expected-score terms, then solves the
volatility step with an **Illinois (regula falsi) root finder** — pinned in
tests to the canonical worked example (a 1500/200/0.06 player against three
opponents converging to ~1464.06 / ~151.52 / ~0.05999). The queue is a **Redis
ZSET** banded by `v5:mm:cell:region:mode:skillBand:pingBand` (200-wide skill
bands, three ping bands), ordered by enqueue time; `formMatch` uses an **atomic
`ZPOPMIN`** so two formation passes never claim the same player, and re-enqueues
what it took if a rare race leaves it short. The per-cell ranked rules, season
ladder, and crossplay opt-out _consequences_ are surfaced in
[Modes & Multiplayer](./modes-and-multiplayer.md); this service is the fairness
engine they sit on.

## Anti-cheat: defense in depth, labelled by tier

Anti-cheat is defended in depth. The client `UV5_Online_AntiCheatClient` builds
eight distinct, dedicated-endpoint requests: an EAC session signal, aim
plausibility (impossible-angle turn rate), sub-tick aimbot snap, line-of-sight
wall-hack history, position-delta speed-hack, auto-fire interval pattern, a
weighted classifier carrying `modelId: V5_AntiCheat_ML`, and an appeal routed to
the companion-app support path. Each detector's scoring math is real and lives
in `runtime.ts` — the speed-delta detector, for instance, divides observed
distance by elapsed time and flags when the ratio exceeds the weapon/character
max speed by more than 15%.

Server-side, `AntiCheatStore` (`domain/anti-cheat.ts`) makes this **stateful**:
`recordSignal` journals each flagged detector to Postgres and recomputes the
account's running strike count over verdicts in `('strike', 'ban-review')`,
driving the three-strike ladder (`warning` → `ranked-suspension` →
`ban-review`); `fileAppeal` queues a row on the companion-app path. The
governing posture matches the platform's: model output **prioritises a review
queue** — it never silently disciplines. The `platformPolicy` helper encodes the
integration plan honestly: Windows is a signed kernel driver
(`EasyAntiCheat_EOS.sys`, `blockLaunchIfUnavailable`), Mac/Linux/console/iOS are
userland modules — but, per the label above, that kernel driver is named, not
shipped in this tree. The strike journal is what ships real.

## The ledgers and graphs the social layer rides

Three domains turn "online" into durable, auditable state:

- **`BalanceLedgerDomain`** is a genuine append-only, **hash-chained** public
  ledger. Every entry's hash binds the previous entry's hash, its sequence
  number, and the canonical JSON of its payload; appends serialise through a
  single head row taken `FOR UPDATE` for a gap-free monotonic sequence under
  concurrent writers; and `verifyChain` re-walks the chain, recomputes every
  hash, and returns the first `brokenAtSeq` if a row was edited out from under
  it. That is what makes a designer's economy tune _evidence-linked_ rather than
  trust-me.
- **`MindPalaceCloudSyncDomain`** is a **version-vector CRDT** for the deduction
  graph. `mergeNode` resolves by causal dominance, falling back to a
  deterministic `(updatedUnix, lastWriter)` tiebreak only on genuinely
  concurrent edits, and accusations are an append-only union
  (`ON CONFLICT DO NOTHING`). The merge is commutative and idempotent, so two
  devices syncing in either order — or the same diff twice — converge to one
  graph. This is the service that lets a clue annotated on the companion app
  reach the console save without clobbering it.
- **`FriendsDomain`** and **`PartiesDomain`** back the cross-platform social
  graph: friends pairs and presence heartbeats in Postgres + Redis, party
  create/join/ready-check with cross-cell Bureau-HQ sessions (a 64-player
  dedicated-server hub where account-scoped profile, cosmetics, Bureau XP, Mind
  Palace, and codex are shared while per-cell campaign branches stay isolated).

## The backend infrastructure

Every service is the same shape — a contract, a container, a cluster entry — and
the shape is what makes sixteen of them operable as one fleet.

```mermaid
flowchart TD
  subgraph Client["V5 client · Unreal C++"]
    CAT["UV5_Online_ServiceCatalog<br/>16 endpoints · UV5_Online_* builders"]
    HTTP["FV5OnlineHttp / UV5_Online_HttpRequest<br/>real FHttpModule · Bearer JWT · X-V5-Account/Region"]
  end
  CAT --> HTTP
  HTTP -->|"V5_ONLINE_BASE_URL · else host:4210+idx"| GW[API gateway / DNS failover]
  GW --> SVC["16 × NestJS service<br/>handle() → handleRealRequest → executeRealServiceRequest"]
  SVC -->|resolveBackingClients| CFG{"V5_POSTGRES_URL<br/>+ V5_REDIS_URL set?"}
  CFG -->|no| OUT["503 outage · not_configured<br/>bQueuedOffline per endpoint"]
  CFG -->|yes| DOM["real domain<br/>AuthDomain · MatchmakingDomain · …"]
  DOM --> PG[("Postgres<br/>db-per-service")]
  DOM --> RS[("Redis<br/>ZSET queues · presence")]
  TEL["telemetry · crash-reporting"] -. residency-pinned .-> CH[(ClickHouse)]
  SVC --> K8S["MultiRegionFailover CRD<br/>iad·fra·sin / pdx·dublin·syd"]
```

### One shape per service: contract, container, cluster

Each service's `contract.json` is its machine-readable spec — slug, package
name, port, primary and additional endpoints (with `requiresJwt` /
`supportsOfflineQueue`), declared `storage` (`Postgres` + `Redis`, optional
analytics), `capabilities`, `contractCases`, the `environment` variable names it
reads, and its `docker`/`kubernetes` block. The `Dockerfile` is a real
multi-stage Node 24 build: it installs the workspace with pnpm, builds
`@v5/service-shared` then the service, prunes to production deps, and runs
`node dist/main.js` on the service's catalog port. The `k8s/` directory carries
a `Deployment` (two replicas, `/healthz` readiness and liveness probes,
CPU/memory requests and limits), a `Service`, and a per-service
`MultiRegionFailover` custom resource. Databases are **domain-isolated** — each
service owns its store rather than sharing one schema, the database-per-service
posture the architecture specifies.

### Configuration, health, and graceful degradation

The most important honesty seam is how a service behaves when its stores are
_not_ there. `resolveBackingClients` hands a service live `pg.Pool` / `ioredis`
clients **only** when its catalog-declared `V5_POSTGRES_URL` and `V5_REDIS_URL`
resolve to non-empty values; otherwise it returns `configured: false` with null
clients, and `executeRealServiceRequest` serves the **503 outage** contract —
carrying `retryAfterSeconds`, the failover mode, and fallback regions, with
`queuedOffline` set from the endpoint flag. `buildHealth` reports
`status: not_configured` in that case rather than the hardcoded
`serviceHealthy: true` that once made the outage branch dead code. Absence is
reported loudly, never faked — and that fail-loud health is what the readiness
probe and DNS failover key off.

Region and failover policy is checked-in data. The catalog declares primary
regions `iad / fra / sin` and secondary `pdx / dublin / syd`, with
`compliance-dsar`, `telemetry`, and `crash-reporting` flagged
`dataResidencyPinned`. The shared `services/k8s/multi-region-failover.yaml`
`OnlineServiceFailoverPlan` sets a 45-second health window, a 0.98 success-rate
threshold, a 180 ms regional-latency ceiling, automatic DNS failover, and a
per-service mode (`global-active-passive` for auth/friends,
`regional-active-active` for parties, queue-backed for the offline-tolerant
services). The result is the degradation promise the player feels: a service
outage drops single-player to offline-deterministic campaign play with local
saves that reconcile on reconnect; an unavailable matchmaker shows a clean retry
state, never a hang; and telemetry buffers client-side and flushes on recovery —
because every layer above is built to read a 503 as "retry or queue," not as a
reason to lie.

## Where this connects

- **Sideways to the match:** [Modes & Multiplayer](./modes-and-multiplayer.md) —
  the connection models (dedicated / listen / LAN / server browser), the PvP and
  horde modes that queue through this matchmaker, the ranked-season ladder, and
  the esports surfaces that run on this fleet.
- **Sideways to live service:**
  [Live Service & Community](./live-service-and-community.md) — the seasonal
  calendar feeds, persistent world events, workshop marketplace, and faction-rep
  decay that ride these same services and ledgers.
- **Down to the engine:**
  [Persistence, Online Services & Cross-Play](../architecture/persistence-online-and-crossplay.md)
  — the encrypted save substrate, the `V5Persistence` ledgers and replay
  determinism, the canonical-account cross-play policy, and the Postgres / Redis
  / object-storage substrate these services deploy onto.
- The feature hub: [../V5_features.md](../V5_features.md) </content>