Tier Routing and the Pixel Streaming Edge

Tier routing is the part of V3 that answers a single question at the door — "which of the four client surfaces should this visitor actually get?" — and then, if the answer is the browser-but-premium surface, hands them to a GPU worker near enough to stream Unreal Engine frames over WebRTC. It exists because the premium experience (Lumen, Nanite, MetaHumans, Sequencer concerts) can only run inside UE5, and a single UE project drives every premium surface: native desktop/VR/console/mobile and the browser via server-rendered Pixel Streaming. But a server-rendered frame costs a GPU worker per session and a nearby point-of-presence (POP), and neither is always available — so the router has to degrade fidelity (to a locally-rendered three.js/WebGPU client, or to a static landing page) without ever degrading identity, presence, or safety. The decision is not prose: it is a typed contract (LilithLaunchDecisionSchema in libs/contracts/src/v3/lilith.ts) with four surfaces and twenty-one reason codes (a single decision array carries 1–16), resolved by real code in two Fastify BFFs and serviced by a 4,800-line Rust matchmaker at apps/v3/lilith-pxstream-relay/src/lib.rs.

What makes this subsystem subtle is that the monolith's single numbered "tier-router" is realized as two routers at two layers, plus the relay. The front door is a region/tenant availability gate inside the V1 Oshun shell BFF (apps/oshun/bff/src/routes/v3-lilith-launch.ts, resolveV3LilithLaunchDecision); the surface picker is a device-capability probe inside the dedicated Lilith BFF (apps/lilith/bff/src/routes/v3-lilith-launch-route.ts, resolveLilithLaunchDecision); and the Pixel Streaming edge is the relay's match_pxstream_session. This page walks all three, in the order a real visitor hits them. It is the routing-and-edge companion to the orientation hub ../V3_ARCHITECTURE.md, and the deep dive behind the tier-stack summary in Product Promise and Architecture.

What ships, honestly#

The routing logic is real and tested, on both layers. The Lilith BFF device-probe router resolves a full four-surface decision (native / pxstream / fallback / static) with a discriminated-union target and a deduplicated reason trail, and it is exercised by a weighted synthetic-device-mix gate that asserts per-tier decision shares to within ±1 point (apps/lilith/bff/src/routes/v3-lilith-launch-distribution.ts). The Oshun-shell availability gate is real, registered (apps/oshun/bff/src/app.ts:1142), and unit-tested for waitlist, sanctions, and music-rights paths. The relay matchmaker is genuine Rust — POP scoring, admission control, HMAC-SHA256 session JWTs, Epic-protocol SDP answer construction, autoscale planning that emits real KEDA/Karpenter YAML, an abuse classifier with a precision gate — backed by 28 in-crate tests.

Three honest caveats. First, the relay runs against a hardcoded ga_fleet_snapshot() unless V3_PXSTREAM_FLEET_JSON is injected at boot (lib.rs:1060): there is no live POP-health feed wired in source, so the GA fleet, signaller registry, and admission snapshot are fail-honest env seams, not a streaming telemetry pipe. Second, the capacity-management release gate and its drills are computed over synthetic, deterministic fixtures (simulate_*, scheduled_concert_prewarm_load_test_input) — they prove the math and the control flow, not a real cloud failover; the evidence JSONs under V3/pxstream/ are generated from those simulations. Third, the Epic Pixel Streaming Signaller (Node.js) and the headless UE worker fleet are external infrastructure, not in this repo: the relay leases against signaller endpoints and builds an Epic-compatible SDP answer, but the signaller and the per-session UE process live in the GPU fleet. Where the code only models a provider boundary, this page says so.

One more piece of drift worth flagging up front: the canonical launch contract lives in libs/contracts/src/v3/lilith.ts, but the Lilith BFF route re-declares the same schema locally rather than importing it. The two copies are byte-for-byte equivalent today (and the contracts copy is what libs/contracts/src/v3/fixtures.ts validates against), but they must be kept in lockstep by hand — there is no shared import binding them.

Two routers, two questions#

The visitor crosses two independent gates, in different services, both posting to /api/v3/lilith/launch but answering different questions and returning different shapes.

Region and tenant availability — the front door#

resolveV3LilithLaunchDecision runs first because there is no point probing a device for a region that is not open or a tenant that is not yet cleared. It reads a region profile and a tenant-availability matrix (V3_LILITH_TENANT_REGION_AVAILABILITY). Wave-1 regions are us-ca, de, gb, in, br (status open); everything else short-circuits to a regional-waitlist tier with a reason: kr and gcc are wave-2-plan, cn-mainland is partner-of-record-required, and ru is sanctions-prohibited. Tenancy is gated independently of region: Saraswati Stage is waitlist in both India and Brazil — "Music-rights and lyric-tone review remain gated" and "Brazil music-rights release approval remains gated" — even though Tara Studio and Lilith Commons are GA in those same regions. Only after a region/tenant pair clears does this gate make a coarse tier pick: native if the native client is installed, else pixel-streaming when webRtcSupported && pixelStreamingCapacityPercent < 85, else fallback-web. That < 85 cut is the same 85% utilization line the relay enforces as free-tier backpressure (FREE_TIER_BACKPRESSURE_BASIS_POINTS = 8_500), surfaced here as a percentage so the shell can pre-empt a doomed Pixel Streaming hop. The regional landing is real, but email collection is deliberately disabled: no consent, delivery, DSAR, or durable lead authority is configured. The POST endpoint validates input and then returns 503 regional_waitlist_not_configured with emailStored: false; guessed legacy record ids always return 404. The public form reports that disposition without an optimistic saved confirmation. Tenancy/residency reasoning behind this matrix is covered in Data, Tenancy, and Residency; the per-tenant rights gates (why Saraswati waits on music rights) belong to Persona Policy, Provenance, and Rights.

Device capability — the surface picker#

resolveLilithLaunchDecision is the router the monolith describes as the numbered tier ladder, and it is the one whose contract is canonical. It consumes a LilithLaunchRequest — a strict Zod object carrying a native-client probe, a browser probe (webRtc, codecs, webGpu, webGl2), a network probe (sustainedDownlinkMbps, probeDurationMs), a region probe (pixelStreamingRestricted), and up to 64 candidate POPs each with rttMs, availableWorkers, programmingPriority, and codecs — and walks a fixed ladder:

1. Override wins first. override: 'fallback' jumps straight to the local-render branch with user-selected-lite-mode; override: 'native' forces the deep-link; override: 'pxstream' skips the native preference.
2. Native, if installed (and not overridden to pxstream): emit a native-deep-link target (oshun://lilith/launch?...). This is Tier-4 and carries no backend GPU cost — see Tier-1 UE5 Client.
3. Pixel Streaming, only if all five readiness checks pass: browser.webRtc; a usable codec (AV1 preferred over H264 by selectCodec); sustainedDownlinkMbps >= 8 (the pixelStreamingBandwidthFloorMbps floor); !region.pixelStreamingRestricted; and at least one POP with availableWorkers > 0, rttMs <= 60 (pixelStreamingPopLatencyBudgetMs), and a matching codec. The chosen POP is sorted by programmingPriority desc, then rttMs asc, then availableWorkers desc — so a concert-priority POP outranks a marginally closer idle one. The target is a pxstream-browser carrying matchEndpoint: '/api/v3/pxstream/match', the selectedPopId, the selectedCodec, and expectedFirstFrameBudgetMs: 8000.
4. Fallback otherwise: webgpu if available, else webgl2, as a fallback-browser target with fidelity: 'reduced' — the locally-rendered client in Tier-2 Fallback Web Client.
5. Static landing as the floor: if the browser reports neither WebGPU nor WebGL2, the router refuses to claim a runnable fallback and returns a static-landing target with supportReason: 'local-rendering-unsupported'.

Every branch accumulates reason codes (native-client-missing, bandwidth-insufficient, pop-latency-too-high, region-restricted, …), deduplicated and capped at 16, so analytics can reconstruct why a surface was chosen. mapLilithLaunchDecisionToTelemetryTier then collapses the decision to a five-value telemetry tier (tier_0_static … tier_4_native), splitting the single fallback surface into tier_2_webgpu vs tier_1_webgl by renderer.

The decision-distribution gate#

The router is not just unit-tested for individual cases — it is regression-locked by a distribution. buildLilithLaunchDecisionDistributionReport runs eight weighted synthetic device profiles (total weight 1,000) through the real resolver and asserts the resulting telemetry-tier shares against tight bounds: native 0.26 (±0.01), pixel-streaming 0.34, WebGPU 0.22, WebGL 0.14, static 0.04. The gate fails if any case routes to the wrong surface, if any required reason code is missing, or if any tier is never reached (allTiersCovered). This is what backs the claim in V3/TIER_ROUTER_DECISION_DISTRIBUTION.md: the eval set eval-v3-tier-router-decision-distribution-v1 is a real, deterministic computation over the shipped routing code, not a hand-tuned dashboard number.

The launch decision contract#

LilithLaunchDecisionSchema is the wire shape both the Lilith BFF route and the libs/contracts fixtures speak. A decision carries a stable decisionId (launch: + the first 32 hex of a SHA-256 over a canonicalized request + surface + reasons + timestamp — so identical probes yield identical ids), a nullable v1UserId, the selectedSurface, the reasonCodes array, the discriminated-union target, the clientRegion, decidedAt/expiresAt (default TTL 30 s, defaultDecisionTtlMs), and an embedded LilithLaunchDecisionEvent mirror for the analytics bus. The target union is where the four surfaces become concrete:

Because the union is discriminated on kind, an attendee browser literally cannot be handed a Pixel Streaming target without a selectedPopId and a first-frame budget, and cannot be handed a static landing while claiming a renderer — the schema makes the dishonest combinations unrepresentable.

The Pixel Streaming relay#

A pxstream-browser decision is a promise the relay has to keep. The relay (apps/v3/lilith-pxstream-relay, service port 43103) advertises eight capabilities — pop-match, worker-lease, session-jwt, sdp-exchange, worker-autoscale, scheduled-concert-prewarm, abuse-admission-control, pop-capacity-management — and exposes them as axum routes. The browser's /api/v3/pxstream/match call lands in match_pxstream_session, which runs admission, selects a POP, finds an accepting signaller, mints a session JWT, and returns a PxStreamMatchDecision (POP id, signaller WebSocket URL, codec, queue position, an 8,000 ms first-frame budget, and the JWT). The browser then POSTs its SDP offer to /api/v3/pxstream/signalling/exchange, where complete_sdp_exchange verifies the JWT, checks the offer advertises the agreed codec, and returns an SDP answer.

POP selection and scoring#

select_pxstream_pop filters the fleet through candidate_for_pop and then picks the minimum-score survivor. A POP is a candidate only if its health is Accepting, it supports the requested codec, it has both a free worker and a remaining session slot, it satisfies residency, and — critically — it has a known RTT for the client's region within the latency budget. That last check matters: estimated_rtt_ms reads a per-region map (rtt_ms_by_region), so a POP with no measured path to the client region is silently not a candidate rather than guessed. Free-tier requests get two extra gates: the POP must keep paid headroom (has_paid_headroom) and sit below 85% utilization, so a free walk-up never consumes the last slot a booked attendee is holding.

The score is a deliberate blend of geography, load, and programming priority:

geo_penalty       = rtt_ms * 100
load_penalty      = utilization_bp * 3 + queued * 25 - min(available_workers, 500) * 4
programming_credit = effective_priority * 75
score             = geo_penalty + load_penalty - programming_credit   // lower is better

Geography dominates (×100 per ms), which keeps the floor at the ≤ 60 ms budget, but a busy POP can lose to a slightly-farther idle one, and a concert-priority POP can win back ground through the credit term. Ties break by RTT, then utilization, then POP id for determinism.

Admission control#

evaluate_pxstream_admission runs before selection and is where abuse and fairness live. It enforces, in order: a per-user concurrency cap (default 2, overridable by a PxStreamUserConcurrencyGrant for verified instructors or Saraswati editorial); a per-network cap (default 8, lifted to an institutional cap of 64 for approved tenants, keyed on a /24 IPv4 or /48 IPv6 prefix); a daily free-tier minute budget (120 minutes, which paid booked windows are exempt from); a concert walk-up reservation that routes free-tier latecomers to the fallback when a reserved pool is full; and a regional backpressure check that falls free-tier traffic back when every eligible POP in range is saturated. Each refusal is a typed PxStreamAdmissionDecision whose action maps to an HTTP status so the client can react precisely:

Denied admissions are written to an audit log (audit_pxstream_admission_decision) before the error returns, so capacity refusals are accountable. The free-tier abuse posture is detailed in V3/PIXEL_STREAMING_ABUSE_POSTURE.md; the paid-window and concert-reservation economics connect to Commerce and Royalties.

Session JWTs, SDP, and the Epic seam#

The match mints a real HS256 JWT (issue_session_jwt) over an HMAC-SHA256 secret that must be configured — MatchmakerConfig::from_env fails loud with a Config error if V3_PXSTREAM_JWT_HMAC_SECRET (or the PXSTREAM_SESSION_JWT_SECRET alias) is unset, rather than issuing an unsigned token. Claims pin issuer, audience, the pxstream-session type, the v3:pxstream scope, the POP, the codec, and a default 120 s TTL (clamped 30–900 s). complete_sdp_exchange verifies that JWT, rejects an offer whose m=video section does not advertise the agreed codec marker (H264/90000 or AV1), and returns a deterministic Epic-compatible answer SDP whose ICE ufrag/pwd and DTLS fingerprint are derived from the session id — tagged with x-epic-pixel-streaming-protocol: ue5.5-pixel-streaming-signalling-server. That protocol token is the compatibility seam to Epic's real signaller; the relay constructs a valid answer, but the live SDP is ultimately negotiated against the external signaller and headless UE worker.

Idle lifecycle and abuse#

A streamed worker is expensive, so evaluate_pxstream_idle_lifecycle reclaims it: after 90 s with no input it emits PromptIdle ("Still there? Move, press a key…"); after 150 s total — or 60 s past an unanswered prompt — it emits DisconnectIdle, returns the worker to the pool, and sets reconnect_requires_new_match. Bot traffic is caught earlier by classify_pxstream_abuse, which scores behavioural signals (mouse-only-no- keyboard, zero-inertia avatar, scripted reconnect loops, impossible input cadence) in basis points: ≥ 7,000 routes to safety review, ≥ 9,000 bans. The classifier is itself gated — validate_pxstream_abuse_classifier runs a labelled sample set and asserts precision ≥ 90% (MIN_ABUSE_CLASSIFIER_PRECISION_BASIS_POINTS), so the bot heuristic cannot ship if it would over-flag real users.

Per-POP capacity, GPU quota, and multi-cloud posture#

POP capacity is gated by GPU instance availability, not by control-plane software, and the relay models that explicitly. ga_fleet_snapshot declares the seven GA POPs with the exact concurrency targets the architecture commits to: us-east-1 1,200 sessions, us-west-2 and eu-west-1 800, ap-northeast-1 300, and ap-south-1 / ap-southeast-2 / sa-east-1 250 each — each with a per-region RTT map and a residency scope (us, eu, apac, india, latam, plus global). build_pxstream_pop_capacity_dashboard turns that into the operator payload at /api/v3/pxstream/pop-capacity/dashboard (Grafana target grafana://v3/pxstream/pop-capacity), with a 90-day forecast horizon and a hardcoded GPU-quota table — us-east-1 256, us-west-2 192, eu-west-1 160, ap-northeast-1 / ap-south-1 96, ap-southeast-2 / sa-east-1 80 — that matches the signed quota review at apps/v3/lilith-pxstream-relay/config/pops/quota-review-2026-05-22.md. Note the two units: the monolith's "1,200 concurrent sessions" is max_sessions; the dashboard's "256" is GPU nodes (≈ 6 class-tier or 3 stadium-tier sessions per 24-vCPU node). A POP row is ready only when it has positive quota headroom, a healthy primary, and all four secondary pools (Azure, GCP, CoreWeave, Lambda Cloud) present.

The harder regions are codified. ap-south-1 and ap-northeast-1 are tight-quota POPs: plan_pxstream_capacity_reservation requires a 90-day reservation lead for scheduled events there versus 30 days elsewhere, and the worker autoscaler extends the prewarm window to 60 minutes for stadium-tier events in those POPs. Residency is enforced in selection, not bolted on: residency_allows keeps an eu-scoped user inside EU POPs, prefers ap-south-1 for India, and only permits the India→eu-west-1 cross-border hop when cross_region_consent is set — the exact behaviour the residency drill (evaluate_pxstream_residency_routing_drill) verifies by routing real synthetic requests through select_pxstream_pop.

The capacity-management release gate (evaluate_pxstream_pop_capacity_management_release_gate) bundles the dashboard with four drills: a multi-cloud failover (AWS ap-northeast-1 outage → Azure japaneast NVadsA10 v5, session re-established in 24 s, under the 30 s ceiling), the India equivalent (→ Azure centralindia, 26 s), a tight-quota festival pre-reservation, and a spot-burst interruption that promotes 100 affected sessions to reserved on-demand in 24 s with operator notification. These pass because the modelled numbers clear the thresholds — they validate the gate's logic and the documented commitments, but they run on simulate_* fixtures, not a live cloud cut-over. See V3/PIXEL_STREAMING_POP_CAPACITY_MANAGEMENT.md for the evidence-file map.

Failure modes and edge cases#

How it connects#

Every tier — native UE, Pixel Streaming worker, and Tier-2 fallback — speaks the same multiplayer wire protocol to the same authoritative Rust world server. A Pixel Streaming worker is itself just another world-server client: it renders server-side and forwards multiplayer packets through the realtime gateway, which is why the world-server fan-out math counts workers, not unique users. That shared-authority design is the subject of World Server and Gateway and Netcode Protocol and Physics. Concert pre-warm and reserved walk-up pools tie routing to programming load in Saraswati Stage Pipeline; the launch decision's identity (v1UserId) and the JWT chain originate in V1 Integration and Identity Bridge; and the telemetry tiers, decision-distribution dashboard, and capacity Grafana panels feed Observability, Performance, Security, and Launch. For the catalogue of every service and module named here, see Subsystem Glossary and Layout.

Related#

• Product Promise and Architecture — the tier-stack rationale and the V1-reuse layout
• Subsystem Glossary and Layout — every service, module, and contract named on this page
• Tier-1 UE5 Client and Tier-2 Fallback Web Client — the two render surfaces routing chooses between
• World Server and Gateway and Netcode Protocol and Physics — why a Pixel Streaming worker is a world-server client
• Avatar, Animation, and Audio and Saraswati Stage Pipeline — concert prewarm and the priority that biases POP selection
• Tara Classes, Aja, and Commons and Authoring and Content Pipeline
• Data, Tenancy, and Residency and Persona Policy, Provenance, and Rights — region/tenant gating and the rights holds behind it
• V1 Integration and Identity Bridge, Commerce and Royalties, and Observability, Performance, Security, and Launch
• The section hub: ../V3_ARCHITECTURE.md