# V3 Day-0 Patch Pipeline Runbook runbook-set: `v3-day0-patch-pipeline.v1` Owner: V3 Release Captain (patch captain for the launch window), joint with the platform-release engineer per store Companion gates: `V3/DAY0_PATCH_HEADROOM.md` (size budgets the patch must fit), `V3/COOK_SIZE_REPORTS_CI.md` (size regression checks), `V3/SERVICE_WORKER_CACHE_VALIDATION.md` (web cache invalidation), `V3/launch/store-rejection-contingency.md` (if a patch itself is rejected) `DAY0_PATCH_HEADROOM.md` proves a day-0 patch **fits** — 1 GB reserved on desktop/handheld/console, 500 MB on mobile/standalone VR, delivered as a UE Patcher + Iostore delta with no full installer re-cook. This runbook is the other half: how that patch actually **ships fast**, end to end, per platform, with the server-vs-client decision made before panic sets in. ## Standing Rule: Server-Side First Most launch-day defects are fixable without touching a store. The BFF, world server, realtime gateway, pxstream relay, web client, and CDN-delivered assets all deploy on our own clock. A client binary patch is the **last** resort, taken only when the defect lives in shipped native code. ### Mitigation decision matrix | Defect class | First response (no cert) | Client patch needed? | | ---------------------------------------------------------------- | ----------------------------------------------------------------------------------- | ------------------------------------------------------ | | Server logic (BFF route, world-server rule, gateway, relay) | Server deploy, 30–60 min | No | | Tunable behavior (rates, caps, timers, matchmaking weights) | Config/flag flip, < 15 min | No | | Broken feature reachable from client | Server-side kill switch for the feature; degraded-but-stable beats crashing | Later, batched | | Content/asset defect in CDN-streamed assets (on-demand mobile assets, tenant pak chunks) | CDN asset swap + manifest version bump; clients re-fetch — 1–2 h | No, if the asset is streamed; yes if cooked into the base install | | Web client bug (Tier-1/2 fallback, pxstream front-end) | Web deploy + service-worker cache invalidation, ~1 h | No (web is not store-gated) | | Native client crash / startup failure | Kill switch if feature-scoped; otherwise nothing server-side helps | **Yes — this pipeline, expedited** | | Client security vulnerability | Server-side exploit blocking at BFF/WAF where possible, immediately | **Yes — all platforms, expedited, highest priority** | | Cosmetic / minor | Nothing | Batched into the day-7 planned patch | The patch captain makes this call within 30 minutes of defect confirmation and records it (`operator.release.day0_patch.decision`) with the matrix row cited. ## Client Patch Path: Build → QA Smoke → Cert → Rollout ### Stage 1 — Build (all platforms in parallel, target: 3 h) 1. Branch from the GA release tag; fix lands with mandatory review even under time pressure (two reviewers, one outside the authoring team). 2. Build UE Patcher + Iostore delta against the **shipped** GA paks for every affected platform — the same artifact class as the validated synthetic patch (`day0-localization-audio-texture-hotfix`). 3. Size gate, fail-closed: delta must fit the platform headroom from `DAY0_PATCH_HEADROOM.md` (1 GB desktop/console, 500 MB mobile/standalone VR). The validated synthetic patch sizes (0.38–0.82 GB) show real multi-pak patches fit with margin; a delta that exceeds headroom does not ship — re-scope the patch, never raise the budget mid-incident. 4. No full re-cook, ever, on the day-0 path. If the fix forces a full re-cook, it is by definition not a day-0 patch — escalate to the release captain for a scheduled update instead. ### Stage 2 — QA smoke (target: 2 h, parallel per platform) The day-0 smoke suite is a fixed list, not improvised: - patch-application test: delta applies cleanly over a pristine GA install AND over an install that already took a previous delta; - cold join to the Atrium within the 5 s budget on the patched build; - reconnect within the 3 s budget; - 15-minute soak at target FPS on one representative device per platform (the same devices as the FPS validation gates); - regression check on the specific defect being fixed, plus its nearest neighbors; - cook-size report diff vs the GA baseline (`COOK_SIZE_REPORTS_CI.md`). Any smoke failure stops that platform's submission; platforms pass or fail independently. ### Stage 3 — Cert / store submission (the long pole) Expected turnaround per platform; the patch captain plans comms around the **slowest affected platform**: | Platform | Path | Expected turnaround | Expedite lever | | --------------------- | ----------------------------------------------- | --------------------------------------------- | ----------------------------------------------------- | | Server/web/CDN | Own deploy | 15 min – 2 h | n/a | | Steam (Win64) | SteamPipe update, no per-update review in practice | 2–4 h end to end | n/a (already fast) | | Epic (Win64) | Portal update | same day – 2 business days | Partner manager | | Google Play | Update review | 2–24 h | Staged-rollout track reviews faster | | Apple iOS/iPadOS/visionOS/Mac | Update review | 24–48 h | Expedited review request — hold for genuine day-0 breakage, one ask per incident | | Meta Horizon (Quest) | Update review | 2–5 business days expedited (5–10 standard) | Partner-rep escalation, request filed at GA-minus-7 | | Sony PSVR 2 | Patch cert (TRC delta) | **Assumption: 3–5 business days via the pre-arranged day-0 patch window; 5–10 without** | The day-0 window must be arranged with Sony **before** GA; verify it exists at GA-minus-14 | Consequence the matrix makes explicit: a native-only defect on Quest or PSVR 2 cannot be fully fixed for 3–5 business days. The kill-switch row of the decision matrix is therefore mandatory engineering, not nice-to-have — every launch feature reachable from a console/VR client needs a server-side disable path, because cert latency is the floor on fixing it client-side. ### Stage 4 — Staged rollout Server-side: canary 5% of shards (30 min bake, error-rate and tick-health watch) → 50% (30 min) → 100%. Client, per store mechanics: | Platform | Stages | | ------------ | ------------------------------------------------------------------ | | Steam | `prerelease` branch smoke (internal) → default branch 100% (Steam has no percentage rollout; the branch step is the stage) | | Google Play | 5% (2 h bake) → 25% (4 h) → 50% (6 h) → 100% | | Apple | Phased release **off** for day-0 (7-day phasing defeats the point); manual release to 100% after a 1-h TestFlight-build spot check | | Meta / Sony | 100% on cert clearance (no partial mechanism assumed); server kill switch stays armed as the rollback | | Web | Blue/green at the CDN; service-worker cache invalidation verified per its gate | Bake-stage abort: crash-free-sessions < 99.5% on the patched cohort, or the defect's regression metric not improving, halts promotion and triggers rollback. ### Rollback - Server: previous deploy retained warm; rollback is a redeploy, < 15 min. - Client paks: the previous pak set is retained on disk by the patcher; rollback ships as **another forward delta** that restores prior content — it fits the same reserved headroom (budgets sized for patch-sized deltas, per `DAY0_PATCH_HEADROOM.md`), so rollback never requires a full re-download either. - Steam: repoint default branch at the previous build id (minutes). - Google Play: halt staged rollout; previous version keeps serving the un-rolled-out cohort; full rollback via a new submission of the prior artifact. - Apple/Meta/Sony: no true store rollback — rollback is the server kill switch plus an expedited corrective submission. This asymmetry is another reason client patches are last resort. - Audit event: `operator.release.day0_patch.rolled_back` with the stage, cohort percentage, and trigger metric recorded. ## Communications - Internal: patch case opened in the Operator Console at decision time; status at each stage transition. - Public: status page note when the defect is user-visible (`incident-communications` action SLAs apply: 15 min to first public update); patch notes published at 100% rollout; per-platform "fix is in review with " honesty when cert is the bottleneck. ## Fail-Closed Criteria This pipeline is GA-blocking when any of the following holds: - the Sony day-0 patch window is not confirmed in writing by GA-minus-14 days, or the Meta expedite request channel is unverified by GA-minus-7; - any launch feature reachable from a console/VR client lacks a tested server-side kill switch (drill: flip each switch in staging, verify the client degrades without crashing — evidence `drill-kill-switch-matrix-ga`); - the day-0 smoke suite is not runnable as a single command per platform within the 2 h target; - a dry-run of this pipeline (synthetic defect → patch → smoke → mock submission → staged rollout in staging) has not been executed end to end before GA, with stage timings recorded (evidence `drill-day0-pipeline-ga`); - patch deltas are not verified against the `DAY0_PATCH_HEADROOM.md` budgets in CI for every affected platform.