V1 disaster-recovery acceptance matrix (S10.12.f)

• Owner: UNASSIGNED — no named person or rota owns V1 recovery acceptance. Stated rather than filled with a team noun, because "the platform team" is nobody.
• Last reviewed: 2026-08-14
• Scenario: gd:regional-loss
• Runbook: runbook:v1-regional-loss (v1-disaster-recovery.md)

The seven aspects S10.12.f names, each answered by a step of one executed game day. The machine-readable matrix — outcomes, evidence, every measurement behind them, and the findings — is the newest acceptance-matrix.json under docs/releases/v1/verification/game-day/. This document and that file are graded against each other: libs/oshun/workbench-kit/tools/acceptance-scan.mjs fails if an outcome stated here differs from the one the drill measured, so a row cannot be improved by editing prose.

1. What fills it#

scripts/operations/v1-acceptance-drill.mjs stands up a disposable estate in the topology this runbook describes — one box carrying the relational store, the queue, the blob store and the user-facing surface; an off-box repository holding the bytes; a clean host the recovery runs on — kills the box with SIGKILL, watches two independent observers, publishes a notice, restores onto the clean host in the runbook's declared order, lets the old box come back, captures the recovered primary and fails back.

One incident, seven facets. The aspects are steps of a single timeline and not seven experiments. An aspect measured against its own private incident would be measuring a different estate each time.

What it does not claim. estate: "disposable" is a field in the report, not a caveat in prose. The containers are the drill's own, because the development box's postgres, redis and minio are shared with other worktrees. A run establishes that the procedure works and that the numbers are real. It does not establish the production estate's recovery time.

2. The matrix#

3. Aspect by aspect#

3.1 split-brain-prevention — failed#

What passing would mean here. This runbook promotes: a clean host is restored from the off-box repository and becomes the estate. Passing means that when the original box returns — a partition heals, a host reboots, an operator brings it up to look at it — something stops it writing.

How it is exercised. After the clean host is serving, the drill starts the box's containers again and writes the same primary key on both sides.

Measured. Both accepted it, with different values. Nothing in the procedure sits between them: §8 of the runbook says Fencing — ABSENT and this is what that sentence costs. The same drill shows the estate's one real fence working one layer down — a stale lease_token matches zero rows while the current holder's write lands — so the hole is specifically in the promotion path, not everywhere.

3.2 degraded-operation — passed#

What passing would mean here. That the estate can answer something true while part of it is missing, and that it never answers an empty store as though it were a full one.

How it is exercised. At every step of the restore the drill asks each store a real question and classifies the answer: unreachable, error (there and refusing — the caller knows), empty-success (there, answering, and wrong) or correct. Whether a port is open is not the question; a staged restore always has a window where some ports are open, so that test could never fail.

Measured. The estate served correct answers from part of its dependency set on the middle steps of the restore, and no survey returned an empty store as a success. It is worth being exact about the scope: at the first survey after the fault, none of the four dependencies answered and no surface did — a single-region estate has no degraded mode for a regional loss, because every declared fallback is co-located with what it backs up (finding F-f4). What passes here is degradation DURING the recovery, not during the outage.

3.3 queued-work — failed#

What passing would mean here. That work already in the queue survives the disaster, that in-flight work is reclaimed rather than lost, and that draining it after recovery applies each unit once.

How it is exercised. The queue is a redis stream with a consumer group: 25 entries, five delivered to a worker, three acknowledged and two still in that worker's pending list when the box dies — the state a queue is actually in during an incident. Six more are enqueued after the off-box capture. The restore is run twice: once from the whole redis data directory the archive holds, and once from dump.rdb alone, which is the artifact infra/hetzner/backup/offbox-backup.sh forces with BGSAVE and refuses to finish without.

Measured. The reclaim and the drain are sound: the in-flight entries were reclaimed by a new consumer, deliberate redeliveries applied nothing twice, and no entry was left pending. Two things failed. Entries written after the capture are gone — the queue's recovery point is the checkpoint instant, and those entries are still sitting on the box that came back. And the checkpoint-only arm restores zero of 25 entries: redis-server --appendonly yes, as infra/hetzner/docker-compose.yml deploys it, creates a new empty AOF on start and never reads dump.rdb. It answers PONG — and infra/hetzner/backup/box-loss-restore-drill.sh writes redisRestoreVerified: true on the strength of exactly that PING (finding F-f2).

3.4 user-communication — passed#

What passing would mean here. That the people who need to be told can be told over a route the incident did not remove, inside the window this runbook promises them.

How it is exercised. The notice is published during the outage to a channel that is not on the box, and read back over HTTP. The negative control is the same page served from the box, fetched at the address the box had before it died.

Measured. The off-box channel answered with the notice; the on-box channel did not answer at all; the notice was published well inside the 900000ms this runbook promises actors. What is measured is the out-of-band channel the runbook declares — the estate's automated sender is the BFF, which is why dep:notification-channel is in the gd:regional-loss target set. An automated notice from inside the failure domain is not a channel; nothing here claims it is.

3.5 monitoring-and-alerts — failed#

What passing would mean here. That the estate's own monitoring sees the incident and an alert fires.

How it is exercised. The drill stands up a real Prometheus and a real blackbox exporter, scraping real probes of the box, evaluating a real rule, and polls /api/v1/alerts until it is firing. Then it freezes the same store with docker pause and reads the probe again.

Measured on 2026-08-14. The rule fired, while the deployed estate contained no service that could run it, so nothing would have fired there (finding F-f3). The latency in this retained report is therefore a property of the disposable drill, and the row remains failed; current source-level remediation cannot retroactively turn it into live evidence. The negative control added the second half: with the store FROZEN rather than dead, the TCP-connect probe still reported probe_success=1 and no alert fired, while a probe that required an ANSWER timed out (finding F-f6). Task 13.7's successor uses answer-capable probes and an external dead-man receiver, and must still pass its own live admission before either historical gap closes.

3.6 backup-of-the-recovered-primary — passed#

What passing would mean here. That the rebuilt estate has a recovery point of its own. Until it does, the estate's most recent backup predates the disaster — the runbook says so in its failback step, and it is the reason this aspect exists at all.

How it is exercised. After the recovery, a fresh capture is taken off the clean host: a dump, a forced redis checkpoint, the object bytes, and the position of each store in its own history — a transaction LSN, an inventory version, a checkpoint sequence. It is then restored into a throwaway host and counted.

Measured. The capture was taken after the restore finished and before the failback began, and restoring it returned exactly the census the primary held. The capture is published in the report as a RecoveryPoint and graded by S10.11's recoveryPointFaults, so a marker with no position or a wall clock standing in for an LSN fails the scan rather than passing as a backup.

3.7 failback — passed#

What passing would mean here. That the estate returns to the topology it started in, with everything the recovered estate accumulated while it was the primary.

How it is exercised. "The same procedure with the roles swapped": the recovered estate is stopped, the box is rebuilt from nothing, and the fresh capture is restored into it.

Measured. The rebuilt box came back with the same row count, the same drained jobs, the queue, the objects and a surface answering. Two things are worth reading off it. The recovered estate is stopped FIRST, by hand, because no fence would make any other order safe — the failback is correct only because an operator remembered a step nothing enforces. And the row the unfenced box accepted during the split-brain window is gone: the failback restores the capture, and everything the old box wrote after the disaster goes with it. That is what the missing fence costs, in rows.

4. Findings for the follow-up register (S10.12.g)#

Each was emitted by the drill with the measurement that produced it. Current source now includes the Task 13.7 monitoring profile and answer probes, but that does not rewrite the retained exercise. F-f3 stays open until the exact source candidate produces admitted live incident and dead-man receipts; only then can a new measurement close it rather than carrying either the finding or the fix forward as received wisdom.

5. Re-running it#

pnpm run run:operations:v1-acceptance-drill      # fills the matrix, ~2 minutes
pnpm run verify:operations:v1-acceptance-matrix  # grades it, with 29 negative controls
pnpm run verify:operations:v1-disaster-recovery  # the artifact read back, no docker needed

The drill exits 3 when docker is unreachable — loudly, because a run that could not happen is not a run that passed. It leaves nothing behind: every container it creates carries the oshun.game-day.disposable label and is removed in a finally, and node scripts/operations/v1-fault-control.mjs --reverse-all drains anything a killed process left applied.

6. What this matrix cannot say#

• It cannot say what the production estate's recovery time is. It measures a disposable estate with a few hundred rows.
• It cannot say that the aspects it passes hold for other scenarios. It rehearses gd:regional-loss; the other nine registered scenarios have plans and runbooks but no acceptance matrix of their own.
• It cannot say that anybody would do these steps under pressure. Every step here was performed by a program that already knew the order.