# V1 disaster-recovery acceptance matrix (S10.12.f)

- Owner: **UNASSIGNED** — no named person or rota owns V1 recovery acceptance.
  Stated rather than filled with a team noun, because "the platform team" is
  nobody.
- Last reviewed: 2026-08-14
- Scenario: `gd:regional-loss`
- Runbook: `runbook:v1-regional-loss`
  ([v1-disaster-recovery.md](v1-disaster-recovery.md))

The seven aspects S10.12.f names, each answered by a step of one executed game
day. The machine-readable matrix — outcomes, evidence, every measurement behind
them, and the findings — is the newest `acceptance-matrix.json` under
`docs/releases/v1/verification/game-day/`. **This document and that file are
graded against each other**:
`libs/oshun/workbench-kit/tools/acceptance-scan.mjs` fails if an outcome stated
here differs from the one the drill measured, so a row cannot be improved by
editing prose.

## 1. What fills it

`scripts/operations/v1-acceptance-drill.mjs` stands up a disposable estate in
the topology this runbook describes — one box carrying the relational store, the
queue, the blob store and the user-facing surface; an off-box repository holding
the bytes; a clean host the recovery runs on — kills the box with SIGKILL,
watches two independent observers, publishes a notice, restores onto the clean
host in the runbook's declared order, lets the old box come back, captures the
recovered primary and fails back.

**One incident, seven facets.** The aspects are steps of a single timeline and
not seven experiments. An aspect measured against its own private incident would
be measuring a different estate each time.

**What it does not claim.** `estate: "disposable"` is a field in the report, not
a caveat in prose. The containers are the drill's own, because the development
box's postgres, redis and minio are shared with other worktrees. A run
establishes that the procedure works and that the numbers are real. It does not
establish the production estate's recovery time.

## 2. The matrix

| aspect                            | outcome    | what decided it                                                                         |
| --------------------------------- | ---------- | --------------------------------------------------------------------------------------- |
| `split-brain-prevention`          | **failed** | the returning box and the estate that replaced it both accepted a write on one key      |
| `degraded-operation`              | **passed** | the recovering estate answered real questions correctly with part of its stores missing |
| `queued-work`                     | **failed** | the checkpoint the backup verifies restores an empty queue that answers `PONG`          |
| `user-communication`              | **passed** | the notice was read back off the box while the on-box channel did not answer            |
| `monitoring-and-alerts`           | **failed** | the rule fires, and no service that could run it is deployed                            |
| `backup-of-the-recovered-primary` | **passed** | a fresh capture was taken after the recovery, before the failback, and restored         |
| `failback`                        | **passed** | the rebuilt box came back with everything the recovered estate held                     |

## 3. Aspect by aspect

### 3.1 `split-brain-prevention` — failed

**What passing would mean here.** This runbook promotes: a clean host is
restored from the off-box repository and becomes the estate. Passing means that
when the original box returns — a partition heals, a host reboots, an operator
brings it up to look at it — something stops it writing.

**How it is exercised.** After the clean host is serving, the drill starts the
box's containers again and writes the same primary key on both sides.

**Measured.** Both accepted it, with different values. Nothing in the procedure
sits between them: §8 of the runbook says `Fencing — ABSENT` and this is what
that sentence costs. The same drill shows the estate's one real fence working
one layer down — a stale `lease_token` matches zero rows while the current
holder's write lands — so the hole is specifically in the promotion path, not
everywhere.

### 3.2 `degraded-operation` — passed

**What passing would mean here.** That the estate can answer something true
while part of it is missing, and that it never answers an empty store as though
it were a full one.

**How it is exercised.** At every step of the restore the drill asks each store
a real question and classifies the answer: `unreachable`, `error` (there and
refusing — the caller knows), `empty-success` (there, answering, and wrong) or
`correct`. Whether a port is open is not the question; a staged restore always
has a window where some ports are open, so that test could never fail.

**Measured.** The estate served correct answers from part of its dependency set
on the middle steps of the restore, and no survey returned an empty store as a
success. It is worth being exact about the scope: at the first survey after the
fault, none of the four dependencies answered and no surface did — a
single-region estate has no degraded mode for a regional loss, because every
declared fallback is co-located with what it backs up (finding F-f4). What
passes here is degradation DURING the recovery, not during the outage.

### 3.3 `queued-work` — failed

**What passing would mean here.** That work already in the queue survives the
disaster, that in-flight work is reclaimed rather than lost, and that draining
it after recovery applies each unit once.

**How it is exercised.** The queue is a redis stream with a consumer group: 25
entries, five delivered to a worker, three acknowledged and two still in that
worker's pending list when the box dies — the state a queue is actually in
during an incident. Six more are enqueued after the off-box capture. The restore
is run twice: once from the whole redis data directory the archive holds, and
once from `dump.rdb` alone, which is the artifact
`infra/hetzner/backup/offbox-backup.sh` forces with `BGSAVE` and refuses to
finish without.

**Measured.** The reclaim and the drain are sound: the in-flight entries were
reclaimed by a new consumer, deliberate redeliveries applied nothing twice, and
no entry was left pending. Two things failed. Entries written after the capture
are gone — the queue's recovery point is the checkpoint instant, and those
entries are still sitting on the box that came back. And the checkpoint-only arm
restores **zero of 25** entries: `redis-server --appendonly yes`, as
`infra/hetzner/docker-compose.yml` deploys it, creates a new empty AOF on start
and never reads `dump.rdb`. It answers `PONG` — and
`infra/hetzner/backup/box-loss-restore-drill.sh` writes
`redisRestoreVerified: true` on the strength of exactly that `PING` (finding
F-f2).

### 3.4 `user-communication` — passed

**What passing would mean here.** That the people who need to be told can be
told over a route the incident did not remove, inside the window this runbook
promises them.

**How it is exercised.** The notice is published during the outage to a channel
that is not on the box, and read back over HTTP. The negative control is the
same page served from the box, fetched at the address the box had before it
died.

**Measured.** The off-box channel answered with the notice; the on-box channel
did not answer at all; the notice was published well inside the 900000ms this
runbook promises actors. What is measured is the out-of-band channel the runbook
declares — the estate's automated sender is the BFF, which is why
`dep:notification-channel` is in the `gd:regional-loss` target set. An automated
notice from inside the failure domain is not a channel; nothing here claims it
is.

### 3.5 `monitoring-and-alerts` — failed

**What passing would mean here.** That the estate's own monitoring sees the
incident and an alert fires.

**How it is exercised.** The drill stands up a real Prometheus and a real
blackbox exporter, scraping real probes of the box, evaluating a real rule, and
polls `/api/v1/alerts` until it is firing. Then it freezes the same store with
`docker pause` and reads the probe again.

**Measured on 2026-08-14.** The rule fired, while the deployed estate contained
no service that could run it, so nothing would have fired there (finding F-f3).
The latency in this retained report is therefore a property of the disposable
drill, and the row remains `failed`; current source-level remediation cannot
retroactively turn it into live evidence. The negative control added the second
half: with the store FROZEN rather than dead, the TCP-connect probe still
reported `probe_success=1` and no alert fired, while a probe that required an
ANSWER timed out (finding F-f6). Task 13.7's successor uses answer-capable
probes and an external dead-man receiver, and must still pass its own live
admission before either historical gap closes.

### 3.6 `backup-of-the-recovered-primary` — passed

**What passing would mean here.** That the rebuilt estate has a recovery point
of its own. Until it does, the estate's most recent backup predates the disaster
— the runbook says so in its failback step, and it is the reason this aspect
exists at all.

**How it is exercised.** After the recovery, a fresh capture is taken off the
clean host: a dump, a forced redis checkpoint, the object bytes, and the
position of each store in its own history — a transaction LSN, an inventory
version, a checkpoint sequence. It is then restored into a throwaway host and
counted.

**Measured.** The capture was taken after the restore finished and before the
failback began, and restoring it returned exactly the census the primary held.
The capture is published in the report as a `RecoveryPoint` and graded by
S10.11's `recoveryPointFaults`, so a marker with no position or a wall clock
standing in for an LSN fails the scan rather than passing as a backup.

### 3.7 `failback` — passed

**What passing would mean here.** That the estate returns to the topology it
started in, with everything the recovered estate accumulated while it was the
primary.

**How it is exercised.** "The same procedure with the roles swapped": the
recovered estate is stopped, the box is rebuilt from nothing, and the fresh
capture is restored into it.

**Measured.** The rebuilt box came back with the same row count, the same
drained jobs, the queue, the objects and a surface answering. Two things are
worth reading off it. The recovered estate is stopped FIRST, by hand, because no
fence would make any other order safe — the failback is correct only because an
operator remembered a step nothing enforces. And the row the unfenced box
accepted during the split-brain window is **gone**: the failback restores the
capture, and everything the old box wrote after the disaster goes with it. That
is what the missing fence costs, in rows.

## 4. Findings for the follow-up register (S10.12.g)

| id     | severity   | summary                                                                                   |
| ------ | ---------- | ----------------------------------------------------------------------------------------- |
| `F-f1` | `critical` | a returning box writes into the key space of the estate that replaced it                  |
| `F-f2` | `critical` | the backup verifies a redis checkpoint the restore never reads                            |
| `F-f3` | `major`    | at exercise time the deployed estate ran no telemetry service, so no alert had a producer |
| `F-f4` | `major`    | no degraded mode exists during a regional loss; every fallback is co-located              |
| `F-f5` | `minor`    | work written after the capture is lost, and still sits on the returned box                |
| `F-f6` | `minor`    | a connect-only probe is blind to a frozen dependency                                      |

Each was emitted by the drill with the measurement that produced it. Current
source now includes the Task 13.7 monitoring profile and answer probes, but that
does not rewrite the retained exercise. F-f3 stays open until the exact source
candidate produces admitted live incident and dead-man receipts; only then can a
new measurement close it rather than carrying either the finding or the fix
forward as received wisdom.

## 5. Re-running it

```bash
pnpm run run:operations:v1-acceptance-drill      # fills the matrix, ~2 minutes
pnpm run verify:operations:v1-acceptance-matrix  # grades it, with 29 negative controls
pnpm run verify:operations:v1-disaster-recovery  # the artifact read back, no docker needed
```

The drill exits 3 when docker is unreachable — loudly, because a run that could
not happen is not a run that passed. It leaves nothing behind: every container
it creates carries the `oshun.game-day.disposable` label and is removed in a
`finally`, and `node scripts/operations/v1-fault-control.mjs --reverse-all`
drains anything a killed process left applied.

## 6. What this matrix cannot say

- It cannot say what the production estate's recovery time is. It measures a
  disposable estate with a few hundred rows.
- It cannot say that the aspects it passes hold for other scenarios. It
  rehearses `gd:regional-loss`; the other nine registered scenarios have plans
  and runbooks but no acceptance matrix of their own.
- It cannot say that anybody would do these steps under pressure. Every step
  here was performed by a program that already knew the order.