Disciplines · Runbooks

V1 game-day follow-up register (S10.12.g)

severity ("nothing orders this against the others, so it is worked on last"), no owner, an owner that is a collective ("a collective does not close a finding"), no deadline ("so it is closed by the next drill finding it again"), no backlog

5sections9 minread

On this page
  • Owner: UNASSIGNED — no named person or rota owns this register. Each finding below has an owner; the register itself does not.
  • Last reviewed: 2026-08-14
  • Machine-readable: docs/releases/v1/verification/game-day/2026-08-14/follow-up-register.json
  • Graded by libs/oshun/workbench-kit/tools/follow-up-scan.mjs (pnpm run verify:operations:v1-follow-ups)

Every finding S10.12.a through S10.12.g measured, with a severity, an owner, a deadline and a backlog reference — and, for the ones that are closed, the run that closed them. This document and the JSON are graded against each other: the scan fails if a state stated here differs from the register, so a finding cannot be closed by editing prose.

1. What an entry has to carry, and why each field is refused when empty#

followUpFaults in @oshun/workbench-kit/game-day refuses a finding with no severity ("nothing orders this against the others, so it is worked on last"), no owner, an owner that is a collective ("a collective does not close a finding"), no deadline ("so it is closed by the next drill finding it again"), no backlog link ("the finding lives only in a drill report, which nobody reads on an ordinary Tuesday"), and — the one this cell turns on — a critical finding closed without a retest.

The scan adds what the contract cannot see:

  • The owner is read, not assigned. Every owner is a contact in the shipped dependency register, and where an entry names a dependency, one of that dependency's owners. The register cannot invent a name to satisfy the field.
  • The deadline is arithmetic: raisedAtMs + policy(severity), with the policy stated once — critical 14 days, major 30, minor 90. A row quietly given six more weeks fails.
  • Closure is backed by a run: a closed entry citing a retest must match a record in follow-up-retests.json whose verdict is fixed and whose finishing instant is the register's retestedAtMs.
  • Nothing measured is missing: every finding the acceptance matrix emitted appears here, under its own id or as an alias of the entry that absorbed it.
  • This document is graded against the JSON cell by cell — state, severity, owner, deadline, and the tally sentence under the table. A reader acts on the severity and the date as much as on the word closed, so a row that agrees on one column and lies on another is not a smaller defect. Twenty-seven negative controls mutate the real register and the real document; the scan exits 2 rather than reporting at all if any of them goes undetected.

The limit of the owner field, stated. The rota identifiers above are real — they are what the S10.5 dependency register records — and docs/operations/on-call.md names no person and holds no rotation config, so none of them resolves to somebody carrying a pager. The owner column says which rota is accountable, not who will answer. That gap is A-11's neighbour and is stated in every scenario runbook's §1.

And nobody has agreed to the deadline policy. agreedBy is null in the register. The windows are the ones this cell proposes; they are enforced by arithmetic so that changing them is a commit rather than a drift.

2. The register#

id severity state owner due finding
A-1 critical closed production operations duty officer 2026-08-28 the documented game-day invocation reported a pass having injected nothing
A-2 critical closed production operations duty officer 2026-08-28 all(.[]; .status != "fail") reads a skip as a pass
A-3 major closed production operations duty officer 2026-09-13 DRY_RUN defaults to true in all three drills
A-4 critical closed production operations duty officer 2026-08-28 jq -r '.isReadOnly // true' returns true for every input
A-5 major closed production operations duty officer 2026-09-13 nothing scheduled any drill, so there was no rotation
A-6 minor open workbench-platform on-call 2026-11-12 the scenario library's targets are not dependency ids
A-7 minor open workbench-platform on-call 2026-11-12 four of the ten classes have no scenario in that library
A-8 minor open workbench-platform on-call 2026-11-12 its objectives are prose graded by numbers a facilitator types in
A-9 minor open workbench-platform on-call 2026-11-12 the whole library has no caller
A-10 critical closed production operations duty officer 2026-08-28 neither mutating drill had a trap
A-11 major open production operations duty officer 2026-09-13 ten rollback records carry an RTO and no detection instant
A-13 major open production operations duty officer 2026-09-13 RPO/RTO are declared per substrate rather than per capability
B-1 major closed production operations duty officer 2026-09-13 no runbook mentioned the V1 recovery path
B-2 major closed production operations duty officer 2026-09-13 keyRecoveryVerified: true was a literal with no check behind it
B-3a minor open production operations duty officer 2026-11-12 content-service and computer-use are verified by directory existence
C-1 critical open workbench-platform on-call 2026-08-28 the reconciler every runbook names has no producer for its input
C-2 minor open workbench-platform on-call 2026-11-12 V1 has no outbox; the replay document declares the absence
C-3 major open delivery pipeline on-call 2026-09-13 the lease fence is real and its reclaim loop is deployed nowhere
D-F1 major open observability on-call 2026-09-13 monitoring now exists in source; exact-candidate live alert delivery is not yet admitted
D-F2 major open observability on-call 2026-09-13 external dead-man exists in source; missing-signal delivery is not yet admitted
D-F3 minor open workbench-platform on-call 2026-11-12 the blast-radius comment cites the development compose
D-F4 major open workbench-platform on-call 2026-09-13 a zeroed page vanishes silently, and checksums do not change that
D-F5 minor open production operations duty officer 2026-11-12 a trap-based reversal does not survive SIGKILL
D-F6 minor open production operations duty officer 2026-11-12 five of ten scenario classes cannot be injected on this estate
F-f1 critical open workbench-platform on-call 2026-08-28 a returning box writes into the key space of the estate that replaced it
F-f2 critical closed delivery pipeline on-call 2026-08-28 the backup verifies a redis checkpoint the restore never reads
F-f4 major open workbench-platform on-call 2026-09-13 no degraded mode exists during a regional loss
F-f5 minor open delivery pipeline on-call 2026-11-12 work written after the capture is lost and still sits on the returned box
F-f6 minor open observability on-call 2026-11-12 a connect-only probe is blind to a frozen dependency
G-1 major closed delivery pipeline on-call 2026-09-13 a failed INFO keyspace inside a pipeline recorded a fabricated zero

Seven criticals, twelve majors, eleven minors. Ten closed.

3. The retests#

scripts/operations/v1-follow-up-retest.mjs runs each of these against real containers and writes docs/releases/v1/verification/game-day/2026-08-14/follow-up-retests.json. Every retest carries an arm about the state before the repair, because a retest that only shows the repaired code passing cannot tell a fix from a check that was always going to pass. For three of the four that arm is the code as it was, run against the same input. For retest:key-recovery it cannot be: the pre-fix behaviour was the absence of a check, and running nothing against a bad input accepts it by definition, so that arm measures the artifact instead — the drill no longer writing the field as a literal, and calling the verifier that produces it. An arm asserting a remembered outcome would be a sentence dressed as a measurement, and there are none.

retest:drill-verdictfixed (closes A-1, A-2, A-3, A-4, A-10)#

The invocation docs/reference/ci-cd.md gives verbatim, run again. The old predicate all(.[]; .status != "fail"), applied to the checks that run produced, still returns true — one passing check and one skipped is the strongest pass it can express. The script now returns overall.outcome=inconclusive, overall.pass=false, naming both reasons, and the twelve tests that shipped with the repair are green.

retest:F-f2fixed (closes F-f2, and B-3 for redis and objects)#

Twenty-five keys, a forced BGSAVE, and two restores. Handed only the checkpoint the backup verifies, an --appendonly yes redis comes up with 0 of 25 keys and answers PONG — the pre-fix check accepts it. The repaired check refuses it by name. Handed the whole archived data directory, the same check accepts 25 of 25.

retest:key-recoveryfixed (closes B-2)#

The required set is derived from every ${VAR:?} in the deployed compose. A complete recovery environment is accepted; the same file with MINIO_ROOT_PASSWORD emptied is refused by name. The third arm reads box-loss-restore-drill.sh and measures that the literal keyRecoveryVerified:true is gone and the verifier is called — because the pre-fix code had no check at all to re-run, the field being a constant in a jq template.

retest:D-F4remedy-does-not-fix-it (closes nothing)#

The retest that refuted its own finding. D-F4 was raised as "data_checksums is off, so page corruption is silent", and the four arms say otherwise:

page data data_checksums what count(*) returned
random off ERROR: invalid page in block 1
random on the same ERROR, plus the checksum
zeros off 1815 of 2000 rows, no error
zeros on 1815 of 2000 rows, no error

An all-zero page is a valid unused page to PostgreSQL. The remedy adds a diagnosis to a garbled page — which was already refused — and changes nothing about the fault that was reported. POSTGRES_INITDB_ARGS: --data-checksums is set anyway, because the diagnosis is worth having and costs nothing at initdb; the finding stays open with its statement corrected, and detection of a zeroed page has to come from a row count compared against something outside the store.

4. Re-running it#

bash
pnpm run run:operations:v1-follow-up-retest   # the four retests, ~4 minutes
pnpm run verify:operations:v1-follow-ups      # grades the register, 27 negative controls

Both run in the follow-ups job of .github/workflows/v1-disaster-recovery.yml, retests first — the grading refuses a closure whose retestedAtMs is not the instant a run finished, so grading against a stale retest report would be grading against a claim.

The retest harness exits 3 when docker is unreachable and 1 when a retest cannot decide. A retest that refutes its finding exits 0 — that is a result, and the record is what matters.

The scan fails in two ways and they mean different things. A malformed register is a defect in this document or the JSON: a name that resolves to nobody, a deadline that does not follow the policy, a closure with no run behind it. An overdue register is well-formed and the work is late — which is what a deadline is for, and why it exits non-zero rather than printing a note. The output separates them. Set OSHUN_FOLLOW_UP_NOW_MS to see the overdue path before a deadline arrives; at 2027-01-01 all twenty open findings report late.

5. What this register does not establish#

  • It does not establish that any open finding is being worked on. It records who is accountable and by when.
  • It does not establish that the deadlines are achievable. Nobody has agreed to them; the policy is enforced arithmetically so that changing one is visible.
  • It does not cover findings from outside S10.12. The rollback rehearsal records, the on-call documentation and the reconciler's missing producer are referenced here because game days found them, not because this register owns them.