Runbook: V1 Relational Corruption (gd:relational-corruption)

Owner: workbench-platform on-call (rota:workbench-platform). Last reviewed: 2026-08-14. Version: 1.

The relational store answers, and the rows are wrong. Corruption rather than absence: every liveness check passes, the recovery is a restore to a recovery point rather than a restart, and putting the dependency back does not undo it.

This is the scenario the estate is least equipped to notice. A store that is down announces itself. A store that returns wrong rows does not, and there is no scheduled integrity check on this estate: a search across scripts/, infra/ and .github/ for pg_amcheck, a row-checksum sweep or an invariant scan returns nothing. Detection here is a report from an actor or a reconciliation that refuses, and both of those come after the damage has been read.

Related: database-backups.md documents the pg_dump path this recovery uses; it does not mention that a V1 restore is unfinished until the reconciler runs (§10).

1. Detection and declaration#

Detection signals that exist.

• scripts/operations/v1-restore-reconcile.mjs compares restored provider effects, publications, queues, invoices, artifacts, placements and approvals against what the outside world observed, and returns blockers. It is written for the post-restore path but it is the only thing on this estate that compares stored state to an independent observation.

• scripts/db-backup.sh writes a manifest.txt of SHA-256 digests and scripts/db-restore.sh aborts with exit code 2 when a digest disagrees. That detects corruption of the backup, not of the live store.

• PostgreSQL itself refuses a GARBLED page and cannot see a ZEROED one, and the difference decides what detection is possible here. Measured on 2026-08-14 over four arms (retest:D-F4 in scripts/operations/v1-follow-up-retest.mjs), same table, same 2000 rows, one page overwritten:

  page data	data_checksums	what a count(*) returned
  random	off	ERROR: invalid page in block 1
  random	on	the same ERROR, plus the checksum
  zeros	off	1815 of 2000 rows, no error
  zeros	on	1815 of 2000 rows, no error

  An all-zero page is a valid unused page to PostgreSQL, so a store that lost one answers cheerfully with fewer rows. POSTGRES_INITDB_ARGS: --data-checksums is now set in infra/hetzner/docker-compose.yml and it buys the diagnosis on the first two rows, not detection on the last two — and only for clusters initialised after that change, which means every clean-host restore and not the running production volume. Detection of a zeroed page has to come from a row count compared against something outside the store, which is what the reconciler in §10 is for.

• Detection authority: workbench-platform on-call, contact rota:workbench-platform.

• Declaration authority: production operations duty officer, contact rota:prod-ops.

• Gap, stated. docs/operations/on-call.md describes a weekly primary/secondary rotation and names no person and no rotation config; none of the register's rota: identifiers appears in it or in docs/operations/alert-routing.md. The name above is registered; the route from it to whoever is carrying the pager is not.

• Declare early and read-only first. Because the store answers, every minute between onset and declaration is a minute of reads returning wrong answers and writes built on them. The first action is to stop writes, not to diagnose.

2. Communications#

Both travel over dep:notification-channel, which this scenario leaves standing.

3. Dependency assumptions#

• dep:blob-store is healthy and is not rolled back with the relational store. Restoring rows to an earlier point while the bytes stay current leaves blobs with no row pointing at them — harmless garbage in the direction the reference graph runs (dep:primary-store → dep:blob-store).
• A backup set exists whose manifest verifies. db-restore.sh refuses to run otherwise, which is the correct behaviour and also means an unverifiable backup turns this incident into a data-loss incident.

4. Recovery infrastructure#

In place, and that is a choice with a cost. The restore runs on the same host. Restoring over the corrupt database destroys the evidence of what went wrong, so restore into a scratch database first (scripts/db-restore.sh BACKUP_DIR --databases <name> against a renamed target) when the cause is unknown and the corruption is not spreading.

5. Secrets#

• PostgreSQL credentials (PG_HOST, PG_PORT, PG_USER, PGPASSWORD — the same variables db-backup.sh uses). Held in the operator's environment, not fetched from the estate being repaired.

Both are recoverable out of band: the configuration-and-secrets state class is in the off-box backup (infra/hetzner/backup/offbox-backup.sh).

6. Restore order#

1. dep:primary-store — restore the rows. scripts/db-restore.sh verifies every SHA-256 in manifest.txt before invoking pg_restore and aborts on any mismatch.
2. dep:primary-store-scan — the scan standby reads the same rows and is correct as soon as they are.
3. dep:search-index — reproject last. The index is built from the rows (dep:search-index → dep:primary-store in the reference graph), so an index rebuilt from corrupt rows is corrupt, and one rebuilt before the restore is stale in a way that looks healthy.

dep:blob-store is not in this order because it is not taken away; if it were, it would come first, because a row exposed before its bytes is a dangling reference.

There is no point-in-time recovery on this estate. archive_mode = on appears in infra/yemaya/postgres-ha/primary.conf, and nothing outside infra/yemaya/postgres-ha/ references that directory — the V1 compose does not use it. The recovery point is therefore the last pg_dump, and the interval between dumps is the exposure. You cannot restore to the instant before the corruption; you restore to the last backup that verifies.

7. Validation#

• db-restore.sh manifest verification (exit code 2 on mismatch) — real, and it runs before any data is written.
• SELECT count(*) >= 0 FROM pg_catalog.pg_class per restored database, the floor box-loss-restore-drill.sh applies.
• A named count. Record what the estate held at the recovery point and compare. Every collection the reconciler is handed can be empty, and empty reconciles clean — an empty restore produces the same verdict as a complete one unless the expectation is supplied.
• The invariant that failed. Corruption is specific: whatever wrong answer raised this incident is the query to re-run.

8. Fencing — NOT APPLICABLE#

Nothing is promoted. There is one primary before and after, so there is no window in which two of them are writing. The dangerous window here is a different one: between declaration and the stopping of writes, the application is writing rows derived from wrong reads. Stopping writes is §1's first action for that reason.

9. Failover#

There is nothing to fail over to. infra/yemaya/postgres-ha/failover.sh exists and promotes a standby; the V1 compose has one postgres service and no replica, so the script's own check_standby refuses with "This server is not in recovery mode (not a standby)". See v1-dr-database-loss.md §9 for the detail — it is the same absent standby.

10. Reconciliation — the restore is not finished when the restore finishes#

Run scripts/operations/v1-restore-reconcile.mjs before resuming any autonomous or effectful work. A rollback to a recovery point un-does rows the outside world already saw: a publication that went out, a provider call that was charged, an invoice that was issued. The reconciler compares those against what was observed and returns blockers.

This matters more here than after a box loss. A box loss stops the estate before it can act on wrong data; corruption lets it keep acting.

11. Failback#

There is no alternate topology to return from. Failback is the resumption of writes, and it has one precondition: the reconciler returns no blockers. Take a fresh backup immediately afterwards — the restored estate's most recent backup is, by construction, the one it was restored from, and it predates the incident.

12. What this runbook does not cover#

• A store that is simply gone. That is v1-dr-database-loss.md; the recovery point there is untouched.
• Logical corruption confined to one table. database-backups.md covers partial restores and is faster.
• Detecting corruption. Nothing on this estate does, and this runbook will not pretend a detector exists.

13. Machine-checkable statement#

{
  "id": "runbook:v1-relational-corruption",
  "scenarioId": "gd:relational-corruption",
  "version": "1",
  "detectionAuthority": {
    "role": "service on-call",
    "name": "workbench-platform on-call",
    "reachableOver": ["dep:notification-channel"]
  },
  "declarationAuthority": {
    "role": "operations duty officer",
    "name": "production operations duty officer",
    "reachableOver": ["dep:notification-channel"]
  },
  "communications": [
    {
      "audience": "actors",
      "channelDependencyId": "dep:notification-channel",
      "withinMs": 900000
    },
    {
      "audience": "operators",
      "channelDependencyId": "dep:notification-channel",
      "withinMs": 300000
    }
  ],
  "assumes": ["dep:blob-store"],
  "recoveryInfrastructureDependencyId": null,
  "secrets": [
    {
      "id": "PGPASSWORD",
      "heldInDependencyId": "dep:primary-store",
      "recoverableOutOfBand": true
    }
  ],
  "restoreOrder": [
    "dep:primary-store",
    "dep:primary-store-scan",
    "dep:search-index"
  ],
  "fencing": null,
  "promotesAStandby": false,
  "validation": [
    "db-restore.sh verifies every SHA-256 in manifest.txt and aborts with exit code 2 on mismatch",
    "each restored database answers SELECT count(*) >= 0 FROM pg_catalog.pg_class",
    "recorded counts at the recovery point are compared against the restored counts",
    "the query whose wrong answer raised the incident returns the right one"
  ],
  "failback": "resume writes once v1-restore-reconcile.mjs returns no blockers, then take a fresh backup: the restored estate's newest backup is the one it was restored from"
}