# Runbook: V1 Blob Store Loss (`gd:blob-store-loss`)

Owner: workbench-platform on-call (`rota:workbench-platform`). Last reviewed:
2026-08-14. Version: 1.

> Rows survive and the bytes they point at do not. The reference direction
> decides the damage: the relational store holds keys into the blob store, so
> this is the outage that produces **dangling references** rather than orphans.

Related: [object-storage-backups.md](object-storage-backups.md) covers S3
versioning and cross-region replication for the multi-domain estate's
`oshun-prod-uploads` buckets. **It does not cover this.** The V1 blob store is
MinIO on the V1 host, and its only capture is the `mc mirror` inside
`infra/hetzner/backup/offbox-backup.sh`.

## 1. Detection and declaration

**Detection signal.** Eight capabilities stop and they stop in a distinctive
pattern: reads of items _with media_ fail while reads of the same items without
media succeed. `slo:item-read-availability`, `slo:render-success-rate`,
`slo:export-success-rate` and `slo:save-durability` all rest on it.

The clearest single observable is that a row resolves and its blob key 404s —
which is the dangling reference the reference graph predicts and the reason this
scenario is not symmetric with a relational loss.

- Detection authority: **workbench-platform on-call**, contact
  `rota:workbench-platform`.
- Declaration authority: **production operations duty officer**, contact
  `rota:prod-ops`.
- **Gap, stated.** `docs/operations/on-call.md` describes a weekly
  primary/secondary rotation and names no person and no rotation config; none of
  the register's `rota:` identifiers appears in it or in
  `docs/operations/alert-routing.md`. The name above is registered; the route
  from it to whoever is carrying the pager is not.

## 2. Communications

| Audience  | Within | Content                                                                                                                           |
| --------- | ------ | --------------------------------------------------------------------------------------------------------------------------------- |
| Actors    | 15 min | that media and downloads are unavailable, that item records are intact, and that uploads are refused rather than silently dropped |
| Operators | 5 min  | that this runbook is running, and by whom                                                                                         |

Both travel over `dep:notification-channel`, which this scenario leaves
standing.

## 3. Dependency assumptions

- `dep:primary-store` is healthy and **must not be rolled back**. The rows hold
  the blob keys and the recorded digests, and they are the only index of what
  the blob store should contain. Restoring bytes without them is a bucket of
  unaddressable objects.
- `dep:job-queue` is healthy: renders and exports that failed during the outage
  are re-run through it.

## 4. Recovery infrastructure

**In place.** MinIO is a service in `docker/docker-compose.yml`. The restore is
a mirror back into it.

## 5. Secrets

- `MINIO_ROOT_USER` and `MINIO_ROOT_PASSWORD`, the credentials
  `offbox-backup.sh` uses for `mc alias set v1-source`.
- The restic repository and its password, if the bytes come from the off-box
  snapshot rather than from a surviving local copy.

Both recoverable out of band: the restic repository is required to be off the V1
host — `offbox-backup.sh` refuses a local one outright with
`RESTIC_REPOSITORY must not be local to the V1 host`.

## 6. Restore order

`dep:blob-store` comes back **before** anything that references it. The
reference graph runs `dep:primary-store` → `dep:blob-store` (artifact rows carry
a blob key and a recorded digest), so:

| Bring back | Before        | Because                                                                                          |
| ---------- | ------------- | ------------------------------------------------------------------------------------------------ |
| blob store | primary store | a row exposed before its bytes is a dangling reference; bytes without a row are harmless garbage |

This is the reverse of the capture order, and deliberately so: capture the
referring store first so inserts in the skew window leave orphans, restore the
referenced store first so no row is exposed pointing at bytes that are not back.
`restoreOrderFaults` in `@oshun/workbench-kit/game-day` checks a proposed order
against the same graph.

In practice the relational store is not down in this scenario, which means the
rows are **already exposed** and pointing at bytes that are not back. That is
the dangling-reference window, it opens at the fault and not at the restore, and
nothing closes it except finishing §7.

## 7. Validation

The rows carry a recorded digest, and that is what makes this store verifiable
where Redis and the file state are not:

- read back three known objects **by digest** and compare against the digest in
  the artifact row — not merely that the key resolves;
- the object count is compared against the count of artifact rows with a blob
  key. Record it before the restore or take it from the relational store, which
  survived.

> **Gap, stated.** `box-loss-restore-drill.sh` asserts its MinIO step from an
> `mc mirror` that an **empty** store also passes. Only PostgreSQL has a count
> floor in that drill. Do not read a successful mirror as evidence that the
> contents came back.

## 8. Fencing — NOT APPLICABLE

Nothing is promoted; there is one blob store before and after. The adjacent risk
is not split-brain but **write loss during the window**: uploads attempted while
the store is down must be refused rather than acknowledged, or the rows will
record artifacts whose bytes were never written — a dangling reference the
restore cannot fix, because the bytes never existed.

## 9. Failover

There is no second blob store to fail over to. The V1 estate has one MinIO
instance, and the cross-region replication described in
`object-storage-backups.md` belongs to the S3 deployment, not to this one.

## 10. Reconciliation

Run `scripts/operations/v1-restore-reconcile.mjs`. Artifacts are among the
collections it compares against what the outside world observed, and this
scenario is the one where its artifact reconciliation earns its place: a
published artifact whose bytes are gone was seen by somebody outside the estate.

**Give it expected counts.** An empty collection reconciles clean, and after a
blob loss "no artifacts" is exactly the wrong answer to accept quietly.

## 11. Failback

The estate never left its normal topology. Failback is the confirmation that the
dangling-reference window is closed: every artifact row with a blob key resolves
to bytes whose digest matches. Until that holds, the estate is serving rows that
point at nothing.

Take a fresh backup afterwards — the mirror the restore came from is now the
estate's most recent capture and it predates the incident.

## 12. What this runbook does not cover

- The multi-domain S3 estate. See
  [object-storage-backups.md](object-storage-backups.md); versioning and
  cross-region replication are real there and absent here.
- Per-object recovery of a single deleted key, which that runbook also covers
  and which does not need this procedure.
- A total host loss, which takes MinIO and the rows together — see
  [v1-disaster-recovery.md](v1-disaster-recovery.md).

## 13. Machine-checkable statement

<!-- oshun:runbook -->

```json
{
  "id": "runbook:v1-blob-store-loss",
  "scenarioId": "gd:blob-store-loss",
  "version": "1",
  "detectionAuthority": {
    "role": "service on-call",
    "name": "workbench-platform on-call",
    "reachableOver": ["dep:notification-channel"]
  },
  "declarationAuthority": {
    "role": "operations duty officer",
    "name": "production operations duty officer",
    "reachableOver": ["dep:notification-channel"]
  },
  "communications": [
    {
      "audience": "actors",
      "channelDependencyId": "dep:notification-channel",
      "withinMs": 900000
    },
    {
      "audience": "operators",
      "channelDependencyId": "dep:notification-channel",
      "withinMs": 300000
    }
  ],
  "assumes": ["dep:primary-store", "dep:job-queue"],
  "recoveryInfrastructureDependencyId": null,
  "secrets": [
    {
      "id": "MINIO_ROOT_PASSWORD",
      "heldInDependencyId": "dep:blob-store",
      "recoverableOutOfBand": true
    },
    {
      "id": "restic repository password",
      "heldInDependencyId": "dep:blob-store",
      "recoverableOutOfBand": true
    }
  ],
  "restoreOrder": ["dep:blob-store"],
  "fencing": null,
  "promotesAStandby": false,
  "validation": [
    "three known objects read back by digest and compared against the digest recorded in the artifact row",
    "the object count matches the count of artifact rows carrying a blob key"
  ],
  "failback": "confirm every artifact row with a blob key resolves to bytes whose digest matches, then take a fresh backup"
}
```