# Runbook: V1 Job Queue Loss (`gd:queue-loss`)

Owner: delivery pipeline on-call (`rota:delivery-pipeline`). Last reviewed:
2026-08-14. Version: 1.

> The queue and everything resting on it stop. Work already accepted is the
> question: the interrupted paths finish nothing, and whether accepted work is
> replayed or lost is decided by the durable record rather than by the queue.

**What `dep:job-queue` actually is on this estate, because it decides the whole
recovery.** There is no broker. `apps/oshun/bff/src/generation/jobs-route.ts`
holds jobs in an **in-process map** and persists the whole set as a snapshot
through `createSnapshotSink`
(`libs/oshun/persistence/src/durable-snapshot-store.ts`) into the
`admin_store_snapshot` table — one row per `(storeKey, scopeId)`, the key being
`generation-jobs`.

So the durable record of accepted work is in **`dep:primary-store`**, and a
queue loss on this estate is a process losing its map. Two consequences:

1. **The recovery is a reload, not a replay.** Restarting the BFF re-loads the
   snapshot. Nothing has to be reconstructed from an event log because there is
   no event log in this path.
2. **A queue loss with a healthy relational store loses only in-flight
   transitions** — the work between the last successful `persistJobs()` and the
   stop. `requireDurableGenerationJobs()` exists so that a mutation is not
   acknowledged before it is durable; where it is called, the acknowledged set
   and the durable set agree.

## 1. Detection and declaration

**Detection signal.** Jobs stop reaching terminal states while submission keeps
succeeding: `dep:job-worker-pool` rests on `dep:job-queue` in the S10.5
substrate, so the queue's failure is observed as work that is accepted and never
finishes rather than as a submission error. The objectives that move first are
`slo:job-success-rate` and `slo:gate-run-completion`.

- Detection authority: **delivery pipeline on-call**, contact
  `rota:delivery-pipeline`.
- Declaration authority: **production operations duty officer**, contact
  `rota:prod-ops`.
- **Gap, stated.** `docs/operations/on-call.md` describes a weekly
  primary/secondary rotation and names no person and no rotation config; none of
  the register's `rota:` identifiers appears in it or in
  `docs/operations/alert-routing.md`. The name above is registered; the route
  from it to whoever is carrying the pager is not.

## 2. Communications

| Audience  | Within | Content                                                                                          |
| --------- | ------ | ------------------------------------------------------------------------------------------------ |
| Actors    | 15 min | that submitted work is not running, whether it was accepted, and that it need not be resubmitted |
| Operators | 5 min  | that this runbook is running, and by whom                                                        |

Both travel over `dep:notification-channel`, which this scenario leaves
standing.

**Tell actors not to resubmit.** A resubmission during a queue outage is the
duplicate this estate is least able to collapse: the snapshot is keyed by a
freshly minted `randomUUID()` job id, so two submissions of the same work are
two jobs.

## 3. Dependency assumptions

- `dep:primary-store` is healthy. It holds `admin_store_snapshot`, which is the
  queue's durable record — if it is not healthy, this is not a queue incident,
  it is [v1-dr-database-loss.md](v1-dr-database-loss.md).
- `dep:blob-store` is healthy: job outputs are written there and a replayed job
  writes there again.

## 4. Recovery infrastructure

**In place.** The workers and the queue are the same process tree as the rest of
the BFF; there is nowhere else for them to run and nothing to provision.

## 5. Secrets

- Provider credentials the workers need to finish the work they reload
  (`OSHUN_MESHY_API_KEY` and its siblings, read at the worker and failing closed
  when absent).

Recoverable out of band: the `configuration-and-secrets` state class is in the
off-box backup (`infra/hetzner/backup/offbox-backup.sh`).

## 6. Restore order

1. `dep:job-queue` — bring the queue back first by restarting the process that
   loads the `generation-jobs` snapshot. The workers have nothing to take from
   an empty queue, and a worker pool started first will report itself healthy
   while doing nothing.
2. `dep:job-worker-pool` — the pool rests on the queue in the substrate, so it
   follows it.

No reference edge touches either, so the ordering constraint here is the
substrate one rather than the referential one.

## 7. Validation

- The reloaded snapshot's job count equals the count recorded before the stop.
  **Write the count down before restarting** — this is the check the estate
  cannot perform for you, and without it a snapshot that loaded nothing looks
  exactly like one that loaded everything.
- One synthetic job submitted after recovery reaches a terminal state.
- The queued backlog drains rather than sitting at a constant depth: a worker
  pool that reconnects to a queue it cannot read holds a steady non-zero depth
  and reports healthy.

> **Gap, stated.** `box-loss-restore-drill.sh` asserts `redisRestoreVerified`
> from a `PING`, and an empty Redis answers `PING`. Redis is captured by
> `offbox-backup.sh` (a forced `BGSAVE`, then the `dump.rdb`), but nothing on
> this estate compares its restored contents to a count. If the queue's state
> ever moves from `admin_store_snapshot` into Redis, this section is wrong and
> must be rewritten before that move ships.

## 8. Fencing — NOT APPLICABLE

Nothing is promoted, so there is no old primary to reject. The adjacent risk is
a real one and is **not** covered by anything here: **two workers reloading the
same snapshot**. The snapshot is a whole-set write (`persistJobs` serialises
through a write tail), so a second process that loads and saves concurrently
overwrites rather than merges. Bring back exactly one.

Worth knowing while you do: the estate already fences leases elsewhere.
`live_media_pipeline_job_attempts.lease_token`
(`libs/shared/live-media/src/sql-media-pipeline-job-store.ts`) is checked in the
`UPDATE ... WHERE` of every heartbeat, lease expiry and completion, so a worker
whose lease was reassigned updates zero rows. The generation-job snapshot has no
such token, which is why "bring back exactly one" is an instruction to a person
rather than a property of the store.

## 9. Failover

There is no standby queue and no second worker pool. The estate has one of each.

## 10. Reconciliation

`scripts/operations/v1-restore-reconcile.mjs` compares restored queues against
what the outside world observed, and it is the control that answers the question
this scenario turns on: **was an accepted job's effect already delivered?** A
job that reached a provider before the stop and is reloaded as pending will be
run again.

**Give it expected counts.** Every collection it is handed can be empty, and
empty reconciles clean.

> **The replay procedure is not written.** There is no queue drain or replay
> script on this estate — `ls scripts/` matches nothing for `queue`, `drain` or
> `replay`. S10.12.c is where those procedures belong, and this runbook will not
> invent one. Until then, replay is the reconciler's blocker list read by a
> person.

## 11. Failback

The estate never left its normal topology; failback is the confirmation that it
is back in it — backlog draining, one synthetic job terminal, and the reloaded
count matching what was recorded. Record the instant: recovery time runs from
the fault, not from the restart.

## 12. What this runbook does not cover

- The relational store that holds the snapshot — see
  [v1-dr-database-loss.md](v1-dr-database-loss.md).
- Provider failures the workers hit while draining — see
  [v1-dr-model-provider-outage.md](v1-dr-model-provider-outage.md).
- Dead-letter triage. `libs/oshun/workbench-kit/src/job-queue-views.ts` models
  the `dead-lettered` state; the operator procedure for draining it is not
  written.

## 13. Machine-checkable statement

<!-- oshun:runbook -->

```json
{
  "id": "runbook:v1-queue-loss",
  "scenarioId": "gd:queue-loss",
  "version": "1",
  "detectionAuthority": {
    "role": "service on-call",
    "name": "delivery pipeline on-call",
    "reachableOver": ["dep:notification-channel"]
  },
  "declarationAuthority": {
    "role": "operations duty officer",
    "name": "production operations duty officer",
    "reachableOver": ["dep:notification-channel"]
  },
  "communications": [
    {
      "audience": "actors",
      "channelDependencyId": "dep:notification-channel",
      "withinMs": 900000
    },
    {
      "audience": "operators",
      "channelDependencyId": "dep:notification-channel",
      "withinMs": 300000
    }
  ],
  "assumes": ["dep:primary-store", "dep:blob-store"],
  "recoveryInfrastructureDependencyId": null,
  "secrets": [
    {
      "id": "provider worker credentials",
      "heldInDependencyId": "dep:job-queue",
      "recoverableOutOfBand": true
    }
  ],
  "restoreOrder": ["dep:job-queue", "dep:job-worker-pool"],
  "fencing": null,
  "promotesAStandby": false,
  "validation": [
    "the reloaded generation-jobs snapshot holds the job count recorded before the stop",
    "one synthetic job submitted after recovery reaches a terminal state",
    "the backlog depth falls rather than holding steady"
  ],
  "failback": "confirm the backlog drains, one synthetic job is terminal and the reloaded count matches; the estate never left its normal topology"
}
```