Disciplines · Runbooks

Runbook: V1 Search Index Loss (gd:search-index-loss)

the registered objectives, and freshness is the one that moves first: an index that has stopped accepting projections still answers queries, with answers that get older.

13sections5 minread
Markdown source
On this page

Owner: workbench-platform on-call (rota:workbench-platform). Last reviewed: 2026-09-14. Version: 2.

A derived store, rebuildable from the relational one, which is exactly why its recovery time is long and its recovery point is inherited: nothing is lost that the source does not also lose, and the wait is the reprojection.

Two facts decide this recovery, and both are absences.

  1. The index is in no backup. infra/hetzner/backup/offbox-backup.sh captures PostgreSQL dumps, the Redis RDB, the MinIO mirror, content-service runs, computer-use snapshots and configuration. The Qdrant volume is not among them, and that is correct — a derived store restored from a stale copy is worse than an empty one, because it is wrong instead of obviously missing.
  2. There is an executable reindex boundary. The BFF image contains dist/reproject-personalization-vector-index.js, built from reproject-personalization-vector-index.ts; @oshun/bff also exposes it as reproject:personalization-index. BFF boot runs the same reconciliation before HTTP is constructed.

1. Detection and declaration#

Detection signal. slo:search-availability and slo:index-freshness are the registered objectives, and freshness is the one that moves first: an index that has stopped accepting projections still answers queries, with answers that get older. The two capabilities that stop outright are "search the catalogue" and "filter a board by facet".

dep:search-index carries fallback among its registered mitigations — dep:primary-store-scan, the search standby. The fallback is what makes this incident quiet. With the scan serving, search still works and is slower, so the estate degrades rather than stops, and the outage can run for a long time before anybody declares it.

  • Detection authority: workbench-platform on-call, contact rota:workbench-platform.
  • Declaration authority: production operations duty officer, contact rota:prod-ops.
  • Gap, stated. docs/operations/on-call.md describes a weekly primary/secondary rotation and names no person and no rotation config; none of the register's rota: identifiers appears in it or in docs/operations/alert-routing.md. The name above is registered; the route from it to whoever is carrying the pager is not.

2. Communications#

Audience Within Content
Actors 15 min that search results are incomplete or stale, and which surfaces fall back to a scan
Operators 5 min that this runbook is running, and by whom

Both travel over dep:notification-channel, which this scenario leaves standing.

Tell actors the results are incomplete. A search that returns fewer rows than it should is the failure an actor cannot see: an empty result reads as "no such item" and not as "the index is down".

3. Dependency assumptions#

  • dep:primary-store is healthy. It is the reprojection source, and the reference graph runs dep:search-index → dep:primary-store: projected documents carry the id of the row they were built from. An index rebuilt from a store that is itself impaired is a second incident wearing the first one's clothes.
  • dep:primary-store-scan is healthy and carrying the search load. The scan rests on the same relational store, so this assumption is the one above restated — which is why S10.10 records that "search degrades to a scan" is a plan for a search-index outage and not for a database one.

4. Recovery infrastructure#

In place. Qdrant is a private, health-gated service in infra/hetzner/docker-compose.yml. The BFF is ordered after it and production requires the endpoint, API key, collection, dimensions, and embedding version. Recovery starts Qdrant with an empty volume and fills it from PostgreSQL.

5. Secrets#

  • OSHUN_PERSONALIZATION_QDRANT_API_KEY, dimensions, and embedding-version configuration from the recovered stack environment.

Recoverable out of band: the configuration-and-secrets state class is in the off-box backup.

6. Restore order#

One dependency, dep:search-index, and its single ordering constraint is against the source rather than within the set: the relational store must be healthy and current before the reprojection starts. That is the reference direction — the referenced store first — and it is the reverse of the capture order, in which the referring store is captured first so that the skew leaves orphans instead of dangling references.

Reprojecting from a store that is still restoring produces an index that is internally consistent and wrong, and nothing downstream will notice.

Once PostgreSQL and Qdrant are healthy, run:

bash
docker compose -p oshun-production \
  --env-file /opt/oshun/production/.env \
  -f /opt/oshun/bundle/docker-compose.yml \
  run --rm bff node dist/reproject-personalization-vector-index.js

For a repository-local built image, the equivalent package entry point is pnpm --filter @oshun/bff reproject:personalization-index.

7. Validation#

  • The command exits zero and its JSON reports equal authoritativeCount, projectedCount, and verifiedCount.
  • metadata.dimensions equals OSHUN_PERSONALIZATION_VECTOR_DIMENSIONS, distance is cosine, and all three required payload-index fields are present.
  • A known vector query returns the retained subject control while a tombstoned subject is absent. Task 13.6's isolated production-store drill automates this query and the empty-index reprojection control.
  • BFF boot completes after the command; it independently repeats reconciliation and refuses malformed metadata or mismatched readback.

8. Fencing — NOT APPLICABLE#

Nothing is promoted. There is one index and one scan fallback, and the fallback is a read path rather than a writable replica, so there is no second writer to reject.

9. Failover#

Failover here is the fallback the register already declares: search reads move to dep:primary-store-scan. It is a degradation and not a promotion — the scan answers the same questions more slowly and from the authoritative rows.

Confirm the scan is actually carrying the load rather than assumed to be. The scan carries throttle and bulkhead among its mitigations precisely because a full-table scan serving the search surface is how a search outage becomes a database outage.

10. Reconciliation#

The Qdrant collection publishes nothing to an external party. Reconciliation is still substantive: the command deletes points not present in the PostgreSQL authority, skips tombstoned subjects, rewrites legacy payloads to the configured embedding/schema version, and verifies the complete readback. A clean target is therefore preferred, but a surviving stale volume is repaired rather than trusted.

11. Failback#

Move search reads off dep:primary-store-scan and back onto the index once the document count has converged and stopped changing. Not before: an index that is half-filled answers with a subset, and moving traffic back to it turns a slow search into a wrong one.

Take the load off the scan promptly. It is a supporting-tier dependency doing a core-tier job, and the longer it does it the more the estate's latency profile looks normal for the wrong reason.

12. What this runbook does not cover#

13. Machine-checkable statement#

json
{
  "id": "runbook:v1-search-index-loss",
  "scenarioId": "gd:search-index-loss",
  "version": "2",
  "detectionAuthority": {
    "role": "service on-call",
    "name": "workbench-platform on-call",
    "reachableOver": ["dep:notification-channel"]
  },
  "declarationAuthority": {
    "role": "operations duty officer",
    "name": "production operations duty officer",
    "reachableOver": ["dep:notification-channel"]
  },
  "communications": [
    {
      "audience": "actors",
      "channelDependencyId": "dep:notification-channel",
      "withinMs": 900000
    },
    {
      "audience": "operators",
      "channelDependencyId": "dep:notification-channel",
      "withinMs": 300000
    }
  ],
  "assumes": ["dep:primary-store", "dep:primary-store-scan"],
  "recoveryInfrastructureDependencyId": null,
  "secrets": [
    {
      "id": "search index connection credentials",
      "heldInDependencyId": "dep:search-index",
      "recoverableOutOfBand": true
    }
  ],
  "restoreOrder": ["dep:search-index"],
  "fencing": null,
  "promotesAStandby": false,
  "validation": [
    "the production reprojection command exits zero with authoritativeCount == projectedCount == verifiedCount",
    "collection dimensions, cosine distance and required payload indexes match the configured contract",
    "a known retained vector is queryable and a tombstoned subject is absent",
    "BFF boot repeats reconciliation before HTTP and refuses malformed metadata or mismatched readback"
  ],
  "failback": "move search reads off dep:primary-store-scan and back onto the index once the document count has converged and stopped changing"
}