Operational playbooks for Oshun V1 and V6 launch-critical systems. Each runbook
is owned by one team and lists its applicable services, the incident shape it
handles, and the rough severity that should trigger it. When paging on-call,
link directly to the relevant runbook below — don't ask the responder to hunt
through ls.
When to use this index#
- An alert fires and you don't know which runbook applies — search this table by service or incident type.
- You're writing a new runbook — add a row here in the same PR.
- You're auditing operational readiness — every domain should appear at least once.
Index#
| Runbook | Owner | Applies to | Incident type | Severity |
|---|---|---|---|---|
| assistant-failure.md | iris team | iris/api, lilith/svc-ai | Assistant returns empty / errors at p95 | SEV-2 |
| database-backups.md | platform | every Postgres cluster | Take or restore a logical backup | n/a — scheduled |
| grounding-failure.md | psyche / iris | iris/api, psyche/grounded-answer | Grounded-answer system produces uncited claims | SEV-1 |
| ios-toolchain.md | mobile | iris/mobile, arete/mobile, lilith/mobile, expo apps | Local or CI iOS build fails before producing an .app | n/a — setup |
| metis-disaster-recovery.md | UNASSIGNED | metis/service Postgres and metis-sources bucket | Restore the Metis estate: database, source objects, then reconcile | SEV-1 |
| model-workflow-rollback.md | isis / nous | isis/generation-api, nous/* providers | A model version produces regressions in eval pipeline | SEV-2 |
| moderation-surge.md | trust & safety | svc-moderation, content-* | Sudden moderation queue spike (>2x baseline) | SEV-2 |
| oshun-v1.md | platform | full stack | V1 launch / cutover playbook | SEV-1 |
| persona-rollback.md | iris team | iris/api persona service | Persona update breaks live conversations | SEV-2 |
| postgres-client-migration.md | platform | every service still using raw pg.Pool |
Migrate service to @oshun/database wrapper |
n/a — refactor |
| privacy-incident.md | trust & safety | full stack | Suspected PII leak, unauthorized data access, GDPR escalation | SEV-1 |
| provenance-failure.md | veritas | veritas/* services | Generated content provenance broken or missing | SEV-2 |
| provider-failover.md | platform | lilith/svc-ai, isis/generation-api | Upstream model provider (OpenAI / Anthropic / etc.) is degraded | SEV-2 |
| v1-acceptance-matrix.md | UNASSIGNED | V1 single-host stack | Not an incident — what a gd:regional-loss drill measured, per aspect |
n/a — rehearsal |
| v1-disaster-recovery.md | prod-ops rota | V1 single-host stack | gd:regional-loss — host lost: rebuild from the off-box snapshot |
SEV-1 |
| v1-dr-blob-store-loss.md | workbench-platform rota | V1 MinIO | gd:blob-store-loss — rows survive, the bytes they point at do not |
SEV-1 |
| v1-dr-bridge-host-loss.md | delivery-pipeline rota | V1 DCC bridge, enrolled hosts | gd:bridge-host-loss — the workstation the bridge runs on is gone |
SEV-3 |
| v1-dr-database-loss.md | workbench-platform rota | V1 PostgreSQL | gd:database-loss — the relational store is unavailable |
SEV-1 |
| v1-dr-model-provider-outage.md | workbench-platform rota | V1 model providers | gd:model-provider-outage — the primary provider stops answering |
SEV-2 |
| v1-dr-queue-loss.md | delivery-pipeline rota | V1 generation-job queue and workers | gd:queue-loss — the queue loses its contents |
SEV-2 |
| v1-dr-relational-corruption.md | workbench-platform rota | V1 PostgreSQL | gd:relational-corruption — the store answers with wrong rows |
SEV-1 |
| v1-dr-search-index-loss.md | workbench-platform rota | V1 Elasticsearch | gd:search-index-loss — the search index is lost |
SEV-2 |
| v1-dr-signing-key-compromise.md | delivery-pipeline rota | V1 C2PA signing, trust store | gd:signing-key-compromise — a signing key is disclosed |
SEV-1 |
| v1-dr-signing-service-outage.md | delivery-pipeline rota | V1 C2PA signer | gd:signing-service-outage — releases cannot be signed |
SEV-2 |
| v1-follow-up-register.md | UNASSIGNED | V1 single-host stack | Not an incident — what every game day found, who owns it, and by when | n/a — rehearsal |
| v1-game-day-plan.md | prod-ops rota | V1 single-host stack | Not an incident — the plan a rehearsal of each scenario class follows | n/a — rehearsal |
| v1-observability-game-day.md | prod-ops rota | V1 Hetzner stack and monitoring plane | Alert response and supervised detect→triage→rollback→recover rehearsal | SEV-1/2 |
| v1-reconciliation-and-replay.md | prod-ops rota | V1 jobs, leases, events, outcomes, gates, releases | After any restore: fold back the state that accrued while it was down | SEV-1 |
| shell-outage.md | platform | oshun-shell (the embedding shell) | Shell unavailable / cannot embed apps | SEV-1 |
| typecheck-cleanup.md | shared-infra | per-library typecheck | Per-lib tsc --noEmit fails; root CI is green |
n/a — debt |
| v6-agent-welfare-review.md | trust & safety | egbe operator console, lilith-agent-welfare | V6 agent welfare thresholds require Lilith review | SEV-1/2/3 |
| v6-capacity-management.md | platform | moirai cluster, pxstream relay, world server | V6 cognition / Pixel Streaming capacity pressure | SEV-1/2/3 |
| v6-commons-moderation.md | trust & safety | Commons Heart, realtime gateway, takedown executor | V6 Commons public-surface moderation or takedown review | SEV-1/2/3 |
| v6-incarnation-governance.md | aye bridge | aye bridge, ori service, threshold adapters | V6 Aye incarnation, return journal, or memory merge governance | SEV-1/2/3 |
| v6-production-setup.md | platform | V6 service stack, web/admin, QA handoff | Stand up a new V6 production-like environment | n/a — setup |
| v6-steward-conduct-investigation.md | trust & safety | egbe operator console, cognition audit log | V6 steward coercion / neglect / manipulation claim | SEV-1/2/3 |
| watermark-verification-failure.md | aphrodite | aphrodite/stream-watermarking | Watermark verification fails on viewer playback | SEV-3 |
Severity definitions#
- SEV-1 — User-visible outage or data loss; page on-call immediately, page the secondary at the 15-minute mark.
- SEV-2 — Significant degradation (≥10% of users); page on-call.
- SEV-3 — Internal degradation or non-blocking regression; ticket during business hours.
- n/a — Operational task that doesn't represent an active incident (setup, scheduled work, debt cleanup).
Adding a new runbook#
- Create
docs/runbooks/<short-name>.mdfollowing the structure of an existing runbook (header with owner + last-reviewed; "When to use"; "Step-by-step"; "Common failures"). - Add a row to the table above in the same PR.
- Cross-reference any new env vars or scripts in the runbook itself so future readers don't have to chase them down.
- If the runbook covers a SEV-1 incident, also add it to the
docs/operations/on-call.mdcheat-sheet (once that doc lands).