# Oshun V1 Runbooks

Per-incident runbooks for the V1 launch. Each runbook follows a standard
structure: scope, severity classification, detection signals, initial triage,
diagnosis, mitigation, communication cadence (using canonical incident templates
from `libs/oshun/analytics/src/incident-ownership-manifest.ts`), escalation,
recovery verification, and post-incident requirements.

| #   | Runbook                                                               | Owner                             | Severity-1 trigger summary                                                           |
| --- | --------------------------------------------------------------------- | --------------------------------- | ------------------------------------------------------------------------------------ |
| 1   | [Shell outage](./shell-outage.md)                                     | reliability                       | >5% of sessions cannot bootstrap, or admin shell unreachable in an incident          |
| 2   | [Grounding failure](./grounding-failure.md)                           | platform_engineering / sophia     | Retracted source served, or grounding below confidence floor with publish active     |
| 3   | [Assistant failure](./assistant-failure.md)                           | assistant_product                 | Streaming failure across sessions, disclosure compliance breach, memory cross-tenant |
| 4   | [Provider failover](./provider-failover.md)                           | reliability                       | Launch-critical provider has zero healthy capacity                                   |
| 5   | [Moderation surge](./moderation-surge.md)                             | trust_safety                      | Crisis-tagged review items unactioned for >1 h, regulator-SLA appeals breach         |
| 6   | [Privacy incident](./privacy-incident.md)                             | compliance + security             | Cross-tenant exposure, DSAR scope miss, special-category data exposed                |
| 7   | [Model or workflow rollback](./model-workflow-rollback.md)            | platform_engineering / model-ops  | Safety regression, cost run-away, or evaluation gating failure                       |
| 8   | [Persona rollback](./persona-rollback.md)                             | trust_safety                      | Identity confusion, disclosure failure on regulated surface, refusal bypass          |
| 9   | [Provenance failure](./provenance-failure.md)                         | platform_engineering / compliance | Customer-visible assets published with invalid or missing provenance                 |
| 10  | [Watermark verification failure](./watermark-verification-failure.md) | platform_engineering / security   | Encoder silently no-op, or forged payload accepted by verifier                       |

## Cross-cutting conventions

- On-call schedules and escalation steps are sourced from
  `libs/oshun/analytics/src/incident-ownership-manifest.ts`. Standard escalation
  is primary → secondary → engineering manager → incident commander → executive
  on-call at 5/5/15/30/60 minutes.
- Incident-communication templates are sourced from
  `OSHUN_V1_INCIDENT_TEMPLATES` in the same manifest. Do not improvise
  customer-facing copy; fill the canonical `bodyPlaceholders` and route via the
  documented channels.
- Alerts referenced by each runbook are defined in
  `libs/oshun/analytics/src/alerts-manifest.ts`. Each alert carries a `runbook`
  slug that points to the file in this directory.
- Dashboards referenced by each runbook live under
  `libs/oshun/analytics/src/dashboards-*.ts`. Service-health, queue-health,
  assistant, customer-KPI, satisfaction, and Nisaba dashboards each have a
  dedicated module.
- Privacy incidents always run an evidence-preservation step **before**
  mitigation. Mitigation that destroys evidence creates regulatory exposure.

## Authoring new runbooks

A new runbook is required when a new launch-critical subsystem is added, or when
a tabletop drill identifies a failure class not currently covered. A runbook is
"covered" only when:

1. A dedicated file exists in this directory with the full structure (scope,
   severity, detection, triage, diagnosis, mitigation, communication,
   escalation, recovery, post-incident).
2. The runbook is wired to at least one alert in `alerts-manifest.ts` via the
   alert's `runbook` slug.
3. A dashboard panel in `dashboards-*.ts` surfaces the failure's primary
   detection signal.
4. The on-call team owning the runbook has rehearsed it in a tabletop drill
   within the last 90 days.

Update this index whenever a runbook is added, retired, or has its owner
reassigned.