Disciplines · Runbooks

Oshun V1 Runbooks

is primary → secondary → engineering manager → incident commander → executive on-call at 5/5/15/30/60 minutes.

2sections2 minread
Markdown source
On this page

Per-incident runbooks for the V1 launch. Each runbook follows a standard structure: scope, severity classification, detection signals, initial triage, diagnosis, mitigation, communication cadence (using canonical incident templates from libs/oshun/analytics/src/incident-ownership-manifest.ts), escalation, recovery verification, and post-incident requirements.

# Runbook Owner Severity-1 trigger summary
1 Shell outage reliability >5% of sessions cannot bootstrap, or admin shell unreachable in an incident
2 Grounding failure platform_engineering / sophia Retracted source served, or grounding below confidence floor with publish active
3 Assistant failure assistant_product Streaming failure across sessions, disclosure compliance breach, memory cross-tenant
4 Provider failover reliability Launch-critical provider has zero healthy capacity
5 Moderation surge trust_safety Crisis-tagged review items unactioned for >1 h, regulator-SLA appeals breach
6 Privacy incident compliance + security Cross-tenant exposure, DSAR scope miss, special-category data exposed
7 Model or workflow rollback platform_engineering / model-ops Safety regression, cost run-away, or evaluation gating failure
8 Persona rollback trust_safety Identity confusion, disclosure failure on regulated surface, refusal bypass
9 Provenance failure platform_engineering / compliance Customer-visible assets published with invalid or missing provenance
10 Watermark verification failure platform_engineering / security Encoder silently no-op, or forged payload accepted by verifier

Cross-cutting conventions#

  • On-call schedules and escalation steps are sourced from libs/oshun/analytics/src/incident-ownership-manifest.ts. Standard escalation is primary → secondary → engineering manager → incident commander → executive on-call at 5/5/15/30/60 minutes.
  • Incident-communication templates are sourced from OSHUN_V1_INCIDENT_TEMPLATES in the same manifest. Do not improvise customer-facing copy; fill the canonical bodyPlaceholders and route via the documented channels.
  • Alerts referenced by each runbook are defined in libs/oshun/analytics/src/alerts-manifest.ts. Each alert carries a runbook slug that points to the file in this directory.
  • Dashboards referenced by each runbook live under libs/oshun/analytics/src/dashboards-*.ts. Service-health, queue-health, assistant, customer-KPI, satisfaction, and Nisaba dashboards each have a dedicated module.
  • Privacy incidents always run an evidence-preservation step before mitigation. Mitigation that destroys evidence creates regulatory exposure.

Authoring new runbooks#

A new runbook is required when a new launch-critical subsystem is added, or when a tabletop drill identifies a failure class not currently covered. A runbook is "covered" only when:

  1. A dedicated file exists in this directory with the full structure (scope, severity, detection, triage, diagnosis, mitigation, communication, escalation, recovery, post-incident).
  2. The runbook is wired to at least one alert in alerts-manifest.ts via the alert's runbook slug.
  3. A dashboard panel in dashboards-*.ts surfaces the failure's primary detection signal.
  4. The on-call team owning the runbook has rehearsed it in a tabletop drill within the last 90 days.

Update this index whenever a runbook is added, retired, or has its owner reassigned.