# Runbook — Provider Failover

> A configured AI, search, payment, messaging, or media provider is unhealthy
> and the circuit breaker must trip. Covers the failover decision, the failover
> execution, the customer-visible degradation, and the audit trail.

## Scope

Failover applies to:

- LLM providers serving assistant/Sophia/Metis (multiple model families per
  domain, primary + alternate routes).
- Embedding/retrieval providers serving Sophia.
- Speech and avatar providers serving voice mode and avatar mode.
- Image and video generation providers serving Isis.
- Messaging providers (Telegram, WhatsApp, email/SMS) per
  `docs/oshun/messaging-boundary.md`.
- Payment providers per the billing adapter.

This runbook handles the failover. The deeper subsystem mitigation lives in the
subsystem runbooks: `assistant-failure.md`, `grounding-failure.md`,
`model-workflow-rollback.md`, etc.

## Severity classification

| Sev  | Condition                                                                                                                             |
| ---- | ------------------------------------------------------------------------------------------------------------------------------------- |
| sev1 | A provider tied to a launch-critical journey (assistant, grounded answers, payments, login SMS) has zero healthy capacity for >5 min. |
| sev2 | Provider error ratio > 5% sustained for >10 min OR p95 latency > 2× SLO for >10 min while the alternate has spare capacity.           |
| sev3 | Provider degradation absorbed by the alternate; no customer impact; observation only.                                                 |

Source events: `provider.health.degraded`, `provider.failover.triggered`,
`provider.failover.recovered` from `release-taxonomy.ts`; alert
`assistant-provider-outage` from `alerts-manifest.ts`.

## Detection signals

- Alerts: `assistant-provider-outage`, `generation-pipeline-failover`,
  `model-health-degraded`.
- Dashboards: provider-health panels on `dashboards-service-health.ts`; the
  customer-KPI conversion-success panel for payments.
- Customer signals: spike in `assistant.session.error_shown`,
  `media.generation.failed`, `payment.charge_failed`, `auth.sms_code_failed`.
- Partner/status: provider's own status page (page the reliability lead to ack
  the external status before tripping).

## Failover decision

1. Confirm the provider is the root cause:
   - Error/latency localized to one provider route.
   - Alternate route healthy (latency, success ratio, capacity).
   - Provider's status page corroborates (where available).
2. Confirm capacity for the alternate:
   - Read `dashboards-service-health.ts` capacity headroom for the alternate.
   - If headroom < forecasted shifted load × 1.5, throttle inbound first
     (assistant queue-depth, generation queue priority) before failover.
3. Decide route policy:
   - **Full failover**: 100% traffic moves to alternate.
   - **Partial failover**: cohort or workload-class moves (e.g., grounded
     answers move while non-grounded stays on primary).
   - **Workload pin**: a specific job class pins to alternate while primary
     stays for everything else.

## Failover execution

1. Trip the circuit breaker via the `release-taxonomy.ts` provider hook — the
   hook emits `provider.failover.triggered` to admin audit and moves the active
   route to the alternate per the chosen policy.
2. Verify by sampling 20 in-flight sessions/jobs that they land on the alternate
   route — trace span `provider.route` attribute must reflect the alternate's
   identifier.
3. If the affected provider is a payment provider, follow the billing bridge's
   idempotency-key contract: in-flight charge attempts must be re-keyed against
   the alternate's idempotency space before retry.
4. If the affected provider is a messaging provider, suppress any non-essential
   queued sends across the affected channel until the alternate's send-rate and
   template-compliance baseline is verified.

## Communication cadence

| Phase         | Template ID                                                  | Audience    | Cadence          |
| ------------- | ------------------------------------------------------------ | ----------- | ---------------- |
| Investigating | `incident.customer.status_page.investigating`                | status_page | within 5 min     |
| Internal      | `incident.internal.slack.investigating`                      | internal    | within 2 min     |
| Identified    | `incident.customer.status_page.identified`                   | status_page | within 30 min    |
| Partner       | `incident.partner.email.identified`                          | partner     | within 60 min    |
| Resolved      | `incident.customer.status_page.resolved` + `.email.resolved` | both        | within 60 min    |
| Postmortem    | `incident.customer.status_page.postmortem`                   | status_page | within 5 biz day |

For payment-provider failover, the partner notification is mandatory and must
include the idempotency-key strategy and the cohort estimate.

## Workload-specific guidance

- **Assistant LLM**: failover changes the per-turn cost and may change
  refusal-rate baselines. Pin Lilith policy to the certified joint baseline for
  the alternate before sustained customer traffic; otherwise refusal regressions
  will trigger `persona-policy-violation` alerts.
- **Sophia retrieval/embedding**: re-index drift between providers means saved
  citations may not round-trip — verify a 20-sample retrieval drill before
  declaring the failover complete.
- **Voice/avatar**: alternate provider may have different voice IDs and avatar
  lineages. The assistant must surface the `assistant.voice.identity_changed`
  and `assistant.avatar.identity_changed` disclosure events so the customer
  perceives the change.
- **Generation (Isis)**: model lineage changes mean watermark and provenance
  hashes shift. Use the provenance runbook (`provenance-failure.md`) if
  signatures fail post-failover.
- **Payments**: do not re-issue refunds during a failover — pause refund
  reconciliation until the alternate completes a settlement window.
- **Messaging**: WhatsApp template approvals, A2P 10DLC registrations, and email
  DMARC reputation do not transfer. Verify each compliance gate before resuming
  send.

## Escalation

Standard escalation. For sev1 payments or login-SMS failover, add the compliance
lead in parallel with reliability.

## Recovery verification

- Circuit breaker closed and `provider.failover.recovered` emitted.
- Primary provider error ratio < SLO for 60 min.
- Alternate-provider load returns to baseline (no carry-over throttle).
- Workload-specific verifications: assistant refusal rate within ±1 sigma;
  Sophia retrieval round-trip OK; Isis provenance signatures valid; payment
  idempotency reconciliation complete; messaging compliance re-baselined.

## Post-incident

- Postmortem must include: provider route timeline (open → trip → recover),
  capacity used on alternate, customer-impact estimate per workload class, and
  any compliance/regulatory notices triggered.
- Update `dashboards-service-health.ts` capacity-forecast inputs if the
  alternate hit > 80% utilization during failover.
- File a cost-and-policy review if the alternate's pricing or refusal baseline
  shifted production economics.