# Runbook — Grounding Failure

> Sophia grounded-answer pipeline produces low-confidence, missing-citation,
> stale-evidence, or hallucinated output above SLO. Use when grounding — not the
> upstream assistant model — is the failure boundary.

## Scope

This runbook covers degradation of the grounded-answer path that powers Veritas
evidence rails, Tara reflection citations, Nisaba passage references, Metis
tutor source binding, and the assistant grounded mode. Failure modes:

- Citation flag rate spike (a returned citation does not match the bound
  source).
- Grounding confidence below the `min_confidence` floor for >5% of grounded
  responses.
- Stale-evidence index (vector index lag > 15 min behind the source-of-truth
  store).
- Sophia adapter timeout, rate limit, or 5xx > 1%.
- Retracted/corrected source still served in a published answer.

If the assistant model itself is failing, use `assistant-failure.md`. If a
specific provider is the failure boundary, use `provider-failover.md`.

## Severity classification

| Sev  | Condition                                                                                                              |
| ---- | ---------------------------------------------------------------------------------------------------------------------- |
| sev1 | A retracted or corrected source is being served in customer-visible answers, OR cross-tenant evidence leak suspected.  |
| sev1 | Grounded answers published with `confidence < min_confidence` for >5 min and customer surfaces continue to show them.  |
| sev2 | Citation flag rate above the alert threshold (`citation-flag-rate` alert), grounding fallback rate above 5% sustained. |
| sev3 | Sophia adapter p99 latency above SLO without published-answer impact, single index partition lag.                      |

Thresholds come from the `evaluation-manifest.ts` benchmark scope
`grounded_answers` (exact-match 0.7+, stance accuracy 0.85+) and from the
`grounding-low-confidence-spike` and `citation-flag-rate` alerts in
`alerts-manifest.ts`.

## Detection signals

- Alerts: `grounding-low-confidence-spike`, `citation-flag-rate`,
  `grounding-fallback` from `libs/oshun/analytics/src/alerts-manifest.ts`.
- Dashboards: `dashboards-service-health.ts` Sophia panel; the assistant
  evidence-quality panel from `dashboards-assistant.ts`.
- Customer signals: `evidence.confidence.below_threshold` and
  `evidence.citation.mismatch_reported` events from `customer-taxonomy.ts`.
- Admin signals: a moderation reviewer flags `review.evidence.disputed`.

## Initial triage (first 5 minutes)

1. Acknowledge `oshun-sophia-oncall`.
2. Determine failure boundary:
   - Confidence drop alone → upstream model or retrieval drift.
   - Citation mismatch alone → index or chunker drift.
   - Both → end-to-end pipeline outage (escalate immediately).
3. Read the active Sophia release row in `release-taxonomy.ts`. If a model or
   retriever was promoted in the last 60 min, that is the prime suspect.
4. If retracted-source-served is suspected, **immediately** disable grounded
   publication (mitigation step 1) before investigating further.

## Diagnosis

1. Pull the last hour of `evidence.citation.mismatch_reported` events, grouped
   by source ID. A clustered set of mismatches against the same source bundle
   implicates an index/chunker issue against that bundle.
2. Inspect the Sophia retriever trace
   (`sophia.retriever.query`/`sophia.retriever.rerank`) for p95 latency and the
   per-stage hit rate. A collapsed rerank score distribution implicates the
   reranker.
3. Compare current `evaluation-manifest.ts` `grounded_answers` benchmark scores
   against the last release row in `release-taxonomy.ts`. A score drop below the
   regression threshold blocks the active version and reverts to the previous
   certified retriever.
4. Compare vector-index lag (`dependency_search_index` health target) against
   the source-of-truth store. >15 min lag explains stale evidence.
5. Pull a sample of 20 grounded responses where confidence dropped. Inspect the
   bound chunks against the source-of-truth excerpts. If the chunks no longer
   match the published source text, the chunker or storage has drifted.

## Mitigation

1. **Stop the bleed**: disable grounded publication via the Sophia
   `publication_gate` kill switch. Customer surfaces fall back to the
   source-inspection-only view; assistant grounded mode degrades to
   non-streaming text answers with explicit "evidence unavailable" disclosure
   from `customer-taxonomy.ts` event `evidence.unavailable_shown`.
2. **Revert the model/retriever**: if release-taxonomy implicates a recent
   promotion, run the rollback hook for that target. The `model-rollback` alert
   runbook is the canonical hook; this runbook delegates to it for the actual
   rollback command.
3. **Force a re-index** for the affected bundle: enqueue
   `sophia.reindex.requested` for the source IDs implicated by the mismatch
   cluster; observe `sophia.reindex.completed` and the recovery of citation
   match rate.
4. **Suppress retracted sources**: if the cause is a retraction cascade, verify
   the retraction event was published to the evidence store. The
   `retraction-cascade` validator in Sophia rejects republication of retracted
   source IDs.
5. **Re-grounding queue**: enqueue customer-visible answers from the impact
   window for re-grounding. The publisher pauses customer-visible republish
   until each answer is recertified or marked `evidence_unavailable`.

## Communication cadence

| Phase         | Template ID                                                  | Audience    | Cadence          |
| ------------- | ------------------------------------------------------------ | ----------- | ---------------- |
| Investigating | `incident.customer.status_page.investigating`                | status_page | within 5 min     |
| Internal      | `incident.internal.slack.investigating`                      | internal    | within 2 min     |
| Identified    | `incident.customer.status_page.identified`                   | status_page | within 30 min    |
| Mitigating    | `incident.customer.status_page.mitigating`                   | status_page | every 30 min     |
| Resolved      | `incident.customer.status_page.resolved` + `.email.resolved` | both        | within 60 min    |
| Postmortem    | `incident.customer.status_page.postmortem`                   | status_page | within 5 biz day |

If a retracted source was served, the customer-email resolved template must
identify the affected content scope (no individual user data) and link the
Sophia provenance bundle for the corrected answer.

## Escalation

Standard escalation from `incident-ownership-manifest.ts`. For sev1 (retracted
source served, or confidence floor breached for >5 min), pre-emptively page the
trust-and-safety lead in parallel with reliability — the customer-visible
correction sits with T&S.

## Recovery verification

- Citation flag rate < 0.5% for 15 min.
- Grounding fallback rate < 1% for 15 min.
- `grounded_answers` benchmark passes (`evaluation-manifest.ts` thresholds).
- All re-grounded customer-visible answers either recertified or marked
  evidence-unavailable.
- No new `evidence.citation.mismatch_reported` clusters for 30 min.

## Post-incident

- Postmortem must include: regression sample IDs (anonymized), retraction
  cascade audit, re-grounding queue length and time-to-clear, the model or
  retriever version that regressed, and the recertified version.
- Update `evaluation-manifest.ts` thresholds if the incident revealed a gap in
  the benchmark coverage.
- If the cause was a missed retraction, add the source to the retraction-set
  fixture in the evaluation pipeline so future promotions catch it.