Runbook — Model or Workflow Rollback

Roll back a recently promoted model version, retriever version, evaluator version, or Isis generation workflow that is causing regression in quality, safety, latency, cost, or compliance.

Scope#

Applies when a promotion via release-taxonomy.ts (event release.model.promoted or release.workflow.promoted) needs to revert to the previous certified version. Targets include:

• Assistant LLM versions per persona.
• Sophia retriever, reranker, and grounding-evaluator versions.
• Metis tutor evaluator and assessment evaluator.
• Lilith policy/refusal versions (their joint baselines with the assistant LLM).
• Isis generation workflows (image/video/audio/3D).
• Veritas claim/source/counterclaim evaluator.

This runbook handles the rollback. The persona-version rollback uses persona-rollback.md. Provider-route failover uses provider-failover.md.

When to roll back#

A rollback fires when any one of:

• An evaluation benchmark in evaluation-manifest.ts regresses past its regression-blocking threshold.
• An alert from alerts-manifest.ts tied to the promoted version fires (model-health-degraded, model-rollback, citation-flag-rate, persona-policy-violation).
• The release-captain post-promotion review identifies a customer-visible issue.
• A T&S escalation identifies a safety regression tied to the version.
• A cost or latency budget overshoot exceeds the per-version SLO.

Rollback is the default mitigation. Stay rolled back unless a forward fix clears the original blocker.

Severity classification#

Detection signals#

• Alerts: model-health-degraded, model-rollback, citation-flag-rate, persona-policy-violation, generation-pipeline-failover, grounding-low-confidence-spike.
• Dashboards: dashboards-assistant.ts quality + compliance; dashboards-service-health.ts per-target panels; dashboards-customer-kpi.ts conversion and satisfaction panels.
• Audit signals: admin.model.promoted, admin.model.evaluation_failed, admin.workflow.promoted per admin-taxonomy.ts.

Rollback procedure#

1. Confirm the regression:
   • Pull the current evaluation-manifest run for the affected target.
   • Compare against the pre-promotion baseline run.
   • Confirm the regressed metric is past the regression-blocking threshold from evaluation-manifest.ts.
   • Save the comparison artifact to the audit store under the incident ID — the postmortem will reference it.
2. Identify the previous certified version: read the release-taxonomy.ts release history for the target — the prior release.model.certified record is the rollback destination. If no prior certified record exists (first release of the target), rollback is not possible — fall back to the kill-switch path (mitigation step 4).
3. Invalidate the promoted cache: any cached answers, embeddings, reranks, or generation outputs tagged with the regressed version must be invalidated before the rollback closes — otherwise customers continue to see the bad version's output. The cache-invalidation hook emits release.cache.invalidated with the version tag.
4. Pin the previous version: call the release-taxonomy rollback hook; the hook emits release.model.rolled_back (or .workflow.rolled_back) with the source and destination version, the operator, and the reason code. Customer surfaces and assistant routes pick up the rollback on their next request.
5. Cohort verification: sample 20 in-flight customer sessions or jobs that hit the rolled-back target. Verify the response carries the prior version's metadata (model id, retriever id, workflow id).
6. Re-run the gating evaluation post-rollback to confirm the metric recovered to baseline; the run is added to the audit store.

Kill-switch path (when no prior version exists)#

If the target is a first-time release with no prior certified version:

1. Disable the target entirely via the target's kill-switch flag in release-taxonomy.
2. Customer surfaces fall back to the documented degraded mode (assistant grounded-mode disabled, Isis generation queue paused, etc.).
3. The on-call team for the target follows the matching domain runbook to manage the customer-visible degraded mode.

Communication cadence#

For safety-related rollbacks (refusal bypass, unsafe-pass), the customer notification must specifically state the safety scope and the recertified version. T&S co-authors the copy.

Escalation#

Standard 5/5/15/30/60 escalation. For sev1 safety rollbacks, page T&S lead and the model-ops lead at minute 0.

Recovery verification#

• Evaluation gating metrics within thresholds for the rolled-back version.
• All caches invalidated for the regressed version (no stale serves for 60 min as confirmed by trace-attribute sampling).
• Customer-KPI panels (dashboards-customer-kpi.ts) recovered to baseline for 30 min.
• No new alerts of the same class for 30 min.
• Audit pipeline records the rollback and the recertified version.

Post-incident#

• Postmortem must include: the regression metric and its threshold, the promotion record that introduced it, the rollback record, customer-impact estimate, cost/latency overshoot if any, and the remediation owner for the forward fix.
• File a promotion-process improvement ticket if the regression slipped past the evaluation pipeline — either the threshold was too loose or the benchmark missed the regressed dimension.
• If the regression was safety-class, add the new failure to the safety benchmark fixture so the next promotion catches it.
• The forward fix follows the standard model-promotion gate; do not shortcut promotion to "re-fix" a previously regressed version.