# V3 GA Incident Response Runbooks

runbook-set: `v3-ga-incident-response.v1`

verifier: `pnpm verify:v3 operator-runbooks-on-call`

> **Evidence caveat (added 2026-06-12).** The six drill records below are
> self-attested: each carries only a date and a bare evidence ID, with no
> transcript, timing log, or participant roster behind it. Until each drill is
> rerun and captured with real artifacts (run log with timestamps, named
> participants, observed time-to-acknowledge / time-to-mitigate against the
> SLAs), treat these as tabletop/structural drills — the runbook content is
> real, but "Drill: green" is not operational proof. Incident-commander
> identifiers of the `ic-na-01` style used elsewhere in the operations evidence
> denote on-call roles, not staffed, named humans.

Every launch incident below has a documented procedure, a named incident
commander role, and writes an audit event into the Lilith Operator Console.

## incident: room-overload

- Owner: Lilith-Safety incident commander
- Severity: SEV-2, SEV-1 if paid event admission is blocked
- Trigger: world-server room admission p95 exceeds 2 s or room shard CPU exceeds
  85% for 5 minutes.
- First action SLA: acknowledge within 2 minutes; cap new walk-ups within 5
  minutes.
- Steps: freeze room mutations, enable queue banner, split overflow into
  fallback web, assign operator host, and publish incident note.
- Rollback: reopen walk-ups only after room admission p95 stays below 1 s for 15
  minutes.
- Escalation: Capacity / Pixel Streaming on-call, then world-server lead.
- Audit event: `operator.incident.room_overload.drilled`
- Drill: green, executed 2026-05-20, evidence `drill-room-overload-ga`

## incident: generation-provider-outage

- Owner: Isis generation incident commander
- Severity: SEV-2, SEV-1 if scheduled concert publication is blocked
- Trigger: primary music, motion, or image provider error rate exceeds 10% for 5
  minutes or queue p95 age exceeds 30 minutes.
- First action SLA: acknowledge within 3 minutes; fail over provider lane within
  10 minutes.
- Steps: pause non-paid generation, switch approved jobs to backup provider,
  preserve provenance bundle, and notify editorial owners.
- Rollback: restore primary provider only after 30 minutes of green canary jobs.
- Escalation: Isis platform lead, Rights lead for provenance exceptions.
- Audit event: `operator.incident.generation_provider_outage.drilled`
- Drill: green, executed 2026-05-20, evidence `drill-generation-provider-ga`

## incident: voice-sfu-degradation

- Owner: Spatial audio incident commander
- Severity: SEV-2, SEV-1 if instructors or performers cannot be heard
- Trigger: SFU packet loss exceeds 3%, jitter exceeds 60 ms, or voice join
  success drops below 99%.
- First action SLA: acknowledge within 2 minutes; route affected rooms within 5
  minutes.
- Steps: move rooms to alternate SFU region, mute degraded uplinks, enable text
  fallback banner, and hand instructor host controls to operator.
- Rollback: return rooms to primary SFU after 20 minutes of packet loss below
  1%.
- Escalation: Voice platform on-call, Capacity engineer.
- Audit event: `operator.incident.voice_sfu_degradation.drilled`
- Drill: green, executed 2026-05-21, evidence `drill-voice-sfu-ga`

## incident: royalty-settlement-failure

- Owner: Lilith-Commerce incident commander
- Severity: SEV-3, SEV-2 if artist payouts or refunds are delayed
- Trigger: settlement job failure, payment ledger imbalance, or rights waterfall
  queue p95 exceeds 4 hours.
- First action SLA: acknowledge within 10 minutes; freeze affected settlement
  batch within 15 minutes.
- Steps: stop payout batch, reconcile signed receipts, notify Rights and
  Commerce pools, and publish artist-facing status note.
- Rollback: release settlement only after dual-control ledger reconciliation.
- Escalation: Finance operations, Rights lead, payment-provider TAM.
- Audit event: `operator.incident.royalty_settlement_failure.drilled`
- Drill: green, executed 2026-05-21, evidence `drill-royalty-settlement-ga`

## incident: pixel-streaming-pop-outage

- Owner: Capacity / Pixel Streaming incident commander
- Severity: SEV-2, SEV-1 during scheduled concert window
- Trigger: POP session-start success below 99%, GPU quota exhaustion, or
  provider health check failure.
- First action SLA: acknowledge within 2 minutes; fail over new sessions within
  5 minutes.
- Steps: drain affected POP, preserve paid-attendee priority, redirect free-tier
  traffic to fallback web, and open secondary-cloud capacity.
- Rollback: shift traffic back only after 30 minutes of green synthetic session
  starts.
- Escalation: Capacity engineering, cloud TAM, festival commander.
- Audit event: `operator.incident.pixel_streaming_pop_outage.drilled`
- Drill: green, executed 2026-05-21, evidence `drill-pxstream-pop-ga`

## incident: master-capture-worker-failure-mid-concert

- Owner: Concert capture incident commander
- Severity: SEV-1 for public concert, SEV-2 for rehearsal
- Trigger: master capture worker heartbeat missing for 15 seconds, segment
  checksum mismatch, or C2PA signing failure.
- First action SLA: acknowledge within 1 minute; promote hot standby worker
  within 3 minutes.
- Steps: mark failed worker read-only, promote standby, replay last signed
  segment, verify consent and privacy ledgers, and notify editorial review gate.
- Rollback: keep standby primary until post-show integrity review closes.
- Escalation: Concert engineering lead, Rights lead, editorial producer.
- Audit event: `operator.incident.master_capture_worker_failure.drilled`
- Drill: green, executed 2026-05-22, evidence `drill-master-capture-ga`