Eve library

Isis Creative Pipeline On-Call Playbook

Runbook

Revision 1

On this page

Last Updated: 2026-02-24
Scope: Creative generation pipeline incidents and SLO breaches across submit -> queue -> execute -> output
Primary Owners: @oshun/isis-team, @oshun/devops, @oshun/platform

Purpose#

Provide an operator-grade on-call playbook for incidents that degrade creative pipeline reliability, throughput, or delivery SLOs.

This playbook complements:

  • docs/domains/isis/runbooks/runpod-oncall.md (provider/runtime specialization)
  • docs/domains/isis/remediation/ISIS_SLO_DEFINITIONS.md (SLO contract)
  • docs/reference/runbooks/incident-response.md (cross-platform incident process)

Failure Classes#

| Class ID | Incident Class | Typical Signals | | -------- | --------------------------- | --------------------------------------------------------- | --------------------------- | | CP-001 | Service availability | /health or /ready failures, job submit outages | | CP-002 | Provider outage/degradation | providers.healthy=false, provider failure spikes | | CP-003 | Queue saturation/starvation | saturation/starvation indicator elevated | critical, stuck-job growth | | CP-004 | SLO breach | fast/slow burn alerts, canary SLO/cost gate failure | | CP-005 | Bad deploy regression | post-deploy failure/latency jump, rollback recommendation |

Severity Policy (Creative Pipeline)#

  • SEV-1
    • availability outage (CP-001) or
    • fast-burn breach on isis.execution-success-rate or isis.e2e-completion-time
  • SEV-2
    • provider degradation (CP-002),
    • queue saturation (CP-003), or
    • fast-burn breach on isis.submit-latency/isis.queue-delay
  • SEV-3
    • slow-burn on any pipeline SLO
  • SEV-4
    • no active incident signal; monitoring-only follow-up

First 15 Minutes#

1) Run Triage Automation#

bash
ISIS_API_BASE_URL="https://<isis-api-host>" \
ISIS_ADMIN_TOKEN="..." \
ISIS_DATABASE_URL="postgres://..." \
STRICT=true \
scripts/isis/run_creative_pipeline_oncall_triage.sh

Outputs:

  • tmp/isis-oncall/creative-pipeline-oncall-triage-<timestamp>.json
  • tmp/isis-oncall/creative-pipeline-oncall-triage-<timestamp>.md

2) Confirm Incident Class and Severity#

Use triage report fields:

  • classification.severity
  • classification.classes[]
  • checks failures for health/ready/providers/queue/SLO status

3) Open Incident Comms#

Post in incident channel with:

  • incident class (CP-*)
  • current severity
  • current mitigation action
  • next update ETA (15 minutes)

Playbook by Incident Class#

CP-001 Service Availability#

  1. Validate /health and /ready from triage artifacts.
  2. Roll back latest deployment/config change if failure began post-release.
  3. Confirm submit path recovery (POST /api/v1/jobs success trend).
  4. Keep canary rollout frozen until stability is sustained.

CP-002 Provider Outage/Degradation#

  1. Run provider outage drill playbook procedures:
    • docs/domains/isis/runbooks/provider-outage-drill.md
  2. Validate provider operations:
    • /api/v1/jobs/providers/operations
  3. If needed, force fallback routing and hold ramp progression.
  4. Verify synthetic jobs progress in non-failed states.

CP-003 Queue Saturation/Starvation#

  1. Inspect queue indicators from triage:
    • .queue.indicators.saturation
    • .queue.indicators.starvation
  2. Pause canary promotion and reduce canary percentage when pressure is rising.
  3. Reconcile endpoint drift and check worker capacity allocation.
  4. Replay dead-letter only after root cause is contained.

CP-004 SLO Breach#

  1. Run canary verifier:
bash
ISIS_DATABASE_URL="postgres://..." \
FAIL_ON_BREACH=false \
scripts/isis/verify_canary_slo_cost.sh
  1. Map failed checks to SLO IDs:
    • submit latency -> isis.submit-latency
    • queue delay -> isis.queue-delay
    • execution success -> isis.execution-success-rate
    • e2e completion -> isis.e2e-completion-time
  2. Apply burn-rate severity mapping from docs/domains/isis/remediation/ISIS_SLO_DEFINITIONS.md.
  3. Trigger rollback path if fast-burn criteria are met.

CP-005 Bad Deploy Regression#

  1. Generate rollback decision artifact:
bash
CURRENT_PERCENTAGE=25 \
AUTO_ROLLBACK_ENABLED=true \
FAIL_ON_ROLLBACK=false \
scripts/isis/advance_queue_canary_rollout.sh
  1. If action is rollback, set target rollout to 0% and execute rollback procedure.
  2. Validate post-rollback stabilization before any re-promotion.

SLO Breach Escalation Matrix#

Breach Type Immediate Owner Escalate To
Submit/queue latency fast burn @oshun/devops @oshun/platform
Execution/e2e fast burn @oshun/isis-team @oshun/platform, @oshun/ml-team
Slow-burn persistent drift @oshun/isis-team @oshun/core-team
Cost + SLO coupled degradation @oshun/devops @oshun/platform, @oshun/ml-team

Exit Criteria#

Incident may close only when:

  1. Required triage checks are passing or approved exceptions are documented.
  2. SLO breach condition has stopped burning at incident threshold.
  3. Queue/provider health has stabilized for an agreed observation window.
  4. Follow-up action items are filed with owners and due dates.
  5. Post-incident review is created for SEV-1/SEV-2.
  • scripts/isis/run_creative_pipeline_oncall_triage.sh
  • docs/domains/isis/runbooks/runpod-oncall.md
  • docs/domains/isis/remediation/ISIS_SLO_DEFINITIONS.md
  • docs/domains/isis/remediation/ISIS_CANARY_SLO_COST_VERIFICATION.md
  • docs/domains/isis/remediation/ISIS_POST_INCIDENT_REVIEW_TEMPLATE.md
  • docs/reference/runbooks/incident-response.md