Last Updated: 2026-02-24
Scope: Creative generation pipeline incidents and SLO breaches across
submit -> queue -> execute -> output
Primary Owners: @oshun/isis-team, @oshun/devops, @oshun/platform
Purpose#
Provide an operator-grade on-call playbook for incidents that degrade creative pipeline reliability, throughput, or delivery SLOs.
This playbook complements:
docs/domains/isis/runbooks/runpod-oncall.md(provider/runtime specialization)docs/domains/isis/remediation/ISIS_SLO_DEFINITIONS.md(SLO contract)docs/reference/runbooks/incident-response.md(cross-platform incident process)
Failure Classes#
| Class ID | Incident Class | Typical Signals |
| -------- | --------------------------- | --------------------------------------------------------- | --------------------------- |
| CP-001 | Service availability | /health or /ready failures, job submit outages |
| CP-002 | Provider outage/degradation | providers.healthy=false, provider failure spikes |
| CP-003 | Queue saturation/starvation | saturation/starvation indicator elevated | critical, stuck-job growth |
| CP-004 | SLO breach | fast/slow burn alerts, canary SLO/cost gate failure |
| CP-005 | Bad deploy regression | post-deploy failure/latency jump, rollback recommendation |
Severity Policy (Creative Pipeline)#
SEV-1- availability outage (
CP-001) or - fast-burn breach on
isis.execution-success-rateorisis.e2e-completion-time
- availability outage (
SEV-2- provider degradation (
CP-002), - queue saturation (
CP-003), or - fast-burn breach on
isis.submit-latency/isis.queue-delay
- provider degradation (
SEV-3- slow-burn on any pipeline SLO
SEV-4- no active incident signal; monitoring-only follow-up
First 15 Minutes#
1) Run Triage Automation#
ISIS_API_BASE_URL="https://<isis-api-host>" \
ISIS_ADMIN_TOKEN="..." \
ISIS_DATABASE_URL="postgres://..." \
STRICT=true \
scripts/isis/run_creative_pipeline_oncall_triage.sh
Outputs:
tmp/isis-oncall/creative-pipeline-oncall-triage-<timestamp>.jsontmp/isis-oncall/creative-pipeline-oncall-triage-<timestamp>.md
2) Confirm Incident Class and Severity#
Use triage report fields:
classification.severityclassification.classes[]checksfailures for health/ready/providers/queue/SLO status
3) Open Incident Comms#
Post in incident channel with:
- incident class (
CP-*) - current severity
- current mitigation action
- next update ETA (15 minutes)
Playbook by Incident Class#
CP-001 Service Availability#
- Validate
/healthand/readyfrom triage artifacts. - Roll back latest deployment/config change if failure began post-release.
- Confirm submit path recovery (
POST /api/v1/jobssuccess trend). - Keep canary rollout frozen until stability is sustained.
CP-002 Provider Outage/Degradation#
- Run provider outage drill playbook procedures:
docs/domains/isis/runbooks/provider-outage-drill.md
- Validate provider operations:
/api/v1/jobs/providers/operations
- If needed, force fallback routing and hold ramp progression.
- Verify synthetic jobs progress in non-failed states.
CP-003 Queue Saturation/Starvation#
- Inspect queue indicators from triage:
.queue.indicators.saturation.queue.indicators.starvation
- Pause canary promotion and reduce canary percentage when pressure is rising.
- Reconcile endpoint drift and check worker capacity allocation.
- Replay dead-letter only after root cause is contained.
CP-004 SLO Breach#
- Run canary verifier:
ISIS_DATABASE_URL="postgres://..." \
FAIL_ON_BREACH=false \
scripts/isis/verify_canary_slo_cost.sh
- Map failed checks to SLO IDs:
- submit latency ->
isis.submit-latency - queue delay ->
isis.queue-delay - execution success ->
isis.execution-success-rate - e2e completion ->
isis.e2e-completion-time
- submit latency ->
- Apply burn-rate severity mapping from
docs/domains/isis/remediation/ISIS_SLO_DEFINITIONS.md. - Trigger rollback path if fast-burn criteria are met.
CP-005 Bad Deploy Regression#
- Generate rollback decision artifact:
CURRENT_PERCENTAGE=25 \
AUTO_ROLLBACK_ENABLED=true \
FAIL_ON_ROLLBACK=false \
scripts/isis/advance_queue_canary_rollout.sh
- If action is
rollback, set target rollout to0%and execute rollback procedure. - Validate post-rollback stabilization before any re-promotion.
SLO Breach Escalation Matrix#
| Breach Type | Immediate Owner | Escalate To |
|---|---|---|
| Submit/queue latency fast burn | @oshun/devops |
@oshun/platform |
| Execution/e2e fast burn | @oshun/isis-team |
@oshun/platform, @oshun/ml-team |
| Slow-burn persistent drift | @oshun/isis-team |
@oshun/core-team |
| Cost + SLO coupled degradation | @oshun/devops |
@oshun/platform, @oshun/ml-team |
Exit Criteria#
Incident may close only when:
- Required triage checks are passing or approved exceptions are documented.
- SLO breach condition has stopped burning at incident threshold.
- Queue/provider health has stabilized for an agreed observation window.
- Follow-up action items are filed with owners and due dates.
- Post-incident review is created for
SEV-1/SEV-2.
Related Artifacts#
scripts/isis/run_creative_pipeline_oncall_triage.shdocs/domains/isis/runbooks/runpod-oncall.mddocs/domains/isis/remediation/ISIS_SLO_DEFINITIONS.mddocs/domains/isis/remediation/ISIS_CANARY_SLO_COST_VERIFICATION.mddocs/domains/isis/remediation/ISIS_POST_INCIDENT_REVIEW_TEMPLATE.mddocs/reference/runbooks/incident-response.md