Date: 2026-02-22
Checklist Item: REL-005
Ticket: ISIS-0179
Objective#
Execute and document pre-cutover game-day exercises that validate Isis operational readiness under failure and rollback scenarios.
The game-day is complete only when all required scenarios are executed, evidence is captured, and unresolved failures have owners and remediation tasks.
Scope#
Systems and controls exercised:
- Queue producer rollout controls (
REL-001,REL-002) - Canary SLO/cost guardrails (
REL-003) - Canary ramp decisioning + auto-rollback (
REL-004) - RunPod operational visibility and drift checks
- Incident response and rollback procedures
- Release failure scenario coverage for provider failure, queue saturation, and
bad deploy rollback (
WS18-003)
Participants#
Required roles:
- Incident commander (
@oshun/devops) - Scribe (
@oshun/isis-team) - Platform observer (
@oshun/platform) - Runtime observer (
@oshun/ml-team)
Optional observers:
- Security representative (
@oshun/security) - Program governance (
@oshun/core-team)
Preconditions#
- Canary rollout is active and measurable.
- Runbooks are current:
docs/domains/isis/runbooks/runpod-oncall.mddocs/domains/isis/remediation/ISIS_ROLLBACK_CRITERIA.md
- Access to required systems:
- generation API operations endpoints
- database telemetry for canary verification
- deployment path for queue canary percentage updates
- Evidence artifact directory prepared (
tmp/isis-gameday/or equivalent).
Automated Exercise Harness#
Run the automation harness to collect baseline operational evidence:
bash
ISIS_DATABASE_URL=postgres://... \
ISIS_API_BASE_URL=https://<isis-api-host> \
CURRENT_PERCENTAGE=10 \
STRICT=true \
scripts/isis/run_release_gameday.sh
Primary output artifact:
tmp/isis-gameday/release-gameday-report-<timestamp>.json
For WS18-003 release failure scenarios, run the dedicated harness:
bash
RUN_PROVIDER_FAILURE_SCENARIO=true \
QUEUE_SATURATION_SOURCE=api \
STRICT=true \
scripts/isis/run_release_failure_gameday.sh
Supplemental output artifact:
tmp/isis-release-failure-gameday/release-failure-gameday-<timestamp>.json
Scenario Matrix (Required)#
GD-001: Canary Health Baseline#
- Inject: no artificial fault; validate current canary baseline.
- Action:
- run
scripts/isis/verify_canary_slo_cost.sh
- run
- Expected:
- gate report generated
- failed checks are either zero or documented with owner
- Pass criteria:
- evidence captured
- canary health state clearly classified (
passorfail)
GD-002: Ramp Decision Simulation#
- Inject: evaluate ramp decision from current canary percentage.
- Action:
- run
scripts/isis/advance_queue_canary_rollout.sh
- run
- Expected:
- decision artifact emitted (
promote/hold/rollback/steady)
- decision artifact emitted (
- Pass criteria:
- decision and rationale captured
- if rollback decision occurs, rollback command path confirmed
GD-003: RunPod Operational Snapshot + Drift Check#
- Inject: operational audit under current endpoint state.
- Action:
- capture
/api/v1/jobs/runpod/operations - optionally run
scripts/isis/detect_runpod_endpoint_drift.py
- capture
- Expected:
- health, queue, and cost telemetry captured
- drift count recorded
- Pass criteria:
- unresolved drift has explicit owner and ETA
GD-004: Rollback Path Drill#
- Inject: simulated canary regression (policy-level trigger).
- Action:
- execute rollback decision path from
docs/domains/isis/remediation/ISIS_ROLLBACK_CRITERIA.md - verify target action (
ISIS_QUEUE_PRODUCER_CANARY_PERCENTAGE=0and restart)
- execute rollback decision path from
- Expected:
- rollback owner acknowledgment and command pathway verified
- Pass criteria:
- rollback path executable and documented end-to-end
GD-005: Recovery Validation#
- Inject: post-rollback or degraded-state recovery.
- Action:
- re-check queue/provider operations
- verify readiness signals and queue stabilization
- Expected:
- health snapshots and stabilization window evidence recorded
- Pass criteria:
- system returns to acceptable baseline or blocking issue declared
Exit Criteria#
Game-day is considered successful when all conditions are true:
- All required scenarios (
GD-001..GD-005) are executed. - Each scenario has a pass/fail disposition and evidence artifact.
- Any failed scenario has:
- owner
- remediation ticket
- due date
- No unresolved blocker remains for cutover decision meeting.
Deliverables#
- Automated harness report JSON.
- Completed game-day report template:
docs/domains/isis/remediation/ISIS_GAMEDAY_REPORT_TEMPLATE.md
- Linked remediation tickets for any failures.
Traceability#
REL-005: Conduct game-day exercises before full cutover- Related:
REL-001,REL-002,REL-003,REL-004,REL-006docs/domains/isis/remediation/ISIS_RELEASE_QUALITY_BAR.md