Eve library

ISIS Game-Day Cutover Plan (REL-005)

Guide

Revision 1

On this page

Date: 2026-02-22
Checklist Item: REL-005
Ticket: ISIS-0179

Objective#

Execute and document pre-cutover game-day exercises that validate Isis operational readiness under failure and rollback scenarios.

The game-day is complete only when all required scenarios are executed, evidence is captured, and unresolved failures have owners and remediation tasks.

Scope#

Systems and controls exercised:

  • Queue producer rollout controls (REL-001, REL-002)
  • Canary SLO/cost guardrails (REL-003)
  • Canary ramp decisioning + auto-rollback (REL-004)
  • RunPod operational visibility and drift checks
  • Incident response and rollback procedures
  • Release failure scenario coverage for provider failure, queue saturation, and bad deploy rollback (WS18-003)

Participants#

Required roles:

  • Incident commander (@oshun/devops)
  • Scribe (@oshun/isis-team)
  • Platform observer (@oshun/platform)
  • Runtime observer (@oshun/ml-team)

Optional observers:

  • Security representative (@oshun/security)
  • Program governance (@oshun/core-team)

Preconditions#

  1. Canary rollout is active and measurable.
  2. Runbooks are current:
    • docs/domains/isis/runbooks/runpod-oncall.md
    • docs/domains/isis/remediation/ISIS_ROLLBACK_CRITERIA.md
  3. Access to required systems:
    • generation API operations endpoints
    • database telemetry for canary verification
    • deployment path for queue canary percentage updates
  4. Evidence artifact directory prepared (tmp/isis-gameday/ or equivalent).

Automated Exercise Harness#

Run the automation harness to collect baseline operational evidence:

bash
ISIS_DATABASE_URL=postgres://... \
ISIS_API_BASE_URL=https://<isis-api-host> \
CURRENT_PERCENTAGE=10 \
STRICT=true \
scripts/isis/run_release_gameday.sh

Primary output artifact:

  • tmp/isis-gameday/release-gameday-report-<timestamp>.json

For WS18-003 release failure scenarios, run the dedicated harness:

bash
RUN_PROVIDER_FAILURE_SCENARIO=true \
QUEUE_SATURATION_SOURCE=api \
STRICT=true \
scripts/isis/run_release_failure_gameday.sh

Supplemental output artifact:

  • tmp/isis-release-failure-gameday/release-failure-gameday-<timestamp>.json

Scenario Matrix (Required)#

GD-001: Canary Health Baseline#

  • Inject: no artificial fault; validate current canary baseline.
  • Action:
    • run scripts/isis/verify_canary_slo_cost.sh
  • Expected:
    • gate report generated
    • failed checks are either zero or documented with owner
  • Pass criteria:
    • evidence captured
    • canary health state clearly classified (pass or fail)

GD-002: Ramp Decision Simulation#

  • Inject: evaluate ramp decision from current canary percentage.
  • Action:
    • run scripts/isis/advance_queue_canary_rollout.sh
  • Expected:
    • decision artifact emitted (promote/hold/rollback/steady)
  • Pass criteria:
    • decision and rationale captured
    • if rollback decision occurs, rollback command path confirmed

GD-003: RunPod Operational Snapshot + Drift Check#

  • Inject: operational audit under current endpoint state.
  • Action:
    • capture /api/v1/jobs/runpod/operations
    • optionally run scripts/isis/detect_runpod_endpoint_drift.py
  • Expected:
    • health, queue, and cost telemetry captured
    • drift count recorded
  • Pass criteria:
    • unresolved drift has explicit owner and ETA

GD-004: Rollback Path Drill#

  • Inject: simulated canary regression (policy-level trigger).
  • Action:
    • execute rollback decision path from docs/domains/isis/remediation/ISIS_ROLLBACK_CRITERIA.md
    • verify target action (ISIS_QUEUE_PRODUCER_CANARY_PERCENTAGE=0 and restart)
  • Expected:
    • rollback owner acknowledgment and command pathway verified
  • Pass criteria:
    • rollback path executable and documented end-to-end

GD-005: Recovery Validation#

  • Inject: post-rollback or degraded-state recovery.
  • Action:
    • re-check queue/provider operations
    • verify readiness signals and queue stabilization
  • Expected:
    • health snapshots and stabilization window evidence recorded
  • Pass criteria:
    • system returns to acceptable baseline or blocking issue declared

Exit Criteria#

Game-day is considered successful when all conditions are true:

  1. All required scenarios (GD-001..GD-005) are executed.
  2. Each scenario has a pass/fail disposition and evidence artifact.
  3. Any failed scenario has:
    • owner
    • remediation ticket
    • due date
  4. No unresolved blocker remains for cutover decision meeting.

Deliverables#

  1. Automated harness report JSON.
  2. Completed game-day report template:
    • docs/domains/isis/remediation/ISIS_GAMEDAY_REPORT_TEMPLATE.md
  3. Linked remediation tickets for any failures.

Traceability#

  • REL-005: Conduct game-day exercises before full cutover
  • Related:
    • REL-001, REL-002, REL-003, REL-004, REL-006
    • docs/domains/isis/remediation/ISIS_RELEASE_QUALITY_BAR.md