# ISIS Game-Day Cutover Plan (REL-005)

Date: 2026-02-22  
Checklist Item: `REL-005`  
Ticket: `ISIS-0179`

## Objective

Execute and document pre-cutover game-day exercises that validate Isis
operational readiness under failure and rollback scenarios.

The game-day is complete only when all required scenarios are executed, evidence
is captured, and unresolved failures have owners and remediation tasks.

## Scope

Systems and controls exercised:

- Queue producer rollout controls (`REL-001`, `REL-002`)
- Canary SLO/cost guardrails (`REL-003`)
- Canary ramp decisioning + auto-rollback (`REL-004`)
- RunPod operational visibility and drift checks
- Incident response and rollback procedures
- Release failure scenario coverage for provider failure, queue saturation, and
  bad deploy rollback (`WS18-003`)

## Participants

Required roles:

- Incident commander (`@oshun/devops`)
- Scribe (`@oshun/isis-team`)
- Platform observer (`@oshun/platform`)
- Runtime observer (`@oshun/ml-team`)

Optional observers:

- Security representative (`@oshun/security`)
- Program governance (`@oshun/core-team`)

## Preconditions

1. Canary rollout is active and measurable.
2. Runbooks are current:
   - `docs/domains/isis/runbooks/runpod-oncall.md`
   - `docs/domains/isis/remediation/ISIS_ROLLBACK_CRITERIA.md`
3. Access to required systems:
   - generation API operations endpoints
   - database telemetry for canary verification
   - deployment path for queue canary percentage updates
4. Evidence artifact directory prepared (`tmp/isis-gameday/` or equivalent).

## Automated Exercise Harness

Run the automation harness to collect baseline operational evidence:

```bash
ISIS_DATABASE_URL=postgres://... \
ISIS_API_BASE_URL=https://<isis-api-host> \
CURRENT_PERCENTAGE=10 \
STRICT=true \
scripts/isis/run_release_gameday.sh
```

Primary output artifact:

- `tmp/isis-gameday/release-gameday-report-<timestamp>.json`

For WS18-003 release failure scenarios, run the dedicated harness:

```bash
RUN_PROVIDER_FAILURE_SCENARIO=true \
QUEUE_SATURATION_SOURCE=api \
STRICT=true \
scripts/isis/run_release_failure_gameday.sh
```

Supplemental output artifact:

- `tmp/isis-release-failure-gameday/release-failure-gameday-<timestamp>.json`

## Scenario Matrix (Required)

### GD-001: Canary Health Baseline

- Inject: no artificial fault; validate current canary baseline.
- Action:
  - run `scripts/isis/verify_canary_slo_cost.sh`
- Expected:
  - gate report generated
  - failed checks are either zero or documented with owner
- Pass criteria:
  - evidence captured
  - canary health state clearly classified (`pass` or `fail`)

### GD-002: Ramp Decision Simulation

- Inject: evaluate ramp decision from current canary percentage.
- Action:
  - run `scripts/isis/advance_queue_canary_rollout.sh`
- Expected:
  - decision artifact emitted (`promote`/`hold`/`rollback`/`steady`)
- Pass criteria:
  - decision and rationale captured
  - if rollback decision occurs, rollback command path confirmed

### GD-003: RunPod Operational Snapshot + Drift Check

- Inject: operational audit under current endpoint state.
- Action:
  - capture `/api/v1/jobs/runpod/operations`
  - optionally run `scripts/isis/detect_runpod_endpoint_drift.py`
- Expected:
  - health, queue, and cost telemetry captured
  - drift count recorded
- Pass criteria:
  - unresolved drift has explicit owner and ETA

### GD-004: Rollback Path Drill

- Inject: simulated canary regression (policy-level trigger).
- Action:
  - execute rollback decision path from
    `docs/domains/isis/remediation/ISIS_ROLLBACK_CRITERIA.md`
  - verify target action (`ISIS_QUEUE_PRODUCER_CANARY_PERCENTAGE=0` and restart)
- Expected:
  - rollback owner acknowledgment and command pathway verified
- Pass criteria:
  - rollback path executable and documented end-to-end

### GD-005: Recovery Validation

- Inject: post-rollback or degraded-state recovery.
- Action:
  - re-check queue/provider operations
  - verify readiness signals and queue stabilization
- Expected:
  - health snapshots and stabilization window evidence recorded
- Pass criteria:
  - system returns to acceptable baseline or blocking issue declared

## Exit Criteria

Game-day is considered successful when all conditions are true:

1. All required scenarios (`GD-001`..`GD-005`) are executed.
2. Each scenario has a pass/fail disposition and evidence artifact.
3. Any failed scenario has:
   - owner
   - remediation ticket
   - due date
4. No unresolved blocker remains for cutover decision meeting.

## Deliverables

1. Automated harness report JSON.
2. Completed game-day report template:
   - `docs/domains/isis/remediation/ISIS_GAMEDAY_REPORT_TEMPLATE.md`
3. Linked remediation tickets for any failures.

## Traceability

- `REL-005`: Conduct game-day exercises before full cutover
- Related:
  - `REL-001`, `REL-002`, `REL-003`, `REL-004`, `REL-006`
  - `docs/domains/isis/remediation/ISIS_RELEASE_QUALITY_BAR.md`
