# Isis Endpoint Region Failover Disaster Recovery Drill Runbook

Last Updated: 2026-02-24  
Scope: Endpoint-family regional failover simulation using stable/canary routing
controls  
Primary Owners: `@oshun/devops`, `@oshun/isis-team`

## Purpose

Run a repeatable disaster-recovery drill that simulates regional endpoint loss
by promoting the configured canary endpoint to the stable route for a target
endpoint family, then verifies routing, synthetic execution, and restoration.

## Drill Objectives

1. Validate every critical endpoint family has a secondary (canary) route.
2. Simulate regional failover by rerouting stable traffic to the canary
   endpoint.
3. Verify provider operations and runpod operations snapshots converge to
   expected failover state.
4. Restore feature flags to baseline and verify parity.

## Preconditions

- Execute in non-production where provider feature flags are writable.
- `ISIS_API_BASE_URL` and privileged `ISIS_ADMIN_TOKEN` are available.
- Target family has both stable and canary endpoint IDs configured.
- Change ticket is opened with rollback owner and drill window.

## Execute Drill

### Dry run (no mutations)

```bash
DRY_RUN=true \
STRICT=true \
TARGET_FAMILY=comfyui \
ISIS_API_BASE_URL="https://isis-staging.example.com" \
ISIS_ADMIN_TOKEN="..." \
scripts/isis/run_endpoint_region_failover_drill.sh
```

### Live simulation (with automatic restore)

```bash
DRY_RUN=false \
STRICT=true \
RESTORE_FLAGS=true \
SUBMIT_SYNTHETIC_JOB=true \
TARGET_FAMILY=comfyui \
ISIS_API_BASE_URL="https://isis-staging.example.com" \
ISIS_ADMIN_TOKEN="..." \
scripts/isis/run_endpoint_region_failover_drill.sh
```

## Supported Families

- `comfyui`
- `sd`
- `flux`
- `character-consistency`
- `cinematic-video`
- `three-d`
- `texture`
- `audio-sfx`
- `audio-voice`
- `audio-music`

## What Automation Verifies

1. Baseline feature-flag snapshot capture.
2. Baseline stable/canary endpoint presence for target family.
3. Failover mutation:
   - stable endpoint flag set to canary endpoint ID
   - canary percentage set to `0`
4. Post-mutation convergence in `/api/v1/jobs/runpod/operations`.
5. Optional synthetic job acceptance/progress checks.
6. Full restore of stable endpoint + canary percentage flags.
7. Restore verification parity with baseline.

## Success Criteria

Drill passes only if:

- failover mutation succeeds,
- active stable endpoint equals expected failover endpoint,
- synthetic flow remains non-failed (when enabled),
- baseline routing state is restored and verified.

## Evidence Artifacts

- `tmp/isis-region-drill/endpoint-region-failover-drill-*.json`
- feature-flag baseline/post/restore snapshots
- runpod operations post-drill snapshot
- synthetic submit/status snapshots (if enabled)

## Related Artifacts

- `scripts/isis/run_endpoint_region_failover_drill.sh`
- `docs/domains/isis/runbooks/provider-outage-drill.md`
- `docs/domains/isis/runbooks/runpod-oncall.md`
