Eve library

Isis Endpoint Region Failover Disaster Recovery Drill Runbook

Runbook

Revision 1

On this page

Last Updated: 2026-02-24
Scope: Endpoint-family regional failover simulation using stable/canary routing controls
Primary Owners: @oshun/devops, @oshun/isis-team

Purpose#

Run a repeatable disaster-recovery drill that simulates regional endpoint loss by promoting the configured canary endpoint to the stable route for a target endpoint family, then verifies routing, synthetic execution, and restoration.

Drill Objectives#

  1. Validate every critical endpoint family has a secondary (canary) route.
  2. Simulate regional failover by rerouting stable traffic to the canary endpoint.
  3. Verify provider operations and runpod operations snapshots converge to expected failover state.
  4. Restore feature flags to baseline and verify parity.

Preconditions#

  • Execute in non-production where provider feature flags are writable.
  • ISIS_API_BASE_URL and privileged ISIS_ADMIN_TOKEN are available.
  • Target family has both stable and canary endpoint IDs configured.
  • Change ticket is opened with rollback owner and drill window.

Execute Drill#

Dry run (no mutations)#

bash
DRY_RUN=true \
STRICT=true \
TARGET_FAMILY=comfyui \
ISIS_API_BASE_URL="https://isis-staging.example.com" \
ISIS_ADMIN_TOKEN="..." \
scripts/isis/run_endpoint_region_failover_drill.sh

Live simulation (with automatic restore)#

bash
DRY_RUN=false \
STRICT=true \
RESTORE_FLAGS=true \
SUBMIT_SYNTHETIC_JOB=true \
TARGET_FAMILY=comfyui \
ISIS_API_BASE_URL="https://isis-staging.example.com" \
ISIS_ADMIN_TOKEN="..." \
scripts/isis/run_endpoint_region_failover_drill.sh

Supported Families#

  • comfyui
  • sd
  • flux
  • character-consistency
  • cinematic-video
  • three-d
  • texture
  • audio-sfx
  • audio-voice
  • audio-music

What Automation Verifies#

  1. Baseline feature-flag snapshot capture.
  2. Baseline stable/canary endpoint presence for target family.
  3. Failover mutation:
    • stable endpoint flag set to canary endpoint ID
    • canary percentage set to 0
  4. Post-mutation convergence in /api/v1/jobs/runpod/operations.
  5. Optional synthetic job acceptance/progress checks.
  6. Full restore of stable endpoint + canary percentage flags.
  7. Restore verification parity with baseline.

Success Criteria#

Drill passes only if:

  • failover mutation succeeds,
  • active stable endpoint equals expected failover endpoint,
  • synthetic flow remains non-failed (when enabled),
  • baseline routing state is restored and verified.

Evidence Artifacts#

  • tmp/isis-region-drill/endpoint-region-failover-drill-*.json
  • feature-flag baseline/post/restore snapshots
  • runpod operations post-drill snapshot
  • synthetic submit/status snapshots (if enabled)
  • scripts/isis/run_endpoint_region_failover_drill.sh
  • docs/domains/isis/runbooks/provider-outage-drill.md
  • docs/domains/isis/runbooks/runpod-oncall.md