# ISIS Canary Ramp and Auto-Rollback (REL-004)

Date: 2026-02-22  
Checklist Item: `REL-004`  
Ticket: `ISIS-0178`

## Purpose

Automate percentage ramp decisions for queue canary rollout with rollback-safe
controls tied to objective SLO and cost thresholds.

This flow builds on:

- `REL-001`: queue producer global dispatch toggle
- `REL-002`: queue canary project/percentage controls
- `REL-003`: canary SLO/cost verifier

## Controller Script

- Script: `scripts/isis/advance_queue_canary_rollout.sh`
- Verification dependency: `scripts/isis/verify_canary_slo_cost.sh`
- Output artifacts:
  - verification report JSON
  - rollout decision JSON (`promote`, `hold`, `rollback`, `steady`)

The controller itself does not mutate deployment configuration unless explicit
callbacks are provided.

## Default Ramp Sequence

Default canary progression:

- `5,10,25,50,100`

Override via `RAMP_SEQUENCE`.

## Usage

### Decision-Only (No Mutation)

```bash
ISIS_DATABASE_URL=postgres://... \
CURRENT_PERCENTAGE=10 \
scripts/isis/advance_queue_canary_rollout.sh
```

### Strict Auto-Rollback Mode

```bash
ISIS_DATABASE_URL=postgres://... \
CURRENT_PERCENTAGE=25 \
AUTO_ROLLBACK_ENABLED=true \
FAIL_ON_ROLLBACK=true \
scripts/isis/advance_queue_canary_rollout.sh
```

If verification fails and auto-rollback is enabled:

- decision action becomes `rollback`
- target percentage is `0`
- script exits non-zero when `FAIL_ON_ROLLBACK=true`

### With Apply/Rollback Callbacks

Callbacks run in shell with context env vars (`ACTION`, `CURRENT_PERCENTAGE`,
`TARGET_PERCENTAGE`, `VERIFICATION_REPORT_FILE`, `DECISION_FILE`, and
`FAILED_CHECKS` on rollback).

```bash
ISIS_DATABASE_URL=postgres://... \
CURRENT_PERCENTAGE=10 \
APPLY_NEXT_PERCENTAGE_CALLBACK='echo "apply ${TARGET_PERCENTAGE}"' \
ROLLBACK_CALLBACK='echo "rollback to ${TARGET_PERCENTAGE}"' \
scripts/isis/advance_queue_canary_rollout.sh
```

## Decision Model

### `promote`

- Triggered when verification passes and a higher ramp percentage exists.
- Target set to next percentage in sequence.

### `steady`

- Triggered when verification passes and canary is already at max percentage.

### `rollback`

- Triggered when verification fails and `AUTO_ROLLBACK_ENABLED=true`.
- Target forced to `0`.

### `hold`

- Triggered when verification fails and auto-rollback is disabled.

## Suggested Deployment Integration

1. Run `verify_canary_slo_cost.sh` (or let controller invoke it).
2. Run `advance_queue_canary_rollout.sh` with current percentage.
3. On `promote`, update deployment env:
   - `ISIS_QUEUE_PRODUCER_CANARY_PERCENTAGE=<targetPercentage>`
   - restart generation API
4. On `rollback`, set:
   - `ISIS_QUEUE_PRODUCER_CANARY_PERCENTAGE=0`
   - optionally `ISIS_QUEUE_PRODUCER_CANARY_ENABLED=false`
   - restart generation API
5. Attach verification + decision JSON to release evidence package.

## Core Configuration

- `CURRENT_PERCENTAGE` (required): current queue canary percentage
- `RAMP_SEQUENCE` (optional): comma-separated integer percentages
- `AUTO_ROLLBACK_ENABLED` (default `true`)
- `FAIL_ON_ROLLBACK` (default `true`)
- `APPLY_NEXT_PERCENTAGE_CALLBACK` (optional)
- `ROLLBACK_CALLBACK` (optional)
- `VERIFY_SCRIPT` (optional path override)
- `REPORT_DIR`, `VERIFICATION_REPORT_FILE`, `DECISION_FILE` (artifact paths)

All `REL-003` verifier threshold overrides are supported via inherited env vars
(e.g., `QUEUE_DELAY_SLO_TARGET`, `COST_BURN_REGRESSION_MAX`,
`MIN_CANARY_TERMINAL_JOBS`, `SUBMIT_LATENCY_SLI`).

## Traceability

- `REL-004`: Gradual canary percentage increase with auto-rollback thresholds
- `REL-005`: game-day harness should consume ramp decision artifacts
- Related docs:
  - `docs/domains/isis/remediation/ISIS_CANARY_SLO_COST_VERIFICATION.md`
  - `docs/domains/isis/remediation/ISIS_GAMEDAY_CUTOVER_PLAN.md`
  - `docs/domains/isis/remediation/ISIS_QUEUE_PRODUCER_ROLLOUT_FLAG.md`
  - `docs/domains/isis/remediation/ISIS_ROLLBACK_CRITERIA.md`
