# Oshun V1 Rollback Plan (per substrate) > Gate owner: Release Captain + Engineering Lead. Approval: required signatures > in `docs/releases/v1/verification/rollback-plan/YYYY-MM-DD.md`. Required for > §33.2. Operational details: `docs/reference/rollback-procedures.md`. OSHUN V1 ships across seven substrates. Each substrate has a named rollback procedure with measured RTO and RPO. Every substrate rollback is **rehearsed** in the 14 days preceding GA. The release captain commits the rehearsal evidence to `docs/releases/v1/verification/rollback-plan//.md` before the go/no-go meeting opens. ## 1. Substrates and rollback strategies ### 1.1 Customer web (apps/oshun/web) - **Strategy** — fast-forward to the prior commit on the deployment branch; CDN purges the new bundle hashes; service worker forces update by bumping `version` in the manifest. - **RTO** — 5 min for the bundle; 10 min for service-worker forced refresh. - **RPO** — 0 (web is stateless). - **Trigger** — any §11 trigger in `docs/launch/go-no-go.md` OR any auto-rollback event from `docs/launch/canary-analysis.md`. - **Procedure** — 1. `gh workflow run rollback-web.yml -F ref=`. 2. Verify CDN cache invalidation completed. 3. Force-update PWA manifest version. 4. Watch `dashboards-service-health` for 30 min. ### 1.2 Customer mobile iOS - **Strategy** — halt phased rollout in App Store Connect; remove the in-progress build from sale; pre-staged minimum-version enforcement banner. - **RTO** — halt-rollout: 10 min; minimum-version banner: 30 min; full revert (Apple-controlled): 1-3 d. - **RPO** — 0 for non-installed users; up to D-1 backup for any installed user blocked by force-update. - **Trigger** — same as §1.1. - **Procedure** — 1. App Store Connect → "Pause Release" on the active build. 2. Push the minimum-version enforcement via remote config. 3. Pre-staged restore-prior-binary requires Apple expedited review; pre-coordinated with App Review. ### 1.3 Customer mobile Android - **Strategy** — halt Play Console staged rollout; push minimum- version enforcement; if needed, promote prior internal-track build to production. - **RTO** — halt: 5 min; minimum-version: 30 min; full revert: 1-4 h (Play Console push). - **RPO** — 0 for non-installed users. - **Procedure** — 1. Play Console → "Halt rollout" on the active build. 2. Push minimum-version enforcement. 3. Promote prior production-track build if rollback required. ### 1.4 BFF (apps/oshun/bff, ECS) - **Strategy** — ECS task-definition revert via the `aws ecs update-service` flow in `docs/reference/rollback-procedures.md` §1. - **RTO** — 5-10 min (ALB target-group draining). - **RPO** — 0 (BFF is stateless). - **Trigger** — same as §1.1. - **Procedure** — 1. `aws ecs update-service --cluster oshun-production --service oshun-bff --task-definition oshun-bff:`. 2. Watch service events until `desiredCount == runningCount` for the previous revision. 3. Verify `dashboards-service-health`. ### 1.5 Domain services (per service) Same ECS task-definition flow as BFF (§1.4). Each service maintains its own task-definition history. Rollback procedures named for the service: | Service | Cluster | Task family | | ----------- | ---------------- | ----------------- | | tara-api | oshun-production | oshun-tara-api | | arete-api | oshun-production | oshun-arete-api | | veritas-api | oshun-production | oshun-veritas-api | | nyx-api | oshun-production | oshun-nyx-api | | nisaba-api | oshun-production | oshun-nisaba-api | | metis-api | oshun-production | oshun-metis-api | | iris-api | oshun-production | oshun-iris-api | | sophia-api | oshun-production | oshun-sophia-api | | isis-api | oshun-production | oshun-isis-api | | lilith-api | oshun-production | oshun-lilith-api | | hathor-api | oshun-production | oshun-hathor-api | | bellona-api | oshun-production | oshun-bellona-api | RTO/RPO same as BFF. ### 1.6 Workers (background jobs) - **Strategy** — ECS task-definition revert (same as BFF) plus in-flight job drain. - **RTO** — 10-30 min depending on job duration; queue depth must fall back to baseline before declaring "rolled back". - **RPO** — depends on the job. Idempotent jobs: 0. Non-idempotent jobs: replay from queue with dedup keys; manual retry for jobs that exceeded the retention window. - **Procedure** — 1. `aws ecs update-service` to prior task-definition. 2. Drain in-flight jobs (worker emits `worker:draining` event; ALB stops dispatching new work). 3. Verify `dashboards-queue-health` queue-depth returns to baseline. ### 1.7 ML inference (libs/iris/\* model surfaces) - **Strategy** — model-version pin revert via the model registry; shadow remains on the new model for analysis. - **RTO** — 5 min. - **RPO** — 0 (model surfaces are stateless). - **Trigger** — §11 triggers plus any one of: grounded-answer rate < 0.90 for 15 min, drift KL > 0.10 vs. baseline, persona drift > 2 σ over 1 h. - **Procedure** — 1. `pnpm nx run iris-model-registry:pin --model --version `. 2. Verify `dashboards-assistant` grounded-answer rate restored. 3. Open a model-rollback incident (`docs/runbooks/model-workflow-rollback.md`). ### 1.8 Database migrations - **Strategy** — every migration MUST ship with a paired down- migration; non-reversible migrations are forbidden unless the release captain explicitly approves with a recovery plan. - **RTO** — depends on table size; published with the migration. - **RPO** — 0 for additive migrations; up to 15 min for destructive migrations restoring from PITR (point-in-time-recovery). - **Procedure** — see `docs/reference/rollback-procedures.md` §2. ### 1.9 Persona - **Strategy** — promote prior persona version via the persona registry. - **RTO** — 5 min. - **RPO** — 0 (persona registry retains every version). - **Trigger** — persona drift > 2 σ in any 7-day window, OR safety incident class "persona-drift" opened. - **Procedure** — see `docs/runbooks/persona-rollback.md`. ### 1.10 Content (Studio publishing) - **Strategy** — revert to prior content version; emit public change note via `libs/shared/release-management/src/rollback-plan.ts`. - **RTO** — 5 min. - **RPO** — 0 for additive content; up to 1 h for content edits (revert via the version graph). - **Procedure** — see `docs/runbooks/oshun-v1.md` §rollback. ## 2. Rehearsal requirements Every substrate above is rehearsed in the 14 days preceding GA. Rehearsal evidence: ``` docs/releases/v1/verification/rollback-plan//.md - substrate: | workers | ml-inference | db-migration: | persona | content> - rehearsed-at: YYYY-MM-DDTHH:MM TZ - rehearsed-by: - procedure-ref: - observed-RTO: - observed-RPO: - evidence: - defects: (any deviations from the documented procedure) ``` Failing a rehearsal blocks GA until the procedure is fixed and the rehearsal repeated. ## 3. Approval Captured at `docs/releases/v1/verification/rollback-plan/YYYY-MM-DD.md`: ``` # Rollback Plan Approval date: YYYY-MM-DD release tag: vX.Y.Z per-substrate rehearsal evidence: - web: - mobile-ios: - mobile-android: - bff: - services: - tara-api: - arete-api: - ... (every service) - workers: - ml-inference: - db-migrations: - : - persona: - content: approvers: - Release Captain: APPROVE @ ts - Engineering Lead: APPROVE @ ts - Security Lead (for migrations + ML inference): APPROVE @ ts - Privacy Lead (for migrations + persona): APPROVE @ ts ``` ## 4. Cross-references - Operational rollback procedures: `docs/reference/rollback-procedures.md`. - Persona rollback runbook: `docs/runbooks/persona-rollback.md`. - Model/workflow rollback runbook: `docs/runbooks/model-workflow-rollback.md`. - Shell-outage runbook: `docs/runbooks/shell-outage.md`. - Provider-failover runbook: `docs/runbooks/provider-failover.md`. - Release-management rollback planner: `libs/shared/release-management/src/rollback-plan.ts`. - Platform rollback orchestration: `libs/oshun/platform-foundations/src/rollback/`. - Rollout plan: `docs/launch/rollout-plan.md`. - Canary analysis: `docs/launch/canary-analysis.md`. - Go/no-go: `docs/launch/go-no-go.md`.