# Oshun V1 Rollout Plan > Gate owner: Release Captain. Approval: required signatures in > `docs/releases/v1/verification/rollout-plan/YYYY-MM-DD.md` (release captain + > engineering lead + product lead + support lead). Required for §33.2. OSHUN V1 ships in five canary stages per substrate. Each stage is a **named gate** with a minimum dwell window and explicit promotion criteria. Stages cannot skip; failures auto-rollback to the previous stage and freeze further promotions until the failure is cleared. ## 1. Substrates and stage schedules OSHUN V1 has seven independent substrates. Each substrate has its own schedule. Substrate rollouts are coordinated by the release captain so no two substrates promote into the same stage within a 4-hour window. ### 1.1 Customer web (apps/oshun/web) | Stage | Cohort | Min dwell | Auto-promotion | | ----- | ------------------------------------------------ | --------- | ------------------------ | | 1 % | Internal employee accounts only | 6 h | No | | 5 % | Internal + opt-in pilot tenants | 12 h | No | | 25 % | Geographic rollout (start in low-traffic region) | 24 h | Yes if §3 thresholds met | | 50 % | Multi-region | 24 h | Yes if §3 thresholds met | | 100 % | All regions | n/a | n/a | Routing implementation: CDN-level cookie-based bucketing reading `x-oshun-ramp-bucket` cohort assignment from `libs/oshun/search-discovery/src/experiments/ab-framework.ts`. ### 1.2 Customer mobile iOS (apps/oshun/mobile, iOS) | Stage | Cohort | Min dwell | Auto-promotion | | ----- | -------------------------------------- | --------- | -------------- | | 1 % | TestFlight internal group | 24 h | No | | 5 % | App Store Connect phased release (1 d) | 24 h | No | | 25 % | App Store Connect phased release (3 d) | 48 h | Yes | | 50 % | App Store Connect phased release (5 d) | 24 h | Yes | | 100 % | App Store Connect phased release (7 d) | n/a | n/a | Phased rollout managed via App Store Connect; the team holds an explicit go/no-go before advancing days 3 and 7. ### 1.3 Customer mobile Android (apps/oshun/mobile, Android) | Stage | Cohort | Min dwell | Auto-promotion | | ----- | --------------------------------- | --------- | -------------- | | 1 % | Play Console internal track | 24 h | No | | 5 % | Play Console staged rollout 5 % | 24 h | No | | 25 % | Play Console staged rollout 25 % | 48 h | Yes | | 50 % | Play Console staged rollout 50 % | 24 h | Yes | | 100 % | Play Console staged rollout 100 % | n/a | n/a | ### 1.4 BFF (apps/oshun/bff) | Stage | Cohort | Min dwell | Auto-promotion | | ----- | ---------------------- | --------- | -------------- | | 1 % | One task in one AZ | 30 min | No | | 5 % | Multi-AZ in one region | 30 min | Yes | | 25 % | Single region | 60 min | Yes | | 50 % | Multi-region | 60 min | Yes | | 100 % | All regions | n/a | n/a | ECS service rollout uses CodeDeploy blue/green; cohort selection via ECS task placement and ALB weighted target groups. ### 1.5 Domain services (per-service) Each service in `apps//` follows the BFF schedule **per service**. The release captain MAY parallelize independent services but never chains services that share a hot data path. ### 1.6 Workers (background jobs) | Stage | Cohort | Min dwell | Auto-promotion | | ----- | -------------------------- | --------- | -------------- | | 1 % | One worker pool in one AZ | 30 min | No | | 25 % | One worker pool per region | 60 min | Yes | | 100 % | All worker pools | n/a | n/a | 5 % and 50 % steps are skipped only when the workers are stateless and the queue depth dashboard confirms no backlog. ### 1.7 ML inference (libs/iris/\* model surfaces) | Stage | Cohort | Min dwell | Auto-promotion | | ----- | ------------------------------------ | --------- | -------------- | | 1 % | Shadow traffic (compare to baseline) | 12 h | No | | 5 % | Live traffic — internal accounts | 12 h | No | | 25 % | Live traffic | 24 h | Yes | | 50 % | Live traffic | 24 h | Yes | | 100 % | Live traffic | n/a | n/a | Shadow stage required by `docs/launch/quality-review.md` §1.8 to verify drift KL ≤ 0.05 against the baseline before live traffic. ## 2. Cohort definitions - **Internal employees** — `cohort:emp` cookie applied to every OSHUN employee account (~120 accounts at GA). - **Opt-in pilot tenants** — pre-arranged tenants who agreed in writing to early access; the cohort assignment lives in the tenant-model layer (`libs/oshun/tenant-console/src/tenant-model/`) under the `pilot:` cohort prefix consumed by `buildCanaryGate({ cohortIds: ['pilot:'] })`. - **Geographic rollout** — first ramp region: `us-west-2` (lowest RPS); follow-up: `us-east-1`, `eu-west-1`. - **Canary** — internal employees + pilot tenants, drawn from `libs/oshun/search-discovery/src/experiments/canary.ts` `buildCanaryGate({ cohortIds })`. ## 3. Promotion thresholds A stage promotes only when **all** thresholds below are within bounds on the trailing dwell window: ### 3.1 Reliability | Metric | Threshold | | -------------------------- | --------------- | | API p95 latency | ≤ 400 ms | | API error rate (HTTP 5xx) | ≤ 0.5 % | | Mobile crash-free sessions | ≥ 99.5 % | | Web JS-error rate | ≤ 0.5 % | | Worker job failure rate | ≤ 0.5 % | | Worker queue depth growth | ≤ +5 % / 30 min | ### 3.2 Customer experience | Metric | Threshold | | -------------------------------- | --------- | | Onboarding completion rate | ≥ 85 % | | Domain switch p95 | ≤ 700 ms | | Tara cold launch p95 | ≤ 3.0 s | | Lighthouse Performance (sampled) | ≥ 85 | | Sustained jank events | none | ### 3.3 Quality and grounding | Metric | Threshold | | ------------------------------- | ------------------- | | Assistant grounded-answer rate | ≥ 95 % | | Veritas claim-source bind rate | ≥ 99 % | | Persona-tone drift | ≤ 1 σ from envelope | | Crisis-flow safe-route hit rate | 1.00 | | Watermark offline-verify rate | ≥ 99.9 % | ### 3.4 Safety, privacy, conversion | Metric | Threshold | | ------------------------------------- | --------- | | Open S0 / S1 defects | 0 | | Cross-tenant exposure incidents | 0 | | Moderation backlog p95 dwell | ≤ 4 h | | DSAR completion rate | 100 % | | Conversion (D1 onboarded → D2 active) | ≥ 55 % | ### 3.5 Support load | Metric | Threshold | | ------------------------------------ | -------------------- | | Support contact rate per active user | ≤ 0.04 / week | | New-topic CS volume growth | ≤ +25 % vs. baseline | ## 4. Auto-promotion versus manual-promotion Auto-promotion uses the canary-analysis service (`docs/launch/canary-analysis.md`). The substrate stage table above indicates which transitions are auto-promotable; others require the release captain to manually approve the next stage in the release console with a one-line rationale. Auto-promotion **never** applies to: - 1 % → 5 % (humans confirm baseline behaviour). - ML inference shadow → live. - Mobile iOS / Android phased rollout day advances (Apple / Google require human action). ## 5. Auto-rollback The canary-analysis service performs auto-rollback if **any** §3 threshold is breached for ≥ 5 consecutive minutes within a stage. The substrate is reverted to the previous stage (or pre-stage 1 if the breach occurs at 1 %). A reverted substrate cannot re-promote until the release captain logs `freeze-lift` and re-baselines the metrics. See `docs/launch/rollback-plan.md` for substrate-specific rollback procedures. ## 6. Coordination The release captain runs a **launch war-room** during the rollout. Each substrate has a named on-call (primary + secondary). Substrate on-calls own: - Reading their substrate's dashboard during each stage. - Approving manual promotions. - Escalating auto-rollback events to the release captain. - Maintaining the substrate's status entry in `docs/releases/v1/verification/rollout-plan//status.md`. ## 7. Communication - **Internal** — `#launch-war-room` Slack channel. - **Pilot tenants** — pre-arranged email lists. - **Public status page** — populated when stage ≥ 25 %. - **Release captain hourly summary** — posted to `#launch-war-room` and mirrored to `docs/releases/v1/verification/rollout-plan//hourly/.md`. ## 8. Signed approval Approval is captured at `docs/releases/v1/verification/rollout-plan/YYYY-MM-DD.md`: ``` # Rollout Plan Approval date: YYYY-MM-DD release tag: vX.Y.Z approvers: - Release Captain: APPROVE @ YYYY-MM-DDTHH:MM - Engineering Lead: APPROVE @ YYYY-MM-DDTHH:MM - Product Lead: APPROVE @ YYYY-MM-DDTHH:MM - Support Lead: APPROVE @ YYYY-MM-DDTHH:MM deviations from §1 default schedule: (one line each, with rationale) deviations from §3 default thresholds: (one line each, with rationale) ``` ## 9. Cross-references - Canary code: `libs/oshun/search-discovery/src/experiments/canary.ts`. - A/B framework: `libs/oshun/search-discovery/src/experiments/ab-framework.ts`. - Rollback plan: `docs/launch/rollback-plan.md`. - Canary analysis: `docs/launch/canary-analysis.md`. - Post-deploy monitoring: `docs/launch/post-deploy-monitoring.md`. - Go/no-go: `docs/launch/go-no-go.md`.