Disciplines · Launch

Oshun V1 Rollout Plan

OSHUN V1 has seven independent substrates.

9sections5 minread
Markdown source
On this page

Gate owner: Release Captain. Approval: required signatures in docs/releases/v1/verification/rollout-plan/YYYY-MM-DD.md (release captain + engineering lead + product lead + support lead). Required for §33.2.

OSHUN V1 ships in five canary stages per substrate. Each stage is a named gate with a minimum dwell window and explicit promotion criteria. Stages cannot skip; failures auto-rollback to the previous stage and freeze further promotions until the failure is cleared.

1. Substrates and stage schedules#

OSHUN V1 has seven independent substrates. Each substrate has its own schedule. Substrate rollouts are coordinated by the release captain so no two substrates promote into the same stage within a 4-hour window.

1.1 Customer web (apps/oshun/web)#

Stage Cohort Min dwell Auto-promotion
1 % Internal employee accounts only 6 h No
5 % Internal + opt-in pilot tenants 12 h No
25 % Geographic rollout (start in low-traffic region) 24 h Yes if §3 thresholds met
50 % Multi-region 24 h Yes if §3 thresholds met
100 % All regions n/a n/a

Routing implementation: CDN-level cookie-based bucketing reading x-oshun-ramp-bucket cohort assignment from libs/oshun/search-discovery/src/experiments/ab-framework.ts.

1.2 Customer mobile iOS (apps/oshun/mobile, iOS)#

Stage Cohort Min dwell Auto-promotion
1 % TestFlight internal group 24 h No
5 % App Store Connect phased release (1 d) 24 h No
25 % App Store Connect phased release (3 d) 48 h Yes
50 % App Store Connect phased release (5 d) 24 h Yes
100 % App Store Connect phased release (7 d) n/a n/a

Phased rollout managed via App Store Connect; the team holds an explicit go/no-go before advancing days 3 and 7.

1.3 Customer mobile Android (apps/oshun/mobile, Android)#

Stage Cohort Min dwell Auto-promotion
1 % Play Console internal track 24 h No
5 % Play Console staged rollout 5 % 24 h No
25 % Play Console staged rollout 25 % 48 h Yes
50 % Play Console staged rollout 50 % 24 h Yes
100 % Play Console staged rollout 100 % n/a n/a

1.4 BFF (apps/oshun/bff)#

Stage Cohort Min dwell Auto-promotion
1 % One task in one AZ 30 min No
5 % Multi-AZ in one region 30 min Yes
25 % Single region 60 min Yes
50 % Multi-region 60 min Yes
100 % All regions n/a n/a

ECS service rollout uses CodeDeploy blue/green; cohort selection via ECS task placement and ALB weighted target groups.

1.5 Domain services (per-service)#

Each service in apps/<domain>/ follows the BFF schedule per service. The release captain MAY parallelize independent services but never chains services that share a hot data path.

1.6 Workers (background jobs)#

Stage Cohort Min dwell Auto-promotion
1 % One worker pool in one AZ 30 min No
25 % One worker pool per region 60 min Yes
100 % All worker pools n/a n/a

5 % and 50 % steps are skipped only when the workers are stateless and the queue depth dashboard confirms no backlog.

1.7 ML inference (libs/iris/* model surfaces)#

Stage Cohort Min dwell Auto-promotion
1 % Shadow traffic (compare to baseline) 12 h No
5 % Live traffic — internal accounts 12 h No
25 % Live traffic 24 h Yes
50 % Live traffic 24 h Yes
100 % Live traffic n/a n/a

Shadow stage required by docs/launch/quality-review.md §1.8 to verify drift KL ≤ 0.05 against the baseline before live traffic.

2. Cohort definitions#

  • Internal employees — cohort:emp cookie applied to every OSHUN employee account (~120 accounts at GA).
  • Opt-in pilot tenants — pre-arranged tenants who agreed in writing to early access; the cohort assignment lives in the tenant-model layer (libs/oshun/tenant-console/src/tenant-model/) under the pilot: cohort prefix consumed by buildCanaryGate({ cohortIds: ['pilot:<tenant>'] }).
  • Geographic rollout — first ramp region: us-west-2 (lowest RPS); follow-up: us-east-1, eu-west-1.
  • Canary — internal employees + pilot tenants, drawn from libs/oshun/search-discovery/src/experiments/canary.ts buildCanaryGate({ cohortIds }).

3. Promotion thresholds#

A stage promotes only when all thresholds below are within bounds on the trailing dwell window:

3.1 Reliability#

Metric Threshold
API p95 latency ≤ 400 ms
API error rate (HTTP 5xx) ≤ 0.5 %
Mobile crash-free sessions ≥ 99.5 %
Web JS-error rate ≤ 0.5 %
Worker job failure rate ≤ 0.5 %
Worker queue depth growth ≤ +5 % / 30 min

3.2 Customer experience#

Metric Threshold
Onboarding completion rate ≥ 85 %
Domain switch p95 ≤ 700 ms
Tara cold launch p95 ≤ 3.0 s
Lighthouse Performance (sampled) ≥ 85
Sustained jank events none

3.3 Quality and grounding#

Metric Threshold
Assistant grounded-answer rate ≥ 95 %
Veritas claim-source bind rate ≥ 99 %
Persona-tone drift ≤ 1 σ from envelope
Crisis-flow safe-route hit rate 1.00
Watermark offline-verify rate ≥ 99.9 %

3.4 Safety, privacy, conversion#

Metric Threshold
Open S0 / S1 defects 0
Cross-tenant exposure incidents 0
Moderation backlog p95 dwell ≤ 4 h
DSAR completion rate 100 %
Conversion (D1 onboarded → D2 active) ≥ 55 %

3.5 Support load#

Metric Threshold
Support contact rate per active user ≤ 0.04 / week
New-topic CS volume growth ≤ +25 % vs. baseline

4. Auto-promotion versus manual-promotion#

Auto-promotion uses the canary-analysis service (docs/launch/canary-analysis.md). The substrate stage table above indicates which transitions are auto-promotable; others require the release captain to manually approve the next stage in the release console with a one-line rationale.

Auto-promotion never applies to:

  • 1 % → 5 % (humans confirm baseline behaviour).
  • ML inference shadow → live.
  • Mobile iOS / Android phased rollout day advances (Apple / Google require human action).

5. Auto-rollback#

The canary-analysis service performs auto-rollback if any §3 threshold is breached for ≥ 5 consecutive minutes within a stage. The substrate is reverted to the previous stage (or pre-stage 1 if the breach occurs at 1 %). A reverted substrate cannot re-promote until the release captain logs freeze-lift and re-baselines the metrics.

See docs/launch/rollback-plan.md for substrate-specific rollback procedures.

6. Coordination#

The release captain runs a launch war-room during the rollout. Each substrate has a named on-call (primary + secondary). Substrate on-calls own:

  • Reading their substrate's dashboard during each stage.
  • Approving manual promotions.
  • Escalating auto-rollback events to the release captain.
  • Maintaining the substrate's status entry in docs/releases/v1/verification/rollout-plan/<date>/status.md.

7. Communication#

  • Internal — #launch-war-room Slack channel.
  • Pilot tenants — pre-arranged email lists.
  • Public status page — populated when stage ≥ 25 %.
  • Release captain hourly summary — posted to #launch-war-room and mirrored to docs/releases/v1/verification/rollout-plan/<date>/hourly/<ts>.md.

8. Signed approval#

Approval is captured at docs/releases/v1/verification/rollout-plan/YYYY-MM-DD.md:

text
# Rollout Plan Approval
date: YYYY-MM-DD
release tag: vX.Y.Z

approvers:
- Release Captain: <name> APPROVE @ YYYY-MM-DDTHH:MM
- Engineering Lead: <name> APPROVE @ YYYY-MM-DDTHH:MM
- Product Lead: <name> APPROVE @ YYYY-MM-DDTHH:MM
- Support Lead: <name> APPROVE @ YYYY-MM-DDTHH:MM

deviations from §1 default schedule: (one line each, with rationale)

deviations from §3 default thresholds: (one line each, with rationale)

9. Cross-references#

  • Canary code: libs/oshun/search-discovery/src/experiments/canary.ts.
  • A/B framework: libs/oshun/search-discovery/src/experiments/ab-framework.ts.
  • Rollback plan: docs/launch/rollback-plan.md.
  • Canary analysis: docs/launch/canary-analysis.md.
  • Post-deploy monitoring: docs/launch/post-deploy-monitoring.md.
  • Go/no-go: docs/launch/go-no-go.md.