Canary Analysis Automation — V1 Launch Gate

Gate owner: SRE Lead. Required for §33.2. Verifies the canary-gate decision pipeline is enabled, exercised, and wired to per-substrate rollout-promotion automation.

The canary analysis pipeline auto-decides promote / hold / abort for every rollout stage. Decisions follow the metric-comparison logic in libs/oshun/search-discovery/src/experiments/canary.ts and apply the substrate-specific thresholds in docs/launch/rollout-plan.md §3.

1. Pipeline architecture#

[ rollout stage start ]
        │
        ▼
  registerCanaryGate({ experimentId, cohortIds, minDwellSeconds,
                       minSampleSize, metrics, activatedAtUnixSeconds })
        │
        ▼
  every 60 s during dwell window: collectReadings(cohortIds, metrics)
        │
        ▼
  evaluateCanary({ gate, readings, nowUnixSeconds })
        │
        ▼
  ┌─ promote ─→ applyCanaryToExperiment + emit rollout-promotion event
  │             ↳ rollout controller advances substrate to next stage
  ├─ hold (dwell)        ─→ wait, re-collect
  ├─ hold (sample-size)  ─→ wait, re-collect (alert if > 2× dwell window)
  └─ abort ─→ rollback substrate to prior stage
            ↳ freeze further promotions
            ↳ page launch war-room (sev1)
            ↳ commit decision evidence

2. Code references#

3. Metric registration (per substrate)#

Each substrate registers its own canary gate with metrics from §3 of docs/launch/rollout-plan.md. Sample registration for the BFF substrate (illustrative; concrete configuration lives in the rollout controller config):

import { buildCanaryGate } from '@oshun/search-discovery/experiments/canary';

const bffStage1Gate = buildCanaryGate({
  experimentId: 'rollout:bff:v1:stage-1',
  cohortIds: ['canary'],
  minDwellSeconds: 30 * 60, // 30 minutes
  minSampleSize: 2_000,
  activatedAtUnixSeconds: stageStart,
  metrics: [
    {
      metricId: 'api_p95_latency_ms',
      direction: 'lower-is-better',
      mdeRatio: 0.05,
    },
    { metricId: 'http_5xx_rate', direction: 'lower-is-better', mdeRatio: 0.1 },
    {
      metricId: 'grounded_answer_rate',
      direction: 'higher-is-better',
      mdeRatio: 0.05,
    },
    {
      metricId: 'crisis_safe_route_rate',
      direction: 'higher-is-better',
      mdeRatio: 0.0,
    },
    {
      metricId: 'persona_tone_drift_sigma',
      direction: 'lower-is-better',
      mdeRatio: 0.0,
    },
    // ... see docs/launch/rollout-plan.md §3 for the full list
  ],
});

The MDE ratio is the maximum tolerated relative regression vs. control. A mdeRatio: 0.00 metric (e.g., crisis-flow safe-route hit rate) means any regression aborts the stage.

4. Per-substrate registration#

5. Auto-promotion + auto-abort#

• A promote decision emits a rollout-promotion event that the rollout controller consumes to advance the substrate to the next stage only when the stage table in docs/launch/rollout-plan.md §1 marks the transition as auto-promotable.
• A hold (dwell) decision causes the controller to wait and re-poll.
• A hold (sample-size) decision causes the controller to wait until 2× the dwell window; after that, the SRE lead is paged.
• An abort decision:
  1. Reverts the substrate to the prior stage via the per-substrate procedure in docs/launch/rollback-plan.md §1.
  2. Sets the rollout controller flag frozen: true so further auto-promotion is blocked until the release captain logs freeze-lift.
  3. Pages the launch war-room (sev1 via incident-ownership-manifest.ts).
  4. Commits decision evidence to docs/releases/v1/verification/canary-analysis/<substrate>/<stage>/<ts>.md.

6. End-to-end exercise#

In the 14 days preceding GA, the SRE lead exercises the pipeline:

• Shadow run — fire a synthetic metric regression on a test substrate; confirm abort decision + rollback executed within 90 s.
• Promote run — fire clean metrics; confirm promote decision + rollout-promotion event observed by controller.
• Sample-size run — fire below-minSampleSize readings; confirm hold with reason sample-size and page after 2× dwell window.

Evidence committed at docs/releases/v1/verification/canary-analysis/exercise-<date>.md.

7. Test coverage#

Unit-level test coverage of the canary-gate evaluation logic lives at libs/oshun/search-discovery/src/experiments/canary.test.ts (asserts promote / hold / abort for representative inputs). The dwell window, sample-size, and per-metric MDE assertions cover the three decision branches. Integration-level coverage of the rollout-controller wiring is exercised end-to-end in §6 (shadow / promote / sample-size exercises) with evidence committed under docs/releases/v1/verification/canary-analysis/exercise-<date>.md.

8. Live verification checklist#

Required for §33.2 signoff:

• canary.ts test suite green on the release tag.
• Per-substrate gates registered in the rollout-controller config for every substrate in §4.
• Shadow / promote / sample-size end-to-end exercise (§6) completed and evidence committed.
• Alert wiring verified for canary_rollback sev1 route.
• War-room dashboard renders rollout-progress per-substrate gauge backed by the canary-gate decisions.

9. Cross-references#

• Canary code: libs/oshun/search-discovery/src/experiments/canary.ts.
• Rollout plan: docs/launch/rollout-plan.md.
• Rollback plan: docs/launch/rollback-plan.md.
• Post-deploy monitoring: docs/launch/post-deploy-monitoring.md.
• Go/no-go: docs/launch/go-no-go.md.