# Training — Launch Support

> Launch-support staff run the canary-monitoring rota, the rollback-trigger
> recognition shift, the handoff to the steady-state on-call rotation, and the
> launch-day war room. Active during the canary, GA, and post-GA hyper-care
> windows.

## Audience and prerequisites

- Audience: reliability + platform-engineering staff rotated onto the
  launch-support shift, plus designated launch captains.
- Prerequisites: reliability foundations, on-call onboarding for the primary
  substrate, runbook authorship awareness, audit-scope familiarity.
- Refresh cadence: per-launch; refresh annually for staff retained on the
  steady-state rota.

## Learning objectives

By certification, launch-support staff can independently:

1. Operate the canary cohort schedule (1% → 5% → 25% → 50% → 100%) with
   per-stage monitoring windows.
2. Read the launch dashboards (`dashboards-service-health.ts`,
   `dashboards-customer-kpi.ts`, `dashboards-satisfaction.ts`,
   `dashboards-queue-health.ts`, `dashboards-assistant.ts`,
   `dashboards-nisaba.ts`) and identify a rollback trigger within the documented
   SLO.
3. Trigger the rollback chain: notify, execute, verify, document.
4. Handoff a healthy GA launch to the steady-state on-call rotation with the
   audit-grade handoff packet.
5. Operate the launch-day war room: bridge, situation room, escalation queue,
   comms cadence.
6. Coordinate hyper-care: triage backlog, customer-signal escalation, product
   remediation.

## Curriculum modules

| #   | Module                                            | Duration | Format                               | Assessment                                        |
| --- | ------------------------------------------------- | -------- | ------------------------------------ | ------------------------------------------------- |
| 1   | Canary schedule and monitoring windows            | 90 min   | hands-on with canary tooling         | 1 full canary cycle drill (5 stages)              |
| 2   | Rollback trigger recognition                      | 120 min  | scenario rehearsal                   | 6-scenario rollback-trigger test (≥ 5/6 detected) |
| 3   | Rollback execution and audit                      | 90 min   | hands-on against rollback hooks      | 3-rollback drill                                  |
| 4   | Handoff to steady-state on-call                   | 60 min   | guided handoff with packet authoring | 1 handoff packet pass rubric                      |
| 5   | War-room operations                               | 90 min   | scenario rehearsal                   | 1 full war-room drill                             |
| 6   | Hyper-care triage                                 | 60 min   | hands-on with backlog tooling        | 20-item triage drill                              |
| 7   | Communications cadence with product, T&S, support | 60 min   | scenario rehearsal                   | 3-cadence drill                                   |
| 8   | Post-launch postmortem authorship                 | 60 min   | guided + peer review                 | 1 postmortem draft pass rubric                    |

## Canary monitoring protocol

For each stage (1%, 5%, 25%, 50%, 100%):

1. Confirm the canary advanced via `release.canary.advanced` event.
2. Open the monitoring window per the rollout plan (typically 30–120 min per
   stage; longer for sensitive stages).
3. Watch the launch dashboards: error ratio, latency p95/p99,
   conversion-success, customer-KPI deltas, support-ticket spike, T&S queue
   spike, assistant-quality deltas.
4. Watch the rollback triggers list (see below).
5. At the end of the window, run a structured advance-or-hold decision with the
   launch captain.

## Rollback triggers

A rollback fires if any of these is observed during a canary stage:

- Unresolved critical security issue tied to the release.
- P0 data-loss signal (audit pipeline drop, persistence error class).
- Cross-tenant leakage suspected.
- Failed grounded-answer gate (confidence floor breached, retraction served).
- Elevated crisis-flow defect (crisis routing misses or aged-out).
- Sustained SLO breach (error ratio, latency, conversion).
- Customer-KPI regression beyond the documented bound.
- Cost overshoot beyond the per-stage envelope.
- T&S queue spike beyond the SLA window.

## Handoff protocol

When the launch advances to GA and hyper-care closes:

1. Author the handoff packet: scope, residual risks, observed regressions,
   mitigations in place, owners, ETAs.
2. Brief the steady-state on-call rotation: walk through the packet, the
   runbooks invoked, the dashboards still tagged, and the open tickets.
3. Capture the handoff as an audit event; rotate the launch-support shift down.

## Scenario rehearsal

1. **Canary advance hold**: customer-KPI deltas are within bounds but the
   satisfaction dashboard ticks down; hold the advance and investigate.
2. **Rollback under safety trigger**: a model promotion fires a
   `persona-policy-violation` alert mid-canary; execute rollback, communicate,
   follow `model-workflow-rollback.md`.
3. **War-room cross-domain incident**: a shared dependency fails mid-canary;
   coordinate the war room, route to the right runbook, maintain comms cadence.
4. **Handoff with residual risk**: GA hits, but a non-blocking regression
   remains under remediation; author the handoff packet that scopes the residual
   risk for steady-state on-call.

## Certification criteria

- All 8 modules complete with passing assessment.
- All 4 scenarios passed.
- Launch-support staff has shadowed 1 prior canary cycle.
- Reliability lead and release captain sign-off.

## Tabletop drills

- Per-launch: pre-launch tabletop covering rollback triggers and war- room
  comms.
- Post-launch: postmortem drill with product and reliability.

## Owner

Reliability lead and release captain co-own this training. Updates require both
sign-offs.