Checklist: YSD-22058 (provider change) · drill kind provider-failover
Decision owner role: the service owner for svc-study-workspace Nearest
existing document: docs/runbooks/provider-failover.md
The repository has a provider-failover runbook. Its scope list names the assistant, Sophia, Isis, messaging and payment providers, and not this workspace — and the study provider surface is a different shape from all of them: link-only sources, analyzer backends, and the study adapters.
Three surfaces, three different failures#
Read this table before deciding anything. They are not one outage.
| Surface | If the provider goes | Correct response |
|---|---|---|
| Link-only sources | The workspace never held the bytes. Playback stops; the record, rights state and every annotation anchored to it remain | Suppress playback honestly. Do not substitute another provider's copy — a different edition has different timings, so every anchor silently points at the wrong moment |
| Analyzer backends | Analysis cannot run; existing outputs are unaffected | Queue or refuse, and say which. An analyzer that returns a lower-quality result under the same pipelineVersion makes two different things indistinguishable |
| Study adapters | The connector's material stops arriving | Refuse the connector, keep what was already ingested with its provenance |
⚠️ The failure mode this workspace must not have is silent substitution. A provider swap that keeps the same identifiers while changing the underlying edition, model, or clock turns every existing anchor into a wrong answer that still looks right. If a substitute cannot be shown to be the same edition, it is a new source, not a failover.
Detection#
- The provider's own error rate through the adapter, and readiness for the surfaces that depend on it.
- Not an increase in analysis latency alone: a slow analyzer and a withdrawn one need opposite responses, and only the second is this document.
The steps#
| # | Action | Owner role | Expected |
|---|---|---|---|
| 1 | Identify which of the three surfaces is affected | on-call engineer | 10 min |
| 2 | Confirm what is authorized with the replacement — the terms are per provider and per territory | rights owner decides | — |
| 3 | Suppress or queue the affected surface, honestly labelled | on-call engineer | 20 min |
| 4 | Cut over only if step 2 says the replacement is authorized AND is the same edition | service owner | — |
Step 2 has no default. The provider due-diligence matrix (YSD-20030) that
would answer it is drafted and unapproved, and an undeclared provider capability
resolves as provider-undeclared and is denied. That denial is correct: it is
the difference between "we read the terms and they allow this" and "nobody has
looked."
Containment objective: 40 minutes, from noticing the provider is gone to the affected surface being suppressed or queued and honestly labelled — steps 1 and 3, budgeted at 10 + 20 = 30, plus ten minutes of headroom rounded up to the next five.
There is deliberately no objective for the cutover. Steps 2 and 4 are marked
— above because step 2 waits on a rights decision that has no default, and a
duration promised over a step that waits on a person is a promise about how fast
somebody else will answer. Sizing an objective around the whole procedure would
make it unkeepable by construction, and sizing one around the containment is the
part this team controls. A drill of this procedure is timed to the end of step 3
and records step 2 as outstanding; the register's own verification — that
outputs after the switch name the substituted provider — cannot be reached at
all until the matrix is approved, which is the shortfall below rather than a
slow step.
Communication#
| Audience | Within |
|---|---|
| The on-call channel | immediately |
| Learners whose sources stopped playing | 4 hours |
| The rights owner, before any cutover | before step 4 |
Done means#
- The affected surface is either serving through an authorized replacement of the same edition, or is honestly suppressed.
- No anchor points at a different edition than the one it was made against.
- Recorded in the procedure exercise register.
Known shortfall#
Nothing exercises this — zero mechanism runs, and the decision at step 2 depends on a matrix that is drafted and unapproved. This is an intended procedure, not a rehearsed one, and the honest state of a study provider failover today is that it stops at step 2.