Disciplines · Proposals

Study & Deconstruction Workspace — Incident Runbooks (YSD-4100)

condition has stopped getting worse — access closed, the leaking path shut, the prompt surface disabled, the key rotated — whichever the class's Containment & eradication step names.

10sections14 minread
Markdown source
On this page

Response procedures for the security incident classes named in YSD-4100. Each runbook gives: Detection (the live alert where one exists, or the operational query/check where the signal is not metered), Immediate response, Containment & eradication, Recovery, Tell the people affected, and the owning mechanism (the code that either prevents the incident or supports the response). Prometheus alerts live in apps/yemaya/svc-study-workspace/observability/study-workspace.alerts.yml; every alert runbook: annotation links to a section here.

On the communication step (YSD-18163's tenth leg). Until 2026-08-09 these runbooks had no step in which anybody outside the response team was told anything, and alerting.ts routes every condition to page, ticket or notice — three destinations, all inside operations. So an authorization leakage was closed by rebuilding the projection, quarantining the read surface and re-running the leakage suite, and the person whose material had been reachable was told by nothing. The platform's customer channel was already finished and had only ever carried words somebody remembered to type.

Every class below now carries either a Tell the people affected step or a Nobody outside operations finding with its reasoning. Do not compose the wording during the incident: deriveIncidentNotice (libs/yemaya/study-workspace/src/observability/incident-communication.ts) returns the audience, the severity, the deadline and the message — inside the 400 characters the banner contract allows — for each class, and obligationStanding reads back afterwards whether what went out actually reached the people it was about. The deadline is 30 minutes from detection, not from the moment somebody picked the alert up, borrowed from the SEV-1 cadence in docs/runbooks/v6-capacity-management.md. Publish through POST /v1/admin/communications/banners; a person presses publish, because a customer-facing statement an unattended rule can send is one nobody read before it went out.

Know how the notice is delivered before you rely on the step. The derived notice is scoped to the tenants the incident happened in. GET /v1/communications/banners is public — so that a status page survives an auth outage — and therefore serves untargeted banners only, dropping every one carrying audienceTenantIds or audienceRoles: a public reader has no viewer, and serving a tenant-scoped banner there would announce one institution's incident to every other.

Since 2026-08-15 there is a reader that does have a viewer. GET /v1/communications/banners/for-me is authenticated, takes the audience from the principal's own tenant claim, and the customer shell asks it whenever it holds a token — so a tenant-scoped banner now reaches signed-in members of that tenant without their asking. One thing it does not settle, and it decides whether this step works: the study workspace is single-tenant on a deployment constant (STUDY_TENANT_ID), and nothing in the repository states that that string is the string a learner's platform tid claim carries. Confirm for THIS deployment that the tenant id you are targeting is the one its members' tokens carry, before treating the banner as delivered.

Whatever the answer, also send the same words to the institution administrator by the contact route the tenancy agreement names, and record both in the incident: a signed-out learner is reached by neither reader, and a notice nobody can see is the failure this step exists to prevent rather than a lighter version of it.

The objective#

Containment objective: 60 minutes from detection, for every class below: the condition has stopped getting worse — access closed, the leaking path shut, the prompt surface disabled, the key rotated — whichever the class's Containment & eradication step names. Set 2026-08-15. Recovery may legitimately take longer; containment is the part that decides how much of it there is to do.

The thirty minutes is a different promise and is not this one. "Tell the people affected within 30 minutes of detection" is a deadline for telling somebody, and the drill checks it as a verification — that everyone the plan names was told inside the window it gives. A response that told everybody inside half an hour and was still containing the leak at teatime met that promise and missed this objective, which is why the two numbers are stated apart and neither is derived from the other.

A note on honesty: several incident classes have no live metric because the signal is not (yet) metered — malware escape has no scanner (planned YSD-4093), prompt injection has no model surface (YSD-4096), signing has the not-inspected seam (YSD-4092). For those the Detection step is a real operational procedure — an audit-ledger check, a state query, a re-scan — never a fabricated alert.

authorization-leakage#

A record reachable across a tenant, project, or rights boundary that should have denied it.

  • Detection: alert StudyProjectionPermissionInconsistent (study_projection_permission_inconsistent_works > 0) — a projection row whose visibility disagrees with the rights gate. Supporting: StudyProjectionLagHigh (a revocation may not have propagated yet). Cross-check the tamper-evident audit ledger (YSD-3060) for the offending record's playback-authorized / search-performed events.
  • Immediate response: identify the affected works from the gauge's scope. Force a projection rebuild for those works (requestReprojection, YSD-3062) to re-derive visibility from current rights. If a tenant boundary is implicated, confirm the store binding — every read is tenant-scoped at the store (YSD-4091); a leak here means a caller constructed a store for the wrong tenant.
  • Containment & eradication: quarantine the affected read surface (disable the search/vector/graph route) until the reprojection completes and the gauge returns to zero. Revoke any grant that was resolved incorrectly and let the rights-epoch cache invalidate.
  • Recovery: verify with the zero-leakage integration suite (permission-leakage.integration.spec.ts) against the affected tenant before re-enabling the surface.
  • Tell the people affected: within 30 minutes of detection, publish the authorization-leakage notice from deriveIncidentNotice to every tenant in the gauge's scope — a critical, undismissable banner. It is owed to the people whose material was reachable, to the rights holders whose terms it was held under, and to the institution administrators who may have a reporting duty of their own; it says that search and graph are switched off until every view is re-derived, and asks them to revoke links they shared outside their project, which is the one thing only they can do. Do not wait for the reprojection to finish: the deadline runs from detection.
  • Owning mechanism: tenant-scoped store reads (YSD-4091), the central rights gate (rights-gate.ts), projection health gauges (projection-health.ts), the audit ledger (YSD-3060).

malware-escape#

Malicious bytes reaching a parser or escaping quarantine.

  • Detection: no live scanner yet (planned YSD-4093) — there is no malware_detected metric to alert on, and claiming one would be false. Detection today is operational: (a) StudyServerErrorSpike / StudyProjectionFailingRows firing right after an ingest, and (b) the audit ledger showing an ingest-denied cluster or an unexpected source-ingested for an unusual container. Archives are refused outright (YSD-3022), so the decompression-bomb vector is closed by construction.
  • Immediate response: stop the ingest workers. Do not run the file through any parser again. Isolate the object-store prefix holding the suspect original.
  • Containment & eradication: hold the source in quarantine (quarantine-review), never promote it. Snapshot the file for offline analysis; purge it from the hot object store. Confirm no derivative was generated (representation generation is rights- and review-gated).
  • Recovery: resume ingest only after the file is removed and the container type added to the refusal set if warranted. When the scanner ships (YSD-4093) wire a study_malware_quarantined_total counter and a companion alert.
  • Tell the people affected: within 30 minutes of detection, publish the malware-escape notice from deriveIncidentNotice. Stopping the ingest workers stops everybody from adding sources, so the notice goes to the whole affected scope rather than to the uploader alone, says that everything already here can still be studied, and asks the uploader to check any copy of the file they keep elsewhere — this workspace can only quarantine its own copy.
  • Owning mechanism: archive refusal + media limits (YSD-3022/4095), DRM refusal (YSD-4030), quarantine review, parser timeout (YSD-4095).

parser-crash-loop#

A crafted file repeatedly crashing or hanging the media parser.

  • Detection: alert StudyServerErrorSpike (5xx ratio > 5%). The prober enforces a hard wall-clock timeout and output-buffer ceiling (YSD-4095), so a hostile file surfaces as a bounded error, never an unbounded hang.
  • Immediate response: identify the offending source from the ingest route logs (requestId/traceId join). Pause that source's ingest.
  • Containment & eradication: the timeout already bounds blast radius; quarantine the file and confirm the worker recovers (in-flight gauge drains). If a specific codec/container is implicated, tighten the media limits allowlist.
  • Recovery: resume ingest; watch the 5xx ratio return to baseline.
  • Tell the people affected: within 30 minutes of detection, publish the parser-crash-loop notice from deriveIncidentNotice — a dismissable warning, because there is nothing for the reader to do and every other source is unaffected. It exists so that the person whose upload stopped moving learns that it was stopped deliberately rather than sitting in a queue.
  • Owning mechanism: parserTimeoutMs / parserMaxOutputBytes (media-limits.ts, consumed by media-prober.ts), RED metrics.

prompt-injection#

Untrusted transcript/subtitle/metadata content steering a model.

  • Detection: no model surface exists yet (YSD-4096), so there is no prompt-injection metric — the honest detection is design-time: the analysis-method surface is a fixed safe union validated against the YSD-4052 prohibition at wiring time, and model outputs are always the model-suggestion layer (YSD-4071). Any PR adding a model call is the detection point; review it against the untrusted-content boundary in the threat model.
  • Immediate response (once a model surface ships): if injected content is suspected, disable the affected analysis method (remove it from the safe union — a wiring-time check, not a runtime toggle).
  • Containment & eradication: the isolation seam (YSD-4096, planned) must wrap untrusted content as non-executable data delimited from the system prompt and deny it tool access; until it ships, no untrusted content may reach a model.
  • Tell the people affected: within 30 minutes of detection, publish the prompt-injection notice from deriveIncidentNotice and name the window. This one is not optional and not merely courteous: the epistemic model here lets a person promote a machine suggestion into a claim (YSD-4071), so a steered suggestion may already be carrying somebody's argument. The notice says their own observations are untouched and asks them to re-read anything from the window before promoting it; open the affected-decision reviews for suggestions already promoted, and tell the teachers who set work on them.
  • Owning mechanism: analysis-method safe union (YSD-4052), model-suggestion layering (YSD-4071), safe-language checks (YSD-4070).

provider-compromise#

A linked external provider is compromised or serving hostile content.

  • Detection: no live metric — provider sources are link-only, no bytes are fetched (YSD-4009), so the workspace has no provider-response telemetry to alert on. Detection is external (provider advisory) or via a source-takedown audit event. Signed provider URLs are stripped from exports (YSD-4097), so a compromised provider cannot ride a live credential out of an export.
  • Immediate response: mark affected provider-linked assets as unverified; suppress their deep links from new outputs.
  • Containment & eradication: issue takedowns (source-takedown) for the affected works; rotate any provider credentials held by the deployment. Because exports carry only bare deep links, no re-scrub of past exports is needed for credentials — but re-verify link validity.
  • Tell the people affected: within 30 minutes of detection, publish the provider-compromise notice from deriveIncidentNotice. Because the model is link-only, nothing hostile ever entered this workspace and the danger is entirely on the other side of a link the reader may follow — including links inside bundles exported before today, which is why the notice reaches everybody in scope and not only people who opened one recently.
  • Owning mechanism: link-only provider model (YSD-4009), export credential stripping (YSD-4097), takedown flow.

leaked-export#

An export carrying a secret, token, signed URL, or rights-denied content.

  • Detection: the export path is preventive — signed URLs are credential-stripped and free text is secret-redacted at build time (YSD-4097), and works without an export-permitting grant are excluded (YSD-1035). There is no runtime leak metric; detection of a suspected leak is a re-scan: run containsSecret / stripUrlCredentials over the stored export document, and diff its included works against the current rights decisions.
  • Immediate response: revoke the export object (delete from object store), and if it was distributed, treat any embedded credential as compromised and rotate it.
  • Containment & eradication: if the leak was a live signed URL, rotate the signing key/credential at the storage provider. If it was a rights-denied work, confirm the grant state and re-run the export.
  • Recovery: re-issue a scrubbed export; add any newly-recognized secret shape to SECRET_PATTERNS and any new signing param to the strip list.
  • Tell the people affected: within 30 minutes of detection, publish the leaked-export notice from deriveIncidentNotice — critical and undismissable. It goes to the whole affected scope rather than only to people known to hold a copy, because the store knows who downloaded the bundle and not who they forwarded it to. Revoking the object does nothing about copies already taken, so the notice asks for those to be deleted and says a scrubbed bundle is being issued. The rights holder whose work went out without a grant hears it from us.
  • Owning mechanism: leakage scrub (leakage-scrub.ts, YSD-4097), rights-filtered export (YSD-1035/4032).

signing-failure#

A content-credential or replay-bundle signature that is missing, forged, or unverifiable.

  • Detection: replay-bundle integrity is a real sha256 anchor (bellona adapter) and mismatches fail loud. Content-credential signing and rotation are planned (YSD-4092); until the inspector ships the workspace records the honest contentCredentials: { status: 'not-inspected' } seam, so a "verified" claim is never fabricated. Detection: a bundle whose recomputed sha256 does not match its anchor.
  • Immediate response: reject the bundle; do not present unverified content as signed. Flag the record for review.
  • Containment & eradication: when signing ships (YSD-4092), a signing failure means a key/HSM problem — fail closed (refuse to emit signed outputs) rather than emitting unsigned-as-signed.
  • Tell the people affected: within 30 minutes of detection, publish the signing-failure notice from deriveIncidentNotice, naming the window the unverifiable bundles were issued in. Failing closed keeps a false "verified" off new outputs and says nothing about the bundles already handed over: those still open and still read as citable. The notice asks that none be cited as verified until it is downloaded again.
  • Owning mechanism: replay-bundle sha256 anchor, not-inspected content-credentials seam (YSD-4092).

stuck-deletion#

A deletion saga that has stalled or is reading as complete while data remains.

  • Detection: no gauge yet — the honest detection is a state query on the saga store for rows in the terminal needs-manual-review state or stalled past the service-objective timer. The saga is designed so it can NEVER read as done while incomplete (YSD-3063): a stall lands in needs-manual-review, not a false "deleted".
  • Immediate response: query the deletion-saga table for needs-manual-review and long-running in-progress sagas. For each, read the per-store acknowledgement state to see which store (originals, derivatives, projections, exports) did not acknowledge.
  • Containment & eradication: the saga is resumable — re-drive it for the unacknowledged store. If a legal hold blocked deletion, that outranks deletion by design (YSD-3023); confirm the hold before forcing.
  • Recovery: confirm the tombstone and that every store acknowledged; the saga transitions to complete only then.
  • Tell the people affected: within 30 minutes of detection, publish the stuck-deletion notice from deriveIncidentNotice to the person who asked for the deletion, the rights holder it was promised to, and the institution administrator. This is the class that proves impact is the wrong test: nothing the person can see has changed, no surface is missing, and an impact-only rule would let it pass in silence. What broke is a promise, and the only person who can decide whether the delay matters — a licence that lapses on a date, a subject who withdrew consent — is the one being told. Say the copies are still held; do not let them read a completion into the silence.
  • Owning mechanism: durable resumable deletion saga (YSD-3063), legal hold precedence (YSD-3023), expiry cache/URL invalidation (YSD-3062).

Supporting operational runbooks#

These back the non-security alerts that nonetheless gate incident response (a broken projection pipeline can mask a rights change).

projection-pipeline-failure#

  • Detection: StudyProjectionFailingRows, StudyProjectionStaleRows, StudyProjectionBacklogGrowing.
  • Response: inspect the failing consumer (search/vector/graph) via assessProjections; the outbox is at-least-once, so re-drive the per-consumer drain for the affected tenant. A persistent failure blocks rights/deletion propagation — escalate to authorization-leakage if a revocation is pending.
  • Tell the people affected: within 30 minutes of detection, publish the projection-pipeline-failure notice from deriveIncidentNotice. Stale search is the visible half; the half worth the notice is that a permission change or a deletion somebody just made may not have reached these views yet, so a teacher who revoked access believes it took effect when it has not. The notice says opening the source itself is current, and asks them to do that before acting on a search result.
  • Owning mechanism: transactional outbox + per-consumer drains, projection health (projection-health.ts).

resource-exhaustion#

  • Detection: StudyRequestSaturation, StudyRequestLatencyHigh.
  • Response: identify the slow route; check downstream (Postgres, object store) health via /ready. The prober timeout bounds a single parse, so sustained saturation points at a dependency stall or a request flood.
  • Tell the people affected: within 30 minutes of detection, publish the resource-exhaustion notice from deriveIncidentNotice — a dismissable warning. Load shedding is indistinguishable from a broken workspace to somebody whose request just failed; the notice says work already open keeps working and that a failed request is safe to retry, which is the difference between waiting and starting again.
  • Owning mechanism: RED metrics, readiness checks, parser limits.

capacity#

  • Detection: StudyEventVolumeHigh.
  • Response: review telemetry retention and the per-append batch cap; confirm the dedicated event store remains justified (study_event_dedicated_store_justified).
  • Nobody outside operations: the only class of the eleven where the answer is that no one is owed a word, and it is recorded here so that the silence is a finding rather than an omission. The signal asks whether the dedicated event store is still justified and whether telemetry retention should change; no surface, promise or artifact belonging to a person studying here is touched. A retention change that would alter what they keep is not made here — it is made through the retention policy, which gives its own notice.
  • Owning mechanism: event store volume gauges (YSD-3043).