Disciplines · Runbooks

Runbook: V1 Signing Key Compromise (gd:signing-key-compromise)

from outside it (a leaked build artifact, a public repository, a provider notice).

13sections6 minread
Markdown source
On this page

Owner: delivery pipeline on-call (rota:delivery-pipeline). Last reviewed: 2026-08-14. Version: 1.

A signing key is disclosed. The material is intact and no longer trustworthy, which is why this is a corruption and not an outage. Rotation is not reversal: every signature made under the disclosed version stays disclosed, so this scenario has a recovery procedure and no rollback.

Read this first, because it changes what the rest of the runbook can promise. verifyAudioC2paManifest in apps/oshun/bff/src/generation/audio-c2pa-signer.ts verifies the Ed25519 signature against manifest.publicKeyPem — the public key carried inside the manifest being verified. Nothing on this estate compares signerKeyId or publicKeyPem against an allowlist; dep:trust-store is registered as a core dependency in the S10.5 register and has no implementation.

Two consequences, and both are the incident rather than a footnote to it:

  1. Revocation is not enforceable. Rotating the key does not invalidate a single existing manifest. Nothing reads a revocation list because there is no revocation list.
  2. The disclosed key mints manifests that verify. Anyone holding it produces output indistinguishable from ours, and the estate's own verifier admits it.

Rotation therefore protects future output only. Say so when communicating, because an audience told "the key was rotated" will hear "the old signatures are invalid", and they are not.

1. Detection and declaration#

Detection signal. Disclosure is not observed by the estate — it is reported from outside it (a leaked build artifact, a public repository, a provider notice). There is no in-estate detector, and there cannot be one for a key that still works.

  • Detection authority: delivery pipeline on-call, contact rota:delivery-pipeline.
  • Declaration authority: production operations duty officer, contact rota:prod-ops.
  • Gap, stated. docs/operations/on-call.md describes a weekly primary/secondary rotation and names no person and no rotation config; none of the register's rota: identifiers appears in it or in docs/operations/alert-routing.md. The name above is registered; the route from it to whoever is carrying the pager is not.
  • Record the disclosure instant, not the discovery instant, if they differ. The exposure window is what decides which signatures are suspect, and it opens when the key left, not when somebody noticed.

2. Communications#

Audience Within Content
Actors 15 min that content credentials issued before the rotation cannot be relied on to prove origin
Operators 5 min that this runbook is running, and by whom
Regulator 72 h only where a disclosure is reportable; the estate has no automatic determination of this

All travel over dep:notification-channel, which this scenario leaves standing.

Do not describe rotation as revocation. See the header.

3. Dependency assumptions#

  • dep:primary-store is healthy: release rows carry the key version their signature was made under, and that column is the only record of which output was signed under the disclosed key.
  • dep:blob-store is healthy: re-signing anything requires the bytes.

4. Recovery infrastructure#

In place, with one exception that is the point. The new key material must be generated outside the compromised estate — on an operator-controlled machine — and delivered into the BFF environment. A key generated on the host whose disclosure is under investigation is a key of unknown provenance.

5. Secrets#

  • OSHUN_C2PA_SIGNING_KEY (PKCS#8 Ed25519 private key) and OSHUN_C2PA_SIGNER_KEY_ID (default oshun-audio-c2pa-v1), read by resolveAudioC2paSigner.
  • OSHUN_LIVING_SCENES_C2PA_KEY_ID, read by apps/oshun/bff/src/routes/living-scenes.ts. The estate has more than one signing key id. A rotation that changes one and not the other leaves half the output signed under the disclosed identity.

Both are recoverable out of band, and here that is a requirement rather than a convenience: the replacement is generated off the estate.

6. Restore order#

  1. dep:trust-store — ABSENT. This is where the revocation entry would be published, and there is nothing to publish it to. The step is listed rather than dropped so that the order states what is missing instead of implying it was done.
  2. dep:signing-service — set the new OSHUN_C2PA_SIGNING_KEY, set a new OSHUN_C2PA_SIGNER_KEY_ID (do not reuse the disclosed id — the id is how a later reader tells the two apart), set OSHUN_LIVING_SCENES_C2PA_KEY_ID, and restart the process that reads them.

The order is revoke-then-re-key and not the reverse: re-keying first means a window in which the new key is live and the old one is still advertised as trustworthy. On this estate that window never closes, because step 1 does not exist.

7. Validation#

  • Sign a known media file under the new key and verify it with verifyAudioC2paManifest — media hash, canonical hash, signature.
  • Confirm the new signerKeyId appears on newly issued manifests and the disclosed one does not.
  • Confirm both key ids moved: the generation signer and the living-scenes signer.

The check that cannot be written. "A manifest signed by the disclosed key is refused" is not assertable, because nothing refuses it. Until a trust list exists, treat every pre-rotation manifest as unverified provenance rather than as invalid provenance — the estate cannot tell the two apart.

8. Fencing — ABSENT for keys, and this scenario is what fencing is for#

A key id in an allowlist is the fencing token this scenario needs — the thing a holder of the old key is rejected by, rather than instructed about. There is no such list, which is the reason §7 has a check it cannot write and §2 has a message it cannot soften.

The estate knows how to do this; it just has not done it here. live_media_pipeline_job_attempts.lease_token (libs/shared/live-media/src/sql-media-pipeline-job-store.ts) is a real fencing token: every heartbeat, lease expiry and completion carries AND lease_token = $4 AND status = 'leased' in its UPDATE ... WHERE, so a worker holding a superseded lease updates zero rows. A superseded key has no equivalent, and the shape of the fix is the same — the check has to be on the side that accepts the work, not on the side that produces it.

9. Failover#

Nothing is promoted; there is one signing path before and after. The change is which key it holds.

10. Reconciliation#

Enumerate what was signed under the disclosed key. The release rows carry the key version (dep:primary-store → dep:signing-service in the reference graph), so the population is queryable — that column is what makes this scenario recoverable at all.

Re-signing that population under the new key does not retire the old signatures; both verify. Re-sign only where a consumer will fetch the newer manifest, and record which artifacts were left under the disclosed key.

11. Failback#

There is no failback. The estate stays on the new key; returning to the disclosed one is the incident. The nearest thing to a return is the point at which the last artifact still advertising the old key id has been re-signed or retired, and that is a backlog item rather than a step.

12. What this runbook does not cover#

  • A key that is merely missing. That is v1-dr-signing-service-outage.md, and its recovery is restoring the same key.
  • Provider credentials. scripts/isis/rotate_runpod_credentials.sh rotates those; they are not content-signing material.
  • Building the trust store. That is the gap this runbook names and does not close.

13. Machine-checkable statement#

json
{
  "id": "runbook:v1-signing-key-compromise",
  "scenarioId": "gd:signing-key-compromise",
  "version": "1",
  "detectionAuthority": {
    "role": "service on-call",
    "name": "delivery pipeline on-call",
    "reachableOver": ["dep:notification-channel"]
  },
  "declarationAuthority": {
    "role": "operations duty officer",
    "name": "production operations duty officer",
    "reachableOver": ["dep:notification-channel"]
  },
  "communications": [
    {
      "audience": "actors",
      "channelDependencyId": "dep:notification-channel",
      "withinMs": 900000
    },
    {
      "audience": "operators",
      "channelDependencyId": "dep:notification-channel",
      "withinMs": 300000
    },
    {
      "audience": "regulator",
      "channelDependencyId": "dep:notification-channel",
      "withinMs": 259200000
    }
  ],
  "assumes": ["dep:primary-store", "dep:blob-store"],
  "recoveryInfrastructureDependencyId": null,
  "secrets": [
    {
      "id": "OSHUN_C2PA_SIGNING_KEY",
      "heldInDependencyId": "dep:signing-service",
      "recoverableOutOfBand": true
    },
    {
      "id": "OSHUN_LIVING_SCENES_C2PA_KEY_ID",
      "heldInDependencyId": "dep:signing-service",
      "recoverableOutOfBand": true
    }
  ],
  "restoreOrder": ["dep:trust-store", "dep:signing-service"],
  "fencing": null,
  "promotesAStandby": false,
  "validation": [
    "a file signed under the new key verifies through verifyAudioC2paManifest",
    "newly issued manifests carry the new signerKeyId and not the disclosed one",
    "both signing key ids moved: the generation signer and the living-scenes signer"
  ],
  "failback": "none: the estate stays on the new key and returning to the disclosed one is the incident; the nearest equivalent is retiring the last artifact still advertising the old key id"
}