# Runbook — Provenance Failure

> Provenance generation, signing, verification, or ledger commit failed for
> assets produced or curated by Oshun. Use to block publish/share, regenerate
> provenance bundles, verify the checksum ledger, and coordinate with the rights
> and compliance teams.

## Scope

Provenance covers:

- C2PA / content-credentials manifests for generated images, video, audio, and
  3D assets produced by Isis.
- Source-binding bundles attached to grounded Sophia answers and the
  Veritas/Tara/Nisaba/Metis evidence rails.
- Provenance ledger entries for Nisaba editions, manuscripts, and translations.
- Watermark signatures (handled jointly with
  `watermark-verification-failure.md`).
- DSAR export packages — each export carries an inner provenance manifest
  proving export integrity.

Failure modes:

- Manifest generator throws or produces malformed bundles.
- Provenance signature does not validate against the active signing key.
- Checksum mismatch between manifest and asset.
- Ledger commit fails or the ledger consensus rejects the entry.
- Bound source disappeared from the source-of-truth store (orphaned provenance).
- Cross-version drift — manifest references a version that no longer matches the
  asset bytes.

## Severity classification

| Sev  | Condition                                                                                                                           |
| ---- | ----------------------------------------------------------------------------------------------------------------------------------- |
| sev1 | Customer-visible assets are published with invalid or missing provenance, OR ledger consensus rejects a sustained class of entries. |
| sev1 | A signed manifest is found to be forged or signed with the wrong key.                                                               |
| sev2 | Generator failure rate > 5% sustained; ledger commit lag > 15 min; orphaned-provenance count > 0.                                   |
| sev3 | Single asset rejected by the verifier; no publish/share has occurred for the affected asset.                                        |

Source events: `isis.provenance.bundle_generated`,
`isis.provenance.bundle_failed`, `isis.provenance.ledger_committed`,
`isis.provenance.ledger_rejected`, `sophia.provenance.source_bound`.

## Detection signals

- Alerts: provenance-pipeline panels on `dashboards-service-health.ts`; the
  ledger lag and commit-rejection panels.
- Customer signals: spike in `evidence.provenance.banner_unavailable_shown`,
  `media.provenance.verification_failed_shown`.
- Operator signals: rights/compliance review queue flags a provenance defect.
- External signals: a third-party content-credentials verifier reports an
  invalid manifest publicly.

## Initial triage (first 15 minutes)

1. Acknowledge `oshun-isis-oncall` (and `oshun-compliance-oncall` for ledger
   issues).
2. **Block publish/share immediately** for the affected asset class — enable the
   publish-freeze flag scoped to the class. The freeze emits
   `studio.publish.frozen_for_provenance`.
3. Determine failure boundary:
   - Generator fault → suspect the most recent Isis workflow promotion (delegate
     to `model-workflow-rollback.md` for the workflow rollback).
   - Signing fault → suspect signing-key rotation or HSM availability.
   - Ledger fault → ledger consensus, network, or storage backend.
   - Checksum mismatch → upstream asset corruption or post-sign mutation.
4. Page rights lead if the affected assets are subject to a third-party rights
   agreement; page compliance lead if a regulatory disclosure is in scope (e.g.,
   DSAR export integrity).

## Diagnosis

1. For generator faults: open the Isis provenance pipeline trace for a failing
   job — span `isis.provenance.generate`. The failing sub-step (manifest schema,
   signing call, ledger commit, asset checksum) tells you which mitigation
   applies.
2. For signing faults: confirm the signing key fingerprint matches the active
   key in the key-management record. A rotation drift means the verifier was
   using a stale key — check rotation logs.
3. For ledger faults: inspect ledger consensus health, peer connectivity, and
   commit-batch logs. A peer split implies a consensus partition; a single-peer
   fault implies an availability issue.
4. For checksum mismatches: diff the asset bytes against the bytes recorded in
   the manifest. A mismatch with a recent post-sign step (e.g., transcoder pass)
   implicates that step.
5. For orphaned provenance: a bound source disappeared from the source-of-truth
   store. Inspect the source-store deletion log; a wrongful deletion is the
   prime cause.

## Mitigation

1. **Block publish/share**: already done in triage.
2. **Regenerate provenance bundles** for the affected assets — enqueue
   `isis.provenance.regenerate.requested` for the asset cohort. The regeneration
   uses the certified pipeline; if the certified pipeline is the broken one, the
   rollback in `model-workflow-rollback.md` must precede regeneration.
3. **Verify checksum ledger**: run the ledger reconciliation job over the
   affected window. The job emits `isis.provenance.ledger_reconciled` with the
   per-entry verdict (matched, mismatched, missing). Mismatched entries are
   quarantined and re-signed against the recovered asset bytes.
4. **Signing-key rotation recovery**: if a rotation drift caused the failure,
   re-publish the active key fingerprint to all verifier nodes and re-verify the
   impacted window. Do not roll back the rotation — forward fixes only for
   signing keys (rotation rollback creates a non-repudiation hole).
5. **Ledger fault recovery**: if the ledger backend is partitioned, coordinate
   with the data-platform team to restore consensus. Pending commits queue
   durably; publish remains frozen until commit lag is below SLO.
6. **Customer-visible degraded mode**: while frozen, customer surfaces that
   depend on provenance render the `evidence.provenance.unavailable_shown`
   banner with a "we are verifying" copy block. Source-inspection still works
   against the source-of-truth store; only the cryptographic provenance
   attestation is hidden.

## Communication cadence

| Phase      | Template ID                                | Audience    | Cadence                              |
| ---------- | ------------------------------------------ | ----------- | ------------------------------------ |
| Internal   | `incident.internal.slack.investigating`    | internal    | within 5 min                         |
| Identified | `incident.customer.status_page.identified` | status_page | for customer-visible blocks          |
| Partner    | `incident.partner.email.identified`        | partner     | rights/compliance if external assets |
| Resolved   | `incident.customer.status_page.resolved`   | status_page | within 60 min of unblock             |
| Postmortem | `incident.customer.status_page.postmortem` | status_page | within 5 biz day                     |

The customer-visible copy is owned by compliance; provenance failures touch
disclosure and trust commitments.

## Escalation

- Isis lead is incident commander for generator/signing/checksum failures.
- Compliance lead is incident commander for ledger/regulator-scope failures and
  DSAR-export integrity failures.
- For sev1 forged-manifest cases, page security and legal at minute 0.
- Standard 5/5/15/30/60 escalation otherwise.

## Recovery verification

- Provenance generator failure rate < 0.5% for 30 min.
- Ledger commit lag below SLO for 30 min and consensus peers green.
- Manifest verifier returns valid for a 50-asset post-regeneration sample.
- Publish-freeze lifted; smoke-test 20 freshly-published assets through the full
  verification chain (manifest, signature, checksum, ledger).
- DSAR exports issued during the freeze are reissued where their inner manifest
  was invalid.

## Post-incident

- Postmortem must include: the failing pipeline step, the affected asset cohort,
  the manifest-regeneration record, ledger reconciliation report, signing-key
  history, and any third-party verifier findings.
- Update the provenance evaluation fixtures to cover the failure mode.
- File a rights/compliance notice for any externally-licensed asset whose
  provenance was customer-visible-broken; the notice route depends on the rights
  agreement.
- If the ledger backend partitioned, file a data-platform reliability ticket and
  a ledger-DR drill on the calendar.