# Eve SOTA eval-data governance — 2026-09

Task 12.5 is implemented as a generated, byte-bound registry plus a fail-closed
semantic verifier and per-use admission CLI. The governing decision is
[ADR-0085](../adr/ADR-0085-eve-eval-data-governance.md).

## What is governed

The registry independently discovers and accounts for every tracked Eve
evaluation case, corpus, label sheet, measured response set, evaluator oracle,
and synthetic contract fixture in the bounded
`apps/oshun/bff/src/assistant/evals` and named `docs/audits` namespaces. Each
artifact has byte length, SHA-256, Git blob identity, dataset version, source
and licence posture, privacy/redaction status, split and allowed use,
contamination rules, review/expiry/retirement policy, accountable roles, and
retention requirements.

The registry separates tuning/regression, held-out, graded, calibration,
reference, evaluator-only, and contract-fixture material. A use must be
explicitly allowed. Held-out material may not overlap or permit train/tune use;
evaluator-only oracles remain candidate-hidden. Promotion additionally requires
an admitted, current, exactly hashed, licence-approved, privacy-reviewed,
promotion-eligible dataset.

## Honest holds

- The ordinary deck and operator-memory regression cases are development/tuning
  evidence, never independent graded or held-out evidence.
- The assistant held-out deck remains a non-gating drift set under its existing
  harness contract.
- SMX provider transcripts are quarantined until provider-output rights and the
  actual blinded human labels are recorded. Task 12.3 remains open.
- Operator-memory live output is quarantined because provider-output rights are
  unresolved and its proposed labels explicitly say they are machine-authored
  and ineligible as human evidence.
- The docs relevance record remains quarantined as graded evidence while five
  semantic classes await human labels.
- Synthetic trajectory fixtures prove contracts only and cannot promote a model
  or family.

## Verification boundary

The verifier checks total inventory ownership, live hashes, Git blobs,
source/licence, privacy/redaction, split isolation, known label holds,
sensitive-secret patterns, freshness, ownership, retention, and exact Task 12.5
bindings to the NIST GenAI IP/media and NIST AML poisoning crosswalk entries.
Targeted negative controls demonstrate rejection of omissions, hash/version
drift, unresolved licences, privacy bypass, personal-data admission, split
overlap, held-out tuning, quarantine promotion, expiry, missing owners, weakened
evidence retention, crosswalk drift, and machine-label promotion.

Passing these checks proves the governance contract and current registry state
only. It does not prove evaluator validity, human agreement, real model quality,
or release admission.