# Eval dataset assessment: assistant-deck-regression, version 2026-10-04

- **Task:** `main-tests-green:eval-registry-stale-and-model-registry-vs-probe-2026-10-04`
- **Decision followed:** question `62d9e81b-0de2-420a-9e41-3496685c7e68`, answered by `eve-elikem-delegate-01` on 2026-10-04 (option 2, with a substantive review rather than a scan-only rebind).
- **Assessed on:** 2026-10-04. Revised the same day after Eve review `8c27fbfe-f854-4c63-b88a-4d0b7c819d64` returned the first submission for one finding (advisory cases and the pooled family gate); the section "Which gates the new cases reach" is the correction.
- **Assessor:** Claude Opus 5.5, working as the Eve identity `claude-board-worker` on the Linux dev server. This is an agent's assessment. **No individual owner re-attestation is claimed**: the roles the registry names (Evaluation Science Lead, Privacy Lead, Governance Lead, Security Lead) stay as accountability metadata and none of them has reviewed this version.
- **Candidate commit:** `9d3c9b941ac8e70965492ee0ae922907cab443e1`.
- **Status of the candidate:** registered and **quarantined**. It is admitted for nothing until this assessment and the exact candidate are independently reviewed.

## What changed

The registry (`docs/audits/eve-sota-eval-data-governance/registry.json`) binds each dataset artifact by byte count, SHA-256 and Git blob. One artifact of `assistant-deck-regression` had changed since its binding, so the registry refused to verify and `admit-eval-dataset.mjs` refused every admission, for every dataset.

| | Predecessor (version 2026-09-13) | Candidate (version 2026-10-04) |
| --- | --- | --- |
| Registry | Git commit `a2c4c72fdd8`, registry blob `8df44174e5a` | commit `9d3c9b941ac`, registry blob `40973b915b2` |
| `apps/oshun/bff/src/assistant/evals/deck-builder-cases.ts` bytes | 54,695 | 59,257 |
| SHA-256 | `e6ebbd38d8fdf535aaff275c4c9398f69a4b6da836ec1d9183446aa2187d519e` | `b7f09ab27cfe545fe478612d157f634e98affb7e91c930e62e5faed4d65cce89` |
| Git blob | `1af9cb68fe0a2295d8a46e502a8176b6abe3f335` | `784d88b6567b1207dd8e7b15eb21a0b97f2e9e67` |

The other five artifacts of the dataset are unchanged. The predecessor is not rewritten: it is the registry as Git holds it at the commit above.

## Scope of the assessment

The whole difference between the two blobs of `deck-builder-cases.ts`: 113 added lines and no removed or changed line, made by three commits of 2026-09-30 (`dec5810fe49`, `4a662cd6034`, `8010e0d73b5`). Every added line was read. They are two advisory-reason constants and six eval cases:

- `builder-wbw-draft-outcome-reject`, `builder-wbw-draft-outcome-defer` (G10.1: a decision draft carries an outcome);
- `builder-wbw-approve-plan`, `builder-wbw-decide-not-dispatch` (R06.5: approving and dispatching a plan revision);
- `builder-operations-named-render`, `builder-operations-file-failed-build` (R25.4: reading an operation's status).

## Findings

**Provenance and licence.** All three commits are by the repository's own worker identity (`Claude (board worker)`), written for board tasks G10.1, R06.5 and R25.4. The cases are authored in this repository; nothing is copied from an external corpus, a provider's output or a customer record. The dataset's rights entry (repository-authored, Apache-2.0) describes them. No change.

**Personal or sensitive data.** The six prompts are invented operator requests ("Draft a decision titled "No second job queue"…", "Did the Q3 board deck render finish?"). They name no person, account, address, credential or customer. The identifiers in them (`plan-rev-drawer-cards-2`, `goal-drawer-cards`, the family `eve-gap-analysis`) are made-up plan and goal ids and a board family name. The automated scan of the complete artifact bytes (`privacyScan`, patterns `eval-data-sensitive-v2`) reports: high-confidence secret matches 0, email-like matches 0, personal-data matches 0, encoded-sensitive matches 0. Privacy classification unchanged: synthetic or repository-public, no redaction required.

**Contamination and split exposure.** The dataset is a `tune` split and stays one. The six new case ids and six new prompt texts were searched for in every other artifact the registry binds (21 artifacts in 11 datasets, including both held-out sets, the evaluator-only acceptance set and the graded relevance record): no occurrence. The new cases are marked `advisory` in the file itself ("gating waits on the next k=10 measurement"). That exempts each of them from the per-case assertion only; five of the six still count toward a pooled family gate, which the next section sets out. Nothing here was drawn from, or copied into, a held-out or graded set. The dataset's existing limitation stands: its cases are referenced by assistant skill `evalCaseIds`, so it is not independent release evidence.

## Which gates the new cases reach

The first revision of this assessment said that, being advisory, none of the new cases gates a release. That was wrong, and review 8c27fbfe said so. The runner (`apps/oshun/bff/src/assistant/evals/assistant-golden.eval.ts`, unchanged since the candidate commit) has two different gates:

- **The per-case gate.** It asserts that every case in `aggregates` passed all its runs. An advisory case is put in `advisoryAggregates` instead, and a held-out case in `heldOutAggregates`, so neither can fail this assertion. The six new cases are outside it.
- **The pooled family gate.** `gradedAggregates = [...aggregates, ...advisoryAggregates]` is rolled up by family and compared with the Wilson floors stamped in `docs/audits/eve-smx-ratchet.json` (`builderWilsonFloors`). On a full-deck run a family whose pooled lower bound is under its floor, or which is missing, fails the run ("builder-family Wilson ratchet breached"). Advisory cases are in this pool; only held-out cases are kept out. A family with fewer pooled runs or cases than its floor was stamped on is reported as insufficient and does not fail.

Where the six cases fall, counted from the deck at the candidate commit (263 cases):

| Case | Family | Per-case gate | Pooled family floor |
| --- | --- | --- | --- |
| `builder-wbw-draft-outcome-reject` | workbench-write | not asserted (advisory) | counted |
| `builder-wbw-draft-outcome-defer` | workbench-write | not asserted (advisory) | counted |
| `builder-wbw-approve-plan` | workbench-write | not asserted (advisory) | counted |
| `builder-wbw-decide-not-dispatch` | workbench-write | not asserted (advisory) | counted |
| `builder-operations-file-failed-build` | workbench-write | not asserted (advisory) | counted |
| `builder-operations-named-render` | presentation | not asserted (advisory) | none: no floor is stamped for the presentation family |

Floors are stamped for three families only: docs, workbench-read and workbench-write.

**Contribution to the workbench-write floor.** The floor is 0.875, the Wilson lower bound of 229 passed runs in 250 over 25 cases. The family now has 37 cases, 5 gating and 32 advisory, none held out; the five new cases are five of the 32. At k=10 the pool is 370 runs, of which the five new cases are 50 (13.5%). Holding the other 32 cases at the stamped pass rate (293 of 320, lower bound 0.88) and varying only the five new cases:

| Runs the five new cases fail, of 50 | Pooled runs passed, of 370 | Wilson lower bound | Against the 0.875 floor |
| --- | --- | --- | --- |
| 0 | 343 | 0.8959 | holds |
| 5 | 338 | 0.8804 | holds |
| 6 | 337 | 0.8773 | holds |
| 7 | 336 | 0.8743 | breach |
| 10 | 333 | 0.8652 | breach |

So the five cases can fail a full-deck run on their own: seven failed runs among their fifty, with every other case at its stamped rate, puts the family under its floor, although no case assertion fails. They can also carry the family: if they pass every run the bound rises to 0.8959. This arithmetic is computed with the runner's own `wilsonInterval`; it is a what-if, not a measurement. **No k=10 run of the six cases is recorded in this assessment**; their real pass counts are unknown here.

**What this means for the dataset's use.** These are tune cases: the prompts and tools they exercise are adjusted against them. Because they are pooled into the workbench-write floor, that floor moves with tuning on them, and a full-deck pass or breach of it is not independent evidence about a release. This is the dataset's existing limitation, and it holds with more weight than the first revision gave it: the limitation covers the family floor these cases feed, not only the cases. The allowed uses (development, regression, advisory) and the forbidden uses (graded, held-out, production-training) are right as they stand and are not changed. A change to which pooled gates advisory cases feed would be a change to the runner and the ratchet, outside this dataset's assessment; none is proposed here.

Nothing else found is unresolved.

## What the candidate changes and what it keeps

Changed, in `tools/eve-everywhere/eval-data-governance-policy.mjs` and the registry together: the dataset's `version` (2026-10-04), its `freshness.reviewedOn` (2026-10-04, the date of this assessment), the artifact's bytes, SHA-256 and Git blob, its `status` (quarantined while this is under review) and one added limitation saying so.

Kept: the `tune` split, `promotionEligible: false`, the allowed uses (development, regression, advisory), the forbidden uses (graded, held-out, production-training), the retirement triggers, the expiry (2027-03-13), the named roles, the global governed-on date (2026-09-13) and every other dataset.

## Checks run at the candidate commit

- `node tools/eve-everywhere/verify-eval-data-governance.mjs`: "Eval-data governance verified: 11 datasets, 21 artifacts."
- `node --test tools/eve-everywhere/verify-eval-data-governance.test.mjs`: 21 passed, 0 failed (18 passed, 2 failed before).
- `admit-eval-dataset.mjs --dataset=assistant-deck-regression --use=regression --as-of=2026-10-04`: refused, "dataset status is quarantined".
- `admit-eval-dataset.mjs --dataset=engineering-benchmark-heldout --use=held-out --as-of=2026-10-04`: admitted (it was refused while the registry did not verify).
- With the candidate's status set to admitted in a copy (what approval would do): regression and development are admitted on and after 2026-10-04; refused before that date and after 2027-03-13; held-out, graded and production-training are refused.

## What a reviewer is asked to decide

Whether this assessment and the exact candidate bytes above are sufficient to return `assistant-deck-regression` version 2026-10-04 to `admitted`. That change is a commit of its own after approval. This document asks for no publication or promotion.
