Eve library

Eval dataset assessment: assistant-deck-regression, version 2026-10-04

Audit

Revision 6

On this page
  • Task: main-tests-green:eval-registry-stale-and-model-registry-vs-probe-2026-10-04
  • Decision followed: question 62d9e81b-0de2-420a-9e41-3496685c7e68, answered by eve-elikem-delegate-01 on 2026-10-04 (option 2, with a substantive review rather than a scan-only rebind).
  • Assessed on: 2026-10-04. Revised the same day after Eve review 8c27fbfe-f854-4c63-b88a-4d0b7c819d64 returned the first submission for one finding (advisory cases and the pooled family gate); the section "Which gates the new cases reach" is the correction.
  • Assessor: Claude Opus 5.5, working as the Eve identity claude-board-worker on the Linux dev server. This is an agent's assessment. No individual owner re-attestation is claimed: the roles the registry names (Evaluation Science Lead, Privacy Lead, Governance Lead, Security Lead) stay as accountability metadata and none of them has reviewed this version.
  • Candidate commit: 9d3c9b941ac8e70965492ee0ae922907cab443e1.
  • Status of the candidate: registered and quarantined. It is admitted for nothing until this assessment and the exact candidate are independently reviewed.

What changed#

The registry (docs/audits/eve-sota-eval-data-governance/registry.json) binds each dataset artifact by byte count, SHA-256 and Git blob. One artifact of assistant-deck-regression had changed since its binding, so the registry refused to verify and admit-eval-dataset.mjs refused every admission, for every dataset.

Predecessor (version 2026-09-13) Candidate (version 2026-10-04)
Registry Git commit a2c4c72fdd8, registry blob 8df44174e5a commit 9d3c9b941ac, registry blob 40973b915b2
apps/oshun/bff/src/assistant/evals/deck-builder-cases.ts bytes 54,695 59,257
SHA-256 e6ebbd38d8fdf535aaff275c4c9398f69a4b6da836ec1d9183446aa2187d519e b7f09ab27cfe545fe478612d157f634e98affb7e91c930e62e5faed4d65cce89
Git blob 1af9cb68fe0a2295d8a46e502a8176b6abe3f335 784d88b6567b1207dd8e7b15eb21a0b97f2e9e67

The other five artifacts of the dataset are unchanged. The predecessor is not rewritten: it is the registry as Git holds it at the commit above.

Scope of the assessment#

The whole difference between the two blobs of deck-builder-cases.ts: 113 added lines and no removed or changed line, made by three commits of 2026-09-30 (dec5810fe49, 4a662cd6034, 8010e0d73b5). Every added line was read. They are two advisory-reason constants and six eval cases:

  • builder-wbw-draft-outcome-reject, builder-wbw-draft-outcome-defer (G10.1: a decision draft carries an outcome);
  • builder-wbw-approve-plan, builder-wbw-decide-not-dispatch (R06.5: approving and dispatching a plan revision);
  • builder-operations-named-render, builder-operations-file-failed-build (R25.4: reading an operation's status).

Findings#

Provenance and licence. All three commits are by the repository's own worker identity (Claude (board worker)), written for board tasks G10.1, R06.5 and R25.4. The cases are authored in this repository; nothing is copied from an external corpus, a provider's output or a customer record. The dataset's rights entry (repository-authored, Apache-2.0) describes them. No change.

Personal or sensitive data. The six prompts are invented operator requests ("Draft a decision titled "No second job queue"…", "Did the Q3 board deck render finish?"). They name no person, account, address, credential or customer. The identifiers in them (plan-rev-drawer-cards-2, goal-drawer-cards, the family eve-gap-analysis) are made-up plan and goal ids and a board family name. The automated scan of the complete artifact bytes (privacyScan, patterns eval-data-sensitive-v2) reports: high-confidence secret matches 0, email-like matches 0, personal-data matches 0, encoded-sensitive matches 0. Privacy classification unchanged: synthetic or repository-public, no redaction required.

Contamination and split exposure. The dataset is a tune split and stays one. The six new case ids and six new prompt texts were searched for in every other artifact the registry binds (21 artifacts in 11 datasets, including both held-out sets, the evaluator-only acceptance set and the graded relevance record): no occurrence. The new cases are marked advisory in the file itself ("gating waits on the next k=10 measurement"). That exempts each of them from the per-case assertion only; five of the six still count toward a pooled family gate, which the next section sets out. Nothing here was drawn from, or copied into, a held-out or graded set. The dataset's existing limitation stands: its cases are referenced by assistant skill evalCaseIds, so it is not independent release evidence.

Which gates the new cases reach#

The first revision of this assessment said that, being advisory, none of the new cases gates a release. That was wrong, and review 8c27fbfe said so. The runner (apps/oshun/bff/src/assistant/evals/assistant-golden.eval.ts, unchanged since the candidate commit) has two different gates:

  • The per-case gate. It asserts that every case in aggregates passed all its runs. An advisory case is put in advisoryAggregates instead, and a held-out case in heldOutAggregates, so neither can fail this assertion. The six new cases are outside it.
  • The pooled family gate. gradedAggregates = [...aggregates, ...advisoryAggregates] is rolled up by family and compared with the Wilson floors stamped in docs/audits/eve-smx-ratchet.json (builderWilsonFloors). On a full-deck run a family whose pooled lower bound is under its floor, or which is missing, fails the run ("builder-family Wilson ratchet breached"). Advisory cases are in this pool; only held-out cases are kept out. A family with fewer pooled runs or cases than its floor was stamped on is reported as insufficient and does not fail.

Where the six cases fall, counted from the deck at the candidate commit (263 cases):

Case Family Per-case gate Pooled family floor
builder-wbw-draft-outcome-reject workbench-write not asserted (advisory) counted
builder-wbw-draft-outcome-defer workbench-write not asserted (advisory) counted
builder-wbw-approve-plan workbench-write not asserted (advisory) counted
builder-wbw-decide-not-dispatch workbench-write not asserted (advisory) counted
builder-operations-file-failed-build workbench-write not asserted (advisory) counted
builder-operations-named-render presentation not asserted (advisory) none: no floor is stamped for the presentation family

Floors are stamped for three families only: docs, workbench-read and workbench-write.

Contribution to the workbench-write floor. The floor is 0.875, the Wilson lower bound of 229 passed runs in 250 over 25 cases. The family now has 37 cases, 5 gating and 32 advisory, none held out; the five new cases are five of the 32. At k=10 the pool is 370 runs, of which the five new cases are 50 (13.5%). Holding the other 32 cases at the stamped pass rate (293 of 320, lower bound 0.88) and varying only the five new cases:

Runs the five new cases fail, of 50 Pooled runs passed, of 370 Wilson lower bound Against the 0.875 floor
0 343 0.8959 holds
5 338 0.8804 holds
6 337 0.8773 holds
7 336 0.8743 breach
10 333 0.8652 breach

So the five cases can fail a full-deck run on their own: seven failed runs among their fifty, with every other case at its stamped rate, puts the family under its floor, although no case assertion fails. They can also carry the family: if they pass every run the bound rises to 0.8959. This arithmetic is computed with the runner's own wilsonInterval; it is a what-if, not a measurement. No k=10 run of the six cases is recorded in this assessment; their real pass counts are unknown here.

What this means for the dataset's use. These are tune cases: the prompts and tools they exercise are adjusted against them. Because they are pooled into the workbench-write floor, that floor moves with tuning on them, and a full-deck pass or breach of it is not independent evidence about a release. This is the dataset's existing limitation, and it holds with more weight than the first revision gave it: the limitation covers the family floor these cases feed, not only the cases. The allowed uses (development, regression, advisory) and the forbidden uses (graded, held-out, production-training) are right as they stand and are not changed. A change to which pooled gates advisory cases feed would be a change to the runner and the ratchet, outside this dataset's assessment; none is proposed here.

Nothing else found is unresolved.

What the candidate changes and what it keeps#

Changed, in tools/eve-everywhere/eval-data-governance-policy.mjs and the registry together: the dataset's version (2026-10-04), its freshness.reviewedOn (2026-10-04, the date of this assessment), the artifact's bytes, SHA-256 and Git blob, its status (quarantined while this is under review) and one added limitation saying so.

Kept: the tune split, promotionEligible: false, the allowed uses (development, regression, advisory), the forbidden uses (graded, held-out, production-training), the retirement triggers, the expiry (2027-03-13), the named roles, the global governed-on date (2026-09-13) and every other dataset.

Checks run at the candidate commit#

  • node tools/eve-everywhere/verify-eval-data-governance.mjs: "Eval-data governance verified: 11 datasets, 21 artifacts."
  • node --test tools/eve-everywhere/verify-eval-data-governance.test.mjs: 21 passed, 0 failed (18 passed, 2 failed before).
  • admit-eval-dataset.mjs --dataset=assistant-deck-regression --use=regression --as-of=2026-10-04: refused, "dataset status is quarantined".
  • admit-eval-dataset.mjs --dataset=engineering-benchmark-heldout --use=held-out --as-of=2026-10-04: admitted (it was refused while the registry did not verify).
  • With the candidate's status set to admitted in a copy (what approval would do): regression and development are admitted on and after 2026-10-04; refused before that date and after 2027-03-13; held-out, graded and production-training are refused.

What a reviewer is asked to decide#

Whether this assessment and the exact candidate bytes above are sufficient to return assistant-deck-regression version 2026-10-04 to admitted. That change is a commit of its own after approval. This document asks for no publication or promotion.