Disciplines · Audits

Eve SOTA family evaluation and sample-design contract — 2026-09-13

The prospective version-2 outcome scorecard is the numerical and governance authority.

5sections4 minread
Markdown source
On this page

Task: 12.2
Status: preregistered and intentionally unmeasured
Machine record: eve-sota-evaluation-design/2026-09-13.json
Design schema: eve-sota-evaluation-design.schema.json
Cohort schema: eve-sota-evaluation-cohort.schema.json

Decision#

The prospective version-2 outcome scorecard is the numerical and governance authority. Its operator-authorized amendment applies only to fresh cohorts locked after that amendment; Task 12.2 does not change a target, floor, hard lock, statistical unit, or decision owner.

For each of the exact fifteen Task 12.1 families, the generated record now freezes:

  • four independent case-diversity classes;
  • deterministic, nested stochastic, and human run rules;
  • separate pure, model-advisory, and human-authority rubric lanes;
  • paired negative and benign controls;
  • every applicable hard safety lock and its HOLD escalation;
  • scorecard outcome and floor-criterion ownership;
  • all seven required axis decisions, including explicit source-backed exclusions;
  • every marginal stratum plus every plane × high/critical-risk intersection;
  • a separate denominator for every floor criterion in every stratum.

The expansion contains 15 families, 774 strata, 206 criterion plans, and 10,585 criterion denominators. These are obligations, not results. Every client-facing family conservatively includes all 25 platform ids ratified in the Task 0.8 charter inventory. Narrowing applicability requires a prospective amendment before affected output is visible.

Statistical rules#

The statistical unit is copied verbatim from each Task 0.3 scorecard row. A case id is the independent unit. Seeds, retries, turns, provider attempts, claims, rubric items, and fault locations nested in that case never increase n. Model-dependent cases require at least three fixed seeds, and every required seed must pass.

For every Wilson criterion/stratum, the design requires at least 200 independent units and reproduces the exact binomial probability of clearing the unchanged two-sided 95% Wilson floor at the declared target. The semantic verifier rejects an infeasible floor, a probability below 80%, or a claimed probability that it cannot independently reproduce.

Continuous minima are deliberately not called power claims. Before a graded cohort can lock, a separate non-promoting pilot must provide distribution, tail, cluster, missingness, censoring, failure, and abstention assumptions. Each criterion/stratum must preregister at least 10,000 bootstrap resamples, at least 10,000 power-simulation replicates, at least 80% planned probability of clearing the floor, and the criterion's precision target. The exact graded n is the maximum of the record's denominator minimum, scorecard composition minimum, pilot-derived power size, and pilot-derived precision size. Pilot case ids may not overlap graded case ids.

Human authority#

Every family names an actual human role for subjective labels and a separate review role. The manifest must carry a privacy-safe but attributable human assessor roster. The system under evaluation, evaluation harness, and model judge may randomize or route-blind material, but they may not author, infer, repair, or replace a missing human label. Model rubrics remain advisory until their exact classes pass Task 12.3 calibration.

Fail-closed evaluation CLI#

Lock a cohort before any result:

sh
node tools/eve-everywhere/evaluate-evaluation-cohort.mjs \
  --manifest=path/to/family.cohort-lock.json

Evaluate a result only against the separate immutable lock artifact:

sh
node tools/eve-everywhere/evaluate-evaluation-cohort.mjs \
  --manifest=path/to/family.results.json \
  --lock=path/to/family.cohort-lock.json

The lock binds the design digest, exact primary case ids, nested seed ids, criterion denominators, per-case strata, continuous pilot analyses, human assessor roster, and zero amendments. Result admission compares the complete plan projection to the separate lock and rejects post-result expansion or rewriting.

The CLI also rejects missing or reordered denominators, undersized strata, duplicate case ids, nested seed inflation, cross-stratum pooling, missing case classes, missing or model-owned human assessment, overlapping/underpowered/ imprecise pilots, result sets not matching the locked case ids, failed required seeds, hard-lock events, and aggregate-only results. A family-closed result requires a human-owned label and content-hashed observation evidence for every unit in every denominator; the CLI reads those evidence bytes and verifies their hashes.

Evidence and limitations#

The generator pins byte-level receipts for the Task 12.2 policy, Task 0.3 scorecard, Task 12.1 taxonomy, ratified Task 0.8 charter inventory, threat-plane inventory, model-leg inventory, and shipped-interface baseline. The verifier independently reconstructs the family, axis, stratum, criterion, denominator, hard-lock, Wilson-power, continuous-pilot, and ownership invariants.

Task 12.2 closes only the prospective design and manifest-admission machinery. It records no pilot observation, product result, human label, model-quality measurement, family pass, release admission, phase closure, or closure of G7 or G14. Task 12.3 owns human/judge calibration; Task 12.4 owns long-horizon case artifacts; Task 12.5 owns dataset governance; Task 12.7 owns release admission; Task 12.8 owns charter-completeness decisions.