Task 12.9 now has a deterministic, content-bound benchmark candidate for four separate creative families: reference-to-editable motion, original brief-to-editable motion, measured-house reconstruction, and interactive architectural walkthrough. It remains blocked, rather than preregistered, because Task 17.7 has not supplied and ratified the authoritative creative delivery contracts and coverage map. The absent contract is represented as a null digest, the limitation is explicit, and the verifier refuses a promoted status while that dependency is absent.
Each family fixes its independent case identity before variants or output inspection, prohibits derivatives and practiced capstones crossing splits, and requires at least 200 independent cases with at least 40 per case class. Three fixed model seeds are nested under a case rather than counted as extra evidence; deterministic runtime paths require two replays. Decisions must remain visible by case class, complexity, native/runtime format, provider route or no-model route, and fresh versus corrected input.
The benchmark measures quality at the boundary that can establish it. Reference motion uses decoded frames and native-timeline readback for layout, timing, and Bezier fidelity. Original motion adds blinded brief, brand, composition, and originality review. Reconstruction compares native geometry and topology with independent dimensional ground truth and hard-locks unsupported load-bearing claims. Walkthroughs are exercised in the actually served runtime, including input-to-motion latency, frame rate, navigation containment, reconnect behavior, accessible alternatives, and dimensional readback after engine import. Native editability is checked after save and reopen; an unopenable project or unserved build is a zero-tolerance failure.
Every family also carries source-to-verified-delivery measures for success, failure or abandonment, paired human labor, revisions, wall-clock latency, and provider/render budget. Route identity comes from runtime receipts joined to the current approved model registry. Asset metadata, UI/marketing labels, and a headline clip duration are expressly inadmissible. Model, rendering, storage, streaming, retry, queue, and revision legs remain in their respective cost and latency boundaries, and any unpriced provider or render leg hard-locks a family. Ratios use the same held-out case completed by an independently assigned qualified professional in conventional native tools without Eve or model help. Lane order is randomized across at least three professionals, acceptance inputs and limits are identical and blinded, and a preregistered stopping rule retains all attempts instead of selecting the best output after inspection.
Visual-judge calibration is itself held out by source identity and requires at least 100 independent cases per criterion and case class, three blinded human raters per case, Krippendorff alpha of at least 0.67, Spearman correlation of at least 0.80, false accepts no greater than 0.05, and false rejects no greater than 0.10. The model judge is advisory. Humans remain required for missing or failed calibration, drift, unresolved appeals, dimensional decisions, high-risk cases, and hard-lock-relevant decisions. Human ratings use fixed anchors for unusable, substantively revision-requiring, and delivery-ready work.
The decision rule is conjunctive: every applicable target and floor must pass in
every preregistered stratum, all hard locks must remain zero, and all four
families must pass before release. Missing cases, prices, receipts, native
readbacks, labels, repeats, failed attempts, or denominators are INCOMPLETE
and block. Amendments are prospective, independently approved, content-hashed,
and made before affected outputs are visible. Criterion/stratum denominators are
locked before execution; Wilson 95% intervals govern binary measures, while
continuous measures use a separately held non-promoting pilot and at least
10,000 case-clustered BCa resamples.
The schema and verifier enforce the exact family membership, case classes, sample and repeat rules, criteria, numeric thresholds, measurement boundaries, judge calibration, routing and costing rules, source hashes, dependency status, and record digest. The focused suite includes a deterministic regeneration check and real-CLI negative controls for a missing family, pooled repeats, undersized samples, missing dimensions, weakened thresholds, nonzero hard locks, unvalidated judges, metadata attribution, unpriced cost, hidden dependency status, stale source hashes, fabricated ratification, and output-visible amendments, an assisted comparator, and unanchored human scores.
Verify the retained blocked candidate with:
node tools/eve-everywhere/verify-creative-workflow-benchmark.mjs
Regenerate it deterministically with:
node tools/eve-everywhere/generate-creative-workflow-benchmark.mjs
This artifact contains no held-out creative case, human label, model run, native project, runtime receipt, quality score, cost, labor, latency, or operator decision. It is preregistration machinery awaiting Task 17.7, not evidence that any creative family, Task 12.9, Phase 12, G7, or G14 passes.