Eve library

EVAL-004 progress evidence — human-quality launch criteria

Evidence

Revision 1

On this page

Status: partially complete, not closed. The repository now enforces the statistical thresholds, non-compensatory gates, preregistration shape, raw observation integrity, and signed release binding required for a defensible human-quality launch decision. It does not claim that a real target-audience study has occurred. EVAL-002 still records zero acquired corpus/cohort units, and test fixtures are not launch evidence.

Work-package charter#

  • Task ID: EVAL-004
  • Owner: evaluation governance, target-audience research, creative-autonomy quality, and V8/V10 case release; repository implementation owner: Codex
  • Audit finding: release surfaces had automated quality and calibrated-judge boundaries, but no canonical target-audience study record bound a frozen baseline, candidate identity, raw human observations, editorial effort, subgroup/tail analysis, hard gates, and signed provenance.
  • Repository objective: make authored-parity, “amazing,” and efficiency claims fail closed unless recomputed evidence for the exact candidate passes the declared policy; prevent preference scores from compensating for safety, rights, consent, provenance, accessibility, or runtime failures.
  • External objective: preregister and run the real study with qualified, consented target-audience participants and editors against a rights-cleared best-authored baseline.
  • Rollback strategy: revert source commit 401b5496b6a00aa1a2f3a2f3d53caef16a60a14a and, if the concurrent BFF extraction is also being rolled back, compatibility commit a432f8f8915431e1062218e598760f9896a15d34.
  • Evidence-pack location: this file.

Canonical human-quality launch evidence#

HumanQualityLaunchEvidence and its supporting records now require:

  • a signed, frozen preregistration created before observation collection;
  • the target audience and eligibility rules, exact candidate system/version, candidate configuration digest, best-authored corpus/version/manifest, and rights/author-qualification evidence;
  • the selection protocol, sampling frame, randomization, blinding, power analysis, minimum detectable effect, exclusions, fixed stopping rule, and planned subgroup analyses;
  • at least two distinct, non-overlapping sampling-frame waves;
  • raw comparison observations that bind candidate and baseline digests, baseline-corpus membership, participant consent/eligibility, assignment, blinding, subgroup labels, exclusions, ties, and catastrophic outcomes;
  • raw editorial observations that retain active seconds, interactions, token edit distance, normalized edit distance, acceptance/regeneration outcome, and independent quality-hold outcome;
  • per-artifact safety, rights, consent, provenance, accessibility, and runtime gates for the exact candidate digest.

The evaluator derives all aggregates from raw observations. It does not accept caller-supplied win rates, confidence intervals, acceptance rates, or tail summaries. Wilson 95% bounds are recomputed using the canonical evaluation implementation, ties count as candidate losses, fixed sample/wave rules are enforced, post-hoc or ineligible observations are rejected, and weakened preregistered thresholds cannot receive a passing decision.

The report exposes overall, independent-wave, and planned-subgroup comparison statistics plus editorial active-time and edit-distance p50/p90/p95/max tails. Any catastrophic outcome blocks release; a high mean cannot hide it. Minimum active-time and interaction floors prevent zero-effort rubber stamps from qualifying as efficient acceptance.

Exact launch policy and non-compensatory decisions#

The frozen policy implements the ledger thresholds exactly:

  • authored parity requires a lower 95% preference bound strictly above 0.50;
  • “amazing” requires the overall and every independent wave to have a win rate strictly above 0.60 and lower 95% bound strictly above 0.55;
  • efficiency requires at least 0.80 acceptance without full regeneration, after the independent quality hold passes;
  • every hard gate must pass independently, regardless of preference score.

The decision separately records evidenceReady, releaseEligible, authoredParityClaimEligible, amazingClaimEligible, and efficiencyClaimEligible, with typed blockers. A preferred candidate with a rights, consent, provenance, accessibility, runtime, or safety failure remains ineligible.

Runtime and signed-release integration#

Creative autonomy requires an exact human-quality binding in DirectorConfig. Director construction recomputes the decision and fails closed unless release, parity, and every configured public claim pass. Gate and review provenance retain the evidence/study identity, candidate system/version/configuration digest, and claim verdicts. BFF composition requires semantic diversity, judge calibration, and human-quality launch evidence; absent real launch evidence it remains deliberately unconfigured.

Yemaya case release adds G8. decideRelease recomputes the raw launch evidence, binds it to the exact case, and requires parity plus configured claims. Loom, Daedalus, the V10 Case Files cook, and the scheduler require the governed evidence file rather than manufacturing a pass. Daedalus accepts --human-quality-evidence-file; V10 uses the same CLI option and CASE_FILES_HUMAN_QUALITY_EVIDENCE_FILE for scheduled cooks.

V8 authenticated release context and V10 fairness-receipt provenance include the human-quality study/evidence/candidate identity inside the signed payload. Verification recomputes the binding. Adversarial tests replace the study after signing and prove that both V8 and V10 reject the substituted provenance.

Checklist disposition#

Three repository-enforceable EVAL-004 requirements are complete:

  • authored-parity claims require a lower 95% confidence bound above 0.50;
  • sustained “amazing” claims require overall and per-wave win rates above 0.60 with lower bounds above 0.55;
  • hard safety, rights, consent, provenance, accessibility, and runtime gates remain non-compensatory.

Three items remain open because completion requires real humans and observations, not schemas or generated fixtures:

  • preregister the actual target audience, best-authored baseline, sampling, exclusions, statistical test, minimum effect, and stopping rule;
  • collect actual active human minutes, interaction counts, edit distances, regeneration outcomes, and independent quality holds;
  • publish real tail and subgroup distributions and demonstrate zero catastrophic outliers on the launch sample.

No test fixture or synthetic observation is counted toward those open boxes.

Automated verification#

The integrated EVAL-004 source passed 4,486/4,486 non-duplicated owning-package tests:

Surface Result
Canonical contracts full package 4,080/4,080
Creative autonomy full package 268/268
Affected BFF autonomy route/bindings/E2E 50/50
Yemaya case-eval full package 14/14
Yemaya signed case-verifier full package 26/26
V8 Loom service full package 29/29
V10 Case Files cook/scheduler full package 9/9
V8 Daedalus compiler full package 3/3
V10 fairness-receipt focused boundary 7/7

Nine EVAL-004 typechecks passed: contracts, creative autonomy, the targeted BFF autonomy graph, case-eval, case-verifier, Loom, V10 Case Files cook, Daedalus, and the targeted V10 receipt boundary. The canonical contracts Nx build and BFF 29.7 MiB production bundle passed.

The final source rebase exposed a concurrent broken @oshun/bff-kit extraction: implementations had been replaced by recursive self-reexport shims while the legacy app shims were empty. Compatibility commit a432f8f891... restores all nine implementations to the shared package and all nine thin app reexports. Its own 3/3 tests, strict typecheck, emitted build, lint, formatting, the BFF 50/50 suite, and production bundle pass. After the final remote sync, the kit 3/3, BFF 50/50, V10 receipt 7/7, and their three overlap-specific typechecks were repeated and passed.

Root ESLint over all EVAL-004 tracked/new TypeScript completed with zero errors; the only six warnings are existing non-null assertions in legacy creative director/rubric code. The compatibility files pass with zero warnings. Prettier and git diff --check pass.

Representative commands:

text
pnpm --dir libs/contracts exec vitest run --testTimeout=120000
pnpm exec vitest run --config libs/oshun/creative-autonomy/vitest.config.ts
pnpm --dir apps/oshun/bff exec vitest run <three affected autonomy files>
pnpm --dir libs/yemaya/case-eval exec vitest run --config vitest.config.ts
pnpm --dir libs/yemaya/case-verifier exec vitest run --config vitest.config.ts
pnpm --dir apps/v8/loom-service exec vitest run --config vitest.config.ts
pnpm --dir apps/v10/case-files-cook exec vitest run --config vitest.config.ts
pnpm --dir apps/v8/daedalus-compiler exec vitest run --config vitest.config.ts
pnpm --dir apps/v10/web exec vitest run --config vitest.config.ts <two receipt files>
pnpm exec tsc --noEmit -p <each owning package tsconfig>
pnpm exec tsc --noEmit -p apps/oshun/bff/tsconfig.eval003.json
pnpm exec tsc --noEmit -p apps/v10/web/tsconfig.eval004.json
NX_DAEMON=false NX_TUI=false pnpm nx run @oshun/contracts:build --parallel=1 --skip-nx-cache
pnpm --dir apps/oshun/bff run build
pnpm exec tsc -p libs/shared/bff-kit/tsconfig.lib.json
pnpm exec eslint --no-warn-ignored <all changed TypeScript>
pnpm exec prettier --check <all changed non-lock files>
git diff --check

Bounded verification notes#

  • The V10 app-wide typecheck was attempted and is blocked by the checkout's pre-existing missing package links (next, Tauri, Axe, and numerous workspace packages), with downstream implicit-any diagnostics. The permanent tsconfig.eval004.json checks the exact changed receipt/schema boundary and passes; both focused receipt test files pass 7/7.
  • The first focused V10 receipt run exposed the app config's bare-contract alias precedence. An exact @oshun/contracts/v10 alias was added; the clean rerun passed 7/7.
  • The BFF's full application typecheck is intentionally not repeated after the previously observed 4 GiB heap exhaustion. The permanent affected-graph typecheck, 50 tests, strict shared-kit typecheck, and production bundle pass.
  • A case-eval Nx build is not repeated because it expands into 37 dependencies and previously stopped on unrelated Iris/Isis failures. The owning case-eval test/typecheck and every downstream changed app test/typecheck pass.
  • Loom and the shared kit emit existing missing source-map warnings from shared HTTP-client generated JavaScript. They do not change test results.
  • The 15 GiB RAM/31 GiB swap host was checked before expensive verification; broad workspace gates were not run, checks were serialized, and memory stayed above the mandated stop thresholds.
  • The unrelated binary pnpm-lock.yaml modification was preserved byte-for-byte through both rebases (Git blob a37431842e330b3a06b030947c25bebdc72bc2b1). Generated Cargo locks and libs/urania/ were not staged.
  • No customer UI, PWA, admin, mobile, Unreal, or visual behavior changed. The V10 web change is a receipt fixture/resolver/type boundary only. Frontend skill, Playwright, mobile automation, screenshots, and Unreal verification were inapplicable.

Artifact identity#

Artifact SHA-256
Canonical human-quality launch contract c6345f38dee164e09ca5d7e0fc933ea9d89fe893b767f9dcd7dadf73ada1f5c1
Creative director launch boundary 7f283656d76a89acace66c4050fb02acedc1e5374914e1842acf6c3c8d6ac086
Yemaya G8 release boundary 338d95fba990907b984467627ddb8a064a5e11dc84d6db8384d66037c16f1630
V8 signed release authentication 297dfb3cde1e354ab77b07a0e6ba955a6c1ce67166b7088f7ad87f225faebddf
V10 signed Case Files receipt contract 92b9d9d34f785df789496d786946c4baa73b6d4e50f958b84cfcf911e67f2295
BFF fail-closed composition 9208b59d0a1308a46c75f74499ed9dc856b1661c0766c7fa475294a302e2a5e8
Restored shared BFF auth implementation 91d2b0ceea1f5b4b690f889fa86927aa8e4553211304fc73f0c44952e7f17296

Publication identity#

  • Repository source commit: 401b5496b6a00aa1a2f3a2f3d53caef16a60a14a
  • Required compatibility commit: a432f8f8915431e1062218e598760f9896a15d34
  • Repository source publication tip: a432f8f8915431e1062218e598760f9896a15d34
  • EVAL-004 closure evidence commit: not applicable; EVAL-004 remains open.