Status: partially complete, not closed. The repository now enforces the statistical thresholds, non-compensatory gates, preregistration shape, raw observation integrity, and signed release binding required for a defensible human-quality launch decision. It does not claim that a real target-audience study has occurred. EVAL-002 still records zero acquired corpus/cohort units, and test fixtures are not launch evidence.
Work-package charter#
- Task ID:
EVAL-004 - Owner: evaluation governance, target-audience research, creative-autonomy quality, and V8/V10 case release; repository implementation owner: Codex
- Audit finding: release surfaces had automated quality and calibrated-judge boundaries, but no canonical target-audience study record bound a frozen baseline, candidate identity, raw human observations, editorial effort, subgroup/tail analysis, hard gates, and signed provenance.
- Repository objective: make authored-parity, “amazing,” and efficiency claims fail closed unless recomputed evidence for the exact candidate passes the declared policy; prevent preference scores from compensating for safety, rights, consent, provenance, accessibility, or runtime failures.
- External objective: preregister and run the real study with qualified, consented target-audience participants and editors against a rights-cleared best-authored baseline.
- Rollback strategy: revert source commit
401b5496b6a00aa1a2f3a2f3d53caef16a60a14aand, if the concurrent BFF extraction is also being rolled back, compatibility commita432f8f8915431e1062218e598760f9896a15d34. - Evidence-pack location: this file.
Canonical human-quality launch evidence#
HumanQualityLaunchEvidence and its supporting records now require:
- a signed, frozen preregistration created before observation collection;
- the target audience and eligibility rules, exact candidate system/version, candidate configuration digest, best-authored corpus/version/manifest, and rights/author-qualification evidence;
- the selection protocol, sampling frame, randomization, blinding, power analysis, minimum detectable effect, exclusions, fixed stopping rule, and planned subgroup analyses;
- at least two distinct, non-overlapping sampling-frame waves;
- raw comparison observations that bind candidate and baseline digests, baseline-corpus membership, participant consent/eligibility, assignment, blinding, subgroup labels, exclusions, ties, and catastrophic outcomes;
- raw editorial observations that retain active seconds, interactions, token edit distance, normalized edit distance, acceptance/regeneration outcome, and independent quality-hold outcome;
- per-artifact safety, rights, consent, provenance, accessibility, and runtime gates for the exact candidate digest.
The evaluator derives all aggregates from raw observations. It does not accept caller-supplied win rates, confidence intervals, acceptance rates, or tail summaries. Wilson 95% bounds are recomputed using the canonical evaluation implementation, ties count as candidate losses, fixed sample/wave rules are enforced, post-hoc or ineligible observations are rejected, and weakened preregistered thresholds cannot receive a passing decision.
The report exposes overall, independent-wave, and planned-subgroup comparison statistics plus editorial active-time and edit-distance p50/p90/p95/max tails. Any catastrophic outcome blocks release; a high mean cannot hide it. Minimum active-time and interaction floors prevent zero-effort rubber stamps from qualifying as efficient acceptance.
Exact launch policy and non-compensatory decisions#
The frozen policy implements the ledger thresholds exactly:
- authored parity requires a lower 95% preference bound strictly above
0.50; - “amazing” requires the overall and every independent wave to have a win rate
strictly above
0.60and lower 95% bound strictly above0.55; - efficiency requires at least
0.80acceptance without full regeneration, after the independent quality hold passes; - every hard gate must pass independently, regardless of preference score.
The decision separately records evidenceReady, releaseEligible,
authoredParityClaimEligible, amazingClaimEligible, and
efficiencyClaimEligible, with typed blockers. A preferred candidate with a
rights, consent, provenance, accessibility, runtime, or safety failure remains
ineligible.
Runtime and signed-release integration#
Creative autonomy requires an exact human-quality binding in DirectorConfig.
Director construction recomputes the decision and fails closed unless release,
parity, and every configured public claim pass. Gate and review provenance
retain the evidence/study identity, candidate system/version/configuration
digest, and claim verdicts. BFF composition requires semantic diversity, judge
calibration, and human-quality launch evidence; absent real launch evidence it
remains deliberately unconfigured.
Yemaya case release adds G8. decideRelease recomputes the raw launch evidence,
binds it to the exact case, and requires parity plus configured claims. Loom,
Daedalus, the V10 Case Files cook, and the scheduler require the governed
evidence file rather than manufacturing a pass. Daedalus accepts
--human-quality-evidence-file; V10 uses the same CLI option and
CASE_FILES_HUMAN_QUALITY_EVIDENCE_FILE for scheduled cooks.
V8 authenticated release context and V10 fairness-receipt provenance include the human-quality study/evidence/candidate identity inside the signed payload. Verification recomputes the binding. Adversarial tests replace the study after signing and prove that both V8 and V10 reject the substituted provenance.
Checklist disposition#
Three repository-enforceable EVAL-004 requirements are complete:
- authored-parity claims require a lower 95% confidence bound above
0.50; - sustained “amazing” claims require overall and per-wave win rates above
0.60with lower bounds above0.55; - hard safety, rights, consent, provenance, accessibility, and runtime gates remain non-compensatory.
Three items remain open because completion requires real humans and observations, not schemas or generated fixtures:
- preregister the actual target audience, best-authored baseline, sampling, exclusions, statistical test, minimum effect, and stopping rule;
- collect actual active human minutes, interaction counts, edit distances, regeneration outcomes, and independent quality holds;
- publish real tail and subgroup distributions and demonstrate zero catastrophic outliers on the launch sample.
No test fixture or synthetic observation is counted toward those open boxes.
Automated verification#
The integrated EVAL-004 source passed 4,486/4,486 non-duplicated owning-package tests:
| Surface | Result |
|---|---|
| Canonical contracts full package | 4,080/4,080 |
| Creative autonomy full package | 268/268 |
| Affected BFF autonomy route/bindings/E2E | 50/50 |
| Yemaya case-eval full package | 14/14 |
| Yemaya signed case-verifier full package | 26/26 |
| V8 Loom service full package | 29/29 |
| V10 Case Files cook/scheduler full package | 9/9 |
| V8 Daedalus compiler full package | 3/3 |
| V10 fairness-receipt focused boundary | 7/7 |
Nine EVAL-004 typechecks passed: contracts, creative autonomy, the targeted BFF autonomy graph, case-eval, case-verifier, Loom, V10 Case Files cook, Daedalus, and the targeted V10 receipt boundary. The canonical contracts Nx build and BFF 29.7 MiB production bundle passed.
The final source rebase exposed a concurrent broken @oshun/bff-kit extraction:
implementations had been replaced by recursive self-reexport shims while the
legacy app shims were empty. Compatibility commit a432f8f891... restores all
nine implementations to the shared package and all nine thin app reexports. Its
own 3/3 tests, strict typecheck, emitted build, lint, formatting, the BFF 50/50
suite, and production bundle pass. After the final remote sync, the kit 3/3, BFF
50/50, V10 receipt 7/7, and their three overlap-specific typechecks were
repeated and passed.
Root ESLint over all EVAL-004 tracked/new TypeScript completed with zero errors;
the only six warnings are existing non-null assertions in legacy creative
director/rubric code. The compatibility files pass with zero warnings. Prettier
and git diff --check pass.
Representative commands:
pnpm --dir libs/contracts exec vitest run --testTimeout=120000
pnpm exec vitest run --config libs/oshun/creative-autonomy/vitest.config.ts
pnpm --dir apps/oshun/bff exec vitest run <three affected autonomy files>
pnpm --dir libs/yemaya/case-eval exec vitest run --config vitest.config.ts
pnpm --dir libs/yemaya/case-verifier exec vitest run --config vitest.config.ts
pnpm --dir apps/v8/loom-service exec vitest run --config vitest.config.ts
pnpm --dir apps/v10/case-files-cook exec vitest run --config vitest.config.ts
pnpm --dir apps/v8/daedalus-compiler exec vitest run --config vitest.config.ts
pnpm --dir apps/v10/web exec vitest run --config vitest.config.ts <two receipt files>
pnpm exec tsc --noEmit -p <each owning package tsconfig>
pnpm exec tsc --noEmit -p apps/oshun/bff/tsconfig.eval003.json
pnpm exec tsc --noEmit -p apps/v10/web/tsconfig.eval004.json
NX_DAEMON=false NX_TUI=false pnpm nx run @oshun/contracts:build --parallel=1 --skip-nx-cache
pnpm --dir apps/oshun/bff run build
pnpm exec tsc -p libs/shared/bff-kit/tsconfig.lib.json
pnpm exec eslint --no-warn-ignored <all changed TypeScript>
pnpm exec prettier --check <all changed non-lock files>
git diff --check
Bounded verification notes#
- The V10 app-wide typecheck was attempted and is blocked by the checkout's
pre-existing missing package links (
next, Tauri, Axe, and numerous workspace packages), with downstream implicit-any diagnostics. The permanenttsconfig.eval004.jsonchecks the exact changed receipt/schema boundary and passes; both focused receipt test files pass 7/7. - The first focused V10 receipt run exposed the app config's bare-contract alias
precedence. An exact
@oshun/contracts/v10alias was added; the clean rerun passed 7/7. - The BFF's full application typecheck is intentionally not repeated after the previously observed 4 GiB heap exhaustion. The permanent affected-graph typecheck, 50 tests, strict shared-kit typecheck, and production bundle pass.
- A case-eval Nx build is not repeated because it expands into 37 dependencies and previously stopped on unrelated Iris/Isis failures. The owning case-eval test/typecheck and every downstream changed app test/typecheck pass.
- Loom and the shared kit emit existing missing source-map warnings from shared HTTP-client generated JavaScript. They do not change test results.
- The 15 GiB RAM/31 GiB swap host was checked before expensive verification; broad workspace gates were not run, checks were serialized, and memory stayed above the mandated stop thresholds.
- The unrelated binary
pnpm-lock.yamlmodification was preserved byte-for-byte through both rebases (Git bloba37431842e330b3a06b030947c25bebdc72bc2b1). Generated Cargo locks andlibs/urania/were not staged. - No customer UI, PWA, admin, mobile, Unreal, or visual behavior changed. The V10 web change is a receipt fixture/resolver/type boundary only. Frontend skill, Playwright, mobile automation, screenshots, and Unreal verification were inapplicable.
Artifact identity#
| Artifact | SHA-256 |
|---|---|
| Canonical human-quality launch contract | c6345f38dee164e09ca5d7e0fc933ea9d89fe893b767f9dcd7dadf73ada1f5c1 |
| Creative director launch boundary | 7f283656d76a89acace66c4050fb02acedc1e5374914e1842acf6c3c8d6ac086 |
| Yemaya G8 release boundary | 338d95fba990907b984467627ddb8a064a5e11dc84d6db8384d66037c16f1630 |
| V8 signed release authentication | 297dfb3cde1e354ab77b07a0e6ba955a6c1ce67166b7088f7ad87f225faebddf |
| V10 signed Case Files receipt contract | 92b9d9d34f785df789496d786946c4baa73b6d4e50f958b84cfcf911e67f2295 |
| BFF fail-closed composition | 9208b59d0a1308a46c75f74499ed9dc856b1661c0766c7fa475294a302e2a5e8 |
| Restored shared BFF auth implementation | 91d2b0ceea1f5b4b690f889fa86927aa8e4553211304fc73f0c44952e7f17296 |
Publication identity#
- Repository source commit:
401b5496b6a00aa1a2f3a2f3d53caef16a60a14a - Required compatibility commit:
a432f8f8915431e1062218e598760f9896a15d34 - Repository source publication tip:
a432f8f8915431e1062218e598760f9896a15d34 - EVAL-004 closure evidence commit: not applicable; EVAL-004 remains open.