# EVAL-004 progress evidence — human-quality launch criteria

> **Status: partially complete, not closed.** The repository now enforces the
> statistical thresholds, non-compensatory gates, preregistration shape, raw
> observation integrity, and signed release binding required for a defensible
> human-quality launch decision. It does not claim that a real target-audience
> study has occurred. EVAL-002 still records zero acquired corpus/cohort units,
> and test fixtures are not launch evidence.

## Work-package charter

- **Task ID:** `EVAL-004`
- **Owner:** evaluation governance, target-audience research, creative-autonomy
  quality, and V8/V10 case release; repository implementation owner: Codex
- **Audit finding:** release surfaces had automated quality and calibrated-judge
  boundaries, but no canonical target-audience study record bound a frozen
  baseline, candidate identity, raw human observations, editorial effort,
  subgroup/tail analysis, hard gates, and signed provenance.
- **Repository objective:** make authored-parity, “amazing,” and efficiency
  claims fail closed unless recomputed evidence for the exact candidate passes
  the declared policy; prevent preference scores from compensating for safety,
  rights, consent, provenance, accessibility, or runtime failures.
- **External objective:** preregister and run the real study with qualified,
  consented target-audience participants and editors against a rights-cleared
  best-authored baseline.
- **Rollback strategy:** revert source commit
  `401b5496b6a00aa1a2f3a2f3d53caef16a60a14a` and, if the concurrent BFF
  extraction is also being rolled back, compatibility commit
  `a432f8f8915431e1062218e598760f9896a15d34`.
- **Evidence-pack location:** this file.

## Canonical human-quality launch evidence

`HumanQualityLaunchEvidence` and its supporting records now require:

- a signed, frozen preregistration created before observation collection;
- the target audience and eligibility rules, exact candidate system/version,
  candidate configuration digest, best-authored corpus/version/manifest, and
  rights/author-qualification evidence;
- the selection protocol, sampling frame, randomization, blinding, power
  analysis, minimum detectable effect, exclusions, fixed stopping rule, and
  planned subgroup analyses;
- at least two distinct, non-overlapping sampling-frame waves;
- raw comparison observations that bind candidate and baseline digests,
  baseline-corpus membership, participant consent/eligibility, assignment,
  blinding, subgroup labels, exclusions, ties, and catastrophic outcomes;
- raw editorial observations that retain active seconds, interactions, token
  edit distance, normalized edit distance, acceptance/regeneration outcome, and
  independent quality-hold outcome;
- per-artifact safety, rights, consent, provenance, accessibility, and runtime
  gates for the exact candidate digest.

The evaluator derives all aggregates from raw observations. It does not accept
caller-supplied win rates, confidence intervals, acceptance rates, or tail
summaries. Wilson 95% bounds are recomputed using the canonical evaluation
implementation, ties count as candidate losses, fixed sample/wave rules are
enforced, post-hoc or ineligible observations are rejected, and weakened
preregistered thresholds cannot receive a passing decision.

The report exposes overall, independent-wave, and planned-subgroup comparison
statistics plus editorial active-time and edit-distance p50/p90/p95/max tails.
Any catastrophic outcome blocks release; a high mean cannot hide it. Minimum
active-time and interaction floors prevent zero-effort rubber stamps from
qualifying as efficient acceptance.

## Exact launch policy and non-compensatory decisions

The frozen policy implements the ledger thresholds exactly:

- authored parity requires a lower 95% preference bound strictly above `0.50`;
- “amazing” requires the overall and every independent wave to have a win rate
  strictly above `0.60` and lower 95% bound strictly above `0.55`;
- efficiency requires at least `0.80` acceptance without full regeneration,
  after the independent quality hold passes;
- every hard gate must pass independently, regardless of preference score.

The decision separately records `evidenceReady`, `releaseEligible`,
`authoredParityClaimEligible`, `amazingClaimEligible`, and
`efficiencyClaimEligible`, with typed blockers. A preferred candidate with a
rights, consent, provenance, accessibility, runtime, or safety failure remains
ineligible.

## Runtime and signed-release integration

Creative autonomy requires an exact human-quality binding in `DirectorConfig`.
Director construction recomputes the decision and fails closed unless release,
parity, and every configured public claim pass. Gate and review provenance
retain the evidence/study identity, candidate system/version/configuration
digest, and claim verdicts. BFF composition requires semantic diversity, judge
calibration, and human-quality launch evidence; absent real launch evidence it
remains deliberately unconfigured.

Yemaya case release adds G8. `decideRelease` recomputes the raw launch evidence,
binds it to the exact case, and requires parity plus configured claims. Loom,
Daedalus, the V10 Case Files cook, and the scheduler require the governed
evidence file rather than manufacturing a pass. Daedalus accepts
`--human-quality-evidence-file`; V10 uses the same CLI option and
`CASE_FILES_HUMAN_QUALITY_EVIDENCE_FILE` for scheduled cooks.

V8 authenticated release context and V10 fairness-receipt provenance include the
human-quality study/evidence/candidate identity inside the signed payload.
Verification recomputes the binding. Adversarial tests replace the study after
signing and prove that both V8 and V10 reject the substituted provenance.

## Checklist disposition

Three repository-enforceable EVAL-004 requirements are complete:

- authored-parity claims require a lower 95% confidence bound above `0.50`;
- sustained “amazing” claims require overall and per-wave win rates above `0.60`
  with lower bounds above `0.55`;
- hard safety, rights, consent, provenance, accessibility, and runtime gates
  remain non-compensatory.

Three items remain open because completion requires real humans and
observations, not schemas or generated fixtures:

- preregister the actual target audience, best-authored baseline, sampling,
  exclusions, statistical test, minimum effect, and stopping rule;
- collect actual active human minutes, interaction counts, edit distances,
  regeneration outcomes, and independent quality holds;
- publish real tail and subgroup distributions and demonstrate zero catastrophic
  outliers on the launch sample.

No test fixture or synthetic observation is counted toward those open boxes.

## Automated verification

The integrated EVAL-004 source passed **4,486/4,486** non-duplicated
owning-package tests:

| Surface                                    |      Result |
| ------------------------------------------ | ----------: |
| Canonical contracts full package           | 4,080/4,080 |
| Creative autonomy full package             |     268/268 |
| Affected BFF autonomy route/bindings/E2E   |       50/50 |
| Yemaya case-eval full package              |       14/14 |
| Yemaya signed case-verifier full package   |       26/26 |
| V8 Loom service full package               |       29/29 |
| V10 Case Files cook/scheduler full package |         9/9 |
| V8 Daedalus compiler full package          |         3/3 |
| V10 fairness-receipt focused boundary      |         7/7 |

Nine EVAL-004 typechecks passed: contracts, creative autonomy, the targeted BFF
autonomy graph, case-eval, case-verifier, Loom, V10 Case Files cook, Daedalus,
and the targeted V10 receipt boundary. The canonical contracts Nx build and BFF
29.7 MiB production bundle passed.

The final source rebase exposed a concurrent broken `@oshun/bff-kit` extraction:
implementations had been replaced by recursive self-reexport shims while the
legacy app shims were empty. Compatibility commit `a432f8f891...` restores all
nine implementations to the shared package and all nine thin app reexports. Its
own 3/3 tests, strict typecheck, emitted build, lint, formatting, the BFF 50/50
suite, and production bundle pass. After the final remote sync, the kit 3/3, BFF
50/50, V10 receipt 7/7, and their three overlap-specific typechecks were
repeated and passed.

Root ESLint over all EVAL-004 tracked/new TypeScript completed with zero errors;
the only six warnings are existing non-null assertions in legacy creative
director/rubric code. The compatibility files pass with zero warnings. Prettier
and `git diff --check` pass.

Representative commands:

```text
pnpm --dir libs/contracts exec vitest run --testTimeout=120000
pnpm exec vitest run --config libs/oshun/creative-autonomy/vitest.config.ts
pnpm --dir apps/oshun/bff exec vitest run <three affected autonomy files>
pnpm --dir libs/yemaya/case-eval exec vitest run --config vitest.config.ts
pnpm --dir libs/yemaya/case-verifier exec vitest run --config vitest.config.ts
pnpm --dir apps/v8/loom-service exec vitest run --config vitest.config.ts
pnpm --dir apps/v10/case-files-cook exec vitest run --config vitest.config.ts
pnpm --dir apps/v8/daedalus-compiler exec vitest run --config vitest.config.ts
pnpm --dir apps/v10/web exec vitest run --config vitest.config.ts <two receipt files>
pnpm exec tsc --noEmit -p <each owning package tsconfig>
pnpm exec tsc --noEmit -p apps/oshun/bff/tsconfig.eval003.json
pnpm exec tsc --noEmit -p apps/v10/web/tsconfig.eval004.json
NX_DAEMON=false NX_TUI=false pnpm nx run @oshun/contracts:build --parallel=1 --skip-nx-cache
pnpm --dir apps/oshun/bff run build
pnpm exec tsc -p libs/shared/bff-kit/tsconfig.lib.json
pnpm exec eslint --no-warn-ignored <all changed TypeScript>
pnpm exec prettier --check <all changed non-lock files>
git diff --check
```

## Bounded verification notes

- The V10 app-wide typecheck was attempted and is blocked by the checkout's
  pre-existing missing package links (`next`, Tauri, Axe, and numerous workspace
  packages), with downstream implicit-any diagnostics. The permanent
  `tsconfig.eval004.json` checks the exact changed receipt/schema boundary and
  passes; both focused receipt test files pass 7/7.
- The first focused V10 receipt run exposed the app config's bare-contract alias
  precedence. An exact `@oshun/contracts/v10` alias was added; the clean rerun
  passed 7/7.
- The BFF's full application typecheck is intentionally not repeated after the
  previously observed 4 GiB heap exhaustion. The permanent affected-graph
  typecheck, 50 tests, strict shared-kit typecheck, and production bundle pass.
- A case-eval Nx build is not repeated because it expands into 37 dependencies
  and previously stopped on unrelated Iris/Isis failures. The owning case-eval
  test/typecheck and every downstream changed app test/typecheck pass.
- Loom and the shared kit emit existing missing source-map warnings from shared
  HTTP-client generated JavaScript. They do not change test results.
- The 15 GiB RAM/31 GiB swap host was checked before expensive verification;
  broad workspace gates were not run, checks were serialized, and memory stayed
  above the mandated stop thresholds.
- The unrelated binary `pnpm-lock.yaml` modification was preserved byte-for-byte
  through both rebases (Git blob `a37431842e330b3a06b030947c25bebdc72bc2b1`).
  Generated Cargo locks and `libs/urania/` were not staged.
- No customer UI, PWA, admin, mobile, Unreal, or visual behavior changed. The
  V10 web change is a receipt fixture/resolver/type boundary only. Frontend
  skill, Playwright, mobile automation, screenshots, and Unreal verification
  were inapplicable.

## Artifact identity

| Artifact                                | SHA-256                                                            |
| --------------------------------------- | ------------------------------------------------------------------ |
| Canonical human-quality launch contract | `c6345f38dee164e09ca5d7e0fc933ea9d89fe893b767f9dcd7dadf73ada1f5c1` |
| Creative director launch boundary       | `7f283656d76a89acace66c4050fb02acedc1e5374914e1842acf6c3c8d6ac086` |
| Yemaya G8 release boundary              | `338d95fba990907b984467627ddb8a064a5e11dc84d6db8384d66037c16f1630` |
| V8 signed release authentication        | `297dfb3cde1e354ab77b07a0e6ba955a6c1ce67166b7088f7ad87f225faebddf` |
| V10 signed Case Files receipt contract  | `92b9d9d34f785df789496d786946c4baa73b6d4e50f958b84cfcf911e67f2295` |
| BFF fail-closed composition             | `9208b59d0a1308a46c75f74499ed9dc856b1661c0766c7fa475294a302e2a5e8` |
| Restored shared BFF auth implementation | `91d2b0ceea1f5b4b690f889fa86927aa8e4553211304fc73f0c44952e7f17296` |

## Publication identity

- **Repository source commit:** `401b5496b6a00aa1a2f3a2f3d53caef16a60a14a`
- **Required compatibility commit:** `a432f8f8915431e1062218e598760f9896a15d34`
- **Repository source publication tip:**
  `a432f8f8915431e1062218e598760f9896a15d34`
- **EVAL-004 closure evidence commit:** not applicable; EVAL-004 remains open.
