Eve library

EVAL-003 progress evidence — calibrated model-judge release boundary

Evidence

Revision 1

On this page

Status: partially complete, not closed. The repository now prevents lexical diversity and uncalibrated/PCA model scores from making release or search decisions. It does not claim that any real human-gold calibration has occurred. EVAL-002 still records zero acquired corpus/cohort units.

Work-package charter#

  • Task ID: EVAL-003
  • Owner: evaluation governance, model-judge calibration, creative-autonomy quality, and V8/V10 case release; repository implementation owner: Codex
  • Audit finding: position-mitigated pairwise judging and several isolated calibration statistics existed, but no canonical record bound heterogeneous judge identities, human holdouts, bias probes, metrics, routing, approvals, and change regression. Creative-autonomy production still selected with lexical similarity, while V8's batch-dependent PCA/heuristic G5 verdict was a hard release gate.
  • Repository objective: make uncalibrated judge scores and lexical/PCA diversity diagnostic-only; fail closed unless exact, current, human-approved calibration and validated semantic evidence are present.
  • External objective: acquire the EVAL-002 human gold, recruit independent raters, run the blinded calibration study and bias probes, and issue the first real approved certificates.
  • Rollback strategy: revert source commit 47438840440d658eeb7f6b4185fcb1ed68f384fb.
  • Evidence-pack location: this file.

Canonical judge-calibration evidence#

ModelJudgeCalibrationCertificate and its supporting records now require:

  • exact judge, provider, provider-version, model, model-version, model-family, prompt digest/version, rubric version, policy version, and response-privacy identity;
  • declared provider/model-family minima, computed diversity shortfall, and an explicit degraded reason; degraded diversity cannot receive release approval;
  • blinded aliases, sealed identity-map digest, both candidate orders, both raw order picks, per-order confidence/evidence, and a winner derived from order agreement rather than claimed by the caller;
  • position, verbosity, style, self-family, reference-leakage, contamination, and judge-pleasing bias probes, with exact heldout items and threshold-derived pass/fail;
  • per-item, per-dimension, per-judge human observations with multiple human rating references, confidence, subgroup labels, and OOD status;
  • tie-aware Spearman rank correlation, confusion by dimension, MAE, ECE, Brier score, ICC(C,k), within-item disagreement range, and subgroup slices;
  • mandatory human routing for high disagreement, low confidence, or OOD input;
  • paired heldout champion/challenger error regression with a lower 95% confidence bound; approved certificates require both human-alignment and champion/challenger policy passes;
  • sealed release-holdout corpus identity/digest, clean contamination status, signed human approval, validity window, revocation state, and exact evidence references.

The contract/Prisma alignment manifest records this evidence as code_only. Generic model, provider, evaluation, or feedback rows cannot be projected as a calibrated release certificate.

Blinded runtime judging and semantic diversity#

Creative-autonomy now requires both a ValidatedSemanticDiversity binding and an approved JudgeCalibrationBinding before constructing a director. The director verifies that:

  1. semantic provider/model/version, dimensions, validation corpus/evidence, and validity window are complete and current;
  2. the certificate is approved, current, non-degraded, human-aligned, bias- probe clean, and champion/challenger passing;
  3. the certificate's judge IDs exactly equal the runtime pointwise and pairwise panel—neither an uncertified live judge nor an absent certified judge is accepted;
  4. every release tournament returns blinded forward and reverse identity picks plus evidence references. Single-order verdicts fail loud.

Cycle and review provenance retain the raw pairwise comparison evidence and the gate verdict retains the calibration certificate and panel version.

Production novelty, candidate admission, failed-topic cooldown, and portfolio clustering use only the evidence-bound semantic model. The semantic model and version are recorded in the score. Token/trigram and hashing-bag-of-words values remain available only as explicitly labeled lexical-diagnostic-only features. BFF composition remains fail-closed until a real semantic binding and real judge certificate are injected; test certificates and embedders live only in named test-support files.

V8/V10 diagnostic-only G5#

JudgeVerdict and judgePanel still expose V8's deterministic/LLM refinements and batch-dependent PCA aggregate for diagnosis and regression analysis. They can no longer turn release G5 green.

decideRelease now requires a separate CalibratedG5ReleaseEvidence record for the exact case. It checks verdict/certificate/corpus SHA-256 digests, approved human authority, heterogeneous-panel state, human-alignment and bias passes, heldout champion/challenger pass, clean contamination status, human routing, future dating, expiry, and the final calibrated verdict. Missing evidence blocks release while naming the diagnostic PCA aggregate without promoting it.

The boundary is threaded through Loom, Daedalus, and V10 Case Files. Loom accepts governed composition evidence; Daedalus and the V10 cook CLI require an explicit evidence file; the tomorrow scheduler requires CASE_FILES_G5_EVIDENCE_FILE. Positive tests inject conspicuously named test fixtures. Negative tests prove that a 0.99/green heuristic score still blocks without calibration, and also cover wrong-case, expiry, failed heldout regression, and missing semantic/judge evidence.

Checklist disposition#

Three repository-enforceable EVAL-003 requirements are complete:

  • bag-of-words and V8 PCA/G5 are diagnostic-only for production decisions;
  • creative candidate admission/selection uses a validated semantic model and records exact model/version while retaining lexical diagnostics separately;
  • model, prompt, rubric, examples, weights, thresholds, provider, and policy changes require approved heldout champion/challenger evidence.

Five items remain open because their completion requires real EVAL-002 human data and observations, not schemas or fixtures:

  • operate a real heterogeneous provider/model panel and record any degradation;
  • run real blinded human/model comparisons in both orders;
  • measure all seven declared bias families on heldout human gold;
  • publish the real human-judge correlation, dimensional confusion, calibration, reliability, ICC, disagreement, and subgroup report;
  • exercise the high-disagreement, low-confidence, and OOD human route with consented independent raters.

No test fixture is counted as evidence for those five open boxes.

Automated verification#

The final integrated source passed 4,440/4,440 non-duplicated owning-package tests:

Surface Result
Canonical contracts full package 4,067/4,067
Creative autonomy full package 267/267
Affected BFF autonomy route/bindings/E2E/poller 52/52
Yemaya case-eval full package 13/13
V8 Loom service full package 29/29
V10 Case Files cook/scheduler full package 9/9
V8 Daedalus compiler full package 3/3

The final post-rebase verification repeated V10's 9/9 suite, the 16 focused contract/alignment tests, and the affected contracts, BFF, and case-eval typechecks.

Production typechecks passed for contracts, creative autonomy, case-eval, Loom, V10 Case Files, Daedalus, and the resource-bounded BFF autonomy graph. The canonical contracts Nx build and the BFF 29.7 MiB production bundle passed. Prettier and git diff --check passed. New contracts/test-support files pass the repository ESLint policy with zero warnings; the remaining modified legacy files pass ESLint with the repository's existing non-null-assertion warning disabled and no other warnings or errors.

Representative commands:

text
pnpm --dir libs/contracts exec vitest run --testTimeout=120000
pnpm exec vitest run --config libs/oshun/creative-autonomy/vitest.config.ts --testTimeout=120000
pnpm exec vitest run <four affected BFF files> --config apps/oshun/bff/vitest.config.ts --testTimeout=120000
pnpm --dir libs/yemaya/case-eval exec vitest run --testTimeout=120000
pnpm --dir apps/v8/loom-service exec vitest run --testTimeout=120000
pnpm --dir apps/v10/case-files-cook exec vitest run --testTimeout=120000
pnpm --dir apps/v8/daedalus-compiler exec vitest run --testTimeout=120000
pnpm exec tsc -p <each owning package tsconfig> --noEmit
pnpm exec tsc -p apps/oshun/bff/tsconfig.eval003.json --noEmit
NX_DAEMON=false NX_TUI=false pnpm nx run @oshun/contracts:build --parallel=1 --skip-nx-cache
pnpm --dir apps/oshun/bff run build
pnpm exec prettier --check <all changed non-lock files>
git diff --check

Bounded verification notes#

  • The first full contracts run passed 4,066 tests and failed only because the new contract was absent from the exhaustive contract/Prisma alignment table. After adding the honest code_only entry, the focused suites passed 16/16 and the clean full rerun passed all 176 files and 4,067 tests.
  • The app-wide BFF typecheck exceeded V8's 4 GiB heap while another Codex held about 6 GiB and an active Nx daemon held about 0.9 GiB on the 15 GiB RAM/31 GiB swap host. The permanent tsconfig.eval003.json checks the exact affected BFF autonomy graph; it found a real response-digest property mismatch, which was fixed, and then passed. The BFF production bundle and 52 affected tests also passed.
  • A case-eval Nx build expanded to 37 dependencies. It stopped after unrelated existing failures in Iris agents (implicit-any/module resolution) and Isis client DTS generation (TypeScript 6 baseUrl deprecation). The owning case-eval typecheck/test passed, contracts built, and the remaining app-owned build/typecheck paths passed; no unrelated dependency source was altered.
  • Loom emits existing missing source-map warnings from shared HTTP-client generated JavaScript; contracts emit the existing missing libs/contracts/src/index.js.map warning. Neither changes test results.
  • The source rebase incorporated concurrent Daily Wonder schedule assertions; V10's full suite and all affected typechecks passed afterward.
  • The unrelated binary pnpm-lock.yaml modification was restored byte-for-byte after the rebase (Git blob a37431842e330b3a06b030947c25bebdc72bc2b1). Generated Cargo locks and libs/urania/ were not staged.
  • No customer UI, PWA, admin, mobile, Unreal, or visual surface changed. Frontend skill, Playwright, mobile automation, screenshots, and Unreal verification were inapplicable.

Artifact identity#

Artifact SHA-256
Canonical judge-calibration contract 47692dc759f5ccd4cdeeaa1b7ca1709ce00b7715fa47dfa5e9074b5e09f1e768
Creative director release boundary 7bf18a1521abb42438bd09891024f50a4aeb0db2ba2454556afe4be40d435290
Validated semantic-diversity boundary e48103e48525c7e51213aa841ae2ef8bd22733f63c159f48200772924c0d0541
Blinded dual-order tournament record b2c84bef727c6bd7fc862bba2dd1c511f6e2ed5ee397882f3e59b2c52379f660
V8 calibrated G5 release boundary 74beb64b72d398a5d28160bc5d69aed9a1f12318598ab2e7abfe2a25713e80bf
BFF fail-closed composition d5823c131a264d029bd171a0f57ee823932fad582b8932f3d320ee3ab422a9d9

Publication identity#

  • Repository source commit: 47438840440d658eeb7f6b4185fcb1ed68f384fb
  • Repository source publication tip: 47438840440d658eeb7f6b4185fcb1ed68f384fb
  • EVAL-003 closure evidence commit: not applicable; EVAL-003 remains open.