Eve library

EVAL-001 evidence pack — unified evaluation record and harness

Evidence

Revision 1

On this page

Status: repository-complete / closed. The canonical contract and central Nous entry point preserve versioned tasks, trials, evidence, grader authority, statistics, slices, and fail-loud resource preflight. This status does not claim that every legacy evaluator has migrated, that live providers or human cohorts are available, or that production evaluation runs have occurred.

Work-package charter#

  • Task ID: EVAL-001
  • Owner: canonical contracts, evaluation governance, and Nous continuous evaluation maintainers; implementation owner: Elikem Agudogo
  • Accountable product: one non-substitutable portfolio evaluation record and execution boundary for deterministic, model, provider, engine, human, and live-outcome grading
  • Priority: P0/P1 cross-product evaluation-integrity gap
  • Target milestone: 2026-07-15 SOTA gap-closure program
  • Risk tier: Critical evaluation-integrity and release-evidence risk
  • Audit finding: evaluation implementations used incompatible result shapes, usually represented one attempt, blurred grader authority, omitted exact resource requirements, and could silently continue without required engines, providers, source bundles, or human cohorts
  • Dependencies: canonical contract/Prisma alignment, deterministic statistics, injected grader/executor registries, exact versioned resources, and Nous continuous-evaluation composition
  • Affected schemas and content slots: suite, task, transcript, trajectory, artifact, outcome, grader, score, uncertainty, cost, latency, failure, trial, summary, slice, and run records; Nous canonical runner output. No rendered UI, media bytes, or database migration changed in this refresh.
  • Linked task IDs: EVAL-001, EVAL-002, EVAL-003, EVAL-004, EVAL-005, PROOF-001, LEARN-001, and OBS-001
  • Rollout strategy: add a strict versioned cross-product contract and injected execution harness, then make it the canonical entry point on the central Nous runner while retaining legacy projections for existing callers
  • Rollback strategy: revert the original contract/harness source and central Nous adapter together. Partial rollback can leave consumers compiled against an absent or weaker evaluation envelope.
  • Evidence-pack location: this file

Closure matrix#

Ledger requirement Repository closure proof
Versioned canonical records Schema version 1 defines strict suite, task, transcript, trajectory, artifact, outcome, grader, score, uncertainty, cost, latency, failure, trial, summary, and run records. IDs and subject bindings are checked across the entire envelope.
Distinct grader authorities A discriminated grader union preserves deterministic code, solver, model grader, provider probe, engine/runtime, expert rater, target-user rater, and live-outcome identities. Model, provider, engine, prompt, rubric, policy, sandbox, bundle, cohort, and evidence versions remain explicit where applicable.
Distinct suite kinds Every suite declares exactly one of capability, regression, adversarial, safety, reliability, subgroup, cost, or live-canary. Tests exercise all eight kinds and reject unknown kinds.
Repeated nondeterministic trials and statistics Nondeterministic suites require at least two repetitions and one unique seed per repetition. Trial evidence preserves sample index, seed, model/provider references and versions, temperature, scores, uncertainties, cost, latency, and failures. Summaries include Wilson 95% intervals, unbiased pass@k, all-pass pass^k, seed/version inventories, and failure clusters.
Required slice dimensions Every task carries product, slot, modality, locale, audience, model, provider, prompt, rubric, policy, autonomy tier, and risk tier. Run summaries aggregate every dimension, and model/provider task labels must match the actual execution configuration.
Fail loud for unavailable evaluation resources Preflight resolves every explicitly required and grader-inferred dependency, source bundle, engine, provider, human cohort, grader, and executor before dispatching a single trial. Missing resources raise a typed UnifiedEvaluationConfigurationError with the exact absent inventory.

Canonical evidence envelope#

UnifiedEvaluationRunRecordSchema is a closed, versioned envelope rather than an untyped benchmark result. It contains immutable suite/task snapshots, every trial, aggregate summaries, and exact timestamps. Each trial retains:

  • task, suite, run, sample, and seed identity;
  • effective model/provider references and versions plus temperature;
  • transcript and agent trajectory;
  • zero or more exact artifacts with media type, locator, and SHA-256;
  • observed outcome;
  • full grader descriptors and their score records;
  • uncertainty, cost, latency, and typed failure records.

Cross-record refinement rejects trial/run/task substitution, mismatched artifact or outcome subjects, score/grader substitution, duplicate trial identity, duplicate nondeterministic seeds, missing repetitions, and a runtime model/provider that disagrees with the task slice. Strict schemas reject unrecorded fields instead of silently discarding evidence.

Cost has explicit measured and unavailable variants. An unavailable cost names the reason and cannot carry a fabricated amount. External grader versions are mandatory on model graders, provider probes, and engine/runtime graders, so an evaluation cannot be reproduced from a provider or engine name alone.

Harness and fail-loud execution proof#

runUnifiedEvaluation validates the complete request and derives its preflight inventory before invoking the executor. The inventory spans:

  • registered grader implementations;
  • the trial executor;
  • suite- and task-level dependencies;
  • suite- and task-level source bundles;
  • engines and engine versions;
  • providers and provider versions;
  • expert and target-user cohorts.

Grader descriptors infer their operational requirements. A deterministic-code grader requires its sandbox and test bundle, a provider probe requires the named provider version, an engine/runtime grader requires the engine version, and human graders require their declared cohort. Explicit and inferred requirements are combined and deduplicated.

For nondeterministic suites, the harness executes every task once for every declared seed. Per-trial overrides are accepted only when they retain the required model/provider references and versions and temperature. Executor and grader failures become typed per-trial failure evidence; missing infrastructure is a configuration error and stops before dispatch. This prevents an unavailable evaluation system from being reported as a tested product failure or passing empty run.

Statistics and slicing proof#

summarizeUnifiedEvaluation computes results only from canonical trials. It reports successful and failed counts, a Wilson 95% confidence interval for observed pass rate, unbiased pass@k from the finite sample, and pass^k as the probability that every one of k attempts passes. It retains every seed, model/provider version, temperature, and normalized failure cluster.

The run emits slice summaries over all twelve mandatory dimensions: product, slot, modality, locale, audience, model, provider, prompt, rubric, policy, autonomy tier, and risk tier. Consumers can compare products or providers without reconstructing labels from prompts or losing subgroup/risk context.

Central Nous adoption#

UnifiedBenchmarkRunner.runUnified(...) delegates directly to the canonical contracts harness and returns its validated run record. The adapter test invokes the real central runner with repeated trials and proves that transcripts, trajectories, artifacts, outcome, graders, costs, latency, model/provider versions, seeds, and statistics survive the boundary. Existing run and runAll methods remain documented compatibility projections; new portfolio evaluation code has one canonical entry point.

Adversarial fixture matrix#

Fixture Result
Any of the eight suite kinds Parses and executes with its distinct kind retained
Nondeterministic suite has one repetition or duplicate seeds Schema rejects the suite before execution
Trial override erases model/provider/version/temperature Harness rejects the request before dispatch
Task slice labels another model or provider Run schema rejects the substituted execution record
Trial, score, artifact, or outcome names another subject Cross-record refinement rejects the envelope
Required grader is unregistered Typed configuration error; executor call count remains zero
Required dependency or source bundle is absent at task level Typed configuration error; executor call count remains zero
Deterministic sandbox or test bundle is absent Inferred missing-resource error before dispatch
Provider or engine version is unavailable Inferred missing-resource error before dispatch
Expert or target-user cohort is absent Inferred missing-resource error before dispatch
Executor throws after successful preflight Typed trial failure retained in the run and failure cluster
Some repeated trials pass and others fail Exact sample/seeds retained; pass@k, pass^k, and confidence interval computed

Automated verification#

Current admitted value ledger#

Both owning packages were rerun sequentially with VITEST_MAX_WORKERS=1 from published origin/main tip 795b41f0a703312ae5e9ca82e3650d4c45047a8e.

UTC interval Files Result Surface
2026-07-29T14:46:22.576Z–2026-07-29T14:48:29.908Z 263 5386/5386 full canonical contracts package
2026-07-29T14:48:36.024Z–2026-07-29T14:48:56.010Z 87 490/490 full Nous core package and canonical runner adapter
Unique total 350 5876/5876 distinct package files; no focused assertion counted twice

The unchanged EVAL-001-focused subset also passed 23/23 across its two canonical contract files, and the Nous adapter passed 1/1. Those assertions are already included in the full-package rows and are not added again. The former 4,527/4,527 total correctly described the smaller historical package suites.

Representative commands:

text
VITEST_MAX_WORKERS=1 pnpm --dir libs/contracts exec vitest run
VITEST_MAX_WORKERS=1 pnpm --dir libs/nous/core exec vitest run
pnpm exec vitest run --config libs/contracts/vitest.config.ts <two EVAL-001 files>
pnpm exec vitest run --config libs/nous/core/vitest.config.ts <adapter file>
pnpm exec tsc -p libs/contracts/tsconfig.lib.json --noEmit
pnpm exec tsc -p libs/nous/core/tsconfig.lib.json --noEmit
NX_DAEMON=false NX_TUI=false pnpm exec nx run @oshun/contracts:build --parallel=1 --skip-nx-cache
NX_DAEMON=false NX_TUI=false pnpm exec nx run nous-core:build --parallel=1 --skip-nx-cache

Type, lint, build, and compatibility gates#

  • production library typechecks passed for contracts and Nous core;
  • exact ESLint with ignored test files included passed over all six EVAL-001 implementation and fixture files with zero warnings; Prettier and git diff --check passed;
  • the uncached contracts production build passed;
  • the uncached Nous core production build and its three sequential dependencies (@oshun/contracts, @oshun/resilience, and @oshun/math) passed;
  • artifact hashes remain byte-for-byte identical to the original implementation closure.

Bounded red gates#

Red static gates remain explicit and are not represented as passing:

  • contracts tsconfig.spec.json emits 178 lines of unrelated legacy fixture debt. No diagnostic names unified-evaluation.test.ts or unified-evaluation-harness.test.ts;
  • Nous tsconfig.spec.json emits 19 lines from shared Vitest/Vite config typing and the pre-existing inference-engine.spec.ts ONNX mock. No diagnostic names unified-evaluation-adapter.spec.ts;
  • the full runtime suites compile and execute the affected specs successfully, but runtime success is not substituted for the red broad TypeScript configs;
  • no full-workspace sweep or AUD-003's separate 54-config refresh was attempted or claimed.

Artifact identity#

These hashes were recomputed from current published source.

Artifact SHA-256
Canonical record and statistics 3e0c4844264170bba441cfe30ac6d07f5a0cfb20b15a9f6df141d755d7ab3d2c
Fail-loud repeated-trial harness e46e565506cf4427c1c12caacdd18791570b7a38cc8e709ca6c397e8f446c587
Canonical schema/statistics fixtures 8f2a4ee63af7f3fb900b55c27af554792ee9c4d79745e73e649e683ffa293689
Harness adversarial fixtures ee114a7e72becd35ab58fd422d75058e90c8b3dc6579567ddf41da529454f9f1
Central Nous runner fe002d64aacbf5b4e5143f9b4850d2f75e9c00151fdc82863402f89c53ae576e
Nous canonical adapter fixture c40163cd5eefa3bd0f3676ebd94acda80124872a07bff2b98632cc0c637ebdad
Workspace dependency lock 08b055786e9b8a6104d5313601d7eef3205dedcf395066fd9a8264c9859d690f

Human authority and external deployment boundary#

Repository publication of this deterministic contract and its fixtures requires no fabricated human signoff, and none is claimed. Passing tests prove record, preflight, execution, and summary behavior; they are not evidence that a live provider, engine, model grader, expert cohort, target-user cohort, or production canary was available or executed.

  • providers, engines, solvers, model graders, source bundles, and human cohorts remain injected, versioned deployment resources; absence fails before trial dispatch;
  • this package does not claim every legacy domain benchmark has migrated from compatibility result types;
  • durable cross-product evaluation-ledger persistence remains code_only in contract/Prisma alignment because no authoritative run/trial/grader database schema exists; the harness does not fabricate storage;
  • licensed human-gold acquisition, judge-bias measurement, task-specific batteries, and live canaries remain EVAL-002 through EVAL-005/external-state scope;
  • no production evaluation, live cohort, provider call, population outcome, or deployment is claimed.

Resource-safe verification and UI scope#

The refresh host was app-dev-server, Ubuntu 24.04.4 LTS, Linux 6.8.0-134-generic x86_64, with 15 GiB RAM, 31 GiB swap, and 16 GiB free disk. The toolchain was Node.js v24.15.0, pnpm 10.25.0, TypeScript 6.0.3, and Vitest 4.1.7.

An unrelated checkout's long-running Vitest worker and this checkout's Nx daemon were active. Package tests, typechecks, and builds ran sequentially; available memory remained near 11–12 GiB during the supervised contracts test/build intervals. No full workspace gate was started.

This refresh changes evidence only. It does not change customer/admin/PWA/ mobile/Unreal UI, so the frontend skill, Playwright, mobile automation, screenshots, and Unreal verification are inapplicable.

Publication identity#

  • Original source commit on the current graph: 078e72990b264fb2a48d644026ab9f3e1aa30c75
  • Historical equivalent source identity (identical tree): 2ce7dca60b2e2dfe9a5484884efff1195a47ab45
  • Historical evidence-pack commit on the current graph: b950989e7243a53a86edcaf370edd4f256e92fa6
  • Historical equivalent evidence identity (identical tree): 1d99982533499f26e878a21ee73d1d09c953baab
  • Historical ledger closure commit: 4aaec3d86f2ce63ab2f0e02e0244fde1d74e0056
  • Current admitted verification tip: 795b41f0a703312ae5e9ca82e3650d4c45047a8e
  • Current evidence-refresh commit: recorded by the task-scoped commit that contains this file.