Eve library

AUD-007 evidence pack — canonical test coherence

Evidence

Revision 1

On this page

Work-package charter#

  • Task ID: AUD-007
  • Owner: Oshun portfolio audit-integrity maintainer; implementation owner: Codex
  • Accountable product: V1–V10 autonomous-content portfolio
  • Priority: P0
  • Target milestone: 2026-07-15 SOTA gap-closure program
  • Risk tier: High audit-evidence and release-claim integrity
  • Dependencies: AUD-003 canonical verification manifest and authenticated record, AUD-006 generated capability truth, and the 54 credited Vitest configurations
  • Affected schemas and content slots: New oshun.test-coherence-manifest.v1 audit schema and generated coherence view; no shipped content slot or customer flow changed
  • Rollout strategy: Register one source-bound domain-value assertion per credited configuration, classify the proof actually exercised, require explicit external evidence before any high-evidence classification, and publish limitations beside the authenticated counts
  • Rollback strategy: Revert the schema, manifest, generator, report, audit section, package scripts, and workflow changes together. Do not leave the canonical counts without their proof-class boundary.
  • Evidence-pack location: This file and evidence/v1-v9/AUD-007/

Implemented coherence contract#

2026-07-22 scope amendment#

The active contract now samples 54 credited configurations from scripts/audit/v1-v10-verification-manifest.json. V10 contributes 19 configurations with 782/785 tests passed, 3 explicitly pending, and 0 failed. The current canonical commands are pnpm generate:audit:v1-v10-verification-record and pnpm verify:audit:v1-v10-verification-record. Publication identities, distribution counts, and command results below retain the initial 2026-07-15 evidence window as history.

  1. scripts/audit/test-coherence-manifest.json maps exactly one sampled test file and domain-value assertion to each of the 54 Vitest configurations credited by scripts/audit/v1-v10-verification-manifest.json. The generator rejects missing, duplicate, out-of-configuration, or out-of-filter samples.
  2. Every sample is classified as unit, contract, integration, e2e, real-provider, engine, or live. Non-unit classes require a source-resident classification marker. Real-provider, engine, and live classes additionally require an external evidence reference, so a local suite cannot impersonate them.
  3. The initial publication evidence distribution was 17 unit, 5 contract, 9 integration, and 4 local/in-process E2E configurations. The generated view carries the active distribution, including V10's 18 unit and 1 contract samples. Real-provider, engine, and live counts remain zero.
  4. Twenty-seven configuration samples bind a negative or deliberately broken fixture. This includes the V8 case evaluation mutation that drops a required deduction edge and must fail G4. The adversarial contract suite also removes a registered domain assertion in memory and proves the coherence gate fails, and it proves that assigning real-provider status without external evidence fails.
  5. evidence/v1-v9/AUD-007/test-coherence.generated.md publishes the sampled claim, assertion marker, proof class, negative-fixture marker, weakness, and uncovered claims beside each recorded pass count. Canonical audit section 2.2 is generated from the same data.
  6. .github/workflows/test-coherence.yml enforces source-marker, record, and generated-report drift on the full credited source boundary. The existing audit-record workflow also runs the coherence verifier.

Publication identity#

  • Coherence implementation source commit: d76869b2ab861c4958f89c7447c2236c1ea7920f
  • Canonical verification-record publication commit: 977b335bbcb04a409c466c45d8c804e2505dd509
  • Verification-record run interval: 2026-07-15T17:21:30.977Z through 2026-07-15T17:26:12.076Z
  • Host/runtime: dev-server, Linux 6.8.0-110-generic x64, Node.js v24.14.1, pnpm 10.25.0, Vitest 4.1.7, git 2.43.0
  • Provider/model gates: no OpenAI, Anthropic, RunPod, or database credential was present; no real-provider, engine, or live execution is claimed
Artifact SHA-256
scripts/audit/test-coherence-manifest.schema.json 4240ee120dc660b937d6ba9670f6f6694474053581eb2615a5e4b68f5ecf1536
scripts/audit/test-coherence-manifest.json ffe203876c3dd1eafd5dd9f1d2c4d5d1ab526eafffbcdc1434ad3fc60ea95928
scripts/audit/generate-test-coherence.mjs 8d324634d531a4dd392b60c0e33e3406e6ca7bd2fcbcc8af3bf090a00267606a
scripts/audit/test-coherence-manifest.test.mjs 06d328749b270e50c0fcf3b6e98220bba845066aa4d4edac0ad2f037c12d98c8
.github/workflows/test-coherence.yml 996e391659544a29232d956a82d8357d5117e1a00248f492f56163f3f484f2fd
evidence/v1-v9/AUD-007/test-coherence.generated.md 560876f8ad46a1a3cb58c03580c856638c36f1a018e39e211c59b34c2c2c432f
evidence/v1-v9/AUD-003/verification-record.json 0356552d4184dec3e27eff215d3f447f70248cda3cf3334708b353aa83ab5689

Automated and adversarial verification#

  • pnpm generate:audit:v1-v9-verification-record ran sequentially with Vitest --maxWorkers=1 from the clean implementation source. It recorded V1 counts of 248/248, 289/289, 138/138, and 40/40; V8 at 351/351 across 18 configurations; and V9 at 176/176 across 13 configurations. The known V8 generated-contract drift remained measured red with exit 1.
  • pnpm verify:audit:v1-v9-verification-record passed 8/8 adversarial tests and record/table/command/boundary/log/source coherence.
  • pnpm verify:audit:test-coherence passed 6/6 adversarial tests plus the live workspace drift check.
  • pnpm verify:audit:capability-truth passed 7/7 adversarial tests and all five generated-view checks after the new section-order integration.
  • pnpm verify:audit:machine-adjacent-docs passed 3/3 tests and its workspace contract check.
  • JSON Schema Draft 2020-12 validation of the coherence manifest, targeted ESLint with --no-ignore --max-warnings=0, targeted Prettier, JavaScript syntax checks, all three affected workflow YAML parses, and git diff --check passed.
  • Resource supervision stayed near 12 GiB available memory with roughly 30 GiB free swap; no safety floor was approached.

Claim boundary and inapplicable avenues#

  • This is one load-bearing assertion sample per credited configuration, not an assertion-by-assertion audit of every test in those suites. The generated weakness and uncovered-claim columns preserve that boundary.
  • Passing local unit, contract, integration, or in-process E2E tests does not prove a deployed provider, Unreal placement, production release, player or learner outcome, efficacy, or live customer behavior.
  • The negative-fixture count records source-bound, canonically executed fixtures; it does not claim exhaustive mutation coverage or a mutation score.
  • No customer UI, mobile flow, media artifact, API contract, or persistence behavior changed. Playwright, mobile automation, accessibility scans, Unreal execution, provider probes, and media-quality evaluation are inapplicable.
  • No human approval is required or manufactured by this work package.