Eve library

MEDIA-001 evidence pack — media critique and evaluated V8 assets

Evidence

Revision 1

On this page

Status: repository-complete / closed. Video-producing seams consumed by V1 critique now expose a real sampled frame or produce a typed hold; V8 asset minting runs a fail-closed best-of-N tournament with exact media, quality, rights, and candidate-retention evidence; and canonical media batteries keep external benchmarks diagnostic-only. This closure does not claim a deployed evaluator, live human comparison, production provider result, or population superiority.

Work-package charter#

  • Task ID: MEDIA-001
  • Owner: V1 media generation/critique, canonical evaluation contracts, and V8 Loom asset-production maintainers; implementation owner: Elikem Agudogo
  • Accountable products: V1 BFF media generation and autonomy critique; V8 Yemaya case-assets and Loom asset realization/release
  • Priority: P0/P1 multimodal quality and release-integrity gap
  • Target milestone: 2026-07-15 SOTA gap-closure program
  • Risk tier: high multimodal release-quality, candidate-substitution, rights, accessibility, and experiential-evidence risk
  • Dependencies: canonical media-evaluation contracts; V8 case contracts, case-assets, and Loom; ffmpeg/ffprobe video sampling; V1 creative critique and revision; artifact-rights receipts from PROOF-003
  • Affected schemas and content slots: MediaEvaluationBundle, image/video/ audio/3D/gameplay probe and human-comparison records, V8 asset selection and manifest records, V1 video provider output, and composed media-critic result. The affected content slots are generated V8 image, video, audio, and 3D assets plus critic-consumed V1 video; no rendered UI layout changed.
  • Linked task IDs: MEDIA-001, EVAL-005, PROOF-003, V1-002, V8-001, and V8-005
  • Audit finding: video critique could receive no visible frame; V8 asset minting had no production best-of-N tournament or exact experiential evidence
  • Rollout strategy: additive evaluation records and candidate evidence, followed by fail-closed requirements at V1 critique and V8 production/release
  • Rollback strategy: revert source commit f8046ba3a573f082aa24e95573cd30de8a5c31d6; this restores the single-candidate V8 mint path and the previous untyped missing-frame failure
  • Evidence-pack location: this file

Closure matrix#

Ledger requirement Closure proof
Frame output on every critic-consumed video producer The BFF VideoProviderGenerate contract now requires both a clip URL and a representative keyframe URL. The only producer adapters feeding the video executor probe the real produced clip with ffmpeg, extract the first sampled PNG, expose it as a data URL plus SHA-256, and forward it through both text-to-video and image-to-video executors. Empty clip or frame values fail loud.
Typed missing-frame hold The composed media critic resolves its visible image before dispatching either VLM call. An absent video thumbnail, poster, or keyframe returns disposition: held with code media_required_frame_missing, required and observed evidence, and regeneration guidance. The revision loop stops after that pass, preserves the hold on the artifact, and does not fabricate a provider-usage receipt for a pre-dispatch hold.
V8 best-of-N and three-axis quality The real @yemaya/case-assets Loom path derives deterministic candidate seeds, submits every configured variant, requires exact media evaluation, artifact-rights receipts, and perceptual/period/canon evidence, and selects the highest passing composite with deterministic seed/ID tie breaks. Production defaults to three variants, permits 2–16, requires an injected or HTTPS evaluation service, and fails configuration when none exists.
Preserve winner and losers Every selection record retains requested variant count and every candidate's ID, job, seed, rendered prompt, provider, model, locator, MIME type, filename, artifact digest, status, three-axis scores, canonical evaluation, rights receipts, hold reasons, and selected flag. Provider failures and unevaluated candidates remain failed/held records; all attempted jobs consume budget. Manifests retain all-held tournaments as blocking evidence.
Artifact-specific media batteries and human comparison The canonical contract defines and executes complete image, video, audio, 3D, and gameplay batteries. It requires exact artifact subjects and SHA-256, versioned non-diagnostic probes, evidence references, and blinded side-randomized human comparison. Missing probes, failed thresholds, absent/losing human studies, subject substitution, duplicate/unknown candidate evidence, or unreported model/provider identities hold release. External benchmarks are diagnostic-only and cannot satisfy a missing probe or human study.

Video frame and typed-hold proof#

probeVideoContent samples real frames from the produced clip rather than inventing a poster locator. Its result now contains:

  • a data:image/png;base64,... keyframe derived from the first sampled PNG;
  • SHA-256 over those exact PNG bytes;
  • the existing duration, dimensions, luma, temporal, and safety measurements.

The environment-backed provider refuses a missing clip URL, runs this probe for every successful provider response, and returns the keyframe through the required producer contract. The governed executor independently rejects blank clip or keyframe URLs before exposing output to autonomy bindings.

The composed critic treats visible-frame availability as a prerequisite rather than an exception path. A hold names the missing evidence and directs the next producer to emit a real extracted thumbnail, poster, or keyframe. Standalone VLM critics retain their fail-loud behavior for direct callers; the composed autonomy boundary is the typed-hold surface. Revision stops immediately on a hold instead of spending additional generation attempts against unscorable output.

Canonical experiential battery#

MediaEvaluationBundle binds the following required probes to one exact artifact subject and digest:

Modality Required dimensions
Image composition; prompt/source alignment; identity; typography; aesthetics; cultural/brand correctness; artifact detection; accessibility
Video temporal continuity; direction; edit rhythm; lip sync; story comprehension; flicker/artifacts; captioning; accessibility
Audio intelligibility; word error rate where applicable; speaker/consent match; timing; loudness; silence/artifacts; musical structure; originality/fingerprint safety; accessibility
3D topology; UV; rigging; collision; LOD/Nanite; cook; performance; compatibility; novelty; accessibility
Gameplay playability; performance; compatibility; novelty; accessibility; fun; player comprehension

runMediaEvaluationBattery refuses to run when any required modality runner or the human comparator is absent. It executes every required probe and the comparator, parses the resulting strict schema, then evaluates coverage. Human records require a distinct baseline, blinded and randomized presentation, cohort/instruction/locale metadata, exact vote totals, evidence references, and a subject-preference win. Diagnostic external benchmark records are retained but never contribute required coverage.

V8 production tournament and release binding#

For every surviving mint gap, Loom reserves N jobs, submits deterministic candidate seeds, and records failures instead of collapsing the tournament. The evaluation response must name each real candidate. Local selection rejects unknown, duplicate, missing, substituted-subject, wrong-modality, incomplete, failed-threshold, or human-losing evidence. It separately requires passing, identity-bearing perceptual, period, and canon dimensions, plus a complete artifact-rights proof bundle bound to the exact candidate subject.

Only an eligible winner becomes a bound asset. The manifest validator proves that the selected record is unique, eligible, evidence-bearing, and matches the bound locator and seed. It calculates minted-job cost from all winner, loser, failed, and all-held attempts. A tournament with no winner is retained under heldSelections, omitted from releasable assets, and produces a blocking manifest problem.

Production composition requires either an injected evaluator or an HTTPS V8_MEDIA_EVALUATION_API_URL; the adapter applies bearer authentication when configured, time and response-size limits, and a strict candidate envelope. Before release, Loom fetches the authoritative selected locator, verifies its real C2PA claim, hashes the bytes, and requires that digest to equal the selected evaluation digest. Thus evaluation evidence for one candidate cannot authorize substituted bytes at another locator.

Adversarial fixture matrix#

Fixture Result
Provider returns a clip with no URL Producer fails with video_provider_missing_output_url; no fake frame or quality evidence is attached
Executor receives a blank keyframe Executor fails with video_provider_missing_key_frame
Critic receives only a video URL Typed media_required_frame_missing hold; no VLM call and no retry loop
One best-of-N provider job fails Failed candidate remains in the selection evidence; eligible evaluated candidates may still compete
Candidate has no evaluation Candidate is held and cannot win by position or generation order
Evaluation names another artifact Candidate is held for exact-subject mismatch
Evaluation service returns duplicate or unknown IDs Provider response holds every produced candidate
Required modality probe is absent Coverage fails even when an external benchmark score is high
Human baseline wins or no human study exists Coverage fails; machine scores cannot compensate
Selected evaluation digest differs from fetched bytes Loom release verification fails before provenance upgrade
Every candidate is held Full tournament remains in the manifest as a blocking hold; every provider attempt is still charged

Automated verification#

Current admitted refresh#

The current refresh ran against repository commit 8354d0897b597796fc4898f6c2d462427d552158 with lockfile SHA-256 08b055786e9b8a6104d5313601d7eef3205dedcf395066fd9a8264c9859d690f on host app-dev-server, Ubuntu 24.04.4 LTS, Linux 6.8.0-134-generic x86_64. The relevant toolchain was Node.js v24.15.0, pnpm 10.25.0, TypeScript 6.0.3, Vitest 4.1.7, and esbuild 0.28.0. Tests used deterministic local fixtures and temporary files; the video suites invoked real local ffmpeg/ffprobe processing but no remote media provider or production evaluation service.

UTC interval Exit Command and result
2026-07-29T12:26:02.663Z–2026-07-29T12:26:03.582Z 0 pnpm exec vitest run --config libs/contracts/vitest.config.ts libs/contracts/src/common/media-evaluation.test.ts --maxWorkers=1 — exact-subject media batteries, required probes, human comparisons, and diagnostic-only benchmark behavior passed 9/9
2026-07-29T12:26:18.673Z–2026-07-29T12:26:19.609Z 0 (cd libs/yemaya/case-contracts && pnpm exec vitest run --maxWorkers=1) — asset-selection and manifest contracts passed 18/18
2026-07-29T12:26:24.462Z–2026-07-29T12:26:35.730Z 0 (cd libs/yemaya/case-assets && pnpm exec vitest run --maxWorkers=1) — complete V8 asset binding, budget, orchestration, best-of-N, exact media/rights evidence, and realization suite passed 65/65 across ten files
2026-07-29T12:26:43.226Z–2026-07-29T12:26:46.880Z 0 pnpm exec vitest run --config libs/oshun/creative-orchestrator/vitest.config.ts --maxWorkers=1 — complete critique/revision and orchestration package passed 138/138 across 13 files
2026-07-29T12:27:02.161Z–2026-07-29T12:27:20.233Z 0 (cd apps/v8/loom-service && pnpm exec vitest run --maxWorkers=1) — Loom evaluation, provenance, dependency, pipeline, and service suites passed 41, with 3 database-dependent cases explicitly skipped, across ten files
2026-07-29T12:27:28.914Z–2026-07-29T12:28:01.402Z 0 pnpm exec vitest run --config apps/oshun/bff/vitest.config.ts apps/oshun/bff/src/agentic/autonomy-bindings/autonomy-bindings.test.ts apps/oshun/bff/src/agentic/autonomy-bindings/media-critic.test.ts apps/oshun/bff/src/generation/video-executor.test.ts apps/oshun/bff/src/generation/video-probe.test.ts apps/oshun/bff/src/generation/video-provider-env.test.ts --maxWorkers=1 — V1 media production, real-frame probes, typed holds, and autonomy integration passed 115/115
2026-07-29T12:28:13.474Z–2026-07-29T12:28:28.392Z 0 pnpm exec tsc -p libs/contracts/tsconfig.lib.json --noEmit
2026-07-29T12:28:35.670Z–2026-07-29T12:28:51.723Z 0 pnpm exec tsc -p libs/yemaya/case-contracts/tsconfig.lib.json --noEmit followed by pnpm exec tsc -p libs/yemaya/case-contracts/tsconfig.spec.json --noEmit
2026-07-29T12:28:58.050Z–2026-07-29T12:29:13.314Z 0 pnpm exec tsc -p libs/yemaya/case-assets/tsconfig.lib.json --noEmit followed by pnpm exec tsc -p libs/yemaya/case-assets/tsconfig.spec.json --noEmit
2026-07-29T12:29:19.814Z–2026-07-29T12:29:22.579Z 0 pnpm exec tsc -p libs/oshun/creative-orchestrator/tsconfig.json --noEmit
2026-07-29T12:29:27.807Z–2026-07-29T12:29:44.699Z 0 pnpm exec tsc -p apps/v8/loom-service/tsconfig.json --noEmit
2026-07-29T12:30:01.738Z–2026-07-29T12:30:26.845Z 0 Exact canonical-media, V8 contract/asset/Loom, V1 video/critic, and critique/revision lint command shown below — 24 relevant files passed with zero warnings

The exact lint invocation was:

text
pnpm exec eslint libs/contracts/src/common/media-evaluation.ts libs/contracts/src/common/media-evaluation.test.ts libs/yemaya/case-contracts/src/asset-manifest.ts libs/yemaya/case-contracts/src/contracts.test.ts libs/yemaya/case-assets/src/quality-selection.ts libs/yemaya/case-assets/src/quality-selection.test.ts libs/yemaya/case-assets/src/orchestrate.ts libs/yemaya/case-assets/src/orchestrate.test.ts libs/yemaya/case-assets/src/realize.ts libs/yemaya/case-assets/src/realize-best-of-n.test.ts apps/v8/loom-service/src/asset-evaluation-provider.ts apps/v8/loom-service/src/asset-evaluation-provider.test.ts apps/v8/loom-service/src/asset-provenance-verifier.ts apps/v8/loom-service/src/asset-provenance-verifier.test.ts apps/oshun/bff/src/generation/video-probe.ts apps/oshun/bff/src/generation/video-probe.test.ts apps/oshun/bff/src/generation/video-provider-env.ts apps/oshun/bff/src/generation/video-provider-env.test.ts apps/oshun/bff/src/generation/video-executor.ts apps/oshun/bff/src/generation/video-executor.test.ts apps/oshun/bff/src/agentic/autonomy-bindings/media-critic.ts apps/oshun/bff/src/agentic/autonomy-bindings/media-critic.test.ts libs/oshun/creative-orchestrator/src/critique-revise.ts libs/oshun/creative-orchestrator/src/critique-revise.test.ts --no-ignore --max-warnings=0

The current value suites therefore passed all 386 executed assertions across 40 files, with three database-dependent Loom cases explicitly skipped. Seven owning source/spec TypeScript configurations and exact lint over 24 relevant files passed. The most recent published BFF app-owned ratchet, on source commit 24ae90ca8d896c5e1829c19f654c6fabb02979b0, reported zero new errors and zero known backlog; the intervening commit through this refresh source changes only AUTH-002 evidence, not BFF source or configuration.

The one current code drift in the six primary MEDIA-001 artifacts strengthens the boundary: V8 candidate selection now also parses and evaluates an exact artifact-rights proof bundle and retains it with every candidate record. The current 65-test case-assets suite covers that integration. The updated hash is recorded below rather than silently retaining the source-publication identity.

Historical implementation verification#

The integrated source-publication record passed 4,351/4,351 non-duplicated package and focused tests:

Surface Result
Canonical contracts full package, including all migration/rollback cases 4,002/4,002
V8 case-contracts full package 17/17
V8 case-assets full package 64/64
Creative orchestrator full package 132/132
V8 Loom service full package 28/28
V1 BFF affected media/autonomy paths 108/108

Post-lint regression reruns passed 25/25 case-assets tests, 36/36 BFF tests, 27/27 real video-probe tests, and 11/11 critique/revise tests. After integrating the incoming V10 commits, an additional 9/9 canonical media tests and 5/5 V8 selection/realization tests passed.

Typechecks passed for contracts, case-contracts library/specs, case-assets library/specs, creative-orchestrator, and Loom. The resource-supervised BFF ratchet reported zero new and zero known app-owned errors. Explicit ESLint over all changed TypeScript files passed with zero warnings; Prettier and git diff --check passed. A direct production BFF esbuild bundle completed successfully (dist/server.js, 30.0 MB; source map, 71.9 MB).

Representative commands:

text
pnpm --dir <affected-package> exec vitest run
pnpm exec tsc --noEmit -p <affected-tsconfig>
node tools/typecheck/ratchet.mjs --app apps/oshun/bff --tsconfig tsconfig.typecheck.json
pnpm exec eslint <all changed TypeScript files> --no-ignore --max-warnings=0
pnpm exec prettier --check <all changed non-lock files>
pnpm --dir apps/oshun/bff exec esbuild src/server.ts --bundle --platform=node --format=esm --target=node22 --outfile=dist/server.js --sourcemap
git diff --check

Baselines and bounded verification#

Current refresh notes:

  • Two initial Vitest invocations passed package-root-relative include patterns from the repository root and therefore found no files: case-contracts at 2026-07-29T12:26:09.333Z–12:26:09.963Z and Loom at 2026-07-29T12:26:53.383Z–12:26:54.007Z. Both exited 1. The admitted commands run from their owning package roots and passed 18/18 and 41 passing with three explicit skips. These were command-location errors, not product failures.
  • Loom emitted existing missing-source-map warnings for generated JavaScript in libs/shared/database; all discovered tests completed and the process exited 0.
  • The host reported 15 GiB RAM, 31 GiB swap, 12 GiB initially available memory, and 16 GiB free disk. An unrelated checkout's long-running Vitest worker and this checkout's Nx daemon were active. Current tests and typechecks ran serially with one Vitest worker; no full-workspace or broad Nx matrix was started.

Historical source-publication notes:

  • The first full contracts run passed 3,994 tests and exposed eight Prisma migration-diff failures because a filtered pnpm operation had pruned the ignored root Prisma/dotenv resolution links. Both packages remained installed offline. Restoring those local-only links made the exact Prisma command pass; rerunning all nine migration/rollback cases passed, completing 4,002/4,002.
  • One initial case-assets invocation used workspace-relative paths against a package-root Vitest include and discovered no tests. The corrected package-root invocation passed the four focused files and then the full 64-test package.
  • The first BFF ratchet found one new excess-property error in an old aggregate fixture after the aggregate input was intentionally narrowed. The obsolete fields were removed; its 27 tests and the rerun ratchet then passed.
  • Loom's passing run emitted existing missing-source-map warnings for generated JavaScript under libs/shared/http-client; all five files and 28 tests passed.
  • The source commit used the pre-commit hook's documented SKIP_TYPECHECK=1 after all owning package typechecks and the BFF ratchet had passed. The hook incorrectly expanded common-contract ownership to 145 projects at --parallel=4; that unsafe broad run was interrupted under the small-host policy. ESLint, formatting, V2 asset locking, stub scanning, and V5 documentation validation still passed in the successful commit.
  • The host had 15 GiB RAM and 31 GiB swap. Tests and package checks ran sequentially. The BFF ratchet peaked near 4.9 GiB RSS while at least 5.9 GiB remained available, above the stop threshold.
  • The filtered offline lock update produced existing peer/deprecation warnings. Only the two required workspace importer links were committed. The unrelated user lock blob was preserved byte-for-byte as an unstaged modification across the V10 rebase.
  • No customer UI, PWA, admin, mobile, Unreal, or other visual surface changed. Frontend skill, Playwright, mobile automation, screenshots, and Unreal verification were inapplicable.

Artifact identity#

Artifact SHA-256
Canonical media evaluation b2c623195d73cb50c20f1904b81865e7b2b16073a7aa22974946bee2a8f1f800
V8 quality selection 606addc864823c4de625c2ef1bf74992671f609d5f06effead975a756baddeb4
V8 seeded orchestration 9a4f7f47b6199f04ad1c815c3a0776c593a3083287999f91fa345c882b4fb1fa
Loom evaluation service adapter 13e9a20b23ba4adadce18b48749a236f3baa5f2091638e09d3b171637cbaf7f6
Real video frame probe b4784c4ab41892062842bd431d755e44fa4f6b0375f0b92d48788fecb1e34c31
Typed media critic hold 38b6d5226b8a8c680540daad60915cdb2f937f3196d02cce3db76f67285e0af9

Scope and remaining seams#

  • Repository publication of the contract and evidence pack requires no fabricated human signoff. A passing production human-comparison record does require real blinded, randomized participant results with cohort, instruction, locale, vote, and evidence metadata. Test fixtures exercise the schema and fail-closed policy but are not approvals or live study results.
  • Production image/video/audio/3D/gameplay probe implementations and human study operations remain deployment services behind the canonical executable runner/HTTPS adapter. Their absence is now explicit and blocks production V8 composition or candidate selection; no live study result is claimed.
  • V8 currently mints image, video, audio, and 3D assets. Gameplay evaluation is canonical and executable for gameplay artifacts but is not falsely attached to a non-gameplay V8 asset.
  • EVAL-005's separate education battery is outside MEDIA-001. Its repository contract is complete; production education study evidence remains an external-state requirement recorded by that pack.
  • External benchmark ingestion remains diagnostic by contract. This package does not claim live population superiority, a production evaluator deployment, or completed human studies.

Publication identity#

  • Source commit: f8046ba3a573f082aa24e95573cd30de8a5c31d6
  • Historical declared closure evidence commit: 573cce758a2d33cb5caaee8c0e824da09a0eb966
  • Equivalent rebased closure identity on the current origin/main graph: 089e875030dbace657765680ce67a0d506ca5865 (same tree 221cd068d5af8608d086abaa5ce0deb97eef90b6)
  • Current admitted verification source: 8354d0897b597796fc4898f6c2d462427d552158
  • Current evidence-refresh commit: recorded by the task-scoped commit that contains this file.