Eve library

Media Pipeline Eval Baselines — agentic teaching media (Phase 2)

Other

Revision 1

On this page

Baselines for the Planner → Coder → Critic → loop pipeline (Tasks 2.1–2.6). Two tiers: an offline structural baseline (recorded here, runs with no renderer or VLM) and the rendered-video benchmark (TheoremExplainBench / MMMC-style, deferred to a render box).

Offline structural baseline#

Runs the real MediaPlannerAgent → MediaCoderAgent over a fixed set of 5 concept lessons (physics, mathematics, computer science, biology) and measures structural quality with no render. Harness: @metis/agents runMediaBenchmark / MEDIA_BENCHMARK_LESSONS (libs/metis/agents/src/agents/media-benchmark.ts); values computed by the runner and locked in media-benchmark.spec.ts. Reproduce:

bash
# from libs/metis/agents
npx vitest run src/agents/media-benchmark.spec.ts --reporter=verbose
Metric Value
Lessons 5
Total scenes 13
Mean scenes / lesson 2.60
Mean visual assets / scene 1.46
Code validity rate 1.00
Equation-bearing scene rate 0.308 (4 / 13)

Interpretation. The headline guard is the code validity rate: every one of the 13 generated scenes compiles to structurally valid (renderable-shaped) Manim source (from manim import *, a named Scene subclass, a construct), so the Coder does not regress across diverse lesson shapes (equations, fenced code, lists, prose). The planner segments each lesson into ~2.6 scenes (an intro-from-objectives scene plus one per section) and surfaces an equation asset on ~31% of scenes — the cases the Critic routes to the on-screen math verifier.

Rendered-video benchmark (deferred — render box)#

TheoremExplainBench / MMMC-style metrics evaluate the rendered video (visual-element correctness, layout legibility, narration sync) and a TeachQuiz learning-outcome probe, which need the Manim render service (Python) and a VLM. Manim is not installed on this box (import manim fails), so these are deferred to a render-capable host. The pieces are already wired and fail-loud until then:

  • ManimClient (multimedia/src/animation/manim-client.ts) dispatches renders over an injected transport; RendererNotConfiguredError when absent.
  • MediaCriticAgent.inspectVisualClarity uses an injected VlmCritic seam; VlmNotConfiguredError when absent.
  • runAgenticMediaLoop takes the renderer + learning-outcome probe as seams.

When the render box is available: wire the Manim transport + a VLM critic + the TeachQuiz probe, run runAgenticMediaLoop over MEDIA_BENCHMARK_LESSONS, and record per-concept render success, critic-pass rate, mean repair iterations, and the TeachQuiz score here alongside the structural baseline above.