Baselines for the Planner → Coder → Critic → loop pipeline (Tasks 2.1–2.6). Two tiers: an offline structural baseline (recorded here, runs with no renderer or VLM) and the rendered-video benchmark (TheoremExplainBench / MMMC-style, deferred to a render box).
Offline structural baseline#
Runs the real MediaPlannerAgent → MediaCoderAgent over a fixed set of 5
concept lessons (physics, mathematics, computer science, biology) and measures
structural quality with no render. Harness: @metis/agents runMediaBenchmark
/ MEDIA_BENCHMARK_LESSONS (libs/metis/agents/src/agents/media-benchmark.ts);
values computed by the runner and locked in media-benchmark.spec.ts.
Reproduce:
# from libs/metis/agents
npx vitest run src/agents/media-benchmark.spec.ts --reporter=verbose
| Metric | Value |
|---|---|
| Lessons | 5 |
| Total scenes | 13 |
| Mean scenes / lesson | 2.60 |
| Mean visual assets / scene | 1.46 |
| Code validity rate | 1.00 |
| Equation-bearing scene rate | 0.308 (4 / 13) |
Interpretation. The headline guard is the code validity rate: every one
of the 13 generated scenes compiles to structurally valid (renderable-shaped)
Manim source (from manim import *, a named Scene subclass, a construct),
so the Coder does not regress across diverse lesson shapes (equations, fenced
code, lists, prose). The planner segments each lesson into ~2.6 scenes (an
intro-from-objectives scene plus one per section) and surfaces an equation asset
on ~31% of scenes — the cases the Critic routes to the on-screen math verifier.
Rendered-video benchmark (deferred — render box)#
TheoremExplainBench / MMMC-style metrics evaluate the rendered video
(visual-element correctness, layout legibility, narration sync) and a TeachQuiz
learning-outcome probe, which need the Manim render service (Python) and a VLM.
Manim is not installed on this box (import manim fails), so these are deferred
to a render-capable host. The pieces are already wired and fail-loud until then:
ManimClient(multimedia/src/animation/manim-client.ts) dispatches renders over an injected transport;RendererNotConfiguredErrorwhen absent.MediaCriticAgent.inspectVisualClarityuses an injectedVlmCriticseam;VlmNotConfiguredErrorwhen absent.runAgenticMediaLooptakes the renderer + learning-outcome probe as seams.
When the render box is available: wire the Manim transport + a VLM critic + the
TeachQuiz probe, run runAgenticMediaLoop over MEDIA_BENCHMARK_LESSONS, and
record per-concept render success, critic-pass rate, mean repair iterations, and
the TeachQuiz score here alongside the structural baseline above.