# Media Pipeline Eval Baselines — agentic teaching media (Phase 2)

Baselines for the Planner → Coder → Critic → loop pipeline (Tasks 2.1–2.6). Two
tiers: an **offline structural baseline** (recorded here, runs with no renderer
or VLM) and the **rendered-video benchmark** (TheoremExplainBench / MMMC-style,
deferred to a render box).

## Offline structural baseline

Runs the real `MediaPlannerAgent` → `MediaCoderAgent` over a fixed set of 5
concept lessons (physics, mathematics, computer science, biology) and measures
structural quality with no render. Harness: `@metis/agents` `runMediaBenchmark`
/ `MEDIA_BENCHMARK_LESSONS` (`libs/metis/agents/src/agents/media-benchmark.ts`);
values computed by the runner and locked in `media-benchmark.spec.ts`.
Reproduce:

```bash
# from libs/metis/agents
npx vitest run src/agents/media-benchmark.spec.ts --reporter=verbose
```

| Metric                      | Value          |
| --------------------------- | -------------- |
| Lessons                     | 5              |
| Total scenes                | 13             |
| Mean scenes / lesson        | 2.60           |
| Mean visual assets / scene  | 1.46           |
| **Code validity rate**      | **1.00**       |
| Equation-bearing scene rate | 0.308 (4 / 13) |

**Interpretation.** The headline guard is the **code validity rate**: every one
of the 13 generated scenes compiles to structurally valid (renderable-shaped)
Manim source (`from manim import *`, a named `Scene` subclass, a `construct`),
so the Coder does not regress across diverse lesson shapes (equations, fenced
code, lists, prose). The planner segments each lesson into ~2.6 scenes (an
intro-from-objectives scene plus one per section) and surfaces an equation asset
on ~31% of scenes — the cases the Critic routes to the on-screen math verifier.

## Rendered-video benchmark (deferred — render box)

TheoremExplainBench / MMMC-style metrics evaluate the _rendered video_
(visual-element correctness, layout legibility, narration sync) and a TeachQuiz
learning-outcome probe, which need the Manim render service (Python) and a VLM.
Manim is not installed on this box (`import manim` fails), so these are deferred
to a render-capable host. The pieces are already wired and fail-loud until then:

- `ManimClient` (`multimedia/src/animation/manim-client.ts`) dispatches renders
  over an injected transport; `RendererNotConfiguredError` when absent.
- `MediaCriticAgent.inspectVisualClarity` uses an injected `VlmCritic` seam;
  `VlmNotConfiguredError` when absent.
- `runAgenticMediaLoop` takes the renderer + learning-outcome probe as seams.

When the render box is available: wire the Manim transport + a VLM critic + the
TeachQuiz probe, run `runAgenticMediaLoop` over `MEDIA_BENCHMARK_LESSONS`, and
record per-concept render success, critic-pass rate, mean repair iterations, and
the TeachQuiz score here alongside the structural baseline above.
