# Metis Verification & Agentic-Media Runbooks

Operational guide for the correctness-verification (`@metis/verification`) and
agentic teaching-media subsystems (`@metis/agents` media-\* +
`@metis/multimedia`). See `DECISION.md` for the architecture choice and
`EVAL_BASELINES.md` / `../../multimedia/MEDIA_EVAL_BASELINES.md` for recorded
baselines.

## How to add a verifier

1. Implement the `Verifier` interface (`src/types.ts`): `name`, `required`, and
   `verify(content, context) → VerifierResult` (normalized `score` 0–1,
   calibrated `confidence`, `passed`, optional `critique`). Put model-backed
   dependencies behind an **injected seam** and return `{ notConfigured: true }`
   when the seam is absent — never fabricate a result (fail-loud).
2. Register it in `src/gate/compose.ts` (`composeP0Gate`) with a sensible
   required/advisory default, wiring its seam through `MetisGateConfig`.
3. Record a baseline: add it to `src/eval/baselines.ts` and run
   `npx vitest run src/eval/baselines.spec.ts --reporter=verbose`; transcribe
   the measured P/R/F1/κ/MAE into `EVAL_BASELINES.md`.
4. Per-task gate: `npx tsc -p tsconfig.lib.json --noEmit` clean + targeted
   `npx vitest run <spec>` green + the adversarial stub scan clean.

## How to add a media agent (Planner / Coder / Critic)

1. Media types live in `@metis/agents` `src/agents/media-types.ts` (`ScenePlan`,
   `SceneSpec`, `VisualAssetSpec`, `GeneratedSceneCode`, `SceneLayout`). Reuse
   them — do not fork.
2. Implement the agent as a self-contained class + factory (e.g.
   `MediaPlannerAgent.planScenes`). Keep model dependencies (LLM narration, VLM
   critic) as **injected seams** and fail-loud when absent.
3. Wire it into the loop: `@metis/multimedia`
   `src/lecture-generation/agentic-media-loop.ts` consumes the agents through
   the `MediaPlannerLike` / `MediaCoderLike` / `MediaCriticLike` /
   `SceneRenderer` seams. The loop type-only-imports `@metis/agents` (via the
   `@metis/*` path) — keep it that way so the agents barrel is not pulled into
   multimedia at runtime.
4. The renderer (Manim) is injected; until a render host exists the
   `ManimClient` fails loud (`RendererNotConfiguredError`).

## How to add / refresh a gold set

1. Add JSONL fixtures under `src/eval/fixtures/` — one `LabeledItem` per line
   (`content`, `label`, optional `humanScore`, `domain`); `//` comment lines are
   ignored. Document provenance in the file header.
2. Keep it balanced and small enough for CI. Load with `loadGoldSet(path)`.
3. Re-run the eval harness (`scoreVerifier` / `computeBaselines`) and update
   `EVAL_BASELINES.md` with the new numbers.

## How HITL feedback flows

1. The gate routes **low-confidence / high-disagreement** items to humans
   (`needs-human` decision; the active-learning selector `selectForReview` in
   `src/hitl/active-learning.ts`).
2. Human accept/reject/edit decisions are captured in the versioned
   `FeedbackStore` (gold-label overrides).
3. Those labels feed recalibration: judge calibration (`fitHistogramCalibrator`,
   κ/Pearson), the conformal selective threshold
   (`calibrateSelectiveThreshold`), and per-criterion reliability
   (`diagnoseCriterionReliability`).
4. Drift & promotion: `monitorAgreementDrift` (Welch z) watches
   agreement-with-human over time; `evaluateVerifierPromotion` (two-proportion
   z-test) gates promoting a challenger verifier/judge config over the champion.

## Budgets & kill-switch

- Per-call: `ComputeBudgetMeter` + `MODE_BUDGETS` (fast/balanced/deep) — budget
  exhaustion fails loud.
- Production ceilings: `checkBudgetCeilings(mode, latencyMs[], costCents[])`
  asserts p95 latency + total cost against `MODE_CEILINGS` and recommends the
  kill-switch on breach. Tune the ceiling values from production telemetry.
