Eve library

Metis Verification & Agentic-Media Runbooks

Other

Revision 1

On this page

Operational guide for the correctness-verification (@metis/verification) and agentic teaching-media subsystems (@metis/agents media-* + @metis/multimedia). See DECISION.md for the architecture choice and EVAL_BASELINES.md / ../../multimedia/MEDIA_EVAL_BASELINES.md for recorded baselines.

How to add a verifier#

  1. Implement the Verifier interface (src/types.ts): name, required, and verify(content, context) → VerifierResult (normalized score 0–1, calibrated confidence, passed, optional critique). Put model-backed dependencies behind an injected seam and return { notConfigured: true } when the seam is absent — never fabricate a result (fail-loud).
  2. Register it in src/gate/compose.ts (composeP0Gate) with a sensible required/advisory default, wiring its seam through MetisGateConfig.
  3. Record a baseline: add it to src/eval/baselines.ts and run npx vitest run src/eval/baselines.spec.ts --reporter=verbose; transcribe the measured P/R/F1/κ/MAE into EVAL_BASELINES.md.
  4. Per-task gate: npx tsc -p tsconfig.lib.json --noEmit clean + targeted npx vitest run <spec> green + the adversarial stub scan clean.

How to add a media agent (Planner / Coder / Critic)#

  1. Media types live in @metis/agents src/agents/media-types.ts (ScenePlan, SceneSpec, VisualAssetSpec, GeneratedSceneCode, SceneLayout). Reuse them — do not fork.
  2. Implement the agent as a self-contained class + factory (e.g. MediaPlannerAgent.planScenes). Keep model dependencies (LLM narration, VLM critic) as injected seams and fail-loud when absent.
  3. Wire it into the loop: @metis/multimedia src/lecture-generation/agentic-media-loop.ts consumes the agents through the MediaPlannerLike / MediaCoderLike / MediaCriticLike / SceneRenderer seams. The loop type-only-imports @metis/agents (via the @metis/* path) — keep it that way so the agents barrel is not pulled into multimedia at runtime.
  4. The renderer (Manim) is injected; until a render host exists the ManimClient fails loud (RendererNotConfiguredError).

How to add / refresh a gold set#

  1. Add JSONL fixtures under src/eval/fixtures/ — one LabeledItem per line (content, label, optional humanScore, domain); // comment lines are ignored. Document provenance in the file header.
  2. Keep it balanced and small enough for CI. Load with loadGoldSet(path).
  3. Re-run the eval harness (scoreVerifier / computeBaselines) and update EVAL_BASELINES.md with the new numbers.

How HITL feedback flows#

  1. The gate routes low-confidence / high-disagreement items to humans (needs-human decision; the active-learning selector selectForReview in src/hitl/active-learning.ts).
  2. Human accept/reject/edit decisions are captured in the versioned FeedbackStore (gold-label overrides).
  3. Those labels feed recalibration: judge calibration (fitHistogramCalibrator, κ/Pearson), the conformal selective threshold (calibrateSelectiveThreshold), and per-criterion reliability (diagnoseCriterionReliability).
  4. Drift & promotion: monitorAgreementDrift (Welch z) watches agreement-with-human over time; evaluateVerifierPromotion (two-proportion z-test) gates promoting a challenger verifier/judge config over the champion.

Budgets & kill-switch#

  • Per-call: ComputeBudgetMeter + MODE_BUDGETS (fast/balanced/deep) — budget exhaustion fails loud.
  • Production ceilings: checkBudgetCeilings(mode, latencyMs[], costCents[]) asserts p95 latency + total cost against MODE_CEILINGS and recommends the kill-switch on breach. Tune the ceiling values from production telemetry.