Operational guide for the correctness-verification (@metis/verification) and
agentic teaching-media subsystems (@metis/agents media-* +
@metis/multimedia). See DECISION.md for the architecture choice and
EVAL_BASELINES.md / ../../multimedia/MEDIA_EVAL_BASELINES.md for recorded
baselines.
How to add a verifier#
- Implement the
Verifierinterface (src/types.ts):name,required, andverify(content, context) → VerifierResult(normalizedscore0–1, calibratedconfidence,passed, optionalcritique). Put model-backed dependencies behind an injected seam and return{ notConfigured: true }when the seam is absent — never fabricate a result (fail-loud). - Register it in
src/gate/compose.ts(composeP0Gate) with a sensible required/advisory default, wiring its seam throughMetisGateConfig. - Record a baseline: add it to
src/eval/baselines.tsand runnpx vitest run src/eval/baselines.spec.ts --reporter=verbose; transcribe the measured P/R/F1/κ/MAE intoEVAL_BASELINES.md. - Per-task gate:
npx tsc -p tsconfig.lib.json --noEmitclean + targetednpx vitest run <spec>green + the adversarial stub scan clean.
How to add a media agent (Planner / Coder / Critic)#
- Media types live in
@metis/agentssrc/agents/media-types.ts(ScenePlan,SceneSpec,VisualAssetSpec,GeneratedSceneCode,SceneLayout). Reuse them — do not fork. - Implement the agent as a self-contained class + factory (e.g.
MediaPlannerAgent.planScenes). Keep model dependencies (LLM narration, VLM critic) as injected seams and fail-loud when absent. - Wire it into the loop:
@metis/multimediasrc/lecture-generation/agentic-media-loop.tsconsumes the agents through theMediaPlannerLike/MediaCoderLike/MediaCriticLike/SceneRendererseams. The loop type-only-imports@metis/agents(via the@metis/*path) — keep it that way so the agents barrel is not pulled into multimedia at runtime. - The renderer (Manim) is injected; until a render host exists the
ManimClientfails loud (RendererNotConfiguredError).
How to add / refresh a gold set#
- Add JSONL fixtures under
src/eval/fixtures/— oneLabeledItemper line (content,label, optionalhumanScore,domain);//comment lines are ignored. Document provenance in the file header. - Keep it balanced and small enough for CI. Load with
loadGoldSet(path). - Re-run the eval harness (
scoreVerifier/computeBaselines) and updateEVAL_BASELINES.mdwith the new numbers.
How HITL feedback flows#
- The gate routes low-confidence / high-disagreement items to humans
(
needs-humandecision; the active-learning selectorselectForReviewinsrc/hitl/active-learning.ts). - Human accept/reject/edit decisions are captured in the versioned
FeedbackStore(gold-label overrides). - Those labels feed recalibration: judge calibration (
fitHistogramCalibrator, κ/Pearson), the conformal selective threshold (calibrateSelectiveThreshold), and per-criterion reliability (diagnoseCriterionReliability). - Drift & promotion:
monitorAgreementDrift(Welch z) watches agreement-with-human over time;evaluateVerifierPromotion(two-proportion z-test) gates promoting a challenger verifier/judge config over the champion.
Budgets & kill-switch#
- Per-call:
ComputeBudgetMeter+MODE_BUDGETS(fast/balanced/deep) — budget exhaustion fails loud. - Production ceilings:
checkBudgetCeilings(mode, latencyMs[], costCents[])asserts p95 latency + total cost againstMODE_CEILINGSand recommends the kill-switch on breach. Tune the ceiling values from production telemetry.