Eve library

Metis Verification — Architecture Decision (Task 0.1)

Other

Revision 1

On this page

Date: 2026-06-16 Context: METIS_GAP_CLOSURE_TODOS_2026-06-16.md Phase 0; gap analysis METIS_CORRECTNESS_AND_MEDIA_SOTA_GAP_ANALYSIS_2026-06-16.md.

Decision#

Build a metis-local @metis/verification library that REUSES the shared @oshun/content-quality-judge engine for the generic, already-verified machinery, and ADDS the education-specific verifiers the shared engine does not provide.

What we reuse from @oshun/content-quality-judge (do not re-implement)#

The shared package is real and tested (312+ tests; see V1_V9_AGENTIC_CONTENT_SOTA_ASSESSMENT_2026-06-14.md §2). Metis consumes:

  • JudgePanel / judge-engine — calibrated, position-bias-mitigated LLM-judge panel (Task 1.8 pedagogical-quality dimensions).
  • calibration — κ/Pearson/MAE vs human gold.
  • grounding-gate (ClaimExtractor, GroundingRetriever, createGroundingGate) — claim extraction + grounding primitive (Tasks 1.1–1.2 build on this).
  • best-of-n, self-refine, reflexion — verifier-guided generation (Task 3.4).
  • active-learning, drift, quality-champion-challenger — HITL + drift (Tasks 0.6, 6.2).
  • compute-budget, budget/kill-switch — budgets (Tasks 0.7, 6.1).

What is metis-specific and lives here (the gap)#

Education is the verifiable domain (math/code/facts have ground truth), so metis adds verifiers the creative-content engine has no reason to:

  • Atomic claim decomposition + per-claim factuality vs the metis knowledge graph + research corpus (FActScore / SAFE F1@K). (Tasks 1.1–1.2)
  • RAG faithfulness / TRACe adherence and citation sufficiency (entailment, not format). (Tasks 1.3–1.4)
  • STEM correctness: CAS/symbolic math, sandboxed code execution, and a process reward model for worked-solution steps. (Tasks 1.5–1.7)
  • Contradiction detection, calibration/abstention, pedagogical/curriculum correctness. (Tasks 3.1, 3.3, 5.1)

Integration boundary#

  • @metis/verification exposes a VerificationGate composing Verifiers; education-specific verifiers are implemented here, the pedagogical-quality judge is the shared JudgePanel wrapped as a Verifier.
  • The gate is fail-loud: a required verifier that cannot run (notConfigured) blocks release — it never silently passes.
  • Heavy models (NLI/DeBERTa faithfulness, CAS, code sandbox, PRM, Manim) run in the Python apps/metis/service; the TS lib calls them through typed clients that fail loud when an endpoint is unconfigured.
  • The gate plugs into @metis/llm-client content generation (Task 1.9) and the agentic media loop (Phase 2), mirroring the shared engine's generate → judge-gate → refine loop.

Rejected alternative#

Adopt @oshun/content-service wholesale. Rejected: it is tuned for creative artifacts without ground-truth verifiers; metis needs domain verifiers as first-class gate members. We reuse its judge/calibration/loop primitives rather than its creative pipeline shell.