Eve library

Minerva → Metis Test Coverage Baseline Report

Guide

Revision 1

On this page

Generated: 2026-03-24 Scope: Comprehensive test coverage comparison between the Minerva Python source (minerva/) and the Metis TypeScript port (libs/metis/)


1. Executive Summary#

Metric Minerva (Python) Metis (TypeScript)
Total test files 1,822 188
Total test functions/cases 37,977 15,913
Test directories/libraries 78 (with tests) 22
Source modules 93 22
Root-level test files 380 -
Root-level test functions 5,790 -

Metis currently covers 42% of Minerva's test count across 24% of Minerva's module surface area. The ported libraries have strong test density (averaging 725 tests per library) but significant Minerva modules remain unported.


2. Minerva Python Test Suite Overview#

2.1 Directory Structure#

Minerva's test suite is organized into 78 subdirectories plus 380 root-level test files. The root-level files contain 5,790 test functions covering cross-cutting concerns, integration scenarios, and modules that span multiple domains.

2.2 Test Counts by Module (Top 30 by Test Functions)#

Minerva Module Test Files Test Functions Category
research 96 5,989 Knowledge
verification 65 3,103 Quality
integration 65 2,536 Cross-cutting
validation 54 2,191 Quality
api 69 2,028 Infrastructure
linters 64 1,963 Quality
learning 41 1,502 Adaptive
video 28 1,176 Multimedia
tutor 24 921 Tutoring
models 22 869 Core
audio 26 842 Multimedia
cli 65 721 Infrastructure
content 95 656 Content
agents 54 519 AI Agents
byom 20 499 Custom Models
knowledge 19 496 Knowledge
assessment 20 481 Assessment
experiments 15 460 Experimentation
content_analytics 9 352 Analytics
providers 15 341 LLM Providers
database 30 313 Storage
security 29 305 Security
background 15 261 Infrastructure
accessibility 56 238 Accessibility
storyboard 7 234 Content
prompts 25 204 LLM
guidance 7 200 Pedagogical
common 21 194 Shared Utilities
rag 8 187 Retrieval
llm_integration 9 185 LLM

2.3 Test Counts by Module (Remaining)#

Minerva Module Test Files Test Functions Category
course 62 161 Course
syllabus 5 143 Course
multimedia 14 126 Multimedia
scripts 10 111 Tooling
diff 12 110 Content
vlm 40 106 Vision LM
comfyui_nodes 4 106 Image Gen
templates 3 104 Templates
tools 9 95 Tooling
orchestration 12 93 Orchestration
document_processing 2 92 Content
visualization 8 91 Visualization
observability 44 85 Infrastructure
concept_graph 3 85 Knowledge
performance 8 82 Infrastructure
embeddings 2 71 ML
course_creation_asst 15 69 Course
ui 21 55 Frontend
content_generation 4 43 Content
analytics 15 40 Analytics
e2e 2 39 Integration
settings 2 37 Configuration
monitoring 4 28 Infrastructure
dashboard 11 21 Frontend
export 2 18 Export
benchmarks 3 18 Performance
integrations 7 16 Third-party
data 7 16 Data
unified 2 14 Orchestration
compliance 4 11 Compliance
administration 5 11 Admin
metrics 3 8 Metrics
progress 2 7 Progress
lms 2 7 LMS
tui 3 6 CLI/TUI
analysis 3 5 Analysis
optimization 2 4 Optimization
llm 2 2 LLM

Root-level tests (380 files, 5,790 functions) cover cross-cutting concerns including accessibility compliance, diagram rendering, content moderation, error handling, mathematical verification, multimedia pipelines, provider configuration, syllabus processing, and end-to-end workflows.


3. Metis TypeScript Test Suite Current State#

3.1 Per-Library Test Coverage#

Metis Library Test Files Test Cases Source Files Status
@metis/tutoring 7 1,252 8 Complete
@metis/assessment 19 1,112 10 Complete
@metis/prompt-mgmt 5 1,095 7 Complete
@metis/llm-client 5 1,056 7 Complete
@metis/cost-tracking 5 1,036 7 Complete
@metis/linters-types 6 965 7 Complete
@metis/research 7 964 8 Complete
@metis/ab-testing 5 952 7 Complete
@metis/learning 7 878 8 Complete
@metis/model-registry 5 863 7 Complete
@metis/api-client 7 726 9 Complete
@metis/database-types 5 715 6 Complete
@metis/multimedia 19 710 5 Complete
@metis/quality 15 618 5 Complete
@metis/course 7 577 8 Complete
@metis/agents 19 535 3 Complete
@metis/adaptive 12 437 4 Complete
@metis/knowledge-graph 10 403 4 Complete
@metis/models 9 393 10 Complete
@metis/core 7 322 9 Complete
@metis/agents-types 6 224 7 Complete
@metis/integrations 1 78 7 Partial
Total 188 15,913 157

3.2 Test Density Analysis#

Metric Value
Average tests per library 725
Median tests per library 726
Highest test count 1,252 (@metis/tutoring)
Lowest test count (excl. integ) 224 (@metis/agents-types)
Average tests per test file 85

4. Coverage Gap Analysis#

4.1 Ported Modules: Minerva → Metis Mapping#

These Minerva modules have corresponding Metis TypeScript libraries with tests.

Minerva Module Minerva Tests Metis Library Metis Tests Coverage Ratio
models 869 @metis/models 393 45%
agents 519 @metis/agents + agents-types 759 146%
api 2,028 @metis/api-client 726 36%
assessment 481 @metis/assessment 1,112 231%
course 161 @metis/course 577 358%
learning 1,502 @metis/learning 878 58%
multimedia 126 @metis/multimedia 710 563%
research 5,989 @metis/research 964 16%
linters 1,963 @metis/linters-types 965 49%
knowledge 496 @metis/knowledge-graph 403 81%
tutor 921 @metis/tutoring 1,252 136%
database 313 @metis/database-types 715 228%

Key observations:

  • Over-covered modules (>100%): assessment, course, multimedia, database, tutoring, and agents have MORE tests in Metis than Minerva. This reflects the TypeScript ports being comprehensive type-level libraries with extensive validation tests, while Minerva's tests for these modules may be more integration-focused.
  • Under-covered modules (<50%): research (16%), api (36%), models (45%), and linters (49%) have significantly fewer Metis tests than Minerva. These are the largest and most complex Minerva modules.
  • Research is the biggest gap: 5,989 Minerva tests vs 964 Metis tests. This is expected since Metis research is currently a types library, while Minerva research includes full knowledge acquisition, crawling, embedding, and verification logic.

4.2 Additional Metis Libraries (No Direct Minerva Test Directory)#

These Metis libraries were created from cross-cutting Minerva functionality or new Oshun integration needs rather than mapping 1:1 to a Minerva test directory.

Metis Library Tests Derived From (Minerva)
@metis/llm-client 1,056 llm_utils.py, providers/, llm/
@metis/prompt-management 1,095 prompts/, agents/ prompt patterns
@metis/model-registry 863 providers/, experiments/
@metis/cost-tracking 1,036 analytics/, providers/ cost tracking
@metis/ab-testing 952 experiments/, content_analytics/
@metis/adaptive 437 learning/ adaptive algorithms
@metis/quality 618 verification/, validation/, linters/
@metis/integrations 78 integrations/, integration/

4.3 Unported Minerva Modules#

The following Minerva modules have tests but no corresponding Metis library yet. Ordered by test function count (descending priority).

Minerva Module Test Files Test Functions Priority Notes
verification 65 3,103 P0 Core quality — partially in @metis/quality
integration 65 2,536 P1 Cross-module integration tests
validation 54 2,191 P0 Core quality — partially in @metis/quality
video 28 1,176 P1 Multimedia production
audio 26 842 P1 TTS and audio pipeline
cli 65 721 P2 Command-line interface
content 95 656 P0 Content management core
byom 20 499 P2 Bring-your-own-model
experiments 15 460 P1 Partially in @metis/ab-testing
content_analytics 9 352 P1 Content performance analytics
providers 15 341 P1 Partially in @metis/llm-client
security 29 305 P0 Authentication, authorization
background 15 261 P2 Background job processing
accessibility 56 238 P1 WCAG compliance
storyboard 7 234 P2 Storyboard generation
prompts 25 204 P1 Partially in @metis/prompt-mgmt
guidance 7 200 P2 Pedagogical guidance
common 21 194 P1 Shared utilities
rag 8 187 P1 Retrieval-augmented generation
llm_integration 9 185 P1 Partially in @metis/llm-client
diff 12 110 P2 Content diffing
vlm 40 106 P2 Vision language models
comfyui_nodes 4 106 P2 Image generation
templates 3 104 P2 Template system
orchestration 12 93 P1 Multi-agent orchestration
observability 44 85 P2 Monitoring and tracing
concept_graph 3 85 P2 Partially in @metis/knowledge-graph
performance 8 82 P2 Performance benchmarking
embeddings 2 71 P1 Vector embeddings
visualization 8 91 P2 Data visualization
syllabus 5 143 P1 Partially in @metis/course
document_processing 2 92 P2 Document parsing
Root-level 380 5,790 Mixed Cross-cutting, integration, E2E

Total unported test surface: ~21,684 test functions across 1,254 test files (plus 5,790 root-level tests).


5. Migration Validation Criteria#

5.1 Per-Module Porting Success Criteria#

A Minerva module is considered successfully ported to Metis when:

Criterion Requirement
Type coverage All Pydantic models have TypeScript/Zod equivalents
API parity All public Python functions have TypeScript equivalents
Test coverage ratio >= 80% of Minerva test function count
Domain-specific tests Tests validate domain logic, not just CRUD
Edge case coverage Error paths, boundary conditions, empty inputs
Integration tests Cross-module interactions verified
Build verification tsc --noEmit passes, library builds cleanly

5.2 Overall Migration Completion Thresholds#

Milestone Test Count Target Module Count Description
Alpha 15,000+ 22 (current) Core type libraries ported
Beta 25,000+ 35+ Major modules ported
Release Candidate 32,000+ 50+ All P0/P1 modules ported
GA 38,000+ 60+ Full parity with Minerva

Current status: Alpha (15,913 tests across 22 libraries).

5.3 Test Quality Requirements#

Beyond raw test counts, each ported module must demonstrate:

  1. Algorithm verification — Tests that validate domain-specific algorithms produce correct results (e.g., adaptive difficulty calculations, assessment scoring, knowledge graph traversal)
  2. Serialization roundtrips — Data can be serialized and deserialized without loss
  3. Validation coverage — Invalid inputs are rejected with appropriate errors
  4. Concurrent safety — Async operations handle race conditions where applicable
  5. Performance baselines — Critical path operations have timing assertions

6. Priority Test Migration Order#

Phase 1: Core Quality (P0) — Target: +5,000 tests#

Priority Module Minerva Tests Gap Rationale
1 verification 3,103 ~2,500 Core quality assurance pipeline
2 validation 2,191 ~1,600 Input/output validation
3 content 656 ~656 Content management fundamentals
4 security 305 ~305 Auth and authorization

Phase 2: AI and Knowledge (P1) — Target: +6,000 tests#

Priority Module Minerva Tests Gap Rationale
5 research (expand) 5,989 ~5,025 Knowledge acquisition engine
6 api (expand) 2,028 ~1,302 Full API coverage
7 learning (expand) 1,502 ~624 Adaptive learning algorithms
8 providers 341 ~341 LLM provider abstraction
9 rag 187 ~187 Retrieval-augmented generation
10 embeddings 71 ~71 Vector search

Phase 3: Multimedia Production (P1) — Target: +2,500 tests#

Priority Module Minerva Tests Gap Rationale
11 video 1,176 ~1,176 Video production pipeline
12 audio 842 ~842 TTS and audio processing
13 accessibility 238 ~238 WCAG compliance
14 storyboard 234 ~234 Storyboard generation

Phase 4: Infrastructure and Integration (P1-P2) — Target: +4,000 tests#

Priority Module Minerva Tests Gap Rationale
15 integration 2,536 ~2,536 Cross-module integration
16 experiments 460 ~460 Experiment framework
17 content_analytics 352 ~352 Content performance metrics
18 common 194 ~194 Shared utilities
19 orchestration 93 ~93 Multi-agent orchestration
20 cli 721 ~721 Command-line tooling

Phase 5: Specialized and Remaining (P2) — Target: +3,000 tests#

Covers: byom, diff, vlm, comfyui_nodes, templates, background, guidance, observability, concept_graph, performance, visualization, document_processing, syllabus, dashboard, monitoring, and root-level cross-cutting tests.


7. Appendix: Complete Module Inventory#

7.1 Minerva Source Modules (93 total)#

Organized by Python source file count (descending):

Module Source Files Test Dir Exists Tests
integration 336 Yes 2,536
visualization 248 Yes 91
validation 142 Yes 2,191
research 142 Yes 5,989
agents 120 Yes 519
learning 118 Yes 1,502
course 101 Yes 161
assessment 99 Yes 481
content 97 Yes 656
verification 91 Yes 3,103
tutor 88 Yes 921
api 81 Yes 2,028
byom 80 Yes 499
diff 65 Yes 110
linters 63 Yes 1,963
accessibility 58 Yes 238
security 57 Yes 305
orchestration 57 Yes 93
cli 57 Yes 721
observability 54 Yes 85
vlm 52 Yes 106
prompts 52 Yes 204
video 43 Yes 1,176
common 37 Yes 194
audio 35 Yes 842
knowledge 30 Yes 496
database 28 Yes 313
optimization 27 Yes 4
experiments 26 Yes 460
models 25 Yes 869
multimedia 23 Yes 126
tui 20 Yes 6
course_creation_asst 19 Yes 69
dependency_health 18 No -
compliance_automation 18 No -
background 15 Yes 261
analytics 15 Yes 40
administration 15 Yes 11
retry 14 No -
generation 14 No -
document_processing 14 Yes 92
devops 14 No -
moderation 13 Yes 0
integrations 13 Yes 16
reasoning 12 No -
rag 12 Yes 187
concept_graph 12 Yes 85
caching 11 Yes 0
analysis 11 Yes 5
diversity 11 No -
performance 11 Yes 82
providers 11 Yes 341
tools 11 Yes 95
version_control 10 No -
dashboard 10 Yes 21
guidance 9 Yes 200
templates 8 Yes 104
storyboard 8 Yes 234
refinement 8 No -
collaboration 8 No -
autonomous_tour 8 No -
tracking 7 No -
metrics 7 Yes 8
data 7 Yes 16
progress 6 Yes 7
io 6 No -
flow 6 No -
export 6 Yes 18
curriculum 6 No -
core 6 No -
diagram 6 No -
unified 5 Yes 14
schemas 5 No -
attribution 5 No -
compliance 4 Yes 11
assessment_tools 4 No -
service_discovery 3 No -
llm 3 Yes 2
generators 3 No -
embeddings 3 Yes 71
documentation 3 Yes 0
content_standards 3 No -
cache 3 Yes 0

7.2 Metis Libraries (22 total)#

All libraries reside under libs/metis/ in the Oshun monorepo.

Library Source Files Test Files Test Cases Phase
@metis/core 9 7 322 25.4.1
@metis/models 10 9 393 25.4.2
@metis/agents-types 7 6 224 25.4.3
@metis/api-client 9 7 726 25.4.4
@metis/course 8 7 577 25.4.5
@metis/assessment 10 19 1,112 25.4.6
@metis/multimedia 5 19 710 25.4.7
@metis/tutoring 8 7 1,252 25.4.8
@metis/linters-types 7 6 965 25.4.9
@metis/research 8 7 964 25.4.10
@metis/learning 8 7 878 25.4.11
@metis/database-types 6 5 715 25.4.12
@metis/llm-client 7 5 1,056 25.2.2
@metis/prompt-mgmt 7 5 1,095 25.2.2
@metis/model-registry 7 5 863 25.2.2
@metis/cost-tracking 7 5 1,036 25.2.2
@metis/ab-testing 7 5 952 25.2.2
@metis/agents 3 19 535 25.2.4
@metis/adaptive 4 12 437 -
@metis/quality 5 15 618 -
@metis/knowledge-graph 4 10 403 -
@metis/integrations 7 1 78 -