Eve library

ML Sovereignty Gap Analysis Report

Guide

Revision 1

On this page

Date: 2026-04-11 Scope: Complete audit of ML/AI infrastructure across the Oshun monorepo Goal: Identify all missing tasks required for full model sovereignty — the ability to self-host, fine-tune, train, evaluate, and deploy custom open-source models for every ML use case in the system.


Executive Summary#

The Oshun monorepo has substantial ML infrastructure already built — a sovereign inference engine (@nous/core), comprehensive fine-tuning framework (@nous/training, @isis/model-fine-tuning), GPU serving via RunPod, and domain-specific ML services across 15+ domains. However, full ML sovereignty is not yet achievable because critical gaps exist in:

  1. Data Pipeline Infrastructure — No training data loading, preprocessing, or curriculum systems
  2. Model Serving Optimization — No vLLM/TGI/TensorRT-LLM integration for production LLM serving
  3. Continuous Training Pipelines — No automated retraining, drift detection, or model refresh workflows
  4. Evaluation & Benchmarking — No unified cross-domain evaluation framework
  5. Dataset Sovereignty — No centralized dataset registry, versioning, or synthetic data generation at scale
  6. Edge/On-Device Deployment — No model compilation pipeline for mobile, browser, or embedded targets
  7. Human-in-the-Loop — No annotation platform, feedback collection, or RLHF data pipeline
  8. MLOps Maturity — No experiment tracking dashboard, model lineage graph, or automated rollback

The report below catalogs every gap organized by functional area, with specific task recommendations for each.


Table of Contents#

  1. Current ML Infrastructure Inventory
  2. Domain-by-Domain ML Requirements
  3. Gap Analysis by Functional Area
  4. Recommended New Tasks
  5. Priority Matrix

1. Current ML Infrastructure Inventory#

1.1 What's Built and Working#

Component Location Status Maturity
Sovereign Inference Engine (ONNX, CUDA, Metal, WebGPU) @nous/core Complete (279/279 tasks) Production
LLM Serving (KV cache, paged attention, parallelism) @nous/core Complete Production
Fine-tuning Framework (LoRA, QLoRA, DoRA, IA3, PEFT) @nous/training Complete (47.5.1-47.5.2) Production
RLHF Pipeline (PPO, DPO, IPO, KTO, ORPO, Constitutional AI) @nous/training Complete (47.5.3) Production
Model Fine-tuning Toolkit (optimizer, curator, tracker, merger) @isis/model-fine-tuning Complete (1038 tests) Production
RunPod Serverless GPU Workers (SD, Flux, ComfyUI, inference) docker/runpod/ Complete (Phase 19) Production
3D Model Inference (TRELLIS.2, Hunyuan3D, SF3D, SPAR3D) @isis/3d-inference-local Complete (Phase 71) Production
Embedding Service (sentence-transformers, Qdrant) docker/psyche/embedding Complete Production
LLM Provider Adapters (Anthropic, OpenAI, Google, local) @isis/llm-providers, @iris/conversation-providers-* Complete Production
S3 Model Storage (versioned, tiered, cross-region) infra/terraform/s3/models.tf Complete Production
CI/CD for GPU Images .github/workflows/deploy-runpod.yml Complete Production
3D Quality Continuous Evaluation .github/workflows/3d-quality-continuous-evaluation.yml Complete Production
Avatar Training (3DGS, blendshapes) serwaa/avatar-engine/training/ Complete (Python) Production
VLM Fine-tuning (educational) minerva/vlm/educational_vlm_finetuning/ Complete (Python) Production
Neural Prover Fine-tuning minerva/verification/autoformalization/ Complete (Python) Production
Conversation Fine-tuning @iris/conversation-finetuning Complete Production
Model Registry Phase 19.6, @metis/model-registry Complete Production
LoRA Training Pipeline (image + video) Phase 17.3.2 Complete Production
Knowledge Distillation @nous/training Complete Production
Distributed Training (ZeRO, tensor/pipeline parallelism) @nous/training Complete Production
Safety & Content Moderation @nous/safety Files exist (42 files) Needs verification

1.2 What's Partially Built#

Component Location Status Gap
Data Pipeline @nous/training 47.5.4 0/20 tasks No data loading, preprocessing, augmentation
Safety Guardrails @nous/safety 47.6 0/60 tasks marked Files exist but tasks unchecked
Vision Models @nous/vision 47.3 0 tasks marked 81 files exist but tasks unchecked
Audio Models @nous/audio 47.4 0 tasks marked 81 files exist but tasks unchecked
Neith AI Runtime Phase 43 0/200+ tasks Game engine ML inference entirely unstarted
LLM-Powered NPCs Phase 53.2 0/45 tasks Local LLM inference for games unstarted

1.3 What's Entirely Missing#

Component Impact Affected Domains
vLLM / TGI / TensorRT-LLM integration Cannot serve LLMs at scale efficiently All LLM-consuming domains
Unified dataset registry & versioning Cannot manage training data across domains All training workflows
Synthetic data generation pipeline Cannot create domain-specific training data All fine-tuning workflows
Annotation/labeling platform Cannot collect human feedback at scale RLHF, safety, quality
MLOps dashboard (experiment tracking) Cannot compare experiments across teams All training workflows
Model drift detection & monitoring Cannot detect model degradation in production All deployed models
Edge model compilation (ONNX→CoreML/TFLite) Cannot deploy to mobile/browser Oya, Psyche, Iris
Automated retraining pipeline Cannot refresh models on new data All domains
Model A/B testing infrastructure Cannot safely roll out model changes All serving endpoints
Federated learning orchestration Cannot train across privacy boundaries Kuan Yin, Airmid, Psyche
Hardware-aware NAS (Neural Architecture Search) Cannot optimize architectures for target hardware Edge deployment
Model compression beyond quantization Cannot optimize model size (pruning, distillation pipelines) Edge, mobile
Training cost estimation & budgeting Cannot plan GPU spend for training runs All training
Multi-tenant model serving Cannot serve fine-tuned models per customer Iris, Yemaya, Psyche
Prompt optimization pipeline Cannot systematically optimize prompts All LLM-consuming domains

2. Domain-by-Domain ML Requirements#

2.1 Domains Requiring Custom Fine-Tuned Models#

Each domain below needs at least one fine-tuned model for sovereignty. The "Model Type" column indicates what must be trained/fine-tuned locally rather than relying on third-party APIs.

Domain Use Case Model Type Base Model Training Data Source Priority
Iris Conversational AI LLM (chat) Llama 3.x / Mistral / DeepSeek Conversation logs, preference data Critical
Iris Intent classification Classifier BERT/DeBERTa Annotated intent datasets High
Iris Embedding model Embedding BGE / E5 / GTE Domain-specific text pairs Critical
Psyche Avatar lip-sync Audio2Face Wav2Lip / SadTalker Video-audio pairs High
Psyche Voice cloning TTS XTTS v2 / Coqui Voice samples High
Psyche Emotion recognition Classifier FER / AffectNet Facial expression data High
Psyche Gaze tracking Regression GazeML Eye tracking data Medium
Isis Image generation Diffusion SDXL / Flux Domain-specific images Critical
Isis Video generation Video diffusion AnimateDiff / SVD Video clips High
Isis 3D generation 3D diffusion TRELLIS.2 / Hunyuan3D 3D scan data High
Isis Texture generation Diffusion SD-inpainting Texture/material images Medium
Isis Audio generation Audio diffusion AudioLDM / MusicGen Audio samples Medium
Sophia Document embeddings Embedding BGE / E5 Document corpus Critical
Sophia Entity extraction NER spaCy / BERT Annotated documents High
Sophia Reranking Cross-encoder BGE-reranker Query-document pairs High
Sophia Translation Seq2Seq NLLB / MarianMT Parallel corpora Medium
Hathor NPC dialogue LLM Llama 3.x fine-tuned Character dialogue scripts High
Hathor World simulation Classifier/LLM Domain-specific Simulation data Medium
Veritas/Yaa NER (Ghanaian entities) NER spaCy + BERT Ghanaian news corpus Critical
Veritas/Yaa Topic classification Classifier BERT / DeBERTa News article corpus High
Veritas/Yaa Bias detection Classifier Custom Annotated political coverage High
Veritas/Yaa Fact verification LLM Fine-tuned LLM Claim-evidence pairs High
Veritas/Yaa Ghana language TTS TTS Coqui / XTTS Twi/Ewe/Ga audio Critical
Veritas/Yaa Sentiment analysis Classifier BERT multilingual Ghanaian social media Medium
Kuan Yin Harm detection Classifier BERT / DeBERTa Moderation datasets Critical
Kuan Yin Intent analysis Classifier Custom Annotated harm data High
Kuan Yin Deepfake detection CNN EfficientNet Deepfake datasets High
Nous Safety guardrails Classifier Custom Red-team data Critical
Nous Jailbreak detection Classifier Custom Jailbreak attempts Critical
Minerva/Metis Lecture generation LLM Fine-tuned LLM Educational content High
Minerva/Metis Assessment generation LLM Fine-tuned LLM Q&A pairs Medium
Minerva/Metis Diagram generation VLM Fine-tuned VLM Diagram-description pairs Medium
Euterpe Music generation Audio diffusion MusicGen / AudioCraft Music samples Medium
Euterpe Audio classification Classifier AST / BEATs Audio classification data Medium
Aphrodite Content moderation Classifier NSFW detector Moderation datasets Critical
Aphrodite Recommendation RecSys Custom User interaction data High
Oya Aerial object detection YOLO/DETR YOLOv8/v9 Aerial imagery Critical
Oya Pose estimation Pose model Sapiens / ViTPose Human pose data High
Oya Path planning RL Custom PPO Simulation data High
Airmid Plant identification CNN EfficientNet / ConvNeXt Plant image datasets High
Airmid Compound analysis GNN SchNet / DimeNet Molecular data Medium
Seshat Interior design Diffusion SDXL fine-tuned Interior design images Medium
Galatea Robot control RL + Vision Custom Simulation + real data High
Neith NPC inference (on-device) Quantized LLM Llama 3.x 7B-13B Game dialogue data High
Themis Originality detection Similarity Custom embedding Content fingerprints High
Themis Plagiarism detection Cross-encoder BGE-reranker variant Text pairs Medium

Total: 45+ distinct fine-tuned models needed for full sovereignty.


3. Gap Analysis by Functional Area#

GAP 1: Training Data Pipeline (CRITICAL)#

Current state: Phase 47.5.4 defines 20 data pipeline tasks — all unchecked. No data loading, preprocessing, tokenization, augmentation, filtering, or curriculum learning infrastructure exists for training.

What's needed:

Task Description Why Critical
Distributed data loading Stream training data across multi-GPU setups Cannot train at scale without it
Data preprocessing & tokenization Prepare raw text/image/audio for model consumption Foundation for all training
Data sharding & streaming Handle datasets larger than memory Required for large-scale training
Packing & dynamic batching Efficient GPU utilization during training 2-5x training speedup
Text augmentation pipeline Back-translation, paraphrasing, synonym replacement Improve model robustness
Data filtering & quality scoring Remove low-quality samples from training data Critical for model quality
Deduplication & near-duplicate removal Prevent data contamination Required for clean training
Data mixing & curriculum design Balance multiple data sources during training Required for multi-task models
Data selection & active learning Prioritize most informative samples Reduce training cost
Format converters (JSONL, Parquet, HuggingFace, Alpaca, ShareGPT) Load data from any format Interop with community datasets

GAP 2: Production Model Serving (CRITICAL)#

Current state: Inference uses ONNX Runtime. No integration with industry-standard LLM serving frameworks optimized for production throughput.

What's needed:

Task Description Why Critical
vLLM integration PagedAttention, continuous batching for LLMs 10-24x throughput vs naive serving
TensorRT-LLM integration NVIDIA-optimized LLM inference Lowest latency on NVIDIA GPUs
Text Generation Inference (TGI) HuggingFace's production LLM server Battle-tested, flash attention
llama.cpp integration CPU-optimized LLM inference Edge/CPU-only deployment
Triton Inference Server Multi-model, multi-framework serving Production orchestration
SGLang integration Structured generation, constrained decoding JSON/code output reliability
Ollama integration Simple local model deployment Developer experience
Multi-LoRA serving Serve multiple LoRA adapters on one base model Per-customer fine-tuning
Speculative decoding pipeline Draft model + verify for faster generation 2-3x speedup for large models
KV cache optimization Prefix caching, cache sharing across requests Reduce TTFT (time to first token)

GAP 3: Dataset Sovereignty (CRITICAL)#

Current state: Datasets are managed ad-hoc per domain. No centralized registry, versioning, or governance.

What's needed:

Task Description Why Critical
Centralized dataset registry Catalog all datasets across domains with metadata Discoverability, deduplication
Dataset versioning (DVC or similar) Track dataset changes over time Reproducibility
Synthetic data generation pipeline Generate training data using LLMs/diffusion Scale training data without manual annotation
Data lineage tracking Track which datasets trained which models Compliance, debugging
Dataset quality dashboard Monitor quality metrics across all datasets Prevent model degradation
PII detection & scrubbing pipeline Remove personal data from training sets Legal/compliance requirement
License compliance checker Verify training data licenses Legal risk mitigation
Cross-domain data sharing protocol Share curated data between domains safely Reduce duplication, improve models
Dataset card generator Auto-generate HuggingFace-style dataset cards Documentation, governance
Annotation schema library Standardized schemas for labeling tasks Consistent data quality

GAP 4: Human-in-the-Loop Pipeline (HIGH)#

Current state: No annotation platform, no feedback collection system, no RLHF data pipeline connecting user feedback to model training.

What's needed:

Task Description Why Critical
Annotation platform (Label Studio integration or custom) Web UI for data labeling Foundation for supervised training
Preference collection UI A/B comparison interface for RLHF data Feed DPO/PPO training
Feedback ingestion pipeline Collect thumbs-up/down from production systems Continuous improvement signal
Inter-annotator agreement tracking Measure labeling consistency Data quality assurance
Active learning loop Route uncertain samples to annotators Efficient annotation budget
Annotation task management Queue, assign, review annotation batches Scale annotation operations
Quality control for annotations Gold standard checks, annotator calibration Prevent label noise
RLHF data pipeline Convert feedback → preference pairs → training data Close the feedback loop
Red team dataset curation Organize adversarial prompts for safety testing Safety training
Evaluation set management Maintain held-out eval sets per domain Prevent contamination

GAP 5: MLOps & Experiment Management (HIGH)#

Current state: Basic experiment tracking exists in @isis/model-fine-tuning and serwaa/avatar-engine. No unified MLOps layer.

What's needed:

Task Description Why Critical
Experiment tracking dashboard (MLflow or W&B self-hosted) Track all training runs centrally Compare experiments, reproduce results
Model lineage graph Visualize base model → fine-tune → merge → deploy Understand model provenance
Automated model evaluation pipeline Run benchmarks on every model checkpoint Catch regressions early
Model drift detection Monitor production model outputs for distribution shift Prevent silent degradation
Automated retraining triggers Retrain when drift exceeds threshold Maintain model quality
Model A/B testing framework Route traffic between model versions Safe model rollout
Shadow deployment Run new model alongside old, compare outputs Risk-free evaluation
Rollback automation One-click model version rollback Incident response
GPU cost tracking per experiment Track $/experiment, $/token for training Budget management
Training job scheduler Queue and prioritize training jobs across GPU pool Resource efficiency

GAP 6: Edge & On-Device Deployment (HIGH)#

Current state: RunPod serves models in the cloud. No pipeline for compiling/deploying models to edge devices, mobile, or browsers.

What's needed:

Task Description Why Critical
ONNX export pipeline Convert PyTorch/HF models to ONNX Cross-platform deployment
CoreML conversion ONNX → CoreML for iOS/macOS Apple device deployment
TFLite conversion ONNX → TensorFlow Lite for Android Android deployment
TensorRT compilation ONNX → TensorRT for NVIDIA edge Jetson/edge GPU deployment
WebGPU/WebNN inference Browser-native ML inference Web app deployment
WASM inference runtime ML models in WebAssembly Portable edge deployment
Model pruning pipeline Structured/unstructured pruning for size reduction Smaller models for edge
Neural Architecture Search (NAS) Find optimal architecture for target hardware Hardware-specific optimization
Quantization-aware training (QAT) Train with quantization in the loop Better quality than post-training quantization
Model benchmark suite per target Latency/accuracy/memory benchmarks per device class Deployment decision support

GAP 7: Continuous Evaluation & Benchmarking (HIGH)#

Current state: 3D quality continuous evaluation exists. No equivalent for LLMs, classifiers, embeddings, or other model types.

What's needed:

Task Description Why Critical
LLM evaluation suite (MMLU, HumanEval, etc.) Standard benchmarks for LLM quality Compare fine-tuned vs base models
RAG evaluation framework (RAGAS) Evaluate retrieval + generation quality Sophia, Iris, Veritas
Embedding quality benchmarks (MTEB) Evaluate embedding models Sophia, Iris knowledge systems
Safety benchmark suite (TruthfulQA, BBQ, etc.) Evaluate model safety/bias Nous, Kuan Yin
Domain-specific eval sets Per-domain held-out evaluation data Each domain's model quality
Automated regression testing Compare model versions on fixed eval sets Prevent quality regressions
Human evaluation framework Structured human ratings on model outputs Ground truth quality signal
Leaderboard dashboard Visual comparison of model versions Track improvement over time
Cost-quality Pareto analysis Plot quality vs cost for model selection Optimal model selection
Cross-lingual evaluation Evaluate models across languages (especially Ghanaian) Veritas, global deployment

GAP 8: Specialized Training Pipelines per Domain (MEDIUM-HIGH)#

Current state: Generic fine-tuning framework exists. No domain-specific training pipelines that automate the full cycle from data → training → evaluation → deployment for each domain's unique needs.

What's needed:

Domain Pipeline Description
Iris Conversation model fine-tuning pipeline Data collection → format → LoRA train → eval → deploy
Iris Embedding model fine-tuning pipeline Triplet mining → contrastive training → MTEB eval → deploy
Veritas NER model training pipeline Annotate Ghanaian entities → train spaCy/BERT → eval → deploy
Veritas Bias detector training pipeline Collect political coverage → annotate → train classifier → deploy
Veritas Ghana language TTS pipeline Collect Twi/Ewe/Ga audio → train XTTS → eval → deploy
Psyche Voice cloning pipeline Collect voice samples → fine-tune XTTS → quality eval → deploy
Psyche Emotion model training pipeline Collect facial data → train classifier → eval → deploy
Kuan Yin Harm detector training pipeline Collect moderation data → train → red-team eval → deploy
Nous Safety classifier pipeline Red-team generation → train guardrails → eval → deploy
Isis LoRA training pipeline (images) Collect style data → train LoRA → quality gate → registry
Isis LoRA training pipeline (video) Collect motion data → train video LoRA → eval → deploy
Sophia Reranker training pipeline Mine hard negatives → train cross-encoder → eval → deploy
Oya Aerial vision training pipeline Collect aerial data → fine-tune YOLO → quantize → deploy
Airmid Plant ID training pipeline Collect plant images → fine-tune CNN → quantize → deploy
Euterpe Music model training pipeline Collect music data → fine-tune AudioCraft → eval → deploy
Minerva Lecture LLM training pipeline Collect educational data → fine-tune LLM → eval → deploy
Hathor NPC dialogue training pipeline Generate character data → fine-tune LLM → eval → deploy

GAP 9: Model Governance & Compliance (MEDIUM)#

Current state: Basic model registry exists. No governance framework for model lifecycle management, compliance, or audit trails.

What's needed:

Task Description Why Critical
Model card generation Auto-generate model documentation (training data, metrics, limitations) Compliance, transparency
Model approval workflow Review/approve models before production deployment Quality gate
License compliance tracking Track licenses of base models and training data Legal risk
Bias audit pipeline Automated bias testing across protected attributes Fairness compliance
Model deprecation workflow Graceful model retirement with migration path Lifecycle management
Audit trail for model changes Who trained what, when, with what data Compliance, debugging
Data retention policies per model How long to keep training data and checkpoints Storage cost, compliance
Model access control Role-based access to model artifacts Security

GAP 10: GPU Infrastructure Scaling (MEDIUM)#

Current state: RunPod for serverless GPU, some Terraform for AWS. No multi-cloud GPU orchestration or training cluster management.

What's needed:

Task Description Why Critical
Multi-cloud GPU orchestration Use RunPod + AWS + GCP GPUs as single pool Cost optimization, availability
Training cluster provisioner Spin up/down multi-GPU training clusters on demand Large-scale training
Spot/preemptible instance management Use cheap spot instances with checkpointing 60-90% cost reduction
GPU memory profiler Profile VRAM usage per model/batch size Right-size GPU allocation
Cost estimator for training jobs Estimate $/run before launching Budget planning
Queue-based training scheduler Priority queue for training jobs across teams Resource fairness
Distributed training launcher One-command multi-node training setup Developer experience
GPU health monitoring Track GPU temperature, utilization, errors Prevent training failures
Auto-scaling for inference Scale GPU workers based on request queue depth Handle traffic spikes
Reserved instance optimization Analyze usage patterns, recommend reservations Cost optimization

This should be a new dedicated phase in TODOS.md focused exclusively on the cross-cutting ML infrastructure that enables sovereignty across all domains.

Section 1: Centralized Dataset Management#

text
- [ ] Design dataset registry schema (name, version, domain, format, size, license, quality score, lineage)
- [ ] Implement dataset registry service with PostgreSQL storage
- [ ] Build dataset versioning system (content-addressable storage, delta tracking)
- [ ] Create dataset upload/download CLI tool
- [ ] Implement dataset format converters (JSONL, Parquet, HuggingFace, Alpaca, ShareGPT, OpenAI)
- [ ] Build dataset quality scoring engine (completeness, diversity, toxicity, duplication)
- [ ] Implement deduplication pipeline (exact + near-duplicate with MinHash/SimHash)
- [ ] Build PII detection and scrubbing pipeline for training data
- [ ] Create license compliance checker for training datasets
- [ ] Implement dataset card auto-generation (HuggingFace format)
- [ ] Build data lineage tracker (which datasets trained which models)
- [ ] Create cross-domain data sharing protocol with access controls
- [ ] Implement dataset split management (train/val/test with contamination prevention)
- [ ] Build streaming dataset loader for datasets larger than memory
- [ ] Create annotation schema library with standard schemas per task type
- [ ] Implement dataset statistics dashboard (size, distribution, quality over time)
- [ ] Build dataset diff tool (compare versions, detect distribution shifts)
- [ ] Create synthetic data generation orchestrator (LLM-based, diffusion-based)
- [ ] Implement data mixing recipe system (specify ratios, curricula)
- [ ] Build dataset export pipeline for external sharing (anonymized, licensed)

Section 2: Synthetic Data Generation#

text
- [ ] Design synthetic data generation framework (pluggable generators per modality)
- [ ] Implement LLM-based text data generator (instruction following, conversation, QA)
- [ ] Build self-instruct pipeline (seed instructions → LLM generation → quality filter)
- [ ] Implement Evol-Instruct pipeline (complexity evolution for instruction data)
- [ ] Create persona-based dialogue generator (varied styles, knowledge levels)
- [ ] Build synthetic preference pair generator (for DPO/RLHF training)
- [ ] Implement synthetic NER data generator (domain-specific entity annotation)
- [ ] Create synthetic code data generator (problems, solutions, explanations)
- [ ] Build diffusion-based image data generator (with controlled variation)
- [ ] Implement TTS-based audio data generator (multi-speaker, multi-language)
- [ ] Create 3D scene synthetic data generator (for 3D model training)
- [ ] Build adversarial data generator (red-team prompts, jailbreak attempts)
- [ ] Implement multi-lingual synthetic data pipeline (Twi, Ewe, Ga, Hausa, Dagbani)
- [ ] Create quality filter for synthetic data (decontamination, deduplication, scoring)
- [ ] Build human-AI collaborative data generation workflow

Section 3: Annotation & Human Feedback Platform#

text
- [ ] Design annotation platform architecture (task management, reviewer UI, quality control)
- [ ] Integrate Label Studio as self-hosted annotation backend
- [ ] Build preference comparison UI (side-by-side A/B for RLHF data collection)
- [ ] Implement thumbs-up/down feedback ingestion from production endpoints
- [ ] Create annotation task queue with priority and assignment management
- [ ] Build inter-annotator agreement calculator (Cohen's kappa, Fleiss' kappa)
- [ ] Implement gold standard quality checks (inject known-answer questions)
- [ ] Create annotator calibration workflow (training + certification)
- [ ] Build active learning loop (route uncertain samples to annotators)
- [ ] Implement RLHF data pipeline (feedback → preference pairs → training format)
- [ ] Create red team dataset curation tool (organize adversarial prompts by category)
- [ ] Build evaluation set management (held-out sets with contamination monitoring)
- [ ] Implement annotation export to multiple training formats
- [ ] Create annotation analytics dashboard (throughput, quality, cost per annotation)
- [ ] Build multi-annotator consensus resolution workflow

Section 4: Production Model Serving#

text
- [ ] Integrate vLLM as primary LLM serving engine (PagedAttention, continuous batching)
- [ ] Build vLLM Docker image with multi-LoRA support for RunPod deployment
- [ ] Integrate TensorRT-LLM for NVIDIA-optimized inference
- [ ] Build TGI (Text Generation Inference) adapter for HuggingFace models
- [ ] Integrate llama.cpp for CPU-only and edge LLM serving
- [ ] Build SGLang integration for structured/constrained generation
- [ ] Implement multi-LoRA serving (serve N fine-tuned adapters on one base model)
- [ ] Create model routing layer (route requests to optimal backend by model/hardware)
- [ ] Build speculative decoding pipeline (draft model selection, verification)
- [ ] Implement KV cache management (prefix caching, cache sharing, eviction policies)
- [ ] Create batching optimizer (dynamic batching by sequence length)
- [ ] Build model warmup and preloading system
- [ ] Implement graceful model swapping (zero-downtime model updates)
- [ ] Create inference cost calculator ($/token by model, backend, GPU)
- [ ] Build latency monitoring per model endpoint with SLA tracking
- [ ] Implement Ollama-compatible API for local development
- [ ] Create model serving benchmark suite (throughput, latency, memory)
- [ ] Build auto-scaling controller for GPU inference workers
- [ ] Implement request priority queuing (interactive > batch > background)
- [ ] Create multi-tenant model isolation (per-customer fine-tuned models)

Section 5: MLOps & Experiment Management#

text
- [ ] Deploy self-hosted MLflow as experiment tracking backend
- [ ] Build experiment tracking SDK (log metrics, params, artifacts from training code)
- [ ] Create model lineage graph (visualize base → fine-tune → merge → deploy chain)
- [ ] Implement model comparison dashboard (side-by-side metrics, outputs)
- [ ] Build automated model evaluation pipeline (run benchmarks on every checkpoint)
- [ ] Create model drift detection service (monitor output distribution over time)
- [ ] Implement automated retraining trigger (retrain when drift exceeds threshold)
- [ ] Build model A/B testing framework (split traffic, collect metrics, auto-promote)
- [ ] Create shadow deployment system (run new model alongside production, compare)
- [ ] Implement one-click model rollback (revert to previous version)
- [ ] Build GPU cost tracking per experiment ($/run, $/token, $/epoch)
- [ ] Create training job scheduler with priority queue
- [ ] Implement training job dashboard (running, queued, completed, failed)
- [ ] Build model artifact storage with deduplication (shared base weights)
- [ ] Create model deployment pipeline (registry → testing → staging → production)
- [ ] Implement canary deployment for model updates
- [ ] Build model performance alerting (latency, error rate, quality degradation)
- [ ] Create reproducibility system (pin data version + code version + config)
- [ ] Implement experiment tagging and search
- [ ] Build model governance approval workflow (review before production)

Section 6: Edge & On-Device Deployment#

text
- [ ] Build ONNX export pipeline (PyTorch/HF → ONNX with optimization)
- [ ] Implement CoreML conversion pipeline (ONNX → CoreML for iOS/macOS)
- [ ] Build TFLite conversion pipeline (ONNX → TensorFlow Lite for Android)
- [ ] Create TensorRT compilation pipeline (ONNX → TensorRT for NVIDIA edge)
- [ ] Implement WebGPU inference runtime for browser deployment
- [ ] Build WebNN inference adapter for cross-browser ML
- [ ] Create WASM inference runtime (ONNX Runtime Web or custom)
- [ ] Implement model pruning pipeline (structured + unstructured)
- [ ] Build quantization-aware training (QAT) pipeline
- [ ] Create post-training quantization pipeline (INT8, INT4, FP8)
- [ ] Implement knowledge distillation pipeline (teacher → student automation)
- [ ] Build target device profiler (benchmark model on target hardware)
- [ ] Create model size/latency/accuracy Pareto optimizer
- [ ] Implement on-device model update mechanism (differential model updates)
- [ ] Build edge model deployment CI/CD (compile → test → package → distribute)
- [ ] Create device compatibility matrix (which models run on which devices)
- [ ] Implement fallback chain (try on-device → edge server → cloud)
- [ ] Build battery/thermal-aware inference scheduler for mobile
- [ ] Create edge model monitoring (collect on-device metrics, aggregate centrally)
- [ ] Implement federated learning orchestrator for privacy-preserving training

Section 7: Continuous Evaluation & Benchmarking#

text
- [ ] Design unified benchmark framework (pluggable evaluators, standard metrics)
- [ ] Implement LLM benchmark suite (MMLU, HumanEval, MT-Bench, Arena-Hard)
- [ ] Build RAG evaluation framework (faithfulness, relevance, context precision)
- [ ] Create embedding benchmark suite (MTEB subset relevant to Oshun domains)
- [ ] Implement safety benchmark suite (TruthfulQA, BBQ, RealToxicityPrompts)
- [ ] Build multi-lingual evaluation suite (focus on Ghanaian languages)
- [ ] Create domain-specific eval sets per domain (Iris, Sophia, Veritas, etc.)
- [ ] Implement automated regression testing (compare model versions on fixed evals)
- [ ] Build human evaluation framework (structured rating tasks, sampling)
- [ ] Create leaderboard dashboard (track model quality over time per domain)
- [ ] Implement cost-quality Pareto analysis tool ($/quality for model selection)
- [ ] Build cross-model comparison tool (compare fine-tuned vs base vs competitors)
- [ ] Create eval set contamination detector (check if eval data leaked into training)
- [ ] Implement benchmark scheduling (run daily/weekly on production models)
- [ ] Build evaluation report generator (PDF/HTML reports per model version)
- [ ] Create custom benchmark authoring tool (define domain-specific test cases)
- [ ] Implement adversarial evaluation (automatically generated edge cases)
- [ ] Build latency-quality tradeoff analyzer
- [ ] Create evaluation set versioning (track eval set changes, normalize scores)
- [ ] Implement multi-judge evaluation (aggregate LLM-as-judge + human ratings)

Section 8: Domain-Specific Training Pipelines#

text
- [ ] Build Iris conversational model fine-tuning pipeline (data→LoRA→eval→deploy)
- [ ] Build Iris embedding model fine-tuning pipeline (triplet mining→train→MTEB→deploy)
- [ ] Build Veritas NER model training pipeline (annotate→train spaCy/BERT→eval→deploy)
- [ ] Build Veritas bias detector training pipeline (annotate→train→eval→deploy)
- [ ] Build Veritas Ghana language TTS pipeline (collect audio→train XTTS→eval→deploy)
- [ ] Build Psyche voice cloning pipeline (samples→fine-tune XTTS→quality→deploy)
- [ ] Build Psyche emotion model training pipeline (facial data→train→eval→deploy)
- [ ] Build Kuan Yin harm detector training pipeline (moderation data→train→red-team→deploy)
- [ ] Build Nous safety classifier training pipeline (red-team→train guardrails→eval→deploy)
- [ ] Build Isis image LoRA training pipeline (style data→train LoRA→quality gate→registry)
- [ ] Build Isis video LoRA training pipeline (motion data→train→eval→deploy)
- [ ] Build Sophia reranker training pipeline (hard negatives→train cross-encoder→eval→deploy)
- [ ] Build Oya aerial vision training pipeline (aerial data→fine-tune YOLO→quantize→deploy)
- [ ] Build Airmid plant ID training pipeline (images→fine-tune CNN→quantize→deploy)
- [ ] Build Euterpe music model training pipeline (music data→fine-tune→eval→deploy)
- [ ] Build Minerva lecture LLM training pipeline (educational data→fine-tune→eval→deploy)
- [ ] Build Hathor NPC dialogue training pipeline (character data→fine-tune LLM→eval→deploy)
- [ ] Build Aphrodite content moderation training pipeline (data→train NSFW detector→eval→deploy)
- [ ] Build Themis originality model training pipeline (fingerprints→train→eval→deploy)
- [ ] Build cross-domain transfer learning pipeline (share learned representations)

Section 9: GPU Infrastructure & Training Clusters#

text
- [ ] Design multi-cloud GPU orchestration layer (RunPod + AWS + GCP as unified pool)
- [ ] Build training cluster provisioner (spin up multi-GPU clusters on demand)
- [ ] Implement spot/preemptible instance manager with automatic checkpointing
- [ ] Create GPU memory profiler for right-sizing GPU allocation per workload
- [ ] Build training cost estimator (predict $/run before launching)
- [ ] Implement queue-based training scheduler with priority and fairness
- [ ] Create distributed training launcher (one-command multi-node setup)
- [ ] Build GPU health monitoring dashboard (temperature, utilization, errors, ECC)
- [ ] Implement auto-scaling for inference (scale workers by queue depth)
- [ ] Create reserved instance optimization analyzer
- [ ] Build training checkpoint management for spot instance recovery
- [ ] Implement NCCL/Gloo configuration optimizer for multi-node training
- [ ] Create GPU utilization alerting (underutilized or overloaded warnings)
- [ ] Build training cost dashboard (per-domain, per-model, per-experiment)
- [ ] Implement GPU sharing for small models (MIG, MPS, time-sharing)

Section 10: Model Governance & Compliance#

text
- [ ] Build model card auto-generator (training data, metrics, limitations, biases)
- [ ] Create model approval workflow (review gate before production deployment)
- [ ] Implement license compliance tracker for base models and training data
- [ ] Build automated bias audit pipeline (test across protected attributes)
- [ ] Create model deprecation workflow (migration path, sunset timeline)
- [ ] Implement audit trail for all model operations (train, eval, deploy, rollback)
- [ ] Build data retention policy engine (auto-expire old checkpoints and data)
- [ ] Create model access control (RBAC for model artifacts and endpoints)
- [ ] Implement model risk assessment (classify models by risk tier)
- [ ] Build compliance reporting dashboard (summarize governance state)

5. Priority Matrix#

P0 — Blocks All Sovereignty (Must Complete First)#

Gap Estimated Tasks Rationale
Training Data Pipeline (Phase 47.5.4) 20 tasks (exist, unchecked) Cannot train ANY model without data loading
Production Model Serving (vLLM/TGI) 20 new tasks Cannot serve fine-tuned LLMs efficiently
Centralized Dataset Registry 20 new tasks Cannot manage training data at scale
Nous Safety (Phase 47.6) 60 tasks (exist, unchecked) Cannot deploy models without safety guardrails

P1 — Enables Continuous Improvement#

Gap Estimated Tasks Rationale
MLOps & Experiment Management 20 new tasks Cannot track/compare/reproduce experiments
Continuous Evaluation & Benchmarking 20 new tasks Cannot measure model quality systematically
Human-in-the-Loop Pipeline 15 new tasks Cannot collect feedback for RLHF/improvement
Synthetic Data Generation 15 new tasks Cannot scale training data without manual effort

P2 — Enables Full Domain Coverage#

Gap Estimated Tasks Rationale
Domain-Specific Training Pipelines 20 new tasks Each domain needs its own end-to-end pipeline
Nous Advanced Models (47.7-47.16) 200+ tasks (exist, unchecked) Video, image control, speech, animation
Neith AI Runtime (Phase 43.1.6) 15 tasks (exist, unchecked) Game engine ML inference
LLM-Powered NPCs (Phase 53.2) 45 tasks (exist, unchecked) On-device game AI

P3 — Enables Edge & Scale#

Gap Estimated Tasks Rationale
Edge & On-Device Deployment 20 new tasks Mobile, browser, embedded deployment
GPU Infrastructure Scaling 15 new tasks Multi-cloud, spot instances, cost optimization
Model Governance & Compliance 10 new tasks Audit trails, bias testing, compliance

Summary Statistics#

Category Existing Unchecked Tasks New Tasks Needed Total
Training Data Pipeline (47.5.4) 20 0 20
Safety Framework (47.6) 60 0 60
Advanced Models (47.7-47.16) ~200 0 ~200
Neith AI Runtime (43, 53) ~260 0 ~260
Centralized Dataset Management 0 20 20
Synthetic Data Generation 0 15 15
Annotation & Human Feedback 0 15 15
Production Model Serving 0 20 20
MLOps & Experiment Management 0 20 20
Edge & On-Device Deployment 0 20 20
Continuous Evaluation 0 20 20
Domain-Specific Pipelines 0 20 20
GPU Infrastructure 0 15 15
Model Governance 0 10 10
TOTAL ~540 ~175 ~715

~540 existing unchecked tasks need to be completed (mostly in Phases 47, 43, 53) and ~175 new tasks need to be added to achieve full ML sovereignty.

The most critical new phase to add is the ML Sovereignty Foundation covering dataset management, model serving, MLOps, evaluation, and domain-specific pipelines — none of which are currently tracked in any TODOS.md.


Appendix: Models to Self-Host (Open Source Recommendations)#

LLMs#

Use Case Recommended Base Model Size License
General chat/reasoning Llama 3.1 / 3.2 8B-70B Meta Community
Code generation DeepSeek-Coder V2 / StarCoder2 7B-33B Apache 2.0
Long context Mistral Nemo / Qwen 2.5 7B-72B Apache 2.0
Multilingual Aya 23 / BLOOM 8B-35B Apache 2.0
Function calling Hermes 3 / Functionary 7B-70B Apache 2.0
Small/fast Phi-3 / Gemma 2 2B-9B MIT/Apache

Embedding Models#

Use Case Model Dimensions License
Text embedding BGE-large / E5-large 1024 MIT
Multilingual embedding BGE-M3 / multilingual-e5 1024 MIT
Code embedding CodeSage / StarEncoder 768-1024 Apache 2.0
Cross-encoder reranking BGE-reranker-large - MIT

Vision Models#

Use Case Model License
Image classification ConvNeXt V2 / EfficientNet V2 Apache 2.0
Object detection YOLOv9 / DETR / RT-DETR AGPL/Apache
Image generation SDXL / Flux.1 Open RAIL/Apache
Video generation AnimateDiff / SVD Apache/Custom
OCR PaddleOCR / TrOCR Apache 2.0

Audio Models#

Use Case Model License
Speech-to-text Whisper large-v3 MIT
Text-to-speech XTTS v2 / Coqui MPL 2.0
Music generation MusicGen / AudioCraft MIT
Audio classification BEATs / AST MIT
Voice cloning RVC / SoVITS MIT

3D Models#

Use Case Model License
Text-to-3D TRELLIS.2 MIT
Image-to-3D Hunyuan3D 2.1 / SF3D Tencent/Stability
Gaussian splatting 3DGS / InstantSplat Various
Mesh generation MeshAnything V2 Apache 2.0

Safety Models#

Use Case Model License
Toxicity detection Detoxify / Perspective Apache 2.0
NSFW detection NSFW detector / OpenNSFW2 MIT
Deepfake detection FaceForensics++ models Academic
PII detection Presidio / GLiNER MIT