# ML Sovereignty Gap Analysis Report

**Date:** 2026-04-11 **Scope:** Complete audit of ML/AI infrastructure across
the Oshun monorepo **Goal:** Identify all missing tasks required for full model
sovereignty — the ability to self-host, fine-tune, train, evaluate, and deploy
custom open-source models for every ML use case in the system.

---

## Executive Summary

The Oshun monorepo has **substantial ML infrastructure already built** — a
sovereign inference engine (@nous/core), comprehensive fine-tuning framework
(@nous/training, @isis/model-fine-tuning), GPU serving via RunPod, and
domain-specific ML services across 15+ domains. However, **full ML sovereignty
is not yet achievable** because critical gaps exist in:

1. **Data Pipeline Infrastructure** — No training data loading, preprocessing,
   or curriculum systems
2. **Model Serving Optimization** — No vLLM/TGI/TensorRT-LLM integration for
   production LLM serving
3. **Continuous Training Pipelines** — No automated retraining, drift detection,
   or model refresh workflows
4. **Evaluation & Benchmarking** — No unified cross-domain evaluation framework
5. **Dataset Sovereignty** — No centralized dataset registry, versioning, or
   synthetic data generation at scale
6. **Edge/On-Device Deployment** — No model compilation pipeline for mobile,
   browser, or embedded targets
7. **Human-in-the-Loop** — No annotation platform, feedback collection, or RLHF
   data pipeline
8. **MLOps Maturity** — No experiment tracking dashboard, model lineage graph,
   or automated rollback

The report below catalogs every gap organized by functional area, with specific
task recommendations for each.

---

## Table of Contents

1. [Current ML Infrastructure Inventory](#1-current-ml-infrastructure-inventory)
2. [Domain-by-Domain ML Requirements](#2-domain-by-domain-ml-requirements)
3. [Gap Analysis by Functional Area](#3-gap-analysis-by-functional-area)
4. [Recommended New Tasks](#4-recommended-new-tasks)
5. [Priority Matrix](#5-priority-matrix)

---

## 1. Current ML Infrastructure Inventory

### 1.1 What's Built and Working

| Component                                                       | Location                                               | Status                   | Maturity           |
| --------------------------------------------------------------- | ------------------------------------------------------ | ------------------------ | ------------------ |
| Sovereign Inference Engine (ONNX, CUDA, Metal, WebGPU)          | @nous/core                                             | Complete (279/279 tasks) | Production         |
| LLM Serving (KV cache, paged attention, parallelism)            | @nous/core                                             | Complete                 | Production         |
| Fine-tuning Framework (LoRA, QLoRA, DoRA, IA3, PEFT)            | @nous/training                                         | Complete (47.5.1-47.5.2) | Production         |
| RLHF Pipeline (PPO, DPO, IPO, KTO, ORPO, Constitutional AI)     | @nous/training                                         | Complete (47.5.3)        | Production         |
| Model Fine-tuning Toolkit (optimizer, curator, tracker, merger) | @isis/model-fine-tuning                                | Complete (1038 tests)    | Production         |
| RunPod Serverless GPU Workers (SD, Flux, ComfyUI, inference)    | docker/runpod/                                         | Complete (Phase 19)      | Production         |
| 3D Model Inference (TRELLIS.2, Hunyuan3D, SF3D, SPAR3D)         | @isis/3d-inference-local                               | Complete (Phase 71)      | Production         |
| Embedding Service (sentence-transformers, Qdrant)               | docker/psyche/embedding                                | Complete                 | Production         |
| LLM Provider Adapters (Anthropic, OpenAI, Google, local)        | @isis/llm-providers, @iris/conversation-providers-\*   | Complete                 | Production         |
| S3 Model Storage (versioned, tiered, cross-region)              | infra/terraform/s3/models.tf                           | Complete                 | Production         |
| CI/CD for GPU Images                                            | .github/workflows/deploy-runpod.yml                    | Complete                 | Production         |
| 3D Quality Continuous Evaluation                                | .github/workflows/3d-quality-continuous-evaluation.yml | Complete                 | Production         |
| Avatar Training (3DGS, blendshapes)                             | serwaa/avatar-engine/training/                         | Complete (Python)        | Production         |
| VLM Fine-tuning (educational)                                   | minerva/vlm/educational_vlm_finetuning/                | Complete (Python)        | Production         |
| Neural Prover Fine-tuning                                       | minerva/verification/autoformalization/                | Complete (Python)        | Production         |
| Conversation Fine-tuning                                        | @iris/conversation-finetuning                          | Complete                 | Production         |
| Model Registry                                                  | Phase 19.6, @metis/model-registry                      | Complete                 | Production         |
| LoRA Training Pipeline (image + video)                          | Phase 17.3.2                                           | Complete                 | Production         |
| Knowledge Distillation                                          | @nous/training                                         | Complete                 | Production         |
| Distributed Training (ZeRO, tensor/pipeline parallelism)        | @nous/training                                         | Complete                 | Production         |
| Safety & Content Moderation                                     | @nous/safety                                           | Files exist (42 files)   | Needs verification |

### 1.2 What's Partially Built

| Component         | Location              | Status            | Gap                                          |
| ----------------- | --------------------- | ----------------- | -------------------------------------------- |
| Data Pipeline     | @nous/training 47.5.4 | 0/20 tasks        | No data loading, preprocessing, augmentation |
| Safety Guardrails | @nous/safety 47.6     | 0/60 tasks marked | Files exist but tasks unchecked              |
| Vision Models     | @nous/vision 47.3     | 0 tasks marked    | 81 files exist but tasks unchecked           |
| Audio Models      | @nous/audio 47.4      | 0 tasks marked    | 81 files exist but tasks unchecked           |
| Neith AI Runtime  | Phase 43              | 0/200+ tasks      | Game engine ML inference entirely unstarted  |
| LLM-Powered NPCs  | Phase 53.2            | 0/45 tasks        | Local LLM inference for games unstarted      |

### 1.3 What's Entirely Missing

| Component                                       | Impact                                                       | Affected Domains          |
| ----------------------------------------------- | ------------------------------------------------------------ | ------------------------- |
| vLLM / TGI / TensorRT-LLM integration           | Cannot serve LLMs at scale efficiently                       | All LLM-consuming domains |
| Unified dataset registry & versioning           | Cannot manage training data across domains                   | All training workflows    |
| Synthetic data generation pipeline              | Cannot create domain-specific training data                  | All fine-tuning workflows |
| Annotation/labeling platform                    | Cannot collect human feedback at scale                       | RLHF, safety, quality     |
| MLOps dashboard (experiment tracking)           | Cannot compare experiments across teams                      | All training workflows    |
| Model drift detection & monitoring              | Cannot detect model degradation in production                | All deployed models       |
| Edge model compilation (ONNX→CoreML/TFLite)     | Cannot deploy to mobile/browser                              | Oya, Psyche, Iris         |
| Automated retraining pipeline                   | Cannot refresh models on new data                            | All domains               |
| Model A/B testing infrastructure                | Cannot safely roll out model changes                         | All serving endpoints     |
| Federated learning orchestration                | Cannot train across privacy boundaries                       | Kuan Yin, Airmid, Psyche  |
| Hardware-aware NAS (Neural Architecture Search) | Cannot optimize architectures for target hardware            | Edge deployment           |
| Model compression beyond quantization           | Cannot optimize model size (pruning, distillation pipelines) | Edge, mobile              |
| Training cost estimation & budgeting            | Cannot plan GPU spend for training runs                      | All training              |
| Multi-tenant model serving                      | Cannot serve fine-tuned models per customer                  | Iris, Yemaya, Psyche      |
| Prompt optimization pipeline                    | Cannot systematically optimize prompts                       | All LLM-consuming domains |

---

## 2. Domain-by-Domain ML Requirements

### 2.1 Domains Requiring Custom Fine-Tuned Models

Each domain below needs at least one fine-tuned model for sovereignty. The
"Model Type" column indicates what must be trained/fine-tuned locally rather
than relying on third-party APIs.

| Domain            | Use Case                  | Model Type      | Base Model                     | Training Data Source               | Priority |
| ----------------- | ------------------------- | --------------- | ------------------------------ | ---------------------------------- | -------- |
| **Iris**          | Conversational AI         | LLM (chat)      | Llama 3.x / Mistral / DeepSeek | Conversation logs, preference data | Critical |
| **Iris**          | Intent classification     | Classifier      | BERT/DeBERTa                   | Annotated intent datasets          | High     |
| **Iris**          | Embedding model           | Embedding       | BGE / E5 / GTE                 | Domain-specific text pairs         | Critical |
| **Psyche**        | Avatar lip-sync           | Audio2Face      | Wav2Lip / SadTalker            | Video-audio pairs                  | High     |
| **Psyche**        | Voice cloning             | TTS             | XTTS v2 / Coqui                | Voice samples                      | High     |
| **Psyche**        | Emotion recognition       | Classifier      | FER / AffectNet                | Facial expression data             | High     |
| **Psyche**        | Gaze tracking             | Regression      | GazeML                         | Eye tracking data                  | Medium   |
| **Isis**          | Image generation          | Diffusion       | SDXL / Flux                    | Domain-specific images             | Critical |
| **Isis**          | Video generation          | Video diffusion | AnimateDiff / SVD              | Video clips                        | High     |
| **Isis**          | 3D generation             | 3D diffusion    | TRELLIS.2 / Hunyuan3D          | 3D scan data                       | High     |
| **Isis**          | Texture generation        | Diffusion       | SD-inpainting                  | Texture/material images            | Medium   |
| **Isis**          | Audio generation          | Audio diffusion | AudioLDM / MusicGen            | Audio samples                      | Medium   |
| **Sophia**        | Document embeddings       | Embedding       | BGE / E5                       | Document corpus                    | Critical |
| **Sophia**        | Entity extraction         | NER             | spaCy / BERT                   | Annotated documents                | High     |
| **Sophia**        | Reranking                 | Cross-encoder   | BGE-reranker                   | Query-document pairs               | High     |
| **Sophia**        | Translation               | Seq2Seq         | NLLB / MarianMT                | Parallel corpora                   | Medium   |
| **Hathor**        | NPC dialogue              | LLM             | Llama 3.x fine-tuned           | Character dialogue scripts         | High     |
| **Hathor**        | World simulation          | Classifier/LLM  | Domain-specific                | Simulation data                    | Medium   |
| **Veritas/Yaa**   | NER (Ghanaian entities)   | NER             | spaCy + BERT                   | Ghanaian news corpus               | Critical |
| **Veritas/Yaa**   | Topic classification      | Classifier      | BERT / DeBERTa                 | News article corpus                | High     |
| **Veritas/Yaa**   | Bias detection            | Classifier      | Custom                         | Annotated political coverage       | High     |
| **Veritas/Yaa**   | Fact verification         | LLM             | Fine-tuned LLM                 | Claim-evidence pairs               | High     |
| **Veritas/Yaa**   | Ghana language TTS        | TTS             | Coqui / XTTS                   | Twi/Ewe/Ga audio                   | Critical |
| **Veritas/Yaa**   | Sentiment analysis        | Classifier      | BERT multilingual              | Ghanaian social media              | Medium   |
| **Kuan Yin**      | Harm detection            | Classifier      | BERT / DeBERTa                 | Moderation datasets                | Critical |
| **Kuan Yin**      | Intent analysis           | Classifier      | Custom                         | Annotated harm data                | High     |
| **Kuan Yin**      | Deepfake detection        | CNN             | EfficientNet                   | Deepfake datasets                  | High     |
| **Nous**          | Safety guardrails         | Classifier      | Custom                         | Red-team data                      | Critical |
| **Nous**          | Jailbreak detection       | Classifier      | Custom                         | Jailbreak attempts                 | Critical |
| **Minerva/Metis** | Lecture generation        | LLM             | Fine-tuned LLM                 | Educational content                | High     |
| **Minerva/Metis** | Assessment generation     | LLM             | Fine-tuned LLM                 | Q&A pairs                          | Medium   |
| **Minerva/Metis** | Diagram generation        | VLM             | Fine-tuned VLM                 | Diagram-description pairs          | Medium   |
| **Euterpe**       | Music generation          | Audio diffusion | MusicGen / AudioCraft          | Music samples                      | Medium   |
| **Euterpe**       | Audio classification      | Classifier      | AST / BEATs                    | Audio classification data          | Medium   |
| **Aphrodite**     | Content moderation        | Classifier      | NSFW detector                  | Moderation datasets                | Critical |
| **Aphrodite**     | Recommendation            | RecSys          | Custom                         | User interaction data              | High     |
| **Oya**           | Aerial object detection   | YOLO/DETR       | YOLOv8/v9                      | Aerial imagery                     | Critical |
| **Oya**           | Pose estimation           | Pose model      | Sapiens / ViTPose              | Human pose data                    | High     |
| **Oya**           | Path planning             | RL              | Custom PPO                     | Simulation data                    | High     |
| **Airmid**        | Plant identification      | CNN             | EfficientNet / ConvNeXt        | Plant image datasets               | High     |
| **Airmid**        | Compound analysis         | GNN             | SchNet / DimeNet               | Molecular data                     | Medium   |
| **Seshat**        | Interior design           | Diffusion       | SDXL fine-tuned                | Interior design images             | Medium   |
| **Galatea**       | Robot control             | RL + Vision     | Custom                         | Simulation + real data             | High     |
| **Neith**         | NPC inference (on-device) | Quantized LLM   | Llama 3.x 7B-13B               | Game dialogue data                 | High     |
| **Themis**        | Originality detection     | Similarity      | Custom embedding               | Content fingerprints               | High     |
| **Themis**        | Plagiarism detection      | Cross-encoder   | BGE-reranker variant           | Text pairs                         | Medium   |

**Total: 45+ distinct fine-tuned models needed for full sovereignty.**

---

## 3. Gap Analysis by Functional Area

### GAP 1: Training Data Pipeline (CRITICAL)

**Current state:** Phase 47.5.4 defines 20 data pipeline tasks — all unchecked.
No data loading, preprocessing, tokenization, augmentation, filtering, or
curriculum learning infrastructure exists for training.

**What's needed:**

| Task                                                              | Description                                         | Why Critical                      |
| ----------------------------------------------------------------- | --------------------------------------------------- | --------------------------------- |
| Distributed data loading                                          | Stream training data across multi-GPU setups        | Cannot train at scale without it  |
| Data preprocessing & tokenization                                 | Prepare raw text/image/audio for model consumption  | Foundation for all training       |
| Data sharding & streaming                                         | Handle datasets larger than memory                  | Required for large-scale training |
| Packing & dynamic batching                                        | Efficient GPU utilization during training           | 2-5x training speedup             |
| Text augmentation pipeline                                        | Back-translation, paraphrasing, synonym replacement | Improve model robustness          |
| Data filtering & quality scoring                                  | Remove low-quality samples from training data       | Critical for model quality        |
| Deduplication & near-duplicate removal                            | Prevent data contamination                          | Required for clean training       |
| Data mixing & curriculum design                                   | Balance multiple data sources during training       | Required for multi-task models    |
| Data selection & active learning                                  | Prioritize most informative samples                 | Reduce training cost              |
| Format converters (JSONL, Parquet, HuggingFace, Alpaca, ShareGPT) | Load data from any format                           | Interop with community datasets   |

### GAP 2: Production Model Serving (CRITICAL)

**Current state:** Inference uses ONNX Runtime. No integration with
industry-standard LLM serving frameworks optimized for production throughput.

**What's needed:**

| Task                            | Description                                    | Why Critical                       |
| ------------------------------- | ---------------------------------------------- | ---------------------------------- |
| vLLM integration                | PagedAttention, continuous batching for LLMs   | 10-24x throughput vs naive serving |
| TensorRT-LLM integration        | NVIDIA-optimized LLM inference                 | Lowest latency on NVIDIA GPUs      |
| Text Generation Inference (TGI) | HuggingFace's production LLM server            | Battle-tested, flash attention     |
| llama.cpp integration           | CPU-optimized LLM inference                    | Edge/CPU-only deployment           |
| Triton Inference Server         | Multi-model, multi-framework serving           | Production orchestration           |
| SGLang integration              | Structured generation, constrained decoding    | JSON/code output reliability       |
| Ollama integration              | Simple local model deployment                  | Developer experience               |
| Multi-LoRA serving              | Serve multiple LoRA adapters on one base model | Per-customer fine-tuning           |
| Speculative decoding pipeline   | Draft model + verify for faster generation     | 2-3x speedup for large models      |
| KV cache optimization           | Prefix caching, cache sharing across requests  | Reduce TTFT (time to first token)  |

### GAP 3: Dataset Sovereignty (CRITICAL)

**Current state:** Datasets are managed ad-hoc per domain. No centralized
registry, versioning, or governance.

**What's needed:**

| Task                                | Description                                       | Why Critical                                  |
| ----------------------------------- | ------------------------------------------------- | --------------------------------------------- |
| Centralized dataset registry        | Catalog all datasets across domains with metadata | Discoverability, deduplication                |
| Dataset versioning (DVC or similar) | Track dataset changes over time                   | Reproducibility                               |
| Synthetic data generation pipeline  | Generate training data using LLMs/diffusion       | Scale training data without manual annotation |
| Data lineage tracking               | Track which datasets trained which models         | Compliance, debugging                         |
| Dataset quality dashboard           | Monitor quality metrics across all datasets       | Prevent model degradation                     |
| PII detection & scrubbing pipeline  | Remove personal data from training sets           | Legal/compliance requirement                  |
| License compliance checker          | Verify training data licenses                     | Legal risk mitigation                         |
| Cross-domain data sharing protocol  | Share curated data between domains safely         | Reduce duplication, improve models            |
| Dataset card generator              | Auto-generate HuggingFace-style dataset cards     | Documentation, governance                     |
| Annotation schema library           | Standardized schemas for labeling tasks           | Consistent data quality                       |

### GAP 4: Human-in-the-Loop Pipeline (HIGH)

**Current state:** No annotation platform, no feedback collection system, no
RLHF data pipeline connecting user feedback to model training.

**What's needed:**

| Task                                                     | Description                                         | Why Critical                       |
| -------------------------------------------------------- | --------------------------------------------------- | ---------------------------------- |
| Annotation platform (Label Studio integration or custom) | Web UI for data labeling                            | Foundation for supervised training |
| Preference collection UI                                 | A/B comparison interface for RLHF data              | Feed DPO/PPO training              |
| Feedback ingestion pipeline                              | Collect thumbs-up/down from production systems      | Continuous improvement signal      |
| Inter-annotator agreement tracking                       | Measure labeling consistency                        | Data quality assurance             |
| Active learning loop                                     | Route uncertain samples to annotators               | Efficient annotation budget        |
| Annotation task management                               | Queue, assign, review annotation batches            | Scale annotation operations        |
| Quality control for annotations                          | Gold standard checks, annotator calibration         | Prevent label noise                |
| RLHF data pipeline                                       | Convert feedback → preference pairs → training data | Close the feedback loop            |
| Red team dataset curation                                | Organize adversarial prompts for safety testing     | Safety training                    |
| Evaluation set management                                | Maintain held-out eval sets per domain              | Prevent contamination              |

### GAP 5: MLOps & Experiment Management (HIGH)

**Current state:** Basic experiment tracking exists in @isis/model-fine-tuning
and serwaa/avatar-engine. No unified MLOps layer.

**What's needed:**

| Task                                                      | Description                                             | Why Critical                           |
| --------------------------------------------------------- | ------------------------------------------------------- | -------------------------------------- |
| Experiment tracking dashboard (MLflow or W&B self-hosted) | Track all training runs centrally                       | Compare experiments, reproduce results |
| Model lineage graph                                       | Visualize base model → fine-tune → merge → deploy       | Understand model provenance            |
| Automated model evaluation pipeline                       | Run benchmarks on every model checkpoint                | Catch regressions early                |
| Model drift detection                                     | Monitor production model outputs for distribution shift | Prevent silent degradation             |
| Automated retraining triggers                             | Retrain when drift exceeds threshold                    | Maintain model quality                 |
| Model A/B testing framework                               | Route traffic between model versions                    | Safe model rollout                     |
| Shadow deployment                                         | Run new model alongside old, compare outputs            | Risk-free evaluation                   |
| Rollback automation                                       | One-click model version rollback                        | Incident response                      |
| GPU cost tracking per experiment                          | Track $/experiment, $/token for training                | Budget management                      |
| Training job scheduler                                    | Queue and prioritize training jobs across GPU pool      | Resource efficiency                    |

### GAP 6: Edge & On-Device Deployment (HIGH)

**Current state:** RunPod serves models in the cloud. No pipeline for
compiling/deploying models to edge devices, mobile, or browsers.

**What's needed:**

| Task                              | Description                                         | Why Critical                                   |
| --------------------------------- | --------------------------------------------------- | ---------------------------------------------- |
| ONNX export pipeline              | Convert PyTorch/HF models to ONNX                   | Cross-platform deployment                      |
| CoreML conversion                 | ONNX → CoreML for iOS/macOS                         | Apple device deployment                        |
| TFLite conversion                 | ONNX → TensorFlow Lite for Android                  | Android deployment                             |
| TensorRT compilation              | ONNX → TensorRT for NVIDIA edge                     | Jetson/edge GPU deployment                     |
| WebGPU/WebNN inference            | Browser-native ML inference                         | Web app deployment                             |
| WASM inference runtime            | ML models in WebAssembly                            | Portable edge deployment                       |
| Model pruning pipeline            | Structured/unstructured pruning for size reduction  | Smaller models for edge                        |
| Neural Architecture Search (NAS)  | Find optimal architecture for target hardware       | Hardware-specific optimization                 |
| Quantization-aware training (QAT) | Train with quantization in the loop                 | Better quality than post-training quantization |
| Model benchmark suite per target  | Latency/accuracy/memory benchmarks per device class | Deployment decision support                    |

### GAP 7: Continuous Evaluation & Benchmarking (HIGH)

**Current state:** 3D quality continuous evaluation exists. No equivalent for
LLMs, classifiers, embeddings, or other model types.

**What's needed:**

| Task                                           | Description                                            | Why Critical                      |
| ---------------------------------------------- | ------------------------------------------------------ | --------------------------------- |
| LLM evaluation suite (MMLU, HumanEval, etc.)   | Standard benchmarks for LLM quality                    | Compare fine-tuned vs base models |
| RAG evaluation framework (RAGAS)               | Evaluate retrieval + generation quality                | Sophia, Iris, Veritas             |
| Embedding quality benchmarks (MTEB)            | Evaluate embedding models                              | Sophia, Iris knowledge systems    |
| Safety benchmark suite (TruthfulQA, BBQ, etc.) | Evaluate model safety/bias                             | Nous, Kuan Yin                    |
| Domain-specific eval sets                      | Per-domain held-out evaluation data                    | Each domain's model quality       |
| Automated regression testing                   | Compare model versions on fixed eval sets              | Prevent quality regressions       |
| Human evaluation framework                     | Structured human ratings on model outputs              | Ground truth quality signal       |
| Leaderboard dashboard                          | Visual comparison of model versions                    | Track improvement over time       |
| Cost-quality Pareto analysis                   | Plot quality vs cost for model selection               | Optimal model selection           |
| Cross-lingual evaluation                       | Evaluate models across languages (especially Ghanaian) | Veritas, global deployment        |

### GAP 8: Specialized Training Pipelines per Domain (MEDIUM-HIGH)

**Current state:** Generic fine-tuning framework exists. No domain-specific
training pipelines that automate the full cycle from data → training →
evaluation → deployment for each domain's unique needs.

**What's needed:**

| Domain   | Pipeline                                | Description                                                       |
| -------- | --------------------------------------- | ----------------------------------------------------------------- |
| Iris     | Conversation model fine-tuning pipeline | Data collection → format → LoRA train → eval → deploy             |
| Iris     | Embedding model fine-tuning pipeline    | Triplet mining → contrastive training → MTEB eval → deploy        |
| Veritas  | NER model training pipeline             | Annotate Ghanaian entities → train spaCy/BERT → eval → deploy     |
| Veritas  | Bias detector training pipeline         | Collect political coverage → annotate → train classifier → deploy |
| Veritas  | Ghana language TTS pipeline             | Collect Twi/Ewe/Ga audio → train XTTS → eval → deploy             |
| Psyche   | Voice cloning pipeline                  | Collect voice samples → fine-tune XTTS → quality eval → deploy    |
| Psyche   | Emotion model training pipeline         | Collect facial data → train classifier → eval → deploy            |
| Kuan Yin | Harm detector training pipeline         | Collect moderation data → train → red-team eval → deploy          |
| Nous     | Safety classifier pipeline              | Red-team generation → train guardrails → eval → deploy            |
| Isis     | LoRA training pipeline (images)         | Collect style data → train LoRA → quality gate → registry         |
| Isis     | LoRA training pipeline (video)          | Collect motion data → train video LoRA → eval → deploy            |
| Sophia   | Reranker training pipeline              | Mine hard negatives → train cross-encoder → eval → deploy         |
| Oya      | Aerial vision training pipeline         | Collect aerial data → fine-tune YOLO → quantize → deploy          |
| Airmid   | Plant ID training pipeline              | Collect plant images → fine-tune CNN → quantize → deploy          |
| Euterpe  | Music model training pipeline           | Collect music data → fine-tune AudioCraft → eval → deploy         |
| Minerva  | Lecture LLM training pipeline           | Collect educational data → fine-tune LLM → eval → deploy          |
| Hathor   | NPC dialogue training pipeline          | Generate character data → fine-tune LLM → eval → deploy           |

### GAP 9: Model Governance & Compliance (MEDIUM)

**Current state:** Basic model registry exists. No governance framework for
model lifecycle management, compliance, or audit trails.

**What's needed:**

| Task                              | Description                                                             | Why Critical             |
| --------------------------------- | ----------------------------------------------------------------------- | ------------------------ |
| Model card generation             | Auto-generate model documentation (training data, metrics, limitations) | Compliance, transparency |
| Model approval workflow           | Review/approve models before production deployment                      | Quality gate             |
| License compliance tracking       | Track licenses of base models and training data                         | Legal risk               |
| Bias audit pipeline               | Automated bias testing across protected attributes                      | Fairness compliance      |
| Model deprecation workflow        | Graceful model retirement with migration path                           | Lifecycle management     |
| Audit trail for model changes     | Who trained what, when, with what data                                  | Compliance, debugging    |
| Data retention policies per model | How long to keep training data and checkpoints                          | Storage cost, compliance |
| Model access control              | Role-based access to model artifacts                                    | Security                 |

### GAP 10: GPU Infrastructure Scaling (MEDIUM)

**Current state:** RunPod for serverless GPU, some Terraform for AWS. No
multi-cloud GPU orchestration or training cluster management.

**What's needed:**

| Task                                 | Description                                        | Why Critical                    |
| ------------------------------------ | -------------------------------------------------- | ------------------------------- |
| Multi-cloud GPU orchestration        | Use RunPod + AWS + GCP GPUs as single pool         | Cost optimization, availability |
| Training cluster provisioner         | Spin up/down multi-GPU training clusters on demand | Large-scale training            |
| Spot/preemptible instance management | Use cheap spot instances with checkpointing        | 60-90% cost reduction           |
| GPU memory profiler                  | Profile VRAM usage per model/batch size            | Right-size GPU allocation       |
| Cost estimator for training jobs     | Estimate $/run before launching                    | Budget planning                 |
| Queue-based training scheduler       | Priority queue for training jobs across teams      | Resource fairness               |
| Distributed training launcher        | One-command multi-node training setup              | Developer experience            |
| GPU health monitoring                | Track GPU temperature, utilization, errors         | Prevent training failures       |
| Auto-scaling for inference           | Scale GPU workers based on request queue depth     | Handle traffic spikes           |
| Reserved instance optimization       | Analyze usage patterns, recommend reservations     | Cost optimization               |

---

## 4. Recommended New Tasks

### Phase: ML Sovereignty Foundation (New Phase Recommended)

This should be a new dedicated phase in TODOS.md focused exclusively on the
cross-cutting ML infrastructure that enables sovereignty across all domains.

#### Section 1: Centralized Dataset Management

```
- [ ] Design dataset registry schema (name, version, domain, format, size, license, quality score, lineage)
- [ ] Implement dataset registry service with PostgreSQL storage
- [ ] Build dataset versioning system (content-addressable storage, delta tracking)
- [ ] Create dataset upload/download CLI tool
- [ ] Implement dataset format converters (JSONL, Parquet, HuggingFace, Alpaca, ShareGPT, OpenAI)
- [ ] Build dataset quality scoring engine (completeness, diversity, toxicity, duplication)
- [ ] Implement deduplication pipeline (exact + near-duplicate with MinHash/SimHash)
- [ ] Build PII detection and scrubbing pipeline for training data
- [ ] Create license compliance checker for training datasets
- [ ] Implement dataset card auto-generation (HuggingFace format)
- [ ] Build data lineage tracker (which datasets trained which models)
- [ ] Create cross-domain data sharing protocol with access controls
- [ ] Implement dataset split management (train/val/test with contamination prevention)
- [ ] Build streaming dataset loader for datasets larger than memory
- [ ] Create annotation schema library with standard schemas per task type
- [ ] Implement dataset statistics dashboard (size, distribution, quality over time)
- [ ] Build dataset diff tool (compare versions, detect distribution shifts)
- [ ] Create synthetic data generation orchestrator (LLM-based, diffusion-based)
- [ ] Implement data mixing recipe system (specify ratios, curricula)
- [ ] Build dataset export pipeline for external sharing (anonymized, licensed)
```

#### Section 2: Synthetic Data Generation

```
- [ ] Design synthetic data generation framework (pluggable generators per modality)
- [ ] Implement LLM-based text data generator (instruction following, conversation, QA)
- [ ] Build self-instruct pipeline (seed instructions → LLM generation → quality filter)
- [ ] Implement Evol-Instruct pipeline (complexity evolution for instruction data)
- [ ] Create persona-based dialogue generator (varied styles, knowledge levels)
- [ ] Build synthetic preference pair generator (for DPO/RLHF training)
- [ ] Implement synthetic NER data generator (domain-specific entity annotation)
- [ ] Create synthetic code data generator (problems, solutions, explanations)
- [ ] Build diffusion-based image data generator (with controlled variation)
- [ ] Implement TTS-based audio data generator (multi-speaker, multi-language)
- [ ] Create 3D scene synthetic data generator (for 3D model training)
- [ ] Build adversarial data generator (red-team prompts, jailbreak attempts)
- [ ] Implement multi-lingual synthetic data pipeline (Twi, Ewe, Ga, Hausa, Dagbani)
- [ ] Create quality filter for synthetic data (decontamination, deduplication, scoring)
- [ ] Build human-AI collaborative data generation workflow
```

#### Section 3: Annotation & Human Feedback Platform

```
- [ ] Design annotation platform architecture (task management, reviewer UI, quality control)
- [ ] Integrate Label Studio as self-hosted annotation backend
- [ ] Build preference comparison UI (side-by-side A/B for RLHF data collection)
- [ ] Implement thumbs-up/down feedback ingestion from production endpoints
- [ ] Create annotation task queue with priority and assignment management
- [ ] Build inter-annotator agreement calculator (Cohen's kappa, Fleiss' kappa)
- [ ] Implement gold standard quality checks (inject known-answer questions)
- [ ] Create annotator calibration workflow (training + certification)
- [ ] Build active learning loop (route uncertain samples to annotators)
- [ ] Implement RLHF data pipeline (feedback → preference pairs → training format)
- [ ] Create red team dataset curation tool (organize adversarial prompts by category)
- [ ] Build evaluation set management (held-out sets with contamination monitoring)
- [ ] Implement annotation export to multiple training formats
- [ ] Create annotation analytics dashboard (throughput, quality, cost per annotation)
- [ ] Build multi-annotator consensus resolution workflow
```

#### Section 4: Production Model Serving

```
- [ ] Integrate vLLM as primary LLM serving engine (PagedAttention, continuous batching)
- [ ] Build vLLM Docker image with multi-LoRA support for RunPod deployment
- [ ] Integrate TensorRT-LLM for NVIDIA-optimized inference
- [ ] Build TGI (Text Generation Inference) adapter for HuggingFace models
- [ ] Integrate llama.cpp for CPU-only and edge LLM serving
- [ ] Build SGLang integration for structured/constrained generation
- [ ] Implement multi-LoRA serving (serve N fine-tuned adapters on one base model)
- [ ] Create model routing layer (route requests to optimal backend by model/hardware)
- [ ] Build speculative decoding pipeline (draft model selection, verification)
- [ ] Implement KV cache management (prefix caching, cache sharing, eviction policies)
- [ ] Create batching optimizer (dynamic batching by sequence length)
- [ ] Build model warmup and preloading system
- [ ] Implement graceful model swapping (zero-downtime model updates)
- [ ] Create inference cost calculator ($/token by model, backend, GPU)
- [ ] Build latency monitoring per model endpoint with SLA tracking
- [ ] Implement Ollama-compatible API for local development
- [ ] Create model serving benchmark suite (throughput, latency, memory)
- [ ] Build auto-scaling controller for GPU inference workers
- [ ] Implement request priority queuing (interactive > batch > background)
- [ ] Create multi-tenant model isolation (per-customer fine-tuned models)
```

#### Section 5: MLOps & Experiment Management

```
- [ ] Deploy self-hosted MLflow as experiment tracking backend
- [ ] Build experiment tracking SDK (log metrics, params, artifacts from training code)
- [ ] Create model lineage graph (visualize base → fine-tune → merge → deploy chain)
- [ ] Implement model comparison dashboard (side-by-side metrics, outputs)
- [ ] Build automated model evaluation pipeline (run benchmarks on every checkpoint)
- [ ] Create model drift detection service (monitor output distribution over time)
- [ ] Implement automated retraining trigger (retrain when drift exceeds threshold)
- [ ] Build model A/B testing framework (split traffic, collect metrics, auto-promote)
- [ ] Create shadow deployment system (run new model alongside production, compare)
- [ ] Implement one-click model rollback (revert to previous version)
- [ ] Build GPU cost tracking per experiment ($/run, $/token, $/epoch)
- [ ] Create training job scheduler with priority queue
- [ ] Implement training job dashboard (running, queued, completed, failed)
- [ ] Build model artifact storage with deduplication (shared base weights)
- [ ] Create model deployment pipeline (registry → testing → staging → production)
- [ ] Implement canary deployment for model updates
- [ ] Build model performance alerting (latency, error rate, quality degradation)
- [ ] Create reproducibility system (pin data version + code version + config)
- [ ] Implement experiment tagging and search
- [ ] Build model governance approval workflow (review before production)
```

#### Section 6: Edge & On-Device Deployment

```
- [ ] Build ONNX export pipeline (PyTorch/HF → ONNX with optimization)
- [ ] Implement CoreML conversion pipeline (ONNX → CoreML for iOS/macOS)
- [ ] Build TFLite conversion pipeline (ONNX → TensorFlow Lite for Android)
- [ ] Create TensorRT compilation pipeline (ONNX → TensorRT for NVIDIA edge)
- [ ] Implement WebGPU inference runtime for browser deployment
- [ ] Build WebNN inference adapter for cross-browser ML
- [ ] Create WASM inference runtime (ONNX Runtime Web or custom)
- [ ] Implement model pruning pipeline (structured + unstructured)
- [ ] Build quantization-aware training (QAT) pipeline
- [ ] Create post-training quantization pipeline (INT8, INT4, FP8)
- [ ] Implement knowledge distillation pipeline (teacher → student automation)
- [ ] Build target device profiler (benchmark model on target hardware)
- [ ] Create model size/latency/accuracy Pareto optimizer
- [ ] Implement on-device model update mechanism (differential model updates)
- [ ] Build edge model deployment CI/CD (compile → test → package → distribute)
- [ ] Create device compatibility matrix (which models run on which devices)
- [ ] Implement fallback chain (try on-device → edge server → cloud)
- [ ] Build battery/thermal-aware inference scheduler for mobile
- [ ] Create edge model monitoring (collect on-device metrics, aggregate centrally)
- [ ] Implement federated learning orchestrator for privacy-preserving training
```

#### Section 7: Continuous Evaluation & Benchmarking

```
- [ ] Design unified benchmark framework (pluggable evaluators, standard metrics)
- [ ] Implement LLM benchmark suite (MMLU, HumanEval, MT-Bench, Arena-Hard)
- [ ] Build RAG evaluation framework (faithfulness, relevance, context precision)
- [ ] Create embedding benchmark suite (MTEB subset relevant to Oshun domains)
- [ ] Implement safety benchmark suite (TruthfulQA, BBQ, RealToxicityPrompts)
- [ ] Build multi-lingual evaluation suite (focus on Ghanaian languages)
- [ ] Create domain-specific eval sets per domain (Iris, Sophia, Veritas, etc.)
- [ ] Implement automated regression testing (compare model versions on fixed evals)
- [ ] Build human evaluation framework (structured rating tasks, sampling)
- [ ] Create leaderboard dashboard (track model quality over time per domain)
- [ ] Implement cost-quality Pareto analysis tool ($/quality for model selection)
- [ ] Build cross-model comparison tool (compare fine-tuned vs base vs competitors)
- [ ] Create eval set contamination detector (check if eval data leaked into training)
- [ ] Implement benchmark scheduling (run daily/weekly on production models)
- [ ] Build evaluation report generator (PDF/HTML reports per model version)
- [ ] Create custom benchmark authoring tool (define domain-specific test cases)
- [ ] Implement adversarial evaluation (automatically generated edge cases)
- [ ] Build latency-quality tradeoff analyzer
- [ ] Create evaluation set versioning (track eval set changes, normalize scores)
- [ ] Implement multi-judge evaluation (aggregate LLM-as-judge + human ratings)
```

#### Section 8: Domain-Specific Training Pipelines

```
- [ ] Build Iris conversational model fine-tuning pipeline (data→LoRA→eval→deploy)
- [ ] Build Iris embedding model fine-tuning pipeline (triplet mining→train→MTEB→deploy)
- [ ] Build Veritas NER model training pipeline (annotate→train spaCy/BERT→eval→deploy)
- [ ] Build Veritas bias detector training pipeline (annotate→train→eval→deploy)
- [ ] Build Veritas Ghana language TTS pipeline (collect audio→train XTTS→eval→deploy)
- [ ] Build Psyche voice cloning pipeline (samples→fine-tune XTTS→quality→deploy)
- [ ] Build Psyche emotion model training pipeline (facial data→train→eval→deploy)
- [ ] Build Kuan Yin harm detector training pipeline (moderation data→train→red-team→deploy)
- [ ] Build Nous safety classifier training pipeline (red-team→train guardrails→eval→deploy)
- [ ] Build Isis image LoRA training pipeline (style data→train LoRA→quality gate→registry)
- [ ] Build Isis video LoRA training pipeline (motion data→train→eval→deploy)
- [ ] Build Sophia reranker training pipeline (hard negatives→train cross-encoder→eval→deploy)
- [ ] Build Oya aerial vision training pipeline (aerial data→fine-tune YOLO→quantize→deploy)
- [ ] Build Airmid plant ID training pipeline (images→fine-tune CNN→quantize→deploy)
- [ ] Build Euterpe music model training pipeline (music data→fine-tune→eval→deploy)
- [ ] Build Minerva lecture LLM training pipeline (educational data→fine-tune→eval→deploy)
- [ ] Build Hathor NPC dialogue training pipeline (character data→fine-tune LLM→eval→deploy)
- [ ] Build Aphrodite content moderation training pipeline (data→train NSFW detector→eval→deploy)
- [ ] Build Themis originality model training pipeline (fingerprints→train→eval→deploy)
- [ ] Build cross-domain transfer learning pipeline (share learned representations)
```

#### Section 9: GPU Infrastructure & Training Clusters

```
- [ ] Design multi-cloud GPU orchestration layer (RunPod + AWS + GCP as unified pool)
- [ ] Build training cluster provisioner (spin up multi-GPU clusters on demand)
- [ ] Implement spot/preemptible instance manager with automatic checkpointing
- [ ] Create GPU memory profiler for right-sizing GPU allocation per workload
- [ ] Build training cost estimator (predict $/run before launching)
- [ ] Implement queue-based training scheduler with priority and fairness
- [ ] Create distributed training launcher (one-command multi-node setup)
- [ ] Build GPU health monitoring dashboard (temperature, utilization, errors, ECC)
- [ ] Implement auto-scaling for inference (scale workers by queue depth)
- [ ] Create reserved instance optimization analyzer
- [ ] Build training checkpoint management for spot instance recovery
- [ ] Implement NCCL/Gloo configuration optimizer for multi-node training
- [ ] Create GPU utilization alerting (underutilized or overloaded warnings)
- [ ] Build training cost dashboard (per-domain, per-model, per-experiment)
- [ ] Implement GPU sharing for small models (MIG, MPS, time-sharing)
```

#### Section 10: Model Governance & Compliance

```
- [ ] Build model card auto-generator (training data, metrics, limitations, biases)
- [ ] Create model approval workflow (review gate before production deployment)
- [ ] Implement license compliance tracker for base models and training data
- [ ] Build automated bias audit pipeline (test across protected attributes)
- [ ] Create model deprecation workflow (migration path, sunset timeline)
- [ ] Implement audit trail for all model operations (train, eval, deploy, rollback)
- [ ] Build data retention policy engine (auto-expire old checkpoints and data)
- [ ] Create model access control (RBAC for model artifacts and endpoints)
- [ ] Implement model risk assessment (classify models by risk tier)
- [ ] Build compliance reporting dashboard (summarize governance state)
```

---

## 5. Priority Matrix

### P0 — Blocks All Sovereignty (Must Complete First)

| Gap                                   | Estimated Tasks             | Rationale                                      |
| ------------------------------------- | --------------------------- | ---------------------------------------------- |
| Training Data Pipeline (Phase 47.5.4) | 20 tasks (exist, unchecked) | Cannot train ANY model without data loading    |
| Production Model Serving (vLLM/TGI)   | 20 new tasks                | Cannot serve fine-tuned LLMs efficiently       |
| Centralized Dataset Registry          | 20 new tasks                | Cannot manage training data at scale           |
| Nous Safety (Phase 47.6)              | 60 tasks (exist, unchecked) | Cannot deploy models without safety guardrails |

### P1 — Enables Continuous Improvement

| Gap                                  | Estimated Tasks | Rationale                                        |
| ------------------------------------ | --------------- | ------------------------------------------------ |
| MLOps & Experiment Management        | 20 new tasks    | Cannot track/compare/reproduce experiments       |
| Continuous Evaluation & Benchmarking | 20 new tasks    | Cannot measure model quality systematically      |
| Human-in-the-Loop Pipeline           | 15 new tasks    | Cannot collect feedback for RLHF/improvement     |
| Synthetic Data Generation            | 15 new tasks    | Cannot scale training data without manual effort |

### P2 — Enables Full Domain Coverage

| Gap                                | Estimated Tasks               | Rationale                                     |
| ---------------------------------- | ----------------------------- | --------------------------------------------- |
| Domain-Specific Training Pipelines | 20 new tasks                  | Each domain needs its own end-to-end pipeline |
| Nous Advanced Models (47.7-47.16)  | 200+ tasks (exist, unchecked) | Video, image control, speech, animation       |
| Neith AI Runtime (Phase 43.1.6)    | 15 tasks (exist, unchecked)   | Game engine ML inference                      |
| LLM-Powered NPCs (Phase 53.2)      | 45 tasks (exist, unchecked)   | On-device game AI                             |

### P3 — Enables Edge & Scale

| Gap                           | Estimated Tasks | Rationale                                      |
| ----------------------------- | --------------- | ---------------------------------------------- |
| Edge & On-Device Deployment   | 20 new tasks    | Mobile, browser, embedded deployment           |
| GPU Infrastructure Scaling    | 15 new tasks    | Multi-cloud, spot instances, cost optimization |
| Model Governance & Compliance | 10 new tasks    | Audit trails, bias testing, compliance         |

---

## Summary Statistics

| Category                        | Existing Unchecked Tasks | New Tasks Needed | Total    |
| ------------------------------- | ------------------------ | ---------------- | -------- |
| Training Data Pipeline (47.5.4) | 20                       | 0                | 20       |
| Safety Framework (47.6)         | 60                       | 0                | 60       |
| Advanced Models (47.7-47.16)    | ~200                     | 0                | ~200     |
| Neith AI Runtime (43, 53)       | ~260                     | 0                | ~260     |
| Centralized Dataset Management  | 0                        | 20               | 20       |
| Synthetic Data Generation       | 0                        | 15               | 15       |
| Annotation & Human Feedback     | 0                        | 15               | 15       |
| Production Model Serving        | 0                        | 20               | 20       |
| MLOps & Experiment Management   | 0                        | 20               | 20       |
| Edge & On-Device Deployment     | 0                        | 20               | 20       |
| Continuous Evaluation           | 0                        | 20               | 20       |
| Domain-Specific Pipelines       | 0                        | 20               | 20       |
| GPU Infrastructure              | 0                        | 15               | 15       |
| Model Governance                | 0                        | 10               | 10       |
| **TOTAL**                       | **~540**                 | **~175**         | **~715** |

**~540 existing unchecked tasks** need to be completed (mostly in Phases 47,
43, 53) and **~175 new tasks** need to be added to achieve full ML sovereignty.

The most critical new phase to add is the **ML Sovereignty Foundation** covering
dataset management, model serving, MLOps, evaluation, and domain-specific
pipelines — none of which are currently tracked in any TODOS.md.

---

## Appendix: Models to Self-Host (Open Source Recommendations)

### LLMs

| Use Case               | Recommended Base Model         | Size   | License        |
| ---------------------- | ------------------------------ | ------ | -------------- |
| General chat/reasoning | Llama 3.1 / 3.2                | 8B-70B | Meta Community |
| Code generation        | DeepSeek-Coder V2 / StarCoder2 | 7B-33B | Apache 2.0     |
| Long context           | Mistral Nemo / Qwen 2.5        | 7B-72B | Apache 2.0     |
| Multilingual           | Aya 23 / BLOOM                 | 8B-35B | Apache 2.0     |
| Function calling       | Hermes 3 / Functionary         | 7B-70B | Apache 2.0     |
| Small/fast             | Phi-3 / Gemma 2                | 2B-9B  | MIT/Apache     |

### Embedding Models

| Use Case                | Model                    | Dimensions | License    |
| ----------------------- | ------------------------ | ---------- | ---------- |
| Text embedding          | BGE-large / E5-large     | 1024       | MIT        |
| Multilingual embedding  | BGE-M3 / multilingual-e5 | 1024       | MIT        |
| Code embedding          | CodeSage / StarEncoder   | 768-1024   | Apache 2.0 |
| Cross-encoder reranking | BGE-reranker-large       | -          | MIT        |

### Vision Models

| Use Case             | Model                         | License          |
| -------------------- | ----------------------------- | ---------------- |
| Image classification | ConvNeXt V2 / EfficientNet V2 | Apache 2.0       |
| Object detection     | YOLOv9 / DETR / RT-DETR       | AGPL/Apache      |
| Image generation     | SDXL / Flux.1                 | Open RAIL/Apache |
| Video generation     | AnimateDiff / SVD             | Apache/Custom    |
| OCR                  | PaddleOCR / TrOCR             | Apache 2.0       |

### Audio Models

| Use Case             | Model                 | License |
| -------------------- | --------------------- | ------- |
| Speech-to-text       | Whisper large-v3      | MIT     |
| Text-to-speech       | XTTS v2 / Coqui       | MPL 2.0 |
| Music generation     | MusicGen / AudioCraft | MIT     |
| Audio classification | BEATs / AST           | MIT     |
| Voice cloning        | RVC / SoVITS          | MIT     |

### 3D Models

| Use Case           | Model                | License           |
| ------------------ | -------------------- | ----------------- |
| Text-to-3D         | TRELLIS.2            | MIT               |
| Image-to-3D        | Hunyuan3D 2.1 / SF3D | Tencent/Stability |
| Gaussian splatting | 3DGS / InstantSplat  | Various           |
| Mesh generation    | MeshAnything V2      | Apache 2.0        |

### Safety Models

| Use Case           | Model                     | License    |
| ------------------ | ------------------------- | ---------- |
| Toxicity detection | Detoxify / Perspective    | Apache 2.0 |
| NSFW detection     | NSFW detector / OpenNSFW2 | MIT        |
| Deepfake detection | FaceForensics++ models    | Academic   |
| PII detection      | Presidio / GLiNER         | MIT        |
