Date: 2026-04-11 Scope: Complete audit of ML/AI infrastructure across the Oshun monorepo Goal: Identify all missing tasks required for full model sovereignty — the ability to self-host, fine-tune, train, evaluate, and deploy custom open-source models for every ML use case in the system.
Executive Summary#
The Oshun monorepo has substantial ML infrastructure already built — a sovereign inference engine (@nous/core), comprehensive fine-tuning framework (@nous/training, @isis/model-fine-tuning), GPU serving via RunPod, and domain-specific ML services across 15+ domains. However, full ML sovereignty is not yet achievable because critical gaps exist in:
- Data Pipeline Infrastructure — No training data loading, preprocessing, or curriculum systems
- Model Serving Optimization — No vLLM/TGI/TensorRT-LLM integration for production LLM serving
- Continuous Training Pipelines — No automated retraining, drift detection, or model refresh workflows
- Evaluation & Benchmarking — No unified cross-domain evaluation framework
- Dataset Sovereignty — No centralized dataset registry, versioning, or synthetic data generation at scale
- Edge/On-Device Deployment — No model compilation pipeline for mobile, browser, or embedded targets
- Human-in-the-Loop — No annotation platform, feedback collection, or RLHF data pipeline
- MLOps Maturity — No experiment tracking dashboard, model lineage graph, or automated rollback
The report below catalogs every gap organized by functional area, with specific task recommendations for each.
Table of Contents#
- Current ML Infrastructure Inventory
- Domain-by-Domain ML Requirements
- Gap Analysis by Functional Area
- Recommended New Tasks
- Priority Matrix
1. Current ML Infrastructure Inventory#
1.1 What's Built and Working#
| Component | Location | Status | Maturity |
|---|---|---|---|
| Sovereign Inference Engine (ONNX, CUDA, Metal, WebGPU) | @nous/core | Complete (279/279 tasks) | Production |
| LLM Serving (KV cache, paged attention, parallelism) | @nous/core | Complete | Production |
| Fine-tuning Framework (LoRA, QLoRA, DoRA, IA3, PEFT) | @nous/training | Complete (47.5.1-47.5.2) | Production |
| RLHF Pipeline (PPO, DPO, IPO, KTO, ORPO, Constitutional AI) | @nous/training | Complete (47.5.3) | Production |
| Model Fine-tuning Toolkit (optimizer, curator, tracker, merger) | @isis/model-fine-tuning | Complete (1038 tests) | Production |
| RunPod Serverless GPU Workers (SD, Flux, ComfyUI, inference) | docker/runpod/ | Complete (Phase 19) | Production |
| 3D Model Inference (TRELLIS.2, Hunyuan3D, SF3D, SPAR3D) | @isis/3d-inference-local | Complete (Phase 71) | Production |
| Embedding Service (sentence-transformers, Qdrant) | docker/psyche/embedding | Complete | Production |
| LLM Provider Adapters (Anthropic, OpenAI, Google, local) | @isis/llm-providers, @iris/conversation-providers-* | Complete | Production |
| S3 Model Storage (versioned, tiered, cross-region) | infra/terraform/s3/models.tf | Complete | Production |
| CI/CD for GPU Images | .github/workflows/deploy-runpod.yml | Complete | Production |
| 3D Quality Continuous Evaluation | .github/workflows/3d-quality-continuous-evaluation.yml | Complete | Production |
| Avatar Training (3DGS, blendshapes) | serwaa/avatar-engine/training/ | Complete (Python) | Production |
| VLM Fine-tuning (educational) | minerva/vlm/educational_vlm_finetuning/ | Complete (Python) | Production |
| Neural Prover Fine-tuning | minerva/verification/autoformalization/ | Complete (Python) | Production |
| Conversation Fine-tuning | @iris/conversation-finetuning | Complete | Production |
| Model Registry | Phase 19.6, @metis/model-registry | Complete | Production |
| LoRA Training Pipeline (image + video) | Phase 17.3.2 | Complete | Production |
| Knowledge Distillation | @nous/training | Complete | Production |
| Distributed Training (ZeRO, tensor/pipeline parallelism) | @nous/training | Complete | Production |
| Safety & Content Moderation | @nous/safety | Files exist (42 files) | Needs verification |
1.2 What's Partially Built#
| Component | Location | Status | Gap |
|---|---|---|---|
| Data Pipeline | @nous/training 47.5.4 | 0/20 tasks | No data loading, preprocessing, augmentation |
| Safety Guardrails | @nous/safety 47.6 | 0/60 tasks marked | Files exist but tasks unchecked |
| Vision Models | @nous/vision 47.3 | 0 tasks marked | 81 files exist but tasks unchecked |
| Audio Models | @nous/audio 47.4 | 0 tasks marked | 81 files exist but tasks unchecked |
| Neith AI Runtime | Phase 43 | 0/200+ tasks | Game engine ML inference entirely unstarted |
| LLM-Powered NPCs | Phase 53.2 | 0/45 tasks | Local LLM inference for games unstarted |
1.3 What's Entirely Missing#
| Component | Impact | Affected Domains |
|---|---|---|
| vLLM / TGI / TensorRT-LLM integration | Cannot serve LLMs at scale efficiently | All LLM-consuming domains |
| Unified dataset registry & versioning | Cannot manage training data across domains | All training workflows |
| Synthetic data generation pipeline | Cannot create domain-specific training data | All fine-tuning workflows |
| Annotation/labeling platform | Cannot collect human feedback at scale | RLHF, safety, quality |
| MLOps dashboard (experiment tracking) | Cannot compare experiments across teams | All training workflows |
| Model drift detection & monitoring | Cannot detect model degradation in production | All deployed models |
| Edge model compilation (ONNX→CoreML/TFLite) | Cannot deploy to mobile/browser | Oya, Psyche, Iris |
| Automated retraining pipeline | Cannot refresh models on new data | All domains |
| Model A/B testing infrastructure | Cannot safely roll out model changes | All serving endpoints |
| Federated learning orchestration | Cannot train across privacy boundaries | Kuan Yin, Airmid, Psyche |
| Hardware-aware NAS (Neural Architecture Search) | Cannot optimize architectures for target hardware | Edge deployment |
| Model compression beyond quantization | Cannot optimize model size (pruning, distillation pipelines) | Edge, mobile |
| Training cost estimation & budgeting | Cannot plan GPU spend for training runs | All training |
| Multi-tenant model serving | Cannot serve fine-tuned models per customer | Iris, Yemaya, Psyche |
| Prompt optimization pipeline | Cannot systematically optimize prompts | All LLM-consuming domains |
2. Domain-by-Domain ML Requirements#
2.1 Domains Requiring Custom Fine-Tuned Models#
Each domain below needs at least one fine-tuned model for sovereignty. The "Model Type" column indicates what must be trained/fine-tuned locally rather than relying on third-party APIs.
| Domain | Use Case | Model Type | Base Model | Training Data Source | Priority |
|---|---|---|---|---|---|
| Iris | Conversational AI | LLM (chat) | Llama 3.x / Mistral / DeepSeek | Conversation logs, preference data | Critical |
| Iris | Intent classification | Classifier | BERT/DeBERTa | Annotated intent datasets | High |
| Iris | Embedding model | Embedding | BGE / E5 / GTE | Domain-specific text pairs | Critical |
| Psyche | Avatar lip-sync | Audio2Face | Wav2Lip / SadTalker | Video-audio pairs | High |
| Psyche | Voice cloning | TTS | XTTS v2 / Coqui | Voice samples | High |
| Psyche | Emotion recognition | Classifier | FER / AffectNet | Facial expression data | High |
| Psyche | Gaze tracking | Regression | GazeML | Eye tracking data | Medium |
| Isis | Image generation | Diffusion | SDXL / Flux | Domain-specific images | Critical |
| Isis | Video generation | Video diffusion | AnimateDiff / SVD | Video clips | High |
| Isis | 3D generation | 3D diffusion | TRELLIS.2 / Hunyuan3D | 3D scan data | High |
| Isis | Texture generation | Diffusion | SD-inpainting | Texture/material images | Medium |
| Isis | Audio generation | Audio diffusion | AudioLDM / MusicGen | Audio samples | Medium |
| Sophia | Document embeddings | Embedding | BGE / E5 | Document corpus | Critical |
| Sophia | Entity extraction | NER | spaCy / BERT | Annotated documents | High |
| Sophia | Reranking | Cross-encoder | BGE-reranker | Query-document pairs | High |
| Sophia | Translation | Seq2Seq | NLLB / MarianMT | Parallel corpora | Medium |
| Hathor | NPC dialogue | LLM | Llama 3.x fine-tuned | Character dialogue scripts | High |
| Hathor | World simulation | Classifier/LLM | Domain-specific | Simulation data | Medium |
| Veritas/Yaa | NER (Ghanaian entities) | NER | spaCy + BERT | Ghanaian news corpus | Critical |
| Veritas/Yaa | Topic classification | Classifier | BERT / DeBERTa | News article corpus | High |
| Veritas/Yaa | Bias detection | Classifier | Custom | Annotated political coverage | High |
| Veritas/Yaa | Fact verification | LLM | Fine-tuned LLM | Claim-evidence pairs | High |
| Veritas/Yaa | Ghana language TTS | TTS | Coqui / XTTS | Twi/Ewe/Ga audio | Critical |
| Veritas/Yaa | Sentiment analysis | Classifier | BERT multilingual | Ghanaian social media | Medium |
| Kuan Yin | Harm detection | Classifier | BERT / DeBERTa | Moderation datasets | Critical |
| Kuan Yin | Intent analysis | Classifier | Custom | Annotated harm data | High |
| Kuan Yin | Deepfake detection | CNN | EfficientNet | Deepfake datasets | High |
| Nous | Safety guardrails | Classifier | Custom | Red-team data | Critical |
| Nous | Jailbreak detection | Classifier | Custom | Jailbreak attempts | Critical |
| Minerva/Metis | Lecture generation | LLM | Fine-tuned LLM | Educational content | High |
| Minerva/Metis | Assessment generation | LLM | Fine-tuned LLM | Q&A pairs | Medium |
| Minerva/Metis | Diagram generation | VLM | Fine-tuned VLM | Diagram-description pairs | Medium |
| Euterpe | Music generation | Audio diffusion | MusicGen / AudioCraft | Music samples | Medium |
| Euterpe | Audio classification | Classifier | AST / BEATs | Audio classification data | Medium |
| Aphrodite | Content moderation | Classifier | NSFW detector | Moderation datasets | Critical |
| Aphrodite | Recommendation | RecSys | Custom | User interaction data | High |
| Oya | Aerial object detection | YOLO/DETR | YOLOv8/v9 | Aerial imagery | Critical |
| Oya | Pose estimation | Pose model | Sapiens / ViTPose | Human pose data | High |
| Oya | Path planning | RL | Custom PPO | Simulation data | High |
| Airmid | Plant identification | CNN | EfficientNet / ConvNeXt | Plant image datasets | High |
| Airmid | Compound analysis | GNN | SchNet / DimeNet | Molecular data | Medium |
| Seshat | Interior design | Diffusion | SDXL fine-tuned | Interior design images | Medium |
| Galatea | Robot control | RL + Vision | Custom | Simulation + real data | High |
| Neith | NPC inference (on-device) | Quantized LLM | Llama 3.x 7B-13B | Game dialogue data | High |
| Themis | Originality detection | Similarity | Custom embedding | Content fingerprints | High |
| Themis | Plagiarism detection | Cross-encoder | BGE-reranker variant | Text pairs | Medium |
Total: 45+ distinct fine-tuned models needed for full sovereignty.
3. Gap Analysis by Functional Area#
GAP 1: Training Data Pipeline (CRITICAL)#
Current state: Phase 47.5.4 defines 20 data pipeline tasks — all unchecked. No data loading, preprocessing, tokenization, augmentation, filtering, or curriculum learning infrastructure exists for training.
What's needed:
| Task | Description | Why Critical |
|---|---|---|
| Distributed data loading | Stream training data across multi-GPU setups | Cannot train at scale without it |
| Data preprocessing & tokenization | Prepare raw text/image/audio for model consumption | Foundation for all training |
| Data sharding & streaming | Handle datasets larger than memory | Required for large-scale training |
| Packing & dynamic batching | Efficient GPU utilization during training | 2-5x training speedup |
| Text augmentation pipeline | Back-translation, paraphrasing, synonym replacement | Improve model robustness |
| Data filtering & quality scoring | Remove low-quality samples from training data | Critical for model quality |
| Deduplication & near-duplicate removal | Prevent data contamination | Required for clean training |
| Data mixing & curriculum design | Balance multiple data sources during training | Required for multi-task models |
| Data selection & active learning | Prioritize most informative samples | Reduce training cost |
| Format converters (JSONL, Parquet, HuggingFace, Alpaca, ShareGPT) | Load data from any format | Interop with community datasets |
GAP 2: Production Model Serving (CRITICAL)#
Current state: Inference uses ONNX Runtime. No integration with industry-standard LLM serving frameworks optimized for production throughput.
What's needed:
| Task | Description | Why Critical |
|---|---|---|
| vLLM integration | PagedAttention, continuous batching for LLMs | 10-24x throughput vs naive serving |
| TensorRT-LLM integration | NVIDIA-optimized LLM inference | Lowest latency on NVIDIA GPUs |
| Text Generation Inference (TGI) | HuggingFace's production LLM server | Battle-tested, flash attention |
| llama.cpp integration | CPU-optimized LLM inference | Edge/CPU-only deployment |
| Triton Inference Server | Multi-model, multi-framework serving | Production orchestration |
| SGLang integration | Structured generation, constrained decoding | JSON/code output reliability |
| Ollama integration | Simple local model deployment | Developer experience |
| Multi-LoRA serving | Serve multiple LoRA adapters on one base model | Per-customer fine-tuning |
| Speculative decoding pipeline | Draft model + verify for faster generation | 2-3x speedup for large models |
| KV cache optimization | Prefix caching, cache sharing across requests | Reduce TTFT (time to first token) |
GAP 3: Dataset Sovereignty (CRITICAL)#
Current state: Datasets are managed ad-hoc per domain. No centralized registry, versioning, or governance.
What's needed:
| Task | Description | Why Critical |
|---|---|---|
| Centralized dataset registry | Catalog all datasets across domains with metadata | Discoverability, deduplication |
| Dataset versioning (DVC or similar) | Track dataset changes over time | Reproducibility |
| Synthetic data generation pipeline | Generate training data using LLMs/diffusion | Scale training data without manual annotation |
| Data lineage tracking | Track which datasets trained which models | Compliance, debugging |
| Dataset quality dashboard | Monitor quality metrics across all datasets | Prevent model degradation |
| PII detection & scrubbing pipeline | Remove personal data from training sets | Legal/compliance requirement |
| License compliance checker | Verify training data licenses | Legal risk mitigation |
| Cross-domain data sharing protocol | Share curated data between domains safely | Reduce duplication, improve models |
| Dataset card generator | Auto-generate HuggingFace-style dataset cards | Documentation, governance |
| Annotation schema library | Standardized schemas for labeling tasks | Consistent data quality |
GAP 4: Human-in-the-Loop Pipeline (HIGH)#
Current state: No annotation platform, no feedback collection system, no RLHF data pipeline connecting user feedback to model training.
What's needed:
| Task | Description | Why Critical |
|---|---|---|
| Annotation platform (Label Studio integration or custom) | Web UI for data labeling | Foundation for supervised training |
| Preference collection UI | A/B comparison interface for RLHF data | Feed DPO/PPO training |
| Feedback ingestion pipeline | Collect thumbs-up/down from production systems | Continuous improvement signal |
| Inter-annotator agreement tracking | Measure labeling consistency | Data quality assurance |
| Active learning loop | Route uncertain samples to annotators | Efficient annotation budget |
| Annotation task management | Queue, assign, review annotation batches | Scale annotation operations |
| Quality control for annotations | Gold standard checks, annotator calibration | Prevent label noise |
| RLHF data pipeline | Convert feedback → preference pairs → training data | Close the feedback loop |
| Red team dataset curation | Organize adversarial prompts for safety testing | Safety training |
| Evaluation set management | Maintain held-out eval sets per domain | Prevent contamination |
GAP 5: MLOps & Experiment Management (HIGH)#
Current state: Basic experiment tracking exists in @isis/model-fine-tuning and serwaa/avatar-engine. No unified MLOps layer.
What's needed:
| Task | Description | Why Critical |
|---|---|---|
| Experiment tracking dashboard (MLflow or W&B self-hosted) | Track all training runs centrally | Compare experiments, reproduce results |
| Model lineage graph | Visualize base model → fine-tune → merge → deploy | Understand model provenance |
| Automated model evaluation pipeline | Run benchmarks on every model checkpoint | Catch regressions early |
| Model drift detection | Monitor production model outputs for distribution shift | Prevent silent degradation |
| Automated retraining triggers | Retrain when drift exceeds threshold | Maintain model quality |
| Model A/B testing framework | Route traffic between model versions | Safe model rollout |
| Shadow deployment | Run new model alongside old, compare outputs | Risk-free evaluation |
| Rollback automation | One-click model version rollback | Incident response |
| GPU cost tracking per experiment | Track $/experiment, $/token for training | Budget management |
| Training job scheduler | Queue and prioritize training jobs across GPU pool | Resource efficiency |
GAP 6: Edge & On-Device Deployment (HIGH)#
Current state: RunPod serves models in the cloud. No pipeline for compiling/deploying models to edge devices, mobile, or browsers.
What's needed:
| Task | Description | Why Critical |
|---|---|---|
| ONNX export pipeline | Convert PyTorch/HF models to ONNX | Cross-platform deployment |
| CoreML conversion | ONNX → CoreML for iOS/macOS | Apple device deployment |
| TFLite conversion | ONNX → TensorFlow Lite for Android | Android deployment |
| TensorRT compilation | ONNX → TensorRT for NVIDIA edge | Jetson/edge GPU deployment |
| WebGPU/WebNN inference | Browser-native ML inference | Web app deployment |
| WASM inference runtime | ML models in WebAssembly | Portable edge deployment |
| Model pruning pipeline | Structured/unstructured pruning for size reduction | Smaller models for edge |
| Neural Architecture Search (NAS) | Find optimal architecture for target hardware | Hardware-specific optimization |
| Quantization-aware training (QAT) | Train with quantization in the loop | Better quality than post-training quantization |
| Model benchmark suite per target | Latency/accuracy/memory benchmarks per device class | Deployment decision support |
GAP 7: Continuous Evaluation & Benchmarking (HIGH)#
Current state: 3D quality continuous evaluation exists. No equivalent for LLMs, classifiers, embeddings, or other model types.
What's needed:
| Task | Description | Why Critical |
|---|---|---|
| LLM evaluation suite (MMLU, HumanEval, etc.) | Standard benchmarks for LLM quality | Compare fine-tuned vs base models |
| RAG evaluation framework (RAGAS) | Evaluate retrieval + generation quality | Sophia, Iris, Veritas |
| Embedding quality benchmarks (MTEB) | Evaluate embedding models | Sophia, Iris knowledge systems |
| Safety benchmark suite (TruthfulQA, BBQ, etc.) | Evaluate model safety/bias | Nous, Kuan Yin |
| Domain-specific eval sets | Per-domain held-out evaluation data | Each domain's model quality |
| Automated regression testing | Compare model versions on fixed eval sets | Prevent quality regressions |
| Human evaluation framework | Structured human ratings on model outputs | Ground truth quality signal |
| Leaderboard dashboard | Visual comparison of model versions | Track improvement over time |
| Cost-quality Pareto analysis | Plot quality vs cost for model selection | Optimal model selection |
| Cross-lingual evaluation | Evaluate models across languages (especially Ghanaian) | Veritas, global deployment |
GAP 8: Specialized Training Pipelines per Domain (MEDIUM-HIGH)#
Current state: Generic fine-tuning framework exists. No domain-specific training pipelines that automate the full cycle from data → training → evaluation → deployment for each domain's unique needs.
What's needed:
| Domain | Pipeline | Description |
|---|---|---|
| Iris | Conversation model fine-tuning pipeline | Data collection → format → LoRA train → eval → deploy |
| Iris | Embedding model fine-tuning pipeline | Triplet mining → contrastive training → MTEB eval → deploy |
| Veritas | NER model training pipeline | Annotate Ghanaian entities → train spaCy/BERT → eval → deploy |
| Veritas | Bias detector training pipeline | Collect political coverage → annotate → train classifier → deploy |
| Veritas | Ghana language TTS pipeline | Collect Twi/Ewe/Ga audio → train XTTS → eval → deploy |
| Psyche | Voice cloning pipeline | Collect voice samples → fine-tune XTTS → quality eval → deploy |
| Psyche | Emotion model training pipeline | Collect facial data → train classifier → eval → deploy |
| Kuan Yin | Harm detector training pipeline | Collect moderation data → train → red-team eval → deploy |
| Nous | Safety classifier pipeline | Red-team generation → train guardrails → eval → deploy |
| Isis | LoRA training pipeline (images) | Collect style data → train LoRA → quality gate → registry |
| Isis | LoRA training pipeline (video) | Collect motion data → train video LoRA → eval → deploy |
| Sophia | Reranker training pipeline | Mine hard negatives → train cross-encoder → eval → deploy |
| Oya | Aerial vision training pipeline | Collect aerial data → fine-tune YOLO → quantize → deploy |
| Airmid | Plant ID training pipeline | Collect plant images → fine-tune CNN → quantize → deploy |
| Euterpe | Music model training pipeline | Collect music data → fine-tune AudioCraft → eval → deploy |
| Minerva | Lecture LLM training pipeline | Collect educational data → fine-tune LLM → eval → deploy |
| Hathor | NPC dialogue training pipeline | Generate character data → fine-tune LLM → eval → deploy |
GAP 9: Model Governance & Compliance (MEDIUM)#
Current state: Basic model registry exists. No governance framework for model lifecycle management, compliance, or audit trails.
What's needed:
| Task | Description | Why Critical |
|---|---|---|
| Model card generation | Auto-generate model documentation (training data, metrics, limitations) | Compliance, transparency |
| Model approval workflow | Review/approve models before production deployment | Quality gate |
| License compliance tracking | Track licenses of base models and training data | Legal risk |
| Bias audit pipeline | Automated bias testing across protected attributes | Fairness compliance |
| Model deprecation workflow | Graceful model retirement with migration path | Lifecycle management |
| Audit trail for model changes | Who trained what, when, with what data | Compliance, debugging |
| Data retention policies per model | How long to keep training data and checkpoints | Storage cost, compliance |
| Model access control | Role-based access to model artifacts | Security |
GAP 10: GPU Infrastructure Scaling (MEDIUM)#
Current state: RunPod for serverless GPU, some Terraform for AWS. No multi-cloud GPU orchestration or training cluster management.
What's needed:
| Task | Description | Why Critical |
|---|---|---|
| Multi-cloud GPU orchestration | Use RunPod + AWS + GCP GPUs as single pool | Cost optimization, availability |
| Training cluster provisioner | Spin up/down multi-GPU training clusters on demand | Large-scale training |
| Spot/preemptible instance management | Use cheap spot instances with checkpointing | 60-90% cost reduction |
| GPU memory profiler | Profile VRAM usage per model/batch size | Right-size GPU allocation |
| Cost estimator for training jobs | Estimate $/run before launching | Budget planning |
| Queue-based training scheduler | Priority queue for training jobs across teams | Resource fairness |
| Distributed training launcher | One-command multi-node training setup | Developer experience |
| GPU health monitoring | Track GPU temperature, utilization, errors | Prevent training failures |
| Auto-scaling for inference | Scale GPU workers based on request queue depth | Handle traffic spikes |
| Reserved instance optimization | Analyze usage patterns, recommend reservations | Cost optimization |
4. Recommended New Tasks#
Phase: ML Sovereignty Foundation (New Phase Recommended)#
This should be a new dedicated phase in TODOS.md focused exclusively on the cross-cutting ML infrastructure that enables sovereignty across all domains.
Section 1: Centralized Dataset Management#
- [ ] Design dataset registry schema (name, version, domain, format, size, license, quality score, lineage)
- [ ] Implement dataset registry service with PostgreSQL storage
- [ ] Build dataset versioning system (content-addressable storage, delta tracking)
- [ ] Create dataset upload/download CLI tool
- [ ] Implement dataset format converters (JSONL, Parquet, HuggingFace, Alpaca, ShareGPT, OpenAI)
- [ ] Build dataset quality scoring engine (completeness, diversity, toxicity, duplication)
- [ ] Implement deduplication pipeline (exact + near-duplicate with MinHash/SimHash)
- [ ] Build PII detection and scrubbing pipeline for training data
- [ ] Create license compliance checker for training datasets
- [ ] Implement dataset card auto-generation (HuggingFace format)
- [ ] Build data lineage tracker (which datasets trained which models)
- [ ] Create cross-domain data sharing protocol with access controls
- [ ] Implement dataset split management (train/val/test with contamination prevention)
- [ ] Build streaming dataset loader for datasets larger than memory
- [ ] Create annotation schema library with standard schemas per task type
- [ ] Implement dataset statistics dashboard (size, distribution, quality over time)
- [ ] Build dataset diff tool (compare versions, detect distribution shifts)
- [ ] Create synthetic data generation orchestrator (LLM-based, diffusion-based)
- [ ] Implement data mixing recipe system (specify ratios, curricula)
- [ ] Build dataset export pipeline for external sharing (anonymized, licensed)
Section 2: Synthetic Data Generation#
- [ ] Design synthetic data generation framework (pluggable generators per modality)
- [ ] Implement LLM-based text data generator (instruction following, conversation, QA)
- [ ] Build self-instruct pipeline (seed instructions → LLM generation → quality filter)
- [ ] Implement Evol-Instruct pipeline (complexity evolution for instruction data)
- [ ] Create persona-based dialogue generator (varied styles, knowledge levels)
- [ ] Build synthetic preference pair generator (for DPO/RLHF training)
- [ ] Implement synthetic NER data generator (domain-specific entity annotation)
- [ ] Create synthetic code data generator (problems, solutions, explanations)
- [ ] Build diffusion-based image data generator (with controlled variation)
- [ ] Implement TTS-based audio data generator (multi-speaker, multi-language)
- [ ] Create 3D scene synthetic data generator (for 3D model training)
- [ ] Build adversarial data generator (red-team prompts, jailbreak attempts)
- [ ] Implement multi-lingual synthetic data pipeline (Twi, Ewe, Ga, Hausa, Dagbani)
- [ ] Create quality filter for synthetic data (decontamination, deduplication, scoring)
- [ ] Build human-AI collaborative data generation workflow
Section 3: Annotation & Human Feedback Platform#
- [ ] Design annotation platform architecture (task management, reviewer UI, quality control)
- [ ] Integrate Label Studio as self-hosted annotation backend
- [ ] Build preference comparison UI (side-by-side A/B for RLHF data collection)
- [ ] Implement thumbs-up/down feedback ingestion from production endpoints
- [ ] Create annotation task queue with priority and assignment management
- [ ] Build inter-annotator agreement calculator (Cohen's kappa, Fleiss' kappa)
- [ ] Implement gold standard quality checks (inject known-answer questions)
- [ ] Create annotator calibration workflow (training + certification)
- [ ] Build active learning loop (route uncertain samples to annotators)
- [ ] Implement RLHF data pipeline (feedback → preference pairs → training format)
- [ ] Create red team dataset curation tool (organize adversarial prompts by category)
- [ ] Build evaluation set management (held-out sets with contamination monitoring)
- [ ] Implement annotation export to multiple training formats
- [ ] Create annotation analytics dashboard (throughput, quality, cost per annotation)
- [ ] Build multi-annotator consensus resolution workflow
Section 4: Production Model Serving#
- [ ] Integrate vLLM as primary LLM serving engine (PagedAttention, continuous batching)
- [ ] Build vLLM Docker image with multi-LoRA support for RunPod deployment
- [ ] Integrate TensorRT-LLM for NVIDIA-optimized inference
- [ ] Build TGI (Text Generation Inference) adapter for HuggingFace models
- [ ] Integrate llama.cpp for CPU-only and edge LLM serving
- [ ] Build SGLang integration for structured/constrained generation
- [ ] Implement multi-LoRA serving (serve N fine-tuned adapters on one base model)
- [ ] Create model routing layer (route requests to optimal backend by model/hardware)
- [ ] Build speculative decoding pipeline (draft model selection, verification)
- [ ] Implement KV cache management (prefix caching, cache sharing, eviction policies)
- [ ] Create batching optimizer (dynamic batching by sequence length)
- [ ] Build model warmup and preloading system
- [ ] Implement graceful model swapping (zero-downtime model updates)
- [ ] Create inference cost calculator ($/token by model, backend, GPU)
- [ ] Build latency monitoring per model endpoint with SLA tracking
- [ ] Implement Ollama-compatible API for local development
- [ ] Create model serving benchmark suite (throughput, latency, memory)
- [ ] Build auto-scaling controller for GPU inference workers
- [ ] Implement request priority queuing (interactive > batch > background)
- [ ] Create multi-tenant model isolation (per-customer fine-tuned models)
Section 5: MLOps & Experiment Management#
- [ ] Deploy self-hosted MLflow as experiment tracking backend
- [ ] Build experiment tracking SDK (log metrics, params, artifacts from training code)
- [ ] Create model lineage graph (visualize base → fine-tune → merge → deploy chain)
- [ ] Implement model comparison dashboard (side-by-side metrics, outputs)
- [ ] Build automated model evaluation pipeline (run benchmarks on every checkpoint)
- [ ] Create model drift detection service (monitor output distribution over time)
- [ ] Implement automated retraining trigger (retrain when drift exceeds threshold)
- [ ] Build model A/B testing framework (split traffic, collect metrics, auto-promote)
- [ ] Create shadow deployment system (run new model alongside production, compare)
- [ ] Implement one-click model rollback (revert to previous version)
- [ ] Build GPU cost tracking per experiment ($/run, $/token, $/epoch)
- [ ] Create training job scheduler with priority queue
- [ ] Implement training job dashboard (running, queued, completed, failed)
- [ ] Build model artifact storage with deduplication (shared base weights)
- [ ] Create model deployment pipeline (registry → testing → staging → production)
- [ ] Implement canary deployment for model updates
- [ ] Build model performance alerting (latency, error rate, quality degradation)
- [ ] Create reproducibility system (pin data version + code version + config)
- [ ] Implement experiment tagging and search
- [ ] Build model governance approval workflow (review before production)
Section 6: Edge & On-Device Deployment#
- [ ] Build ONNX export pipeline (PyTorch/HF → ONNX with optimization)
- [ ] Implement CoreML conversion pipeline (ONNX → CoreML for iOS/macOS)
- [ ] Build TFLite conversion pipeline (ONNX → TensorFlow Lite for Android)
- [ ] Create TensorRT compilation pipeline (ONNX → TensorRT for NVIDIA edge)
- [ ] Implement WebGPU inference runtime for browser deployment
- [ ] Build WebNN inference adapter for cross-browser ML
- [ ] Create WASM inference runtime (ONNX Runtime Web or custom)
- [ ] Implement model pruning pipeline (structured + unstructured)
- [ ] Build quantization-aware training (QAT) pipeline
- [ ] Create post-training quantization pipeline (INT8, INT4, FP8)
- [ ] Implement knowledge distillation pipeline (teacher → student automation)
- [ ] Build target device profiler (benchmark model on target hardware)
- [ ] Create model size/latency/accuracy Pareto optimizer
- [ ] Implement on-device model update mechanism (differential model updates)
- [ ] Build edge model deployment CI/CD (compile → test → package → distribute)
- [ ] Create device compatibility matrix (which models run on which devices)
- [ ] Implement fallback chain (try on-device → edge server → cloud)
- [ ] Build battery/thermal-aware inference scheduler for mobile
- [ ] Create edge model monitoring (collect on-device metrics, aggregate centrally)
- [ ] Implement federated learning orchestrator for privacy-preserving training
Section 7: Continuous Evaluation & Benchmarking#
- [ ] Design unified benchmark framework (pluggable evaluators, standard metrics)
- [ ] Implement LLM benchmark suite (MMLU, HumanEval, MT-Bench, Arena-Hard)
- [ ] Build RAG evaluation framework (faithfulness, relevance, context precision)
- [ ] Create embedding benchmark suite (MTEB subset relevant to Oshun domains)
- [ ] Implement safety benchmark suite (TruthfulQA, BBQ, RealToxicityPrompts)
- [ ] Build multi-lingual evaluation suite (focus on Ghanaian languages)
- [ ] Create domain-specific eval sets per domain (Iris, Sophia, Veritas, etc.)
- [ ] Implement automated regression testing (compare model versions on fixed evals)
- [ ] Build human evaluation framework (structured rating tasks, sampling)
- [ ] Create leaderboard dashboard (track model quality over time per domain)
- [ ] Implement cost-quality Pareto analysis tool ($/quality for model selection)
- [ ] Build cross-model comparison tool (compare fine-tuned vs base vs competitors)
- [ ] Create eval set contamination detector (check if eval data leaked into training)
- [ ] Implement benchmark scheduling (run daily/weekly on production models)
- [ ] Build evaluation report generator (PDF/HTML reports per model version)
- [ ] Create custom benchmark authoring tool (define domain-specific test cases)
- [ ] Implement adversarial evaluation (automatically generated edge cases)
- [ ] Build latency-quality tradeoff analyzer
- [ ] Create evaluation set versioning (track eval set changes, normalize scores)
- [ ] Implement multi-judge evaluation (aggregate LLM-as-judge + human ratings)
Section 8: Domain-Specific Training Pipelines#
- [ ] Build Iris conversational model fine-tuning pipeline (data→LoRA→eval→deploy)
- [ ] Build Iris embedding model fine-tuning pipeline (triplet mining→train→MTEB→deploy)
- [ ] Build Veritas NER model training pipeline (annotate→train spaCy/BERT→eval→deploy)
- [ ] Build Veritas bias detector training pipeline (annotate→train→eval→deploy)
- [ ] Build Veritas Ghana language TTS pipeline (collect audio→train XTTS→eval→deploy)
- [ ] Build Psyche voice cloning pipeline (samples→fine-tune XTTS→quality→deploy)
- [ ] Build Psyche emotion model training pipeline (facial data→train→eval→deploy)
- [ ] Build Kuan Yin harm detector training pipeline (moderation data→train→red-team→deploy)
- [ ] Build Nous safety classifier training pipeline (red-team→train guardrails→eval→deploy)
- [ ] Build Isis image LoRA training pipeline (style data→train LoRA→quality gate→registry)
- [ ] Build Isis video LoRA training pipeline (motion data→train→eval→deploy)
- [ ] Build Sophia reranker training pipeline (hard negatives→train cross-encoder→eval→deploy)
- [ ] Build Oya aerial vision training pipeline (aerial data→fine-tune YOLO→quantize→deploy)
- [ ] Build Airmid plant ID training pipeline (images→fine-tune CNN→quantize→deploy)
- [ ] Build Euterpe music model training pipeline (music data→fine-tune→eval→deploy)
- [ ] Build Minerva lecture LLM training pipeline (educational data→fine-tune→eval→deploy)
- [ ] Build Hathor NPC dialogue training pipeline (character data→fine-tune LLM→eval→deploy)
- [ ] Build Aphrodite content moderation training pipeline (data→train NSFW detector→eval→deploy)
- [ ] Build Themis originality model training pipeline (fingerprints→train→eval→deploy)
- [ ] Build cross-domain transfer learning pipeline (share learned representations)
Section 9: GPU Infrastructure & Training Clusters#
- [ ] Design multi-cloud GPU orchestration layer (RunPod + AWS + GCP as unified pool)
- [ ] Build training cluster provisioner (spin up multi-GPU clusters on demand)
- [ ] Implement spot/preemptible instance manager with automatic checkpointing
- [ ] Create GPU memory profiler for right-sizing GPU allocation per workload
- [ ] Build training cost estimator (predict $/run before launching)
- [ ] Implement queue-based training scheduler with priority and fairness
- [ ] Create distributed training launcher (one-command multi-node setup)
- [ ] Build GPU health monitoring dashboard (temperature, utilization, errors, ECC)
- [ ] Implement auto-scaling for inference (scale workers by queue depth)
- [ ] Create reserved instance optimization analyzer
- [ ] Build training checkpoint management for spot instance recovery
- [ ] Implement NCCL/Gloo configuration optimizer for multi-node training
- [ ] Create GPU utilization alerting (underutilized or overloaded warnings)
- [ ] Build training cost dashboard (per-domain, per-model, per-experiment)
- [ ] Implement GPU sharing for small models (MIG, MPS, time-sharing)
Section 10: Model Governance & Compliance#
- [ ] Build model card auto-generator (training data, metrics, limitations, biases)
- [ ] Create model approval workflow (review gate before production deployment)
- [ ] Implement license compliance tracker for base models and training data
- [ ] Build automated bias audit pipeline (test across protected attributes)
- [ ] Create model deprecation workflow (migration path, sunset timeline)
- [ ] Implement audit trail for all model operations (train, eval, deploy, rollback)
- [ ] Build data retention policy engine (auto-expire old checkpoints and data)
- [ ] Create model access control (RBAC for model artifacts and endpoints)
- [ ] Implement model risk assessment (classify models by risk tier)
- [ ] Build compliance reporting dashboard (summarize governance state)
5. Priority Matrix#
P0 — Blocks All Sovereignty (Must Complete First)#
| Gap | Estimated Tasks | Rationale |
|---|---|---|
| Training Data Pipeline (Phase 47.5.4) | 20 tasks (exist, unchecked) | Cannot train ANY model without data loading |
| Production Model Serving (vLLM/TGI) | 20 new tasks | Cannot serve fine-tuned LLMs efficiently |
| Centralized Dataset Registry | 20 new tasks | Cannot manage training data at scale |
| Nous Safety (Phase 47.6) | 60 tasks (exist, unchecked) | Cannot deploy models without safety guardrails |
P1 — Enables Continuous Improvement#
| Gap | Estimated Tasks | Rationale |
|---|---|---|
| MLOps & Experiment Management | 20 new tasks | Cannot track/compare/reproduce experiments |
| Continuous Evaluation & Benchmarking | 20 new tasks | Cannot measure model quality systematically |
| Human-in-the-Loop Pipeline | 15 new tasks | Cannot collect feedback for RLHF/improvement |
| Synthetic Data Generation | 15 new tasks | Cannot scale training data without manual effort |
P2 — Enables Full Domain Coverage#
| Gap | Estimated Tasks | Rationale |
|---|---|---|
| Domain-Specific Training Pipelines | 20 new tasks | Each domain needs its own end-to-end pipeline |
| Nous Advanced Models (47.7-47.16) | 200+ tasks (exist, unchecked) | Video, image control, speech, animation |
| Neith AI Runtime (Phase 43.1.6) | 15 tasks (exist, unchecked) | Game engine ML inference |
| LLM-Powered NPCs (Phase 53.2) | 45 tasks (exist, unchecked) | On-device game AI |
P3 — Enables Edge & Scale#
| Gap | Estimated Tasks | Rationale |
|---|---|---|
| Edge & On-Device Deployment | 20 new tasks | Mobile, browser, embedded deployment |
| GPU Infrastructure Scaling | 15 new tasks | Multi-cloud, spot instances, cost optimization |
| Model Governance & Compliance | 10 new tasks | Audit trails, bias testing, compliance |
Summary Statistics#
| Category | Existing Unchecked Tasks | New Tasks Needed | Total |
|---|---|---|---|
| Training Data Pipeline (47.5.4) | 20 | 0 | 20 |
| Safety Framework (47.6) | 60 | 0 | 60 |
| Advanced Models (47.7-47.16) | ~200 | 0 | ~200 |
| Neith AI Runtime (43, 53) | ~260 | 0 | ~260 |
| Centralized Dataset Management | 0 | 20 | 20 |
| Synthetic Data Generation | 0 | 15 | 15 |
| Annotation & Human Feedback | 0 | 15 | 15 |
| Production Model Serving | 0 | 20 | 20 |
| MLOps & Experiment Management | 0 | 20 | 20 |
| Edge & On-Device Deployment | 0 | 20 | 20 |
| Continuous Evaluation | 0 | 20 | 20 |
| Domain-Specific Pipelines | 0 | 20 | 20 |
| GPU Infrastructure | 0 | 15 | 15 |
| Model Governance | 0 | 10 | 10 |
| TOTAL | ~540 | ~175 | ~715 |
~540 existing unchecked tasks need to be completed (mostly in Phases 47, 43, 53) and ~175 new tasks need to be added to achieve full ML sovereignty.
The most critical new phase to add is the ML Sovereignty Foundation covering dataset management, model serving, MLOps, evaluation, and domain-specific pipelines — none of which are currently tracked in any TODOS.md.
Appendix: Models to Self-Host (Open Source Recommendations)#
LLMs#
| Use Case | Recommended Base Model | Size | License |
|---|---|---|---|
| General chat/reasoning | Llama 3.1 / 3.2 | 8B-70B | Meta Community |
| Code generation | DeepSeek-Coder V2 / StarCoder2 | 7B-33B | Apache 2.0 |
| Long context | Mistral Nemo / Qwen 2.5 | 7B-72B | Apache 2.0 |
| Multilingual | Aya 23 / BLOOM | 8B-35B | Apache 2.0 |
| Function calling | Hermes 3 / Functionary | 7B-70B | Apache 2.0 |
| Small/fast | Phi-3 / Gemma 2 | 2B-9B | MIT/Apache |
Embedding Models#
| Use Case | Model | Dimensions | License |
|---|---|---|---|
| Text embedding | BGE-large / E5-large | 1024 | MIT |
| Multilingual embedding | BGE-M3 / multilingual-e5 | 1024 | MIT |
| Code embedding | CodeSage / StarEncoder | 768-1024 | Apache 2.0 |
| Cross-encoder reranking | BGE-reranker-large | - | MIT |
Vision Models#
| Use Case | Model | License |
|---|---|---|
| Image classification | ConvNeXt V2 / EfficientNet V2 | Apache 2.0 |
| Object detection | YOLOv9 / DETR / RT-DETR | AGPL/Apache |
| Image generation | SDXL / Flux.1 | Open RAIL/Apache |
| Video generation | AnimateDiff / SVD | Apache/Custom |
| OCR | PaddleOCR / TrOCR | Apache 2.0 |
Audio Models#
| Use Case | Model | License |
|---|---|---|
| Speech-to-text | Whisper large-v3 | MIT |
| Text-to-speech | XTTS v2 / Coqui | MPL 2.0 |
| Music generation | MusicGen / AudioCraft | MIT |
| Audio classification | BEATs / AST | MIT |
| Voice cloning | RVC / SoVITS | MIT |
3D Models#
| Use Case | Model | License |
|---|---|---|
| Text-to-3D | TRELLIS.2 | MIT |
| Image-to-3D | Hunyuan3D 2.1 / SF3D | Tencent/Stability |
| Gaussian splatting | 3DGS / InstantSplat | Various |
| Mesh generation | MeshAnything V2 | Apache 2.0 |
Safety Models#
| Use Case | Model | License |
|---|---|---|
| Toxicity detection | Detoxify / Perspective | Apache 2.0 |
| NSFW detection | NSFW detector / OpenNSFW2 | MIT |
| Deepfake detection | FaceForensics++ models | Academic |
| PII detection | Presidio / GLiNER | MIT |