Eve library

Nous Domain Feature Inventory

Guide

Revision 1

On this page

Generated on May 8, 2026. This document captures domain feature scope from the Oshun repository and the phase checklist in TODOS.md.

Overview#

Nous provides local LLM inference, model management, embedding generation, agent orchestration, document processing, and prompt engineering as consumable library APIs for any application in the Oshun ecosystem.

Coverage Summary#

  • Domain slug: nous
  • TODO phase coverage: 47
  • Checklist-derived feature items: 782
  • Repository status: Repository implementation exists under apps/libs/services.
  • Oshun integration status: No Oshun shell references discovered yet.
  • Source confidence: Implementation-backed and documentation-backed

Source Hierarchy#

  1. Actual implementation footprints in apps/, libs/, services/, and openapi/
  2. TODO-derived capability scope from the mapped phase sections
  3. Existing standalone domain docs in docs/domains/nous/

Capability Pillars#

Capability Pillar Primary Source Status
Hardware-Accelerated Inference docs/domains/nous/features.md implementation-backed
Model Management docs/domains/nous/features.md implementation-backed
Inference Pipeline docs/domains/nous/features.md implementation-backed
Generation Control docs/domains/nous/features.md implementation-backed
Multi-Adapter Serving docs/domains/nous/features.md implementation-backed
Graph Optimization and Quantization docs/domains/nous/features.md implementation-backed
Tensor Operations docs/domains/nous/features.md implementation-backed
Embedding Generation docs/domains/nous/features.md implementation-backed
Similarity Search docs/domains/nous/features.md implementation-backed
LLM Completion APIs docs/domains/nous/features.md implementation-backed
Natural Language Processing docs/domains/nous/features.md implementation-backed
Prompt Engineering docs/domains/nous/features.md implementation-backed
Advanced Reasoning docs/domains/nous/features.md implementation-backed
Persona and Context Management docs/domains/nous/features.md implementation-backed
Agent Architecture docs/domains/nous/features.md implementation-backed
Document Processing and RAG docs/domains/nous/features.md implementation-backed
Computer Vision docs/domains/nous/features.md implementation-backed
Cross-Cutting Capabilities docs/domains/nous/features.md implementation-backed
Capability Summary by Library docs/domains/nous/features.md implementation-backed
Inference Engine TODOS.md phase 47 planned-or-partially-implemented
LLM Serving TODOS.md phase 47 planned-or-partially-implemented
Embedding Engine TODOS.md phase 47 planned-or-partially-implemented
Text Generation TODOS.md phase 47 planned-or-partially-implemented
Agent Framework TODOS.md phase 47 planned-or-partially-implemented
RAG Pipeline TODOS.md phase 47 planned-or-partially-implemented
Image Understanding TODOS.md phase 47 planned-or-partially-implemented
Image Generation TODOS.md phase 47 planned-or-partially-implemented
Video Understanding TODOS.md phase 47 planned-or-partially-implemented
Video Generation TODOS.md phase 47 planned-or-partially-implemented
Speech Recognition TODOS.md phase 47 planned-or-partially-implemented
Speech Synthesis TODOS.md phase 47 planned-or-partially-implemented
Music Generation TODOS.md phase 47 planned-or-partially-implemented
Audio Processing TODOS.md phase 47 planned-or-partially-implemented
Training Framework TODOS.md phase 47 planned-or-partially-implemented
Fine-tuning TODOS.md phase 47 planned-or-partially-implemented
RLHF Pipeline TODOS.md phase 47 planned-or-partially-implemented
Data Pipeline TODOS.md phase 47 planned-or-partially-implemented
Content Moderation TODOS.md phase 47 planned-or-partially-implemented
Guardrails TODOS.md phase 47 planned-or-partially-implemented
Alignment TODOS.md phase 47 planned-or-partially-implemented
VOID Model Integration & Deployment TODOS.md phase 47 planned-or-partially-implemented
Quadmask Conditioning & Physics Consequence Detection TODOS.md phase 47 planned-or-partially-implemented
Multi-Pass Inference & Temporal Consistency TODOS.md phase 47 planned-or-partially-implemented
Multi-Tier Model Routing & Alternatives TODOS.md phase 47 planned-or-partially-implemented
Quality Metrics & Validation TODOS.md phase 47 planned-or-partially-implemented
Synthetic Training Data & Fine-Tuning TODOS.md phase 47 planned-or-partially-implemented
OminiControl for FLUX/DiT TODOS.md phase 47 planned-or-partially-implemented
ControlNeXt & ControlNet++ TODOS.md phase 47 planned-or-partially-implemented
Spatial Layout & Grounding Control TODOS.md phase 47 planned-or-partially-implemented
Attention Manipulation & Zero-Shot Editing TODOS.md phase 47 planned-or-partially-implemented
Instruction-Based & Granular Image Editing TODOS.md phase 47 planned-or-partially-implemented
Reference Insertion & Multi-Image Consistency TODOS.md phase 47 planned-or-partially-implemented
IC-Light & Image Relighting TODOS.md phase 47 planned-or-partially-implemented
Video & Scene Relighting via Diffusion TODOS.md phase 47 planned-or-partially-implemented
Text-Driven 3DGS Editing & Relighting TODOS.md phase 47 planned-or-partially-implemented
VACE Unified Video Creation & Editing TODOS.md phase 47 planned-or-partially-implemented
Dense Motion Trajectory Control TODOS.md phase 47 planned-or-partially-implemented
Real-Time Interactive Motion Control TODOS.md phase 47 planned-or-partially-implemented
Camera Trajectory Control TODOS.md phase 47 planned-or-partially-implemented
Video-to-Video Editing via Image Diffusion TODOS.md phase 47 planned-or-partially-implemented
Subject-Consistent Multi-Modal Video TODOS.md phase 47 planned-or-partially-implemented
DiT-Based Portrait Animation TODOS.md phase 47 planned-or-partially-implemented
Real-Time Portrait Animation TODOS.md phase 47 planned-or-partially-implemented
Audio-Driven Video Dubbing TODOS.md phase 47 planned-or-partially-implemented
Temporally-Aligned Video-to-Audio TODOS.md phase 47 planned-or-partially-implemented
Foley Sound Generation TODOS.md phase 47 planned-or-partially-implemented
Universal Any-to-Audio TODOS.md phase 47 planned-or-partially-implemented
Zero-Shot Speech Editing TODOS.md phase 47 planned-or-partially-implemented
Flow-Matching TTS Architectures TODOS.md phase 47 planned-or-partially-implemented
Real-Time Voice Style Conversion TODOS.md phase 47 planned-or-partially-implemented
Advanced Music Generation Control TODOS.md phase 47 planned-or-partially-implemented
Diffusion-DPO & Preference Optimization TODOS.md phase 47 planned-or-partially-implemented
LyCORIS & LoRA Architecture Variants TODOS.md phase 47 planned-or-partially-implemented
Multi-LoRA Composition & Attribute Control TODOS.md phase 47 planned-or-partially-implemented
Consistency Models & Few-Step Generation TODOS.md phase 47 planned-or-partially-implemented
Training-Free Inference Acceleration TODOS.md phase 47 planned-or-partially-implemented
Real-Time Interactive Diffusion TODOS.md phase 47 planned-or-partially-implemented
High-Resolution Generation TODOS.md phase 47 planned-or-partially-implemented
Flow Matching Distillation TODOS.md phase 47 planned-or-partially-implemented
Joint Audio-Video Diffusion Architecture TODOS.md phase 47 planned-or-partially-implemented
Multi-Asset Conditioning Framework TODOS.md phase 47 planned-or-partially-implemented
Scene Planning & Multi-Shot Generation TODOS.md phase 47 planned-or-partially-implemented
Physics-Aware Video Generation TODOS.md phase 47 planned-or-partially-implemented

Implementation-Derived Capability Surfaces#

Category Path Inferred Capability Surface
Library libs/nous/advanced-speech/ Advanced Speech
Library libs/nous/agreement-search/ Agreement Search
Library libs/nous/audio/ Audio
Library libs/nous/concordia-sealed-memory/ Concordia Sealed Memory
Library libs/nous/cooperative-bargaining/ Cooperative Bargaining
Library libs/nous/core/ Core
Library libs/nous/generative-relighting/ Generative Relighting
Library libs/nous/image-control/ Image Control
Library libs/nous/joint-av-generation/ Joint Av Generation
Library libs/nous/llm/ Llm
Library libs/nous/platform/ Platform
Library libs/nous/portrait-animation/ Portrait Animation
Library libs/nous/preference-inference/ Preference Inference
Library libs/nous/safety/ Safety
Library libs/nous/training/ Training
Library libs/nous/video-control/ Video Control
Library libs/nous/video-removal/ Video Removal
Library libs/nous/video-to-audio/ Video To Audio
Library libs/nous/vision/ Vision

Canonical Standalone Feature Coverage#

Heading Level Feature Family
H2 Hardware-Accelerated Inference
H3 Execution Providers
H3 Provider Routing and Fallback
H3 Session Management
H2 Model Management
H3 Model Registry and Discovery
H3 Model Acquisition
H3 Model Format Support
H3 Model Conversion and Optimization
H3 Model Caching and Lifecycle
H2 Inference Pipeline
H3 Dynamic and Continuous Batching
H3 LLM Serving
H3 Attention Mechanisms and KV Cache
H3 Multi-GPU Parallelism
H3 Prompt and Prefix Caching
H2 Generation Control
H3 Sampling Strategies
H3 Logit Processing
H3 Streaming and Stop Conditions
H2 Multi-Adapter Serving
H2 Graph Optimization and Quantization
H3 Graph Optimization
H3 Quantization
H2 Tensor Operations
H2 Embedding Generation
H3 Embedding Providers
H3 Modality Support
H3 Embedding Processing
H3 Embedding Compression
H2 Similarity Search
H3 Nearest Neighbor Search
H3 Distance Metrics
H3 Reranking
H2 LLM Completion APIs
H3 Completion Interfaces
H3 Function Calling and Tool Use
H3 Structured Output
H3 Response Processing
H2 Natural Language Processing
H2 Prompt Engineering
H3 Template System
H3 Template Management
H3 Prompt Versioning and Testing
H3 Few-Shot and Examples
H2 Advanced Reasoning
H2 Persona and Context Management
H2 Agent Architecture
H3 Agent Lifecycle
H3 Task Execution
H3 Planning and Decomposition
H3 Agent Memory
H3 Multi-Agent Systems
H3 Runtime Events
H2 Document Processing and RAG
H3 Document Ingestion
H3 Content Extraction
H3 Chunking Strategies
H3 Vector Indexing
H3 Context Retrieval
H2 Computer Vision
H3 Object Detection and Classification
H3 Facial Analysis
H3 Document and Text Recognition
H3 Scene Understanding
H3 Image Generation (Diffusion Models)
H3 Image Editing and Transformation
H3 Fine-Tuning and Adaptation
H2 Cross-Cutting Capabilities
H3 Observability
H3 Type Safety
H3 Runtime Flexibility
H2 Capability Summary by Library
H3 @nous/core -- Inference Runtime (84 modules)
H3 @nous/llm -- LLM API Layer (77 modules)
H3 @nous/vision -- Computer Vision (40 modules)

Detailed TODO Phase Coverage#

Phase 47: Nous - Sovereign AI Platform#

  • Checklist items captured: 782
  • Source: TODOS.md

47.1 @nous/core - AI Runtime Core#

47.1.1 Inference Engine#
  • 47.1.1.1 Create ONNX Runtime integration
  • 47.1.1.2 Implement model loading system
  • 47.1.1.3 Build model optimization pipeline
  • 47.1.1.4 Create quantization support (INT8, INT4, FP16)
  • 47.1.1.5 Implement graph optimization
  • 47.1.1.6 Build operator fusion
  • 47.1.1.7 Create memory planning
  • 47.1.1.8 Implement execution providers
  • 47.1.1.9 Build CPU execution provider
  • 47.1.1.10 Create CUDA execution provider
  • 47.1.1.11 Implement ROCm execution provider
  • 47.1.1.12 Build Metal execution provider
  • 47.1.1.13 Create WebGPU execution provider
  • 47.1.1.14 Implement Vulkan compute provider
  • 47.1.1.15 Build DirectML provider
  • 47.1.1.16 Create NPU/accelerator providers
  • 47.1.1.17 Implement batch inference
  • 47.1.1.18 Build dynamic batching
  • 47.1.1.19 Create continuous batching
  • 47.1.1.20 Implement speculative decoding
47.1.2 LLM Serving#
  • 47.1.2.1 Create LLM serving infrastructure
  • 47.1.2.2 Implement KV cache management
  • 47.1.2.3 Build paged attention
  • 47.1.2.4 Create flash attention integration
  • 47.1.2.5 Implement tensor parallelism
  • 47.1.2.6 Build pipeline parallelism
  • 47.1.2.7 Create model sharding
  • 47.1.2.8 Implement expert parallelism (MoE)
  • 47.1.2.9 Build prefix caching
  • 47.1.2.10 Create radix attention
  • 47.1.2.11 Implement prompt caching
  • 47.1.2.12 Build multi-LoRA serving
  • 47.1.2.13 Create adapter switching
  • 47.1.2.14 Implement context extension
  • 47.1.2.15 Build streaming generation
  • 47.1.2.16 Create stop sequences
  • 47.1.2.17 Implement logit processors
  • 47.1.2.18 Build temperature sampling
  • 47.1.2.19 Create top-k/top-p sampling
  • 47.1.2.20 Implement beam search
47.1.3 Model Management#
  • 47.1.3.1 Create model registry
  • 47.1.3.2 Implement model versioning
  • 47.1.3.3 Build model metadata
  • 47.1.3.4 Create model discovery
  • 47.1.3.5 Implement model download
  • 47.1.3.6 Build checksum verification
  • 47.1.3.7 Create model conversion
  • 47.1.3.8 Implement format detection
  • 47.1.3.9 Build safetensors support
  • 47.1.3.10 Create GGUF support
  • 47.1.3.11 Implement GGML support
  • 47.1.3.12 Build PyTorch loading
  • 47.1.3.13 Create TensorFlow loading
  • 47.1.3.14 Implement JAX loading
  • 47.1.3.15 Build model caching
  • 47.1.3.16 Create model eviction
  • 47.1.3.17 Implement model preloading
  • 47.1.3.18 Build warm model pool
  • 47.1.3.19 Create model benchmarking
  • 47.1.3.20 Implement model profiling
47.1.4 Embedding Engine#
  • 47.1.4.1 Create embedding generation
  • 47.1.4.2 Implement text embeddings
  • 47.1.4.3 Build image embeddings
  • 47.1.4.4 Create audio embeddings
  • 47.1.4.5 Implement multi-modal embeddings
  • 47.1.4.6 Build embedding normalization
  • 47.1.4.7 Create dimensionality reduction
  • 47.1.4.8 Implement batch embedding
  • 47.1.4.9 Build streaming embedding
  • 47.1.4.10 Create embedding caching
  • 47.1.4.11 Implement similarity search
  • 47.1.4.12 Build approximate nearest neighbor
  • 47.1.4.13 Create exact nearest neighbor
  • 47.1.4.14 Implement distance metrics
  • 47.1.4.15 Build hybrid search
  • 47.1.4.16 Create reranking
  • 47.1.4.17 Implement cross-encoder reranking
  • 47.1.4.18 Build embedding compression
  • 47.1.4.19 Create binary embeddings
  • 47.1.4.20 Implement matryoshka embeddings

47.2 @nous/llm - Language Model Framework#

47.2.1 Text Generation#
  • 47.2.1.1 Create completion API
  • 47.2.1.2 Implement chat completion API
  • 47.2.1.3 Build instruction following
  • 47.2.1.4 Create multi-turn conversation
  • 47.2.1.5 Implement system prompts
  • 47.2.1.6 Build function calling
  • 47.2.1.7 Create tool use framework
  • 47.2.1.8 Implement JSON mode
  • 47.2.1.9 Build structured output
  • 47.2.1.10 Create grammar constraints
  • 47.2.1.11 Implement regex constraints
  • 47.2.1.12 Build schema validation
  • 47.2.1.13 Create response formatting
  • 47.2.1.14 Implement markdown rendering
  • 47.2.1.15 Build code extraction
  • 47.2.1.16 Create citation extraction
  • 47.2.1.17 Implement fact extraction
  • 47.2.1.18 Build entity extraction
  • 47.2.1.19 Create sentiment analysis
  • 47.2.1.20 Implement classification
47.2.2 Prompt Engineering#
  • 47.2.2.1 Create prompt template system
  • 47.2.2.2 Implement variable substitution
  • 47.2.2.3 Build conditional sections
  • 47.2.2.4 Create loop constructs
  • 47.2.2.5 Implement prompt versioning
  • 47.2.2.6 Build prompt testing
  • 47.2.2.7 Create prompt optimization
  • 47.2.2.8 Implement few-shot management
  • 47.2.2.9 Build example selection
  • 47.2.2.10 Create dynamic examples
  • 47.2.2.11 Implement chain-of-thought
  • 47.2.2.12 Build tree-of-thought
  • 47.2.2.13 Create graph-of-thought
  • 47.2.2.14 Implement self-consistency
  • 47.2.2.15 Build reflection prompts
  • 47.2.2.16 Create critique prompts
  • 47.2.2.17 Implement persona management
  • 47.2.2.18 Build context injection
  • 47.2.2.19 Create memory injection
  • 47.2.2.20 Implement retrieval injection
47.2.3 Agent Framework#
  • 47.2.3.1 Create agent architecture
  • 47.2.3.2 Implement planning module
  • 47.2.3.3 Build goal decomposition
  • 47.2.3.4 Create task execution
  • 47.2.3.5 Implement action selection
  • 47.2.3.6 Build tool selection
  • 47.2.3.7 Create multi-step reasoning
  • 47.2.3.8 Implement backtracking
  • 47.2.3.9 Build error recovery
  • 47.2.3.10 Create agent memory
  • 47.2.3.11 Implement short-term memory
  • 47.2.3.12 Build long-term memory
  • 47.2.3.13 Create episodic memory
  • 47.2.3.14 Implement semantic memory
  • 47.2.3.15 Build working memory
  • 47.2.3.16 Create agent communication
  • 47.2.3.17 Implement multi-agent systems
  • 47.2.3.18 Build agent coordination
  • 47.2.3.19 Create agent delegation
  • 47.2.3.20 Implement agent supervision
47.2.4 RAG Pipeline#
  • 47.2.4.1 Create document ingestion
  • 47.2.4.2 Implement chunking strategies
  • 47.2.4.3 Build semantic chunking
  • 47.2.4.4 Create sentence chunking
  • 47.2.4.5 Implement token chunking
  • 47.2.4.6 Build recursive chunking
  • 47.2.4.7 Create document parsing
  • 47.2.4.8 Implement PDF parsing
  • 47.2.4.9 Build HTML parsing
  • 47.2.4.10 Create Markdown parsing
  • 47.2.4.11 Implement code parsing
  • 47.2.4.12 Build table extraction
  • 47.2.4.13 Create image extraction
  • 47.2.4.14 Implement metadata extraction
  • 47.2.4.15 Build link extraction
  • 47.2.4.16 Create vector indexing
  • 47.2.4.17 Implement hybrid indexing
  • 47.2.4.18 Build incremental indexing
  • 47.2.4.19 Create index optimization
  • 47.2.4.20 Implement context retrieval

47.3 @nous/vision - Computer Vision Framework#

47.3.1 Image Understanding#
  • 47.3.1.1 Create image classification
  • 47.3.1.2 Implement object detection
  • 47.3.1.3 Build instance segmentation
  • 47.3.1.4 Create semantic segmentation
  • 47.3.1.5 Implement panoptic segmentation
  • 47.3.1.6 Build depth estimation
  • 47.3.1.7 Create surface normal estimation
  • 47.3.1.8 Implement pose estimation
  • 47.3.1.9 Build facial recognition
  • 47.3.1.10 Create emotion recognition
  • 47.3.1.11 Implement age/gender estimation
  • 47.3.1.12 Build face landmark detection
  • 47.3.1.13 Create OCR engine
  • 47.3.1.14 Implement document layout
  • 47.3.1.15 Build table recognition
  • 47.3.1.16 Create handwriting recognition
  • 47.3.1.17 Implement license plate recognition
  • 47.3.1.18 Build barcode/QR recognition
  • 47.3.1.19 Create image captioning
  • 47.3.1.20 Implement visual question answering
47.3.2 Image Generation#
  • 47.3.2.1 Create diffusion model framework
  • 47.3.2.2 Implement Stable Diffusion
  • 47.3.2.3 Build SDXL support
  • 47.3.2.4 Create SD3 support
  • 47.3.2.5 Implement Flux support
  • 47.3.2.6 Build ControlNet integration
  • 47.3.2.7 Create IP-Adapter support
  • 47.3.2.8 Implement LoRA management
  • 47.3.2.9 Build textual inversion
  • 47.3.2.10 Create DreamBooth training
  • 47.3.2.11 Implement inpainting
  • 47.3.2.12 Build outpainting
  • 47.3.2.13 Create image-to-image
  • 47.3.2.14 Implement style transfer
  • 47.3.2.15 Build upscaling (ESRGAN)
  • 47.3.2.16 Create face restoration
  • 47.3.2.17 Implement background removal
  • 47.3.2.18 Build image composition
  • 47.3.2.19 Create variation generation
  • 47.3.2.20 Implement image editing
47.3.3 Video Understanding#
  • 47.3.3.1 Create video classification
  • 47.3.3.2 Implement action recognition
  • 47.3.3.3 Build temporal segmentation
  • 47.3.3.4 Create object tracking
  • 47.3.3.5 Implement multi-object tracking
  • 47.3.3.6 Build trajectory prediction
  • 47.3.3.7 Create video captioning
  • 47.3.3.8 Implement dense video captioning
  • 47.3.3.9 Build video question answering
  • 47.3.3.10 Create video summarization
  • 47.3.3.11 Implement scene detection
  • 47.3.3.12 Build shot boundary detection
  • 47.3.3.13 Create video indexing
  • 47.3.3.14 Implement moment retrieval
  • 47.3.3.15 Build temporal grounding
  • 47.3.3.16 Create video-text matching
  • 47.3.3.17 Implement re-identification
  • 47.3.3.18 Build anomaly detection
  • 47.3.3.19 Create crowd analysis
  • 47.3.3.20 Implement traffic analysis
47.3.4 Video Generation#
  • 47.3.4.1 Create video diffusion framework
  • 47.3.4.2 Implement text-to-video
  • 47.3.4.3 Build image-to-video
  • 47.3.4.4 Create video-to-video
  • 47.3.4.5 Implement frame interpolation
  • 47.3.4.6 Build video upscaling
  • 47.3.4.7 Create temporal consistency
  • 47.3.4.8 Implement motion transfer
  • 47.3.4.9 Build video inpainting
  • 47.3.4.10 Create video outpainting
  • 47.3.4.11 Implement video editing
  • 47.3.4.12 Build object removal
  • 47.3.4.13 Create object insertion
  • 47.3.4.14 Implement video composition
  • 47.3.4.15 Build temporal super-resolution
  • 47.3.4.16 Create slow motion generation
  • 47.3.4.17 Implement video stylization
  • 47.3.4.18 Build rotoscoping automation
  • 47.3.4.19 Create animation generation
  • 47.3.4.20 Implement facial reenactment

47.4 @nous/audio - Audio AI Framework#

47.4.1 Speech Recognition#
  • 47.4.1.1 Create ASR engine
  • 47.4.1.2 Implement Whisper integration
  • 47.4.1.3 Build streaming ASR
  • 47.4.1.4 Create real-time transcription
  • 47.4.1.5 Implement speaker diarization
  • 47.4.1.6 Build speaker identification
  • 47.4.1.7 Create language detection
  • 47.4.1.8 Implement multi-language ASR
  • 47.4.1.9 Build code-switching support
  • 47.4.1.10 Create punctuation restoration
  • 47.4.1.11 Implement capitalization
  • 47.4.1.12 Build timestamp alignment
  • 47.4.1.13 Create word-level timestamps
  • 47.4.1.14 Implement confidence scores
  • 47.4.1.15 Build VAD integration
  • 47.4.1.16 Create noise robustness
  • 47.4.1.17 Implement accent adaptation
  • 47.4.1.18 Build domain adaptation
  • 47.4.1.19 Create custom vocabulary
  • 47.4.1.20 Implement hotword detection
47.4.2 Speech Synthesis#
  • 47.4.2.1 Create TTS engine
  • 47.4.2.2 Implement neural TTS
  • 47.4.2.3 Build VITS integration
  • 47.4.2.4 Create Tortoise TTS
  • 47.4.2.5 Implement voice cloning
  • 47.4.2.6 Build few-shot cloning
  • 47.4.2.7 Create zero-shot cloning
  • 47.4.2.8 Implement emotion control
  • 47.4.2.9 Build prosody control
  • 47.4.2.10 Create speaking style transfer
  • 47.4.2.11 Implement rate control
  • 47.4.2.12 Build pitch control
  • 47.4.2.13 Create emphasis control
  • 47.4.2.14 Implement SSML support
  • 47.4.2.15 Build phoneme input
  • 47.4.2.16 Create multi-speaker TTS
  • 47.4.2.17 Implement language mixing
  • 47.4.2.18 Build singing synthesis
  • 47.4.2.19 Create audiobook generation
  • 47.4.2.20 Implement podcast generation
47.4.3 Music Generation#
  • 47.4.3.1 Create music generation framework
  • 47.4.3.2 Implement text-to-music
  • 47.4.3.3 Build melody generation
  • 47.4.3.4 Create harmony generation
  • 47.4.3.5 Implement rhythm generation
  • 47.4.3.6 Build chord progression
  • 47.4.3.7 Create music continuation
  • 47.4.3.8 Implement style transfer
  • 47.4.3.9 Build genre classification
  • 47.4.3.10 Create mood classification
  • 47.4.3.11 Implement beat detection
  • 47.4.3.12 Build tempo estimation
  • 47.4.3.13 Create key detection
  • 47.4.3.14 Implement chord recognition
  • 47.4.3.15 Build music transcription
  • 47.4.3.16 Create MIDI generation
  • 47.4.3.17 Implement stem separation
  • 47.4.3.18 Build vocal isolation
  • 47.4.3.19 Create accompaniment generation
  • 47.4.3.20 Implement remix automation
47.4.4 Audio Processing#
  • 47.4.4.1 Create audio enhancement
  • 47.4.4.2 Implement noise reduction
  • 47.4.4.3 Build echo cancellation
  • 47.4.4.4 Create dereverberation
  • 47.4.4.5 Implement source separation
  • 47.4.4.6 Build audio super-resolution
  • 47.4.4.7 Create bandwidth extension
  • 47.4.4.8 Implement declipping
  • 47.4.4.9 Build denoising
  • 47.4.4.10 Create audio restoration
  • 47.4.4.11 Implement audio inpainting
  • 47.4.4.12 Build audio forensics
  • 47.4.4.13 Create audio fingerprinting
  • 47.4.4.14 Implement content identification
  • 47.4.4.15 Build audio classification
  • 47.4.4.16 Create sound event detection
  • 47.4.4.17 Implement audio scene classification
  • 47.4.4.18 Build audio segmentation
  • 47.4.4.19 Create audio tagging
  • 47.4.4.20 Implement audio search

47.5 @nous/training - Model Training Infrastructure#

47.5.1 Training Framework#
  • 47.5.1.1 Create distributed training system
  • 47.5.1.2 Implement data parallelism
  • 47.5.1.3 Build model parallelism
  • 47.5.1.4 Create tensor parallelism
  • 47.5.1.5 Implement pipeline parallelism
  • 47.5.1.6 Build ZeRO optimization
  • 47.5.1.7 Create mixed precision training
  • 47.5.1.8 Implement gradient checkpointing
  • 47.5.1.9 Build gradient accumulation
  • 47.5.1.10 Create learning rate schedulers
  • 47.5.1.11 Implement warmup strategies
  • 47.5.1.12 Build optimizer selection
  • 47.5.1.13 Create AdamW implementation
  • 47.5.1.14 Implement LAMB optimizer
  • 47.5.1.15 Build Adafactor
  • 47.5.1.16 Create 8-bit optimizers
  • 47.5.1.17 Implement loss functions
  • 47.5.1.18 Build metric tracking
  • 47.5.1.19 Create checkpoint management
  • 47.5.1.20 Implement fault tolerance
47.5.2 Fine-tuning#
  • 47.5.2.1 Create supervised fine-tuning
  • 47.5.2.2 Implement instruction tuning
  • 47.5.2.3 Build chat fine-tuning
  • 47.5.2.4 Create LoRA fine-tuning
  • 47.5.2.5 Implement QLoRA
  • 47.5.2.6 Build adapter tuning
  • 47.5.2.7 Create prefix tuning
  • 47.5.2.8 Implement prompt tuning
  • 47.5.2.9 Build P-tuning
  • 47.5.2.10 Create IA3 tuning
  • 47.5.2.11 Implement DoRA
  • 47.5.2.12 Build PEFT integration
  • 47.5.2.13 Create multi-task fine-tuning
  • 47.5.2.14 Implement continual learning
  • 47.5.2.15 Build catastrophic forgetting prevention
  • 47.5.2.16 Create knowledge distillation
  • 47.5.2.17 Implement teacher-student training
  • 47.5.2.18 Build self-distillation
  • 47.5.2.19 Create progressive training
  • 47.5.2.20 Implement curriculum learning
47.5.3 RLHF Pipeline#
  • 47.5.3.1 Create reward modeling
  • 47.5.3.2 Implement preference learning
  • 47.5.3.3 Build comparison data collection
  • 47.5.3.4 Create Bradley-Terry model
  • 47.5.3.5 Implement PPO training
  • 47.5.3.6 Build DPO training
  • 47.5.3.7 Create IPO training
  • 47.5.3.8 Implement KTO training
  • 47.5.3.9 Build ORPO training
  • 47.5.3.10 Create constitutional AI
  • 47.5.3.11 Implement red teaming
  • 47.5.3.12 Build safety training
  • 47.5.3.13 Create helpfulness training
  • 47.5.3.14 Implement harmlessness training
  • 47.5.3.15 Build honesty training
  • 47.5.3.16 Create rejection sampling
  • 47.5.3.17 Implement best-of-n sampling
  • 47.5.3.18 Build iterative refinement
  • 47.5.3.19 Create RLAIF
  • 47.5.3.20 Implement self-play training
47.5.4 Data Pipeline#
  • 47.5.4.1 Create data loading system
  • 47.5.4.2 Implement distributed data loading
  • 47.5.4.3 Build data sharding
  • 47.5.4.4 Create streaming datasets
  • 47.5.4.5 Implement data preprocessing
  • 47.5.4.6 Build tokenization
  • 47.5.4.7 Create packing for efficiency
  • 47.5.4.8 Implement sequence padding
  • 47.5.4.9 Build dynamic batching
  • 47.5.4.10 Create data augmentation
  • 47.5.4.11 Implement text augmentation
  • 47.5.4.12 Build back-translation
  • 47.5.4.13 Create paraphrasing
  • 47.5.4.14 Implement data filtering
  • 47.5.4.15 Build quality scoring
  • 47.5.4.16 Create deduplication
  • 47.5.4.17 Implement near-duplicate removal
  • 47.5.4.18 Build data mixing
  • 47.5.4.19 Create curriculum design
  • 47.5.4.20 Implement data selection

47.6 @nous/safety - AI Safety Framework#

47.6.1 Content Moderation#
  • 47.6.1.1 Create content classifier
  • 47.6.1.2 Implement toxicity detection
  • 47.6.1.3 Build hate speech detection
  • 47.6.1.4 Create harassment detection
  • 47.6.1.5 Implement violence detection
  • 47.6.1.6 Build self-harm detection
  • 47.6.1.7 Create sexual content detection
  • 47.6.1.8 Implement spam detection
  • 47.6.1.9 Build misinformation detection
  • 47.6.1.10 Create deepfake detection
  • 47.6.1.11 Implement synthetic content detection
  • 47.6.1.12 Build watermark detection
  • 47.6.1.13 Create attribution extraction
  • 47.6.1.14 Implement provenance tracking
  • 47.6.1.15 Build content fingerprinting
  • 47.6.1.16 Create abuse detection
  • 47.6.1.17 Implement coordinated behavior detection
  • 47.6.1.18 Build bot detection
  • 47.6.1.19 Create manipulation detection
  • 47.6.1.20 Implement influence operation detection
47.6.2 Guardrails#
  • 47.6.2.1 Create input validation
  • 47.6.2.2 Implement jailbreak detection
  • 47.6.2.3 Build prompt injection detection
  • 47.6.2.4 Create adversarial input detection
  • 47.6.2.5 Implement PII detection
  • 47.6.2.6 Build PII redaction
  • 47.6.2.7 Create secrets detection
  • 47.6.2.8 Implement code injection detection
  • 47.6.2.9 Build SQL injection detection
  • 47.6.2.10 Create XSS detection
  • 47.6.2.11 Implement output validation
  • 47.6.2.12 Build hallucination detection
  • 47.6.2.13 Create factuality checking
  • 47.6.2.14 Implement consistency checking
  • 47.6.2.15 Build relevance filtering
  • 47.6.2.16 Create topic enforcement
  • 47.6.2.17 Implement style enforcement
  • 47.6.2.18 Build length enforcement
  • 47.6.2.19 Create format enforcement
  • 47.6.2.20 Implement safety filtering
47.6.3 Alignment#
  • 47.6.3.1 Create value alignment system
  • 47.6.3.2 Implement preference modeling
  • 47.6.3.3 Build intent detection
  • 47.6.3.4 Create goal inference
  • 47.6.3.5 Implement uncertainty quantification
  • 47.6.3.6 Build calibration analysis
  • 47.6.3.7 Create selective prediction
  • 47.6.3.8 Implement abstention
  • 47.6.3.9 Build confidence estimation
  • 47.6.3.10 Create error detection
  • 47.6.3.11 Implement self-correction
  • 47.6.3.12 Build external verification
  • 47.6.3.13 Create human-in-the-loop
  • 47.6.3.14 Implement approval workflows
  • 47.6.3.15 Build escalation system
  • 47.6.3.16 Create oversight mechanisms
  • 47.6.3.17 Implement audit trails
  • 47.6.3.18 Build explanation generation
  • 47.6.3.19 Create interpretability tools
  • 47.6.3.20 Implement attribution analysis

47.7 @nous/video-removal - Physics-Aware Video Object & Interaction Removal#

Netflix VOID (Video Object and Interaction Deletion) equivalent. The first model to perform physics-aware object removal — removing objects AND their causal physical effects (dents, deformations, displaced objects, shadows). Based on VOID (Apache 2.0, github.com/Netflix/void-model) fine-tuned from CogVideoX-Fun-V1.5-5b-InP (5B parameter 3D Transformer video diffusion model).

47.7.1 VOID Model Integration & Deployment#
  • 47.7.1.1 Create @nous/video-removal library with project.json (scope:nous, layer:domain)
  • 47.7.1.2 Implement VOID model loader supporting Pass 1 (void_pass1.safetensors, 11.1 GB) and Pass 2 (void_pass2.safetensors, 11.1 GB) weight files
  • 47.7.1.3 Create CogVideoX-Fun-V1.5-5b-InP base model integration (HuggingFace Diffusers pipeline with 3D Transformer backbone)
  • 47.7.1.4 Implement VOID inference pipeline with configurable denoising steps (default 50), CFG scale (1.0-6.0), DDIM scheduler, and BF16 precision
  • 47.7.1.5 Create memory management system supporting model_full_load, model_cpu_offload, sequential_cpu_offload, and model_cpu_offload_and_qfloat8 modes for varying GPU VRAM availability (40GB+ optimal, 24GB minimum with offloading)
  • 47.7.1.6 Implement RunPod serverless endpoint for VOID inference with A100 80GB GPU worker, persistent model weight volume, and auto-scaling
  • 47.7.1.7 Create VOID Docker image with PyTorch, HuggingFace Diffusers, Transformers, SAM2, OpenCV, imageio-ffmpeg, and all VOID dependencies
  • 47.7.1.8 Implement batch inference scheduling for multi-shot removal jobs with GPU queue management via BullMQ
47.7.2 Quadmask Conditioning & Physics Consequence Detection#
  • 47.7.2.1 Implement quadmask encoding system with 4-value mask semantics: 0 (primary object to remove), 63 (overlap region), 127 (affected/consequence region), 255 (background to preserve)
  • 47.7.2.2 Create SAM2 video segmentation integration for interactive object selection via point prompts, box prompts, and text prompts with mask propagation across all video frames at 44+ FPS
  • 47.7.2.3 Implement SAM2 mask refinement pipeline: Stage 0 (GUI point selector) → Stage 1 (SAM2 segmentation producing primary mask) with temporal tracking and occlusion handling
  • 47.7.2.4 Create VLM-driven physics consequence analyzer using Gemini API (via @isis/ai-providers) to identify affected regions — objects that would move, fall, change, or lose deformation if the removed object never existed
  • 47.7.2.5 Implement grey mask generation for VLM-identified affected regions using SAM2 re-segmentation of consequence areas
  • 47.7.2.6 Create quadmask compositor merging primary mask (black), overlap regions (dark grey), affected regions (mid grey), and background (white) into final 4-value mask video
  • 47.7.2.7 Implement automatic physics consequence classification: contact deformation (dents, compression), shadow/reflection removal, displaced object trajectories, fluid disturbance, and weight-bearing structural effects
  • 47.7.2.8 Create quadmask validation system verifying mask completeness, temporal consistency, and correct value assignment before inference
47.7.3 Multi-Pass Inference & Temporal Consistency#
  • 47.7.3.1 Implement VOID Pass 1 (base inpainting): video + quadmask + text prompt fed into fine-tuned CogVideoX transformer producing initial inpainted output with VAE mask encoding
  • 47.7.3.2 Create VOID Pass 2 (warped-noise refinement): compute optical flow from Pass 1 output, warp latent representations, use warped noise as initialization for second diffusion run with configurable degradation factor (default 0.3)
  • 47.7.3.3 Implement multidiffusion temporal windowing with 85-frame windows and configurable overlap for temporal consistency across the full video length
  • 47.7.3.4 Create optical flow computation engine for warped-noise generation supporting RAFT, FlowFormer, and VideoFlow algorithms
  • 47.7.3.5 Implement object morphing artifact detection and correction — detect when diffusion model gradually changes object shape across frames and trigger Pass 2 refinement
  • 47.7.3.6 Create long-video temporal stitching pipeline: segment video into overlapping windows (max 197 frames each), process independently, blend at window boundaries using cross-fade in latent space
  • 47.7.3.7 Implement seam detection and correction at window boundaries to prevent visible transitions in long-form content
  • 47.7.3.8 Create progressive preview system showing low-resolution inpainting result before committing to full-quality inference
47.7.4 Multi-Tier Model Routing & Alternatives#
  • 47.7.4.1 Implement multi-tier video inpainting router with quality/speed/ cost-based model selection: Premium (VOID, physics-aware, 40GB+, slow), Standard (DiffuEraser/VideoPainter, no physics, 12-33GB, moderate), Fast (STTN/AVID, no physics, low VRAM, near-real-time)
  • 47.7.4.2 Create DiffuEraser integration as Apache 2.0-licensed mid-tier backend (denoising UNet + BrushNet with temporal attention, 12-33GB VRAM)
  • 47.7.4.3 Implement AVID (Any-length Video Inpainting with Diffusion) integration as MIT-licensed alternative with Temporal MultiDiffusion
  • 47.7.4.4 Create STTN (Spatial-Temporal Transformer Network) integration as MIT-licensed ultra-fast fallback for previews and drafts
  • 47.7.4.5 Implement SAM2 + backend modular pipeline allowing any segmentation frontend with any inpainting backend via standard mask interface
  • 47.7.4.6 Create quality-based automatic tier selection: analyze video complexity, object interaction density, and physics consequence presence to route to the appropriate model tier
  • 47.7.4.7 Implement fallback chain: attempt VOID → on failure/timeout fall back to DiffuEraser → on failure fall back to STTN, with quality degradation notification
  • 47.7.4.8 Create A/B comparison tool generating side-by-side outputs from multiple model tiers for quality evaluation
47.7.5 Quality Metrics & Validation#
  • 47.7.5.1 Implement temporal consistency scoring measuring frame-to-frame coherence of inpainted regions using optical flow warping error
  • 47.7.5.2 Create removal completeness metric detecting residual artifacts, ghosting, or incomplete object removal in output frames
  • 47.7.5.3 Implement physics plausibility validation scoring whether consequence regions (dents removed, objects repositioned) look physically correct
  • 47.7.5.4 Create background reconstruction quality metric comparing inpainted background against surrounding unmasked regions for texture/color consistency
  • 47.7.5.5 Implement perceptual quality scoring using VMAF, LPIPS, and FID on inpainted vs surrounding regions
  • 47.7.5.6 Create edge artifact detector identifying halos, seams, or unnatural boundaries at mask edges in output frames
  • 47.7.5.7 Implement human evaluation pipeline with A/B preference testing interface (replicating VOID paper's 25-person evaluation methodology)
  • 47.7.5.8 Create automated quality gate that blocks output delivery if any metric falls below configurable thresholds
47.7.6 Synthetic Training Data & Fine-Tuning#
  • 47.7.6.1 Implement HUMOTO-style synthetic data pipeline: motion capture data → Blender physics simulation → render paired videos (scene with object, scene without object with physics re-simulated) for counterfactual training
  • 47.7.6.2 Create Kubric synthetic data pipeline using Google's Kubric framework to generate object interaction videos with ground-truth removal targets (launched objects altering trajectories of other objects)
  • 47.7.6.3 Implement training data format: rgb_full.mp4 (original), rgb_removed.mp4 (ground truth counterfactual), mask.mp4 (quadmask), metadata.json per sample
  • 47.7.6.4 Create VOID fine-tuning pipeline on 8x A100 80GB with DeepSpeed ZeRO Stage 2, lr=1e-5, quadmask conditioning, VAE mask encoding
  • 47.7.6.5 Implement domain-specific fine-tuning workflows: fashion (remove accessories preserving fabric physics), architecture (remove furniture preserving shadows), sports (remove players preserving ball trajectories)
  • 47.7.6.6 Create per-production fine-tuning using production dailies (integrate with @isis/post-production-ai 70.1.2) for production-specific visual consistency
  • 47.7.6.7 Implement fine-tuning quality validation using held-out test set with FID, LPIPS, and temporal consistency metrics
  • 47.7.6.8 Create model versioning and checkpoint management for fine-tuned VOID variants with rollback capability

47.8 @nous/image-control — DiT-Native & Advanced Image Generation Control#

OminiControl (ICCV 2025) provides DiT-native control using only 0.1% extra parameters for FLUX/SD3 architectures. ControlNeXt replaces zero-convolution with Cross Normalization (90% fewer params). ControlNet++ adds cycle-consistency loss. OmniGen (CVPR 2025) unifies generation eliminating adapter modules. These address the U-Net→DiT architectural shift that renders original ControlNet incompatible with modern diffusion transformer backbones.

47.8.1 OminiControl for FLUX/DiT#
  • 47.8.1.1 Create @nous/image-control library with project.json (scope:nous, layer:domain), package.json, tsconfig.json, and barrel exports
  • 47.8.1.2 Implement OminiControl model loader using FLUX's own VAE encoder and transformer blocks with 0.1% additional parameters — no separate control encoder required
  • 47.8.1.3 Create spatially-aligned control pipeline (depth, canny, normal) using OminiControl's dynamic position encoding for positional correspondence between condition and generated image
  • 47.8.1.4 Implement non-aligned control pipeline (subject-driven, style reference) using OminiControl's flexible position encoding without spatial locking for appearance transfer
  • 47.8.1.5 Create multi-condition OminiControl compositor allowing simultaneous depth + subject reference with independent weight controls per condition channel
  • 47.8.1.6 Implement OminiControl condition preprocessor chain: raw input → preprocessor (MiDaS/DWPose/Canny) → VAE encoding → token injection into DiT blocks at configurable layer indices
  • 47.8.1.7 Create OminiControl benchmark comparing adherence vs ControlNet on FLUX for depth/edge/pose conditions (FID, condition alignment score, CLIP similarity)
  • 47.8.1.8 Implement OminiControl training pipeline for custom condition types using the model's own features as training signal with configurable condition type registration
47.8.2 ControlNeXt & ControlNet++#
  • 47.8.2.1 Implement ControlNeXt model loader with Cross Normalization replacing zero-convolution (90% fewer learnable parameters than ControlNet)
  • 47.8.2.2 Create ControlNeXt image control pipeline supporting all standard conditions (depth, canny, pose, normal, segmentation) via lightweight adapter injection
  • 47.8.2.3 Implement ControlNeXt video control extension — same architecture applied to video diffusion models for temporally-consistent control across frames
  • 47.8.2.4 Create ControlNet++ cycle-consistency module: discriminative reward model extracts condition from generated image and optimizes against input condition for closed-loop fidelity
  • 47.8.2.5 Implement ControlNet++ training pipeline with pixel-level consistency loss for depth, canny, segmentation, and pose conditions
  • 47.8.2.6 Create ControlNeXt vs ControlNet vs OminiControl A/B comparison harness (measure condition adherence, FID, inference speed, VRAM usage, parameter count)
  • 47.8.2.7 Implement unified control adapter interface abstracting ControlNet/ControlNeXt/OminiControl behind common API with automatic backend selection based on base model architecture
  • 47.8.2.8 Create control condition quality scorer evaluating preprocessor output quality before inference (blur detection, completeness, resolution matching, edge density validation)
47.8.3 Spatial Layout & Grounding Control#
  • 47.8.3.1 Implement GLIGEN (Grounded Language-to-Image Generation) model loader with grounding tokens for bounding-box-conditioned generation
  • 47.8.3.2 Create BoxDiff layout control: specify object positions via bounding boxes with text labels, generate images respecting spatial arrangement via cross-attention constraints
  • 47.8.3.3 Implement Attend-and-Excite semantic binding ensuring each text token receives sufficient cross-attention activation (prevents attribute leakage between subjects in multi-object prompts)
  • 47.8.3.4 Create layout-to-image pipeline: accept JSON layout spec ({object, bbox, prompt}) → inject grounding tokens → generate with spatial fidelity and per-object prompt adherence
  • 47.8.3.5 Implement Dense Diffusion per-region prompt control: different prompts for different image regions with smooth blending at boundaries via masked cross-attention
  • 47.8.3.6 Create layout validation system comparing generated object positions against specified bounding boxes using detection model (Grounding DINO / Florence-2) with IoU scoring
  • 47.8.3.7 Implement multi-object layout composition with collision avoidance and occlusion-aware depth ordering based on object z-index
  • 47.8.3.8 Create interactive layout editor API accepting iterative refinements (move object, resize region, change object prompt) without full regeneration via latent space manipulation
47.8.4 Attention Manipulation & Zero-Shot Editing#
  • 47.8.4.1 Implement Prompt-to-Prompt cross-attention map injection: swap/refine/reweight attention maps between source and target prompts for editing without DDIM inversion
  • 47.8.4.2 Create MasaCtrl mutual self-attention control: replace self-attention keys/values from source image into target generation for structure-preserving editing while changing semantics
  • 47.8.4.3 Implement DragDiffusion drag-based editing pipeline: user specifies handle points and target points → optimize latent via motion supervision loss → generate edited image preserving unedited regions
  • 47.8.4.4 Create Plug-and-Play (PnP) Diffusion feature injection: extract spatial features from source image at intermediate denoising steps → inject into target generation for layout/structure preservation
  • 47.8.4.5 Implement attention heat map visualization showing per-token cross-attention activation maps for debugging prompt-image alignment and identifying ignored tokens
  • 47.8.4.6 Create attention-based region isolation: automatically generate editing masks from cross-attention maps without manual mask creation (threshold + morphological cleanup)
  • 47.8.4.7 Implement combined editing pipeline: P2P for text-guided edits + MasaCtrl for structure preservation + PnP for spatial features in single composable workflow
  • 47.8.4.8 Create editing quality validator comparing structural similarity (SSIM, LPIPS) between source and edit to ensure unedited regions remain unchanged beyond configurable threshold
47.8.5 Instruction-Based & Granular Image Editing#
  • 47.8.5.1 Implement InstructPix2Pix model loader with dual conditioning (image + text instruction) for natural-language-directed editing ("make the sky sunset-colored")
  • 47.8.5.2 Create Differential Diffusion variable-rate editing: per-pixel change map controlling edit strength (strong edits in foreground, subtle/none in background) via spatially-varying noise schedule
  • 47.8.5.3 Implement LayerDiffusion for transparent image generation: produce RGBA output with clean alpha channel for compositing workflows
  • 47.8.5.4 Create LayerDiffusion layer decomposition: generate foreground with transparency + separate background layer from single prompt for independent manipulation
  • 47.8.5.5 Implement OmniGen unified generation pipeline: single VAE+transformer handling text-to-image, image editing, subject-driven, and CV tasks via interleaved text+image input tokens
  • 47.8.5.6 Create instruction parsing module: natural language editing instruction → structured edit specification (region, operation, parameters) for routing to appropriate backend (InstructPix2Pix vs P2P vs Differential)
  • 47.8.5.7 Implement Concept Sliders as LoRA direction vectors: train attribute-specific LoRA pairs (e.g., "more aged"/"less aged") for precise attribute control along semantic axes with configurable slider range
  • 47.8.5.8 Create batch instruction editing pipeline: apply same edit instruction across multiple images with consistency enforcement (same slider values, same P2P weights, same edit strength)
47.8.6 Reference Insertion & Multi-Image Consistency#
  • 47.8.6.1 Implement AnyDoor reference-based object insertion: paste reference object into target scene with automatic perspective, lighting, and shadow adaptation via appearance-geometry decoupling
  • 47.8.6.2 Create StoryDiffusion consistent-character generation: shared self-attention across batch ensuring character identity, clothing, and appearance match across a sequence of images/panels
  • 47.8.6.3 Implement reference-based scene composition: multiple reference objects + scene layout → composed image with correct relative scale and occlusion ordering
  • 47.8.6.4 Create subject consistency scoring comparing identity embeddings (CLIP/FaceID/DINOv2) across generated image sequences to flag identity drift above configurable threshold
  • 47.8.6.5 Implement FreeU inference modification: boost skip-connection features for quality improvement with zero additional training or parameters — configurable backbone/skip scale factors
  • 47.8.6.6 Create Depth Anything v2 integration as improved monocular depth estimator (higher accuracy than MiDaS/ZoeDepth) for all depth-conditioned workflows in the control pipeline
  • 47.8.6.7 Implement Grounding DINO text-to-bounding-box detection as complementary module to SAM for text-prompted segmentation without point/box interaction
  • 47.8.6.8 Create unified control condition preprocessor registry supporting hot-swap of depth (MiDaS/ZoeDepth/Depth Anything), pose (OpenPose/DWPose), edge (Canny/HED/Lineart) estimators with version pinning

47.9 @nous/generative-relighting — Diffusion-Based Relighting Control#

Distinct from production relighting (Phase 70.1.3 — relighting already-shot footage). These techniques use diffusion models to control illumination in generated content, enabling precise light source manipulation, environment-map-based relighting, and physically-based shading in both 2D and 3D generation. IC-Light (image), GenLit (video via video diffusion), LightLab (specific light source control), and LumiGauss (3DGS relighting via per-Gaussian material decomposition).

47.9.1 IC-Light & Image Relighting#
  • 47.9.1.1 Implement IC-Light model loader for single-image diffusion-based relighting with foreground/background conditioning branches
  • 47.9.1.2 Create IC-Light foreground relighting pipeline: extract foreground object + apply target lighting environment (HDRI, directional, point) via conditioned diffusion denoising
  • 47.9.1.3 Implement IC-Light background generation: generate environment consistent with specified lighting on foreground object for seamless compositing
  • 47.9.1.4 Create LightLab specific light source control: add, move, change color/intensity of individual light sources in photographs via diffusion with light-parameterized conditioning
  • 47.9.1.5 Implement IntrinsicEdit intrinsic decomposition editing: decompose image into albedo + shading → edit either independently → recompose with modified lighting preserving material properties
  • 47.9.1.6 Create environment map conditioning: accept HDRI/EXR environment maps as relighting target with spherical harmonic encoding for physically-based integration
  • 47.9.1.7 Implement relighting consistency across multiple views of same object (batch relighting with shared lighting parameters and cross-view shadow coherence)
  • 47.9.1.8 Create relighting quality metrics comparing generated shadows/highlights against physically-based rendering reference (using intrinsic decomposition error, shadow direction accuracy, specular consistency)
47.9.2 Video & Scene Relighting via Diffusion#
  • 47.9.2.1 Implement GenLit video relighting: reformulate relighting as video generation (temporal sequence of lighting changes) using video diffusion model's implicit 3D understanding
  • 47.9.2.2 Create RelightMaster video relighting with Multi-Plane Light Image (MPLI) representation encoding spatial properties of 3D light sources for video-length consistency
  • 47.9.2.3 Implement UniLumos unified image-and-video relighting in single framework with shared model weights and conditional temporal attention
  • 47.9.2.4 Create temporal lighting consistency enforcement for video relighting: lighting changes must be smooth and physically plausible across frames (no per-frame flicker)
  • 47.9.2.5 Implement Text2Relight text-guided portrait relighting: "dramatic side lighting", "soft diffuse overcast" as text prompts controlling light direction, quality, and color temperature
  • 47.9.2.6 Create video relighting preview pipeline: low-resolution fast relighting for interactive adjustment before full-quality render with parameter locking
47.9.3 Text-Driven 3DGS Editing & Relighting#
  • 47.9.3.1 Implement GaussCtrl multi-view consistent text-driven 3DGS editing: propagate text-guided edits across all views maintaining 3D geometric consistency
  • 47.9.3.2 Create InterGSEdit interactive 3DGS editing with 3D geometry-consistent attention: user-interactive interface with geometric consistency enforcement across rendered viewpoints
  • 47.9.3.3 Implement CTRL-D controllable dynamic 3DGS scene editing with personalized 2D diffusion priors for text or reference-image guided edits on time-varying scenes
  • 47.9.3.4 Create Morpheus text-driven 3DGS shape and color stylization (e.g., "make this building gothic") applied consistently to 3D Gaussian scene across all viewing angles
  • 47.9.3.5 Implement LumiGauss per-Gaussian material decomposition: precompute diffuse/specular/transmission coefficients as per-Gaussian buffers for real-time relighting without re-rendering
  • 47.9.3.6 Create LumiGauss relighting inference pipeline: dot product of precomputed material buffers with target lighting → real-time relightable 3DGS at interactive frame rates
  • 47.9.3.7 Implement SyncNoise geometrically consistent noise prediction for text-based 3D scene editing maintaining multi-view coherence via shared noise maps
  • 47.9.3.8 Create 3DGS editing quality validator: verify multi-view consistency of edited scene via rendered novel-view comparison and depth consistency check against original geometry

47.10 @nous/video-control — Advanced Video Motion, Camera & Editing Control#

Wan2.1-VACE (ICCV 2025, Alibaba) is the first open-source unified video creation and editing framework. Motion Prompting (CVPR 2025) and Wan-Move (NeurIPS 2025) provide dense point trajectory control for objects AND camera simultaneously. MotionStream (ICLR 2026) achieves real-time interactive motion control at 29 FPS. PostCam and DepthDirector enable 3D-depth-aware camera trajectory control for novel-view generation. TokenFlow and AnyV2V bridge image editing to video.

47.10.1 VACE Unified Video Creation & Editing#
  • 47.10.1.1 Implement Wan2.1-VACE model loader for both 14B (high quality) and 1.3B (fast preview) variants with Video Condition Unit (VCU) support
  • 47.10.1.2 Create VACE unified conditioning interface: text, image, video, and mask inputs fused into single VCU representation for any creation/editing task
  • 47.10.1.3 Implement VACE Context Adapter for task-concept injection via temporal and spatial dimension representations enabling task-specific behavior from unified model
  • 47.10.1.4 Create VACE reference-to-video pipeline: single reference image → full video maintaining subject identity, style, and scene coherence across generated frames
  • 47.10.1.5 Implement VACE video inpainting with spatiotemporal mask: arbitrary region masking across time → temporally-consistent fill respecting motion and scene dynamics
  • 47.10.1.6 Create VACE video outpainting and spatial extension: extend video canvas in any direction with coherent content generation matching existing motion patterns
  • 47.10.1.7 Implement VACE video-to-video editing: text-guided restyling of existing video with temporal consistency preservation and configurable edit strength
  • 47.10.1.8 Create VACE task router: analyze input modality combination (text only, image+text, video+mask, etc.) and automatically select appropriate VACE mode with optimal VCU configuration
47.10.2 Dense Motion Trajectory Control#
  • 47.10.2.1 Implement Motion Prompting model integration: accept point trajectories (sparse or dense, per-object or full-scene) as conditioning signal for video generation
  • 47.10.2.2 Create Motion Prompting object control: user draws trajectory for specific object → model generates video with object following path while scene responds naturally to object motion
  • 47.10.2.3 Implement Motion Prompting camera control: specify camera movement via dense flow field → generate video from target viewpoint trajectory with parallax and depth-aware rendering
  • 47.10.2.4 Create Motion Prompting simultaneous object+camera control: independent trajectories for objects and camera composed in single generation without interference
  • 47.10.2.5 Implement Wan-Move latent trajectory guidance: propagate first-frame features along user-specified dense point trajectories in latent space for motion-faithful generation
  • 47.10.2.6 Create Wan-Move interactive trajectory editor: draw motion paths on first frame → preview trajectory propagation → generate 5-second 480p video with trajectory fidelity scoring
  • 47.10.2.7 Implement motion trajectory format converter: support drag points, dense flow fields, sparse keypoints, and bezier curves as input → convert to model-specific trajectory format (Motion Prompting vs Wan-Move)
  • 47.10.2.8 Create motion trajectory quality validator: verify physical plausibility (no teleportation, reasonable acceleration, object collision avoidance) before generation
47.10.3 Real-Time Interactive Motion Control#
  • 47.10.3.1 Implement MotionStream real-time motion control pipeline achieving sub-second latency at up to 29 FPS on single GPU via streaming denoising architecture
  • 47.10.3.2 Create MotionStream interactive painting interface: user paints trajectories in real-time → instant video preview updates reflecting motion changes
  • 47.10.3.3 Implement MotionStream camera control mode: real-time camera movement specification with immediate visual feedback at interactive frame rates
  • 47.10.3.4 Create MotionStream motion transfer mode: extract motion from source video → apply to target content in real-time with automatic scale and speed adaptation
  • 47.10.3.5 Implement MotionStream GPU memory optimization for sustained interactive session (model persistence, latent caching, incremental inference, VRAM budget management)
  • 47.10.3.6 Create MotionStream quality-latency tradeoff controls: adjust resolution, frame count, and denoising steps for interactive preview vs final output quality
47.10.4 Camera Trajectory Control#
  • 47.10.4.1 Implement PostCam novel-view video generation: source video + target 6-DoF camera trajectory → video from new viewpoint with depth-consistent parallax
  • 47.10.4.2 Create PostCam query-shared cross-attention module unifying camera pose embeddings and 2D rendered frames into single representation for viewpoint-conditioned generation
  • 47.10.4.3 Implement DepthDirector depth-conditioned camera control: inject 3D depth understanding into video diffusion for previously-unseen camera views with occluded region hallucination
  • 47.10.4.4 Create camera trajectory specification format: 6-DoF poses (position + rotation) per frame, with interpolation between keyframes (linear, bezier, ease-in/out, custom spline)
  • 47.10.4.5 Implement camera trajectory library: pre-built cinematographic camera movements (orbit, dolly, crane, tracking, reveal, flyover, dutch tilt) as reusable trajectory presets
  • 47.10.4.6 Create multi-camera consistency: generate multiple videos from different camera angles of same scene with cross-view coherence via shared scene representation
  • 47.10.4.7 Implement camera trajectory extraction from existing video: estimate camera motion from video using structure-from-motion → use as conditioning for re-rendering or style transfer
  • 47.10.4.8 Create camera trajectory visualization: 3D viewport showing camera path relative to estimated scene geometry for trajectory planning and adjustment
47.10.5 Video-to-Video Editing via Image Diffusion#
  • 47.10.5.1 Implement TokenFlow temporal-consistent video editing: share diffusion features across keyframes → propagate to full sequence for coherent video style transfer preserving motion
  • 47.10.5.2 Create AnyV2V video editing pipeline: apply any image editing technique (InstructPix2Pix, P2P, IP-Adapter) to video with automatic temporal consistency via feature propagation
  • 47.10.5.3 Implement TV-LiVE training-free text-guided video editing via Layer Informed Vitality Exploitation for zero-shot editing without per-video optimization
  • 47.10.5.4 Create video editing temporal consistency scoring: optical flow warping error between adjacent edited frames vs original frame relationship with per-region breakdown
  • 47.10.5.5 Implement keyframe-based video editing workflow: edit selected keyframes with image tools → automatically propagate edits to intermediate frames via optical flow warping and feature interpolation
  • 47.10.5.6 Create video editing undo system: revert edits per-frame or per-region while preserving other modifications via layered edit history
47.10.6 Subject-Consistent Multi-Modal Video#
  • 47.10.6.1 Implement HunyuanCustom multi-modal video generation: condition on images, videos, audio, and text simultaneously for subject-consistent output with cross-modal alignment
  • 47.10.6.2 Create HunyuanCustom dual-stream processing with separate video/text transformer blocks fused in single-stream phase for efficient multi-modal attention
  • 47.10.6.3 Implement subject identity preservation across video: extract subject embeddings from reference → inject into video generation → verify identity consistency per-frame via embedding distance tracking
  • 47.10.6.4 Create multi-subject video generation: multiple reference subjects placed in single video with independent identity preservation and interaction-aware composition
  • 47.10.6.5 Implement subject-aware temporal coherence: ensure subject appearance doesn't drift over video duration (face, clothing, proportions) via periodic identity re-anchoring
  • 47.10.6.6 Create subject consistency benchmark comparing identity preservation across HunyuanCustom, Wan-Move, and VACE backends using FaceID/CLIP/DINOv2 similarity metrics

47.11 @nous/portrait-animation — Audio-Driven Portrait Video Generation#

Fundamentally different from lip-sync animation (which generates phoneme/viseme data for animating existing 3D models, covered in Phase 17/42/78). Portrait animation generates entirely new video frames from a reference photo + audio input. Hallo3 uses Video Diffusion Transformers for high dynamic range. IM-Portrait (CVPR 2025) adds 3D-awareness. RAP achieves real-time generation. LivePortrait uses stitching for efficiency. MuseTalk provides real-time video dubbing.

47.11.1 DiT-Based Portrait Animation#
  • 47.11.1.1 Create @nous/portrait-animation library with project.json (scope:nous, layer:domain), package.json, tsconfig.json, and barrel exports
  • 47.11.1.2 Implement Hallo3 Video DiT portrait animation model loader with support for high dynamic range expression and motion generation
  • 47.11.1.3 Create Hallo3 inference pipeline: reference portrait photo + driving audio → animated video with natural head movement, expression, and lip sync at configurable resolution
  • 47.11.1.4 Implement IM-Portrait 3D-aware video diffusion for photorealistic talking heads with 3D geometric consistency across head poses and viewing angles
  • 47.11.1.5 Create portrait reference preprocessing: face detection, alignment, feature extraction (identity embedding, expression baseline, head pose estimation) with quality gate for unsuitable references
  • 47.11.1.6 Implement expression amplitude control: configure intensity of facial movement, head rotation range, and eye blink frequency via per-channel scaling parameters
  • 47.11.1.7 Create multi-style portrait animation: same reference photo animated in different styles (natural, dramatic, subtle, energetic) via style conditioning vectors
  • 47.11.1.8 Implement portrait animation quality metrics: lip-sync accuracy (LSE-C/LSE-D), identity preservation (ArcFace cosine similarity), motion naturalness (FVD), head pose accuracy
47.11.2 Real-Time Portrait Animation#
  • 47.11.2.1 Implement LivePortrait stitching-based architecture for efficient real-time portrait animation without full diffusion inference per frame via warping + stitching
  • 47.11.2.2 Create LivePortrait expression transfer: source video actor's expressions → applied to target portrait photo in real-time with identity preservation
  • 47.11.2.3 Implement RAP (Real-time Audio-driven Portrait) using Video DiT with hybrid attention + static-dynamic training-inference strategy for low-latency generation at interactive speeds
  • 47.11.2.4 Create real-time audio stream to portrait pipeline: microphone input → phoneme extraction → portrait animation at < 200ms end-to-end latency for live conversation
  • 47.11.2.5 Implement ChatAnyone-style real-time portrait with hierarchical motion diffusion for interactive conversation agents with emotional responsiveness
  • 47.11.2.6 Create multi-person portrait animation: animate multiple different portrait references simultaneously from separate or shared audio sources with independent expression control
  • 47.11.2.7 Implement GPU batching for concurrent portrait animation sessions (multiple portraits animated in parallel on single GPU with VRAM budget allocation)
  • 47.11.2.8 Create portrait animation streaming output: WebRTC/HLS adaptive bitrate stream from generated portrait video for browser-based delivery with latency monitoring
47.11.3 Audio-Driven Video Dubbing#
  • 47.11.3.1 Implement MuseTalk real-time high-fidelity video dubbing via spatio-temporal sampling: replace original lip movements in existing video with new audio-driven motion preserving face identity
  • 47.11.3.2 Create EchoMimic audio-to-facial-motion generation: audio features → detailed facial landmark sequences → video synthesis with landmark-guided deformation
  • 47.11.3.3 Implement dubbing quality validation: compare dubbed lip motion against phoneme timing, check for artifacts at face boundary, verify temporal smoothness across sentence boundaries
  • 47.11.3.4 Create cross-language dubbing pipeline: original video + translated audio (from ElevenLabs/F5-TTS) → MuseTalk lip replacement → quality validation with language-specific viseme verification
  • 47.11.3.5 Implement partial face editing: modify only mouth region while preserving original expression, eye movement, and head pose from source video via region-masked generation
  • 47.11.3.6 Create dubbing batch processor: process entire episodes with automatic scene detection, face tracking, and per-shot dubbing with consistency enforcement across cuts

47.12 @nous/video-to-audio — Video-to-Audio Generation#

Entirely new capability absent from the monorepo. AV-Link (ICCV 2025) provides temporally-aligned diffusion features for cross-modal audio-video generation. FoleyCrafter generates sound effects from video (footsteps, impacts, ambient). Any2Audio provides universal any-to-audio generation via flow matching.

47.12.1 Temporally-Aligned Video-to-Audio#
  • 47.12.1.1 Create @nous/video-to-audio library with project.json (scope:nous, layer:domain), package.json, tsconfig.json, and barrel exports
  • 47.12.1.2 Implement AV-Link model loader with cross-modal attention for temporally-aligned audio-video diffusion features bridging visual and auditory latent spaces
  • 47.12.1.3 Create AV-Link inference pipeline: input video → extract temporal visual features → generate synchronized audio track via audio diffusion with frame-level alignment
  • 47.12.1.4 Implement temporal alignment scoring: measure onset synchronization between visual events (impacts, footfalls, door closes) and generated audio transients with per-event precision metrics
  • 47.12.1.5 Create video feature extraction for audio conditioning: motion magnitude, scene change detection, object interaction detection → audio event trigger map with timestamp annotations
  • 47.12.1.6 Implement audio style conditioning: generate sound effects in different styles (realistic, stylized, cinematic, horror) from same video via style embeddings with blendable style vectors
  • 47.12.1.7 Create multi-track audio generation: separate dialogue, effects, ambient, and music layers from single video input with independent per-track generation and mixing controls
  • 47.12.1.8 Implement audio-video sync quality validation using AV-Sync benchmark metrics (onset precision, energy correlation, spectral coherence, perceptual sync score)
47.12.2 Foley Sound Generation#
  • 47.12.2.1 Implement FoleyCrafter foley generation pipeline: video of physical interaction → corresponding sound effect (footstep on gravel, glass breaking, door creaking) with material-aware synthesis
  • 47.12.2.2 Create foley material detection: classify surface materials visible in video (wood, metal, fabric, concrete, water) → condition sound generation on material acoustic properties
  • 47.12.2.3 Implement foley intensity control: map visual impact force (estimated from motion vectors and acceleration) to audio amplitude and spectral characteristics
  • 47.12.2.4 Create foley library augmentation: generate variations of common foley sounds (different footstep patterns, varying impact intensities) for sound design library population
  • 47.12.2.5 Implement foley temporal precision: align generated sound onset within ±2 frames of visual contact point using onset detection refinement loop
  • 47.12.2.6 Create foley A/B comparison tool generating side-by-side outputs of generated foley vs professional foley recordings for perceptual quality evaluation
47.12.3 Universal Any-to-Audio#
  • 47.12.3.1 Implement Any2Audio model loader with flow matching backbone and chain-of-thought reasoning for multi-modal audio generation from arbitrary input combinations
  • 47.12.3.2 Create text-to-audio pipeline: descriptive text → sound effect or ambient audio generation with semantic accuracy and temporal structure control
  • 47.12.3.3 Implement image-to-audio pipeline: static image → associated ambient soundscape generation (forest image → bird chirps, wind, rustling) with scene understanding
  • 47.12.3.4 Create audio-to-audio editing: modify existing audio based on text instructions ("add reverb", "remove background noise", "change to minor key") with source audio preservation
  • 47.12.3.5 Implement multi-modal audio conditioning: combine video + text + reference audio inputs for precisely controlled audio generation with per-modality weight control
  • 47.12.3.6 Create any-to-audio quality metrics: Frechet Audio Distance (FAD), audio-text alignment (CLAP score), temporal coherence for video-to-audio, onset precision scoring

47.13 @nous/advanced-speech — Advanced TTS, Speech Editing & Music Control#

VoiceCraft enables zero-shot speech editing (change specific words without full resynthesis). F5-TTS, CosyVoice, and Fish Audio S2 are flow-matching TTS architectures with lower latency and better prosody than autoregressive approaches. MusiConGen adds rhythm and chord conditioning to music generation. Activation steering enables inference-time music style transfer without retraining.

47.13.1 Zero-Shot Speech Editing#
  • 47.13.1.1 Implement VoiceCraft model loader for zero-shot speech editing in the wild (Apache 2.0 license) with bidirectional causal masking support
  • 47.13.1.2 Create VoiceCraft transcript-based editing: modify transcript text → VoiceCraft regenerates only the changed words while preserving original voice, prosody, and background audio
  • 47.13.1.3 Implement VoiceCraft insertion: add new words or phrases into existing speech at specified positions with natural-sounding transitions matching surrounding intonation
  • 47.13.1.4 Create VoiceCraft deletion: remove words or phrases from speech with seamless gap filling maintaining speech rhythm and natural pacing
  • 47.13.1.5 Implement VoiceCraft voice preservation validation: compare speaker embedding (ECAPA-TDNN / WavLM) of edited segments against original to ensure identity consistency within configurable threshold
  • 47.13.1.6 Create speech editing undo system: preserve original audio segments → allow selective revert of individual word edits with full edit history
  • 47.13.1.7 Implement context-aware speech editing: VoiceCraft uses surrounding audio context → ensure prosody of edited words matches sentence-level intonation and emphasis patterns
  • 47.13.1.8 Create speech editing quality metrics: naturalness MOS, speaker similarity (EER), intelligibility (CER/WER on edited output), boundary naturalness scoring
47.13.2 Flow-Matching TTS Architectures#
  • 47.13.2.1 Implement F5-TTS model loader with flow matching backbone and non-autoregressive generation for low-latency synthesis with DiT-architecture inference
  • 47.13.2.2 Create CosyVoice (Alibaba) integration with controllable prosody, emotion, and speaking style via semantic tokens and flow matching decoder
  • 47.13.2.3 Implement Fish Audio S2 Pro integration with dual-autoregressive architecture and inline prosody control tags for fine-grained speech control
  • 47.13.2.4 Create cross-TTS voice cloning: clone voice from reference audio using any backend (F5, CosyVoice, Fish) with unified voice profile format and quality scoring
  • 47.13.2.5 Implement TTS latency benchmark: measure time-to-first-audio for F5-TTS, CosyVoice, Fish Audio vs existing ElevenLabs integration (target: < 200ms for interactive applications)
  • 47.13.2.6 Create emotion and prosody control API: unified emotion tags ([excited], [whispers], [angry], [sad]) mapped to backend-specific conditioning with fallback for unsupported emotions
  • 47.13.2.7 Implement multi-language TTS routing: detect language → route to optimal TTS backend per language (CosyVoice for Chinese, F5-TTS for English, etc.) with automatic fallback chain
  • 47.13.2.8 Create TTS quality comparison dashboard: MOS scoring, naturalness, speaker similarity across all TTS backends for A/B evaluation and automatic best-backend selection
47.13.3 Real-Time Voice Style Conversion#
  • 47.13.3.1 Implement StyleStream real-time zero-shot voice style conversion: transform speaking style while preserving content and speaker identity via disentangled style embeddings
  • 47.13.3.2 Create voice style library: capture and parameterize speaking styles (news anchor, whisper, shout, storytelling, monotone) as reusable style embeddings with interpolation support
  • 47.13.3.3 Implement real-time voice conversion pipeline: microphone input → style transfer → output at < 100ms latency for live applications with adaptive buffer management
  • 47.13.3.4 Create voice aging/de-aging: transform voice age characteristics while preserving identity (child → adult, adult → elderly) via age-conditioned conversion
  • 47.13.3.5 Implement accent conversion: transform accent while preserving voice identity and content (e.g., British → American English) via accent-disentangled representation
  • 47.13.3.6 Create voice conversion quality metrics: speaker identity preservation (EER), style transfer accuracy (style classifier), naturalness scoring (MOS), latency measurement
47.13.4 Advanced Music Generation Control#
  • 47.13.4.1 Implement MusiConGen rhythm conditioning: specify rhythm pattern (BPM, time signature, rhythmic pattern as onset grid) as input conditioning for music generation
  • 47.13.4.2 Create MusiConGen chord progression conditioning: specify chord sequence (e.g., "Am-F-C-G" with timing) as structural constraint for generated music with harmonic accuracy validation
  • 47.13.4.3 Implement MusicGen activation steering: inference-time interventions on residual stream/attention activations for timbre transfer, style fusion, and genre mixing without retraining
  • 47.13.4.4 Create multi-track music generation: generate separate stems (drums, bass, melody, harmony, pads) independently with rhythmic synchronization and mix-ready levels
  • 47.13.4.5 Implement music continuation with style transition: generate continuation of input audio that smoothly transitions to target style/genre over configurable crossfade duration
  • 47.13.4.6 Create music-to-music style transfer: apply timbre/instrumentation of reference track to content/melody of source track via activation steering without retraining
  • 47.13.4.7 Implement music generation with MIDI control: accept MIDI input as melody/harmony constraint → generate full audio arrangement matching MIDI structure with configurable instrumentation
  • 47.13.4.8 Create music generation evaluation suite: melody accuracy vs input, rhythmic consistency (beat tracking), harmonic analysis (chord recognition), style classification accuracy

47.14 @nous/diffusion-alignment — Diffusion Preference Alignment & LoRA Variants#

Diffusion-DPO (CVPR 2024) adapts Direct Preference Optimization for diffusion models using the ELBO as differentiable likelihood surrogate — mathematically distinct from LLM DPO (47.5.3.6). SPO provides step-level preference optimization. LyCORIS provides LoRA architecture variants (LoKr, OFT, BOFT) with different decomposition strategies. FreeFuse enables multi-LoRA routing at inference.

47.14.1 Diffusion-DPO & Preference Optimization#
  • 47.14.1.1 Implement Diffusion-DPO training pipeline: adapt DPO to diffusion models using ELBO as differentiable likelihood surrogate with comparison image pairs (chosen/rejected per prompt)
  • 47.14.1.2 Create preference dataset preparation: human preference pairs (chosen/rejected images for same prompt) + AI-feedback pairs from reward model with annotation quality filtering
  • 47.14.1.3 Implement SPO (Step-level Preference Optimization): per-denoising-step preference model adjusting each step individually for finer-grained control than trajectory-level DPO
  • 47.14.1.4 Create RPO (Rich Preference Optimization): LLM-generated detailed critiques of synthesized images → refined pairs with rich feedback signals for improved alignment signal density
  • 47.14.1.5 Implement Curriculum DPO: curriculum learning schedule for stable diffusion/consistency model alignment with improved training stability via progressive difficulty scaling
  • 47.14.1.6 Create Diffusion-DPO application to FLUX: fine-tune FLUX base model with human aesthetic preferences using flow-matching-adapted DPO loss with ELBO reformulation for flow objectives
  • 47.14.1.7 Implement preference-aligned inference: swap base model for DPO-aligned variant at inference with configurable alignment strength via model interpolation (base ↔ aligned)
  • 47.14.1.8 Create alignment quality metrics: win rate against base model in human evaluation, FID impact measurement, diversity preservation (recall metric), reward model score distribution
47.14.2 LyCORIS & LoRA Architecture Variants#
  • 47.14.2.1 Implement LyCORIS framework integration supporting multiple decomposition strategies: LoKr (Kronecker product), OFT (Orthogonal Fine-Tuning), BOFT (Block-Diagonal OFT) with unified training API
  • 47.14.2.2 Create DoRA (Weight-Decomposed LoRA) for diffusion models: decompose pretrained weights into magnitude and direction, fine-tune both for higher quality at low rank (r=8) than standard LoRA
  • 47.14.2.3 Implement LoKr (Low-Rank Kronecker) adapter: Kronecker product decomposition for more parameter-efficient adaptation than standard LoRA with configurable factor dimensions
  • 47.14.2.4 Create OFT (Orthogonal Fine-Tuning) adapter: constrain weight updates to orthogonal transformations preserving pretrained feature magnitudes for stable fine-tuning
  • 47.14.2.5 Implement BOFT (Block-Diagonal OFT): block-diagonal structure for scalable orthogonal fine-tuning on large diffusion models with configurable block size
  • 47.14.2.6 Create LyCORIS training configuration for each variant: optimal rank, learning rate, and regularization settings per architecture with documented hyperparameter recommendations
  • 47.14.2.7 Implement LoRA variant A/B comparison: train same dataset with LoRA, DoRA, LoKr, OFT → compare output quality, training speed, storage size, and inference overhead
  • 47.14.2.8 Create unified adapter loading: detect adapter type from checkpoint metadata → automatically load with correct architecture (LoRA/DoRA/LoKr/OFT/BOFT) without manual specification
47.14.3 Multi-LoRA Composition & Attribute Control#
  • 47.14.3.1 Implement FreeFuse multi-LoRA token-level routing: combine multiple LoRAs at inference without merging weights, route tokens dynamically per LoRA specialization based on attention scores
  • 47.14.3.2 Create Concept Sliders training pipeline: train paired LoRAs representing opposite ends of semantic attribute (age+/age-, lighting-hard/lighting-soft) for continuous control along semantic axes
  • 47.14.3.3 Implement multi-LoRA weight interpolation: blend N LoRAs with configurable per-LoRA weights for combined style/concept application with real-time weight adjustment
  • 47.14.3.4 Create LoRA conflict detection: identify when multiple LoRAs modify same attention layers with opposing directions → warn or auto-resolve via orthogonal projection
  • 47.14.3.5 Implement LoRA composition visualization: show per-layer weight magnitude contribution from each active LoRA for debugging composition artifacts and identifying layer conflicts
  • 47.14.3.6 Create LoRA registry with semantic tags: catalog all available LoRAs with metadata (style, concept, base model, rank, architecture type) for searchable selection and compatibility checking

47.15 @nous/inference-acceleration — Diffusion Inference Acceleration#

Consistency Models (ICLR 2025 simplified training) enable 1-4 step generation without diffusion. DeepCache and PAB provide training-free 2-3x acceleration. StreamDiffusion enables real-time interactive generation at 100+ FPS. DemoFusion/HiDiffusion achieve higher resolution without additional training. SenseFlow/SCoT provide flow-matching-native distillation for FLUX/SD3.

47.15.1 Consistency Models & Few-Step Generation#
  • 47.15.1.1 Implement Consistency Model training pipeline with simplified training recipe (Consistency Models Made Easy, ICLR 2025) removing need for EMA teacher and progressive schedule
  • 47.15.1.2 Create Consistency Model inference: 1-step, 2-step, and 4-step generation modes with quality/speed tradeoff selection and automatic step count recommendation
  • 47.15.1.3 Implement Latent Consistency Model (LCM) LoRA: train consistency distillation as LoRA adapter applicable to any base model for 2-4 step inference without full model fine-tuning
  • 47.15.1.4 Create LCM-LoRA application to FLUX/SD3: few-step inference on flow matching models via consistency distillation adapter with flow-matching-specific training adjustments
  • 47.15.1.5 Implement SCoT (Straight-Consistent Trajectories): unified consistency models + rectified flow for straight and consistent sampling paths reducing discretization error
  • 47.15.1.6 Create few-step generation quality benchmark: compare output quality at 1/2/4/8/50 steps for Consistency Model, LCM-LoRA, and standard scheduler using FID, CLIP, and human preference
  • 47.15.1.7 Implement progressive few-step refinement: start with 1-step preview → refine to 4-step → optionally run full 50-step for final quality with progressive preview display
  • 47.15.1.8 Create consistency training data pipeline: teacher model generates paired trajectories for consistency distillation with configurable trajectory sampling and noise schedule
47.15.2 Training-Free Inference Acceleration#
  • 47.15.2.1 Implement DeepCache training-free acceleration: cache and reuse deep features across denoising steps for 2x speedup without quality loss via feature similarity detection
  • 47.15.2.2 Create PAB (Pyramid Attention Broadcast) for video diffusion: broadcast attention across frames at varying granularity for 2-3x video generation speedup with quality preservation
  • 47.15.2.3 Implement FreeU skip-connection feature boosting: amplify backbone features and attenuate skip features for improved quality at zero additional cost with configurable scale factors per U-Net stage
  • 47.15.2.4 Create Token Merging (ToMe) for diffusion: merge redundant tokens in transformer-based diffusion models (FLUX, SD3) for inference speed improvement via bipartite soft matching
  • 47.15.2.5 Implement FORA (Attention Reuse) for temporal diffusion acceleration: reuse attention computation across similar denoising steps based on attention map similarity thresholds
  • 47.15.2.6 Create SageAttention integration for faster attention computation in DiT models (FLUX, SD3) with quantized attention kernels
  • 47.15.2.7 Implement combined acceleration stack: DeepCache + FreeU + ToMe applied simultaneously with conflict-free composition and automatic compatibility validation
  • 47.15.2.8 Create acceleration benchmark: measure throughput (images/second), quality impact (FID delta), and VRAM savings for each technique and combinations on standardized hardware
47.15.3 Real-Time Interactive Diffusion#
  • 47.15.3.1 Implement StreamDiffusion real-time generation pipeline achieving 100+ FPS for interactive image generation via stream batch processing and residual denoising
  • 47.15.3.2 Create StreamDiffusion input-output streaming: continuous image input → continuous image output with minimal frame latency via pipeline parallelism
  • 47.15.3.3 Implement StreamDiffusion residual denoising: denoising with warm-start from previous frame's latent for temporal coherence in streaming and reduced per-frame compute
  • 47.15.3.4 Create real-time ControlNet with StreamDiffusion: webcam depth/pose input → real-time controlled image generation at interactive frame rates
  • 47.15.3.5 Implement real-time style transfer via StreamDiffusion: live camera feed → styled output at interactive frame rates with configurable style strength
  • 47.15.3.6 Create StreamDiffusion VRAM optimization: minimal memory footprint for deployment on consumer GPUs (8GB VRAM target) via aggressive tensor offloading and precision management
  • 47.15.3.7 Implement StreamDiffusion batch-of-one optimization: single-image latency optimization for responsive interactive applications with pre-warmed pipeline state
  • 47.15.3.8 Create StreamDiffusion integration with @nous/image-control: connect OminiControl/ControlNeXt conditions to StreamDiffusion pipeline for real-time controlled generation
47.15.4 High-Resolution Generation#
  • 47.15.4.1 Implement DemoFusion progressive upscaling: generate at native resolution → progressively upscale with patch-based diffusion maintaining global coherence via dilated sampling
  • 47.15.4.2 Create HiDiffusion training-free high-resolution generation: modified attention and convolution patterns enabling 2048-4096px output from 1024px models without additional training
  • 47.15.4.3 Implement tiled diffusion with overlap blending: divide high-resolution canvas into overlapping tiles → denoise independently → blend at seams with configurable overlap ratio and blending strategy
  • 47.15.4.4 Create multi-scale generation pipeline: coarse generation at low resolution → detail refinement at high resolution with consistency enforcement via cross-scale feature alignment
  • 47.15.4.5 Implement resolution-adaptive inference: automatically select generation strategy based on target resolution (native for ≤1024px, DemoFusion for ≤4096px, tiled for larger) with quality estimation
  • 47.15.4.6 Create high-resolution quality validation: detect tile seam artifacts, check global coherence at full resolution, measure detail preservation vs native resolution output
47.15.5 Flow Matching Distillation#
  • 47.15.5.1 Implement SenseFlow large-scale flow matching distillation for FLUX/SD3.5 with Implicit Distribution Alignment for few-step generation preserving flow matching properties
  • 47.15.5.2 Create SCoT training pipeline: unify consistency models and rectified flow for straight-consistent sampling trajectories with configurable trajectory straightness constraint
  • 47.15.5.3 Implement score distillation for flow matching: distill flow matching models into fewer-step variants preserving flow ODE properties via progressive distillation
  • 47.15.5.4 Create flow matching scheduler optimization: evaluate and tune ODE solvers (Euler, midpoint, Heun) for optimal quality-speed tradeoff on FLUX with per-solver quality profiling
  • 47.15.5.5 Implement distilled model quality comparison: FID, CLIP score, and human preference evaluation of distilled vs full-step FLUX models at 1/2/4/8 step counts
  • 47.15.5.6 Create distilled model deployment: export distilled FLUX variants for production serving with optimized inference configuration, quantization, and batching parameters

47.16 @nous/joint-av-generation — Joint Audio-Video Generation & Physics-Aware Synthesis#

Seedance 2.0 (ByteDance, Feb 2026) is the first model to generate synchronized audio AND video in a single diffusion pass using a dual-branch MMDiT with Temporal-Aligned Cross-Attention (TA-CrossAttn). This is fundamentally different from sequential pipelines (generate video → add audio, covered in 47.12). Joint generation produces inherently synchronized audio-video with no post-hoc alignment needed. This section also covers physics-aware video generation — training objectives that penalize physically implausible motion (gravity, fluid dynamics, fabric draping, collisions) — distinct from physics-aware removal (47.7 VOID).

47.16.1 Joint Audio-Video Diffusion Architecture#
  • 47.16.1.1 Create @nous/joint-av-generation library with project.json, package.json, tsconfig.json, and barrel exports for joint audio-video generation infrastructure
  • 47.16.1.2 Implement dual-branch MMDiT architecture: separate visual and auditory processing pathways with cross-attention synchronization layers enabling independent feature extraction with shared temporal context
  • 47.16.1.3 Create Temporal-Aligned Cross-Attention (TA-CrossAttn) module: synchronize audio and video features across different temporal granularities (audio sample rate vs video frame rate) with learnable alignment offsets
  • 47.16.1.4 Implement unified latent space encoder: project text, image, video, and audio embeddings into shared representation for cross-modal interaction during the diffusion process
  • 47.16.1.5 Create joint denoising pipeline: simultaneous denoising of video and audio latents with cross-modal conditioning at each denoising step, ensuring both modalities converge coherently
  • 47.16.1.6 Implement audio-video temporal alignment validation during generation: measure frame-level synchronization during denoising, not just on final output, enabling early termination of misaligned generations
  • 47.16.1.7 Create joint generation output decoder: split unified latent back into video frames + audio waveform with preserved temporal alignment and modality-specific post-processing
  • 47.16.1.8 Implement joint vs sequential A/B comparison framework: compare joint generation output against sequential pipeline (47.12 video-to-audio) on sync quality, naturalness, and perceptual coherence metrics
47.16.2 Multi-Asset Conditioning Framework#
  • 47.16.2.1 Implement multi-modal reference encoder accepting simultaneous image, video, audio, and text references (up to 12 assets) as conditioning signals with per-asset embedding extraction
  • 47.16.2.2 Create reference attribution router: parse natural language instructions ("use @ref1's style with @ref2's motion") → route each reference to appropriate conditioning pathway via instruction parsing
  • 47.16.2.3 Implement composition extraction from reference video: decompose into camera movement, subject motion, scene dynamics, and pacing as independent conditioning signals for selective reuse
  • 47.16.2.4 Create style extraction from reference images: isolate color palette, composition rules, lighting characteristics as conditioning vectors applicable across generated shots
  • 47.16.2.5 Implement audio reference conditioning: extract tempo, mood, instrumentation from reference audio → condition generated soundtrack to match tonal and rhythmic characteristics
  • 47.16.2.6 Create reference conflict resolution: when multiple references provide contradictory signals (different styles, competing motions), resolve via priority weighting and explicit user-defined precedence rules
  • 47.16.2.7 Implement reference contribution visualization: show how each reference asset influences the final output via attention map decomposition and per-reference attribution scores
  • 47.16.2.8 Create reference template system: save successful multi-asset combinations as reusable conditioning templates with parameterized overrides
47.16.3 Scene Planning & Multi-Shot Generation#
  • 47.16.3.1 Implement narrative decomposition engine: parse text prompt → identify characters, locations, actions, emotional arc → plan multi-shot sequence with scene graph representation
  • 47.16.3.2 Create shot planning module: determine shot types (wide, medium, close-up), durations, and transitions for each scene segment based on narrative analysis and cinematic conventions
  • 47.16.3.3 Implement automatic camera language planning: select appropriate camera movements (dolly, crane, tracking, POV) based on scene intent and emotional tone with cinematography-informed heuristics
  • 47.16.3.4 Create multi-shot consistency enforcement: ensure character identity, wardrobe, lighting, and spatial layout remain consistent across planned shots using cross-shot feature anchoring
  • 47.16.3.5 Implement transition planning: select cut types (hard cut, dissolve, match cut, L-cut) based on narrative pacing and emotional flow with configurable editing style presets
  • 47.16.3.6 Create shot plan preview renderer: generate low-resolution previews of each planned shot for review before committing to full resolution generation
  • 47.16.3.7 Implement shot plan editing: allow human modification of auto-generated shot plan (reorder shots, change camera, adjust duration) before generation with real-time preview updates
  • 47.16.3.8 Create multi-shot output assembly: stitch individually generated shots with transitions into seamless final video with continuous audio, handling cross-shot audio continuity and transition blending
47.16.4 Physics-Aware Video Generation#
  • 47.16.4.1 Implement physics-aware training objectives: loss functions penalizing physically implausible motion during video generation (gravity violations, impossible collisions, non-physical deformations) as auxiliary training signals
  • 47.16.4.2 Create gravity simulation awareness: generated falling objects, jumping characters, and thrown projectiles must follow gravitationally consistent trajectories with configurable gravity magnitude and direction
  • 47.16.4.3 Implement fluid dynamics awareness: water splashes, pouring liquids, smoke, and fire must follow physically plausible flow patterns informed by Navier-Stokes approximations in latent space
  • 47.16.4.4 Create fabric and material dynamics: clothing, curtains, hair, and flexible materials must drape, swing, and deform according to material properties and applied forces with material-type conditioning
  • 47.16.4.5 Implement collision and contact physics: objects must not interpenetrate, impacts must produce appropriate deformation, and contact forces must be consistent with object mass and velocity
  • 47.16.4.6 Create inertia and momentum preservation: moving objects must accelerate, decelerate, and transfer momentum plausibly during interactions with conservation-of-momentum constraints in motion prediction
  • 47.16.4.7 Implement physics plausibility scoring: automated evaluation of generated video for physics violations using motion analysis, trajectory prediction, and optical flow consistency metrics
  • 47.16.4.8 Create physics-conditioned generation pipeline: accept physics constraints as explicit conditioning (material properties, gravity direction, wind forces) alongside text prompts for physically grounded generation