nous library

The Concordia agreement-search substrate (libs/nous/agreement-search; phase:179, layer:domain). src/index.ts describes it as the "search-kernel registry, case-profile ranking, and cost/capability descriptors" for Phase 179.4.2. It implements a registry (kernel-registry.ts) plus a genuinely large set of optimization kernels — Nash-genetic (nash-genetic.ts), NSGA-II with fast-non-dominated-sort and crowding distance (nsga-ii.ts), CP-SAT (cp-sat.ts), MCTS-LATS (mcts-lats.ts), Bayesian optimization with EI/UCB/PI acquisition (bayesian-optimization.ts), PSRO opponent modelling (psro-opponent-model.ts), MAP-Elites (map-elites.ts), and coalition-stability analysis with exact and Monte-Carlo Shapley values plus least-core allocation (coalition-stability.ts). Around the search it adds candidate generation/ filtering/mutation (clause-mutators.ts, proposal-decomposition.ts, domain playbooks), auditing (spec-gaming-audit.ts, coercive-choice-audit.ts), and explanation rendering (shared-neutral, party-private, Pareto). Real, expert code.

Concordia privacy primitives (libs/nous/concordia-sealed-memory; phase:179, layer:domain). Per its src/index.ts it provides §179.5.1.2 sealed memory and §179.5.1.3 zero-retention routing. sealed-memory-store.ts implements real envelope encryption using node:crypto: a per-party 256-bit DEK is wrapped by a per-case KEK through a pluggable KmsProvider, every artifact is AES-256-GCM encrypted with a fresh IV and case/party/artifact AAD binding, DEK rotation re-encrypts and zeroes old key material, and every operation passes an AuthorizeFn guard and emits an AuditRecord. zero-retention-mode.ts adds local-model endpoint selection with HMAC retention attestations, and confidential-compute.ts adds TEE-attested confidential scoring. Real security-grade implementation.

The session-scoped bargaining substrate (libs/nous/cooperative-bargaining; phase:179, layer:domain). Small (two source modules) but real: bargaining-session.ts defines a BargainingSession runtime container with a Zod phase state machine (intake → preference_inference → generation → scoring → frontier_stable → review_pending → awaiting_acceptance → closed), stakeholder authority, budgets, closure reasons, and acceptance logic (allStakeholdersCanAccept, stakeholdersBlockingAcceptance). Its docstring states its purpose plainly: bridge the @concordia/contracts case model to the §179.4 search kernels.

Concordia preference inference (libs/nous/preference-inference; phase:179, layer:domain, ~24 modules). src/index.ts documents five calibrated utility-model estimators sharing a UtilityModelFit interface: Bradley-Terry, Thurstone-Mosteller, Plackett-Luce, a Gaussian-process preference model (Chu-Ghahramani 2005), and a neural utility ranker. bradley-terry.ts is genuine numerical code — MAP estimation under a Gaussian prior by Newton-Raphson with backtracking line search and a Laplace-approximation posterior covariance for credible intervals, with Davidson-style tie handling. It adds pairwise elicitation planning, strict-JSON comparison prompts, calibration metrics (ECE/Brier/stability), BATNA modelling, fairness metrics/profiles, and an abstention gate, adapting to the @concordia/contracts UtilityModel.

Advanced TTS, voice, and music-generation pipelines (libs/nous/advanced-speech, package @nous/advanced-speech; layer:domain). Its ~31 modules wrap named SOTA systems: F5-TTS / CosyVoice / Fish-Audio S2-Pro / VoiceCraft model loaders and editing pipelines (f5-tts-model-loader.ts, voicecraft-*), cross-TTS voice cloning, real-time voice conversion and style transfer, emotion/prosody control, and a MusicGen / MusiConGen family (multi-track generation, activation steering, chord/rhythm conditioning, style transfer). The barrel src/index.ts re-exports each module; the implementations are deterministic orchestration around model seams rather than the raw model weights.

A thin facade barrel (libs/nous/diffusion-alignment, package @nous/diffusion-alignment; layer:domain). Its single src/index.ts declares NOUS_DIFFUSION_ALIGNMENT_PHASE = '47.14' and a NOUS_DIFFUSION_ALIGNMENT_CAPABILITIES manifest (diffusion-DPO, curriculum-DPO, rich/step-aware preference optimization, Flux-DPO, LyCORIS, DoRA, LoKr, OFT, BOFT, multi-LoRA composition, concept sliders), then re-exports the corresponding implementation symbols (DiffusionDPOTraining, LyCORISFrameworkIntegration, ConceptSlidersTraining, …) and their request types from @nous/training. It owns no implementation of its own — it is a curated public face over the training library's alignment family.

Generative relighting and 3D-Gaussian-Splatting editing pipelines (libs/nous/generative-relighting; layer:domain, ~23 modules). The barrel exposes IC-Light foreground/background relighting and model loading, LumiGauss material decomposition and relighting inference, GaussCtrl / SyncNoise / Ctrl-D / InterGSEdit 3DGS editing, Morpheus stylization, Text2Relight portrait relighting, RelightMaster / GenLit video relighting, environment-map conditioning, and multiview / temporal lighting-consistency validators (multiview-relighting-consistency-pipeline.ts, temporal-lighting-consistency-pipeline.ts). Deterministic orchestration plus quality-metric scoring around the relighting model seams.

Controllable image generation and editing (libs/nous/image-control; layer:domain, ~50 modules). It collects ControlNet++ (cycle-consistency and training pipelines), ControlNeXt (image + video extension), Depth-Anything-v2, Grounding-DINO text segmentation, GLIGEN, BoxDiff layout control, Attend-and-Excite semantic binding, InstructPix2Pix, DragDiffusion, differential/ dense-diffusion region control, concept sliders, and a layout-to-image stack (layout-to-image-pipeline.ts, interactive-layout-editor-api.ts, layout-validation-system.ts). A control-condition-preprocessor-registry.ts and a control-backend-comparison-harness.ts tie the conditioning backends together.

A thin facade barrel (libs/nous/inference-acceleration, package @nous/inference-acceleration; layer:domain), the sibling of nous-diffusion-alignment. Its single src/index.ts declares NOUS_INFERENCE_ACCELERATION_PHASE = '47.15' and a NOUS_INFERENCE_ACCELERATION_CAPABILITIES manifest (consistency models, LCM-LoRA, DeepCache, pyramid-attention-broadcast, FreeU, token-merging, FORA, SageAttention, StreamDiffusion, DemoFusion, HiDiffusion, tiled-diffusion, flow-matching distillation), then re-exports the matching classes and create* factories and request types from @nous/training. No local implementation — a phase-scoped public surface over training's acceleration family.

Joint (simultaneous) audio-video generation (libs/nous/joint-av-generation; layer:domain, ~34 modules). src/index.ts leads with a library metadata manifest (joint-av-generation-library.ts returns package/project/tag metadata), then exposes a dual-branch MMDiT architecture, unified latent-space and multimodal-reference encoders, a joint denoising pipeline with temporal-alignment validation, narrative decomposition, a camera-language planner, multi-shot consistency/assembly, and a physics-awareness suite (gravity, fluid, fabric, collision-contact, inertia/momentum) plus physics-aware training objectives. It also ships a joint-vs-sequential A/B comparison framework.

Portrait / talking-head animation and dubbing (libs/nous/portrait-animation; layer:domain, ~23 modules). A portrait-animation-library.ts manifest leads, followed by Hallo3 (video-DiT loader + inference), LivePortrait (expression transfer + stitching), IM-Portrait 3D-aware video diffusion, EchoMimic audio-to- facial-motion, MuseTalk video dubbing, ChatAnyone / RAP real-time audio-driven pipelines, multi-person and multi-style variants, expression-amplitude control, a GPU-batching orchestrator, streaming output, and a cross-language dubbing stack with quality validation. Orchestration plus quality metrics around the animation model seams.

Controllable video generation (libs/nous/video-control; layer:domain, ~43 modules). It centres on camera and motion control: a Wan-VACE model loader, camera-trajectory specification format / preset library / visualization workbench, Motion-Prompting (object, camera, and simultaneous control), MotionStream real-time motion control (interactive painting, motion transfer, GPU-memory optimization, quality/latency controls), DepthDirector depth- conditioned camera control, PostCam novel-view generation, HunyuanCustom multi- subject generation with subject-identity preservation and temporal coherence, and editing pipelines (AnyV2V, TokenFlow, keyframe-based) with temporal-consistency scoring.

Object removal / video inpainting (libs/nous/video-removal; layer:domain, ~48 modules) — and the one library here with a deployable out-of-process backend. The TS surface implements the VOID/quadmask pipeline: SAM2 video segmentation and mask refinement, grey-mask generation (grey-mask-generation.ts materializes either a deterministic bbox-seeded-preview or a runtime-sam2 mask, driven by a Gemini physics-consequence analyzer), quadmask encoding/compositing/validation, DiffuEraser / CogVideoX-Fun / STTN backend integrations, edge-artifact and perceptual-quality scoring, a fallback chain, and per-production fine-tuning. It additionally ships a real RunPod serverless Python backend under docker/ (serverless/handlers/void_inference.py, Dockerfile, config/quadmask_cogvideox.py) and a runpod-serverless-endpoint.ts client, plus a Playwright human-evaluation spec.

Video-to-audio (V2A) and Foley generation (libs/nous/video-to-audio; layer:domain, ~21 modules — the smallest media library). Led by a video-to-audio-library.ts manifest, it implements two model families — Any2Audio (model loader; text-to-audio, image-to-audio, audio-editing, and multimodal-conditioning pipelines; quality metrics) and AV-Link (loader + inference) — plus a FoleyCrafter generation pipeline with foley material detection, intensity control, library augmentation, and temporal-precision scoring, a temporal-alignment scorer, a multitrack generator, and an audio-video sync quality validator.

The audio / speech / music DSP and analysis engine (libs/nous/audio, package @nous/audio; layer:infra). With ~81 modules it spans an ASR engine (asr-engine.ts), enhancement/restoration/denoising/dereverberation/declipping, classification, tagging, fingerprinting and forensics, scene classification, super-resolution and bandwidth extension, plus music-side analysis: beat detection with real onset/tempo logic (beat-detection.ts computes onset frames, beat candidates, and instantaneous tempo), chord recognition, accompaniment generation, and audiobook generation. The barrel src/index.ts re-exports the full surface.

The model-inference runtime (libs/nous/core, package @nous/core; layer:infra) — a large infra engine at ~84 modules. src/index.ts re-exports execution providers for every backend (CPU, CUDA, ROCm, Metal, WebGPU, Vulkan, DirectML, NPU), graph optimization, operator fusion, real quantization (quantization.ts implements fp16/int8/int4 with computed scale/zero-point and calibration ranges), memory planning, dynamic and continuous batching, speculative decoding, KV-cache, paged/flash/radix attention, tensor/pipeline parallelism and model sharding, plus an embeddings stack (ANN, cross-encoder reranking, compression, binary embeddings). It is the serving foundation the rest of the area builds on.

Vendor-neutral LLM orchestration (libs/nous/llm, package @nous/llm; layer:infra, ~79 modules). chat-completion-api.ts / completion-api.ts define provider interfaces (ChatCompletionProvider.complete(request)), a registry with default + fallback provider chains and per-provider stats, and model-support filtering — it binds no specific vendor, so callers register their own provider implementations. On top of that it implements structured output with a validation-retry loop (structured-output.ts), function calling, JSON/grammar/ regex/schema constraints, an agent stack (architecture, memory, coordination, delegation, supervision), reasoning strategies (chain/tree-of-thought, backtracking), a RAG stack (chunking, retrieval, citation extraction), and deterministic NLP extraction utilities (entity/fact/code/citation parsing).

An empty scaffold (libs/nous/platform, package @nous/platform; layer:infra). The project has a project.json (with build/lint/test/typecheck targets pointing at src/index.ts) plus tsconfig* and vitest.config.ts, but no src/ directory or source files exist — zero implementation modules. It is a reserved slot in the catalog, not a working library, and is shown as such rather than overclaimed.

Trust & safety (libs/nous/safety, package @nous/safety; layer:infra, ~64 modules). It implements abuse / harassment / hate-speech / bot / coordinated- behaviour / influence-operation detection, a shared content-classifier.ts, adversarial-input and code-injection detection, deepfake detection (deepfake-detection.ts combines a weighted-signal classifier with optional forensics inputs), hallucination and factuality checking, confidence estimation and calibration analysis, content fingerprinting, attribution analysis, and the operational workflow side — audit trails, approval workflows, escalation, and human-in-the-loop. Deterministic signal-and-threshold classifiers rather than hosted models.

The training and fine-tuning stack (libs/nous/training, package @nous/training; layer:infra) — the largest library in the area at ~138 modules, and the implementation backing the two facade libraries. It covers distributed training and every parallelism axis (data/model/tensor/pipeline, ZeRO), mixed-precision and gradient checkpointing/accumulation, optimizers with real math (adamw-implementation.ts computes bias-corrected moments, global grad-norm clipping with clipScale, AMSGrad; plus LAMB, Adafactor, 8-bit), fine-tuning (SFT, instruction/chat tuning, LoRA/QLoRA/adapter/prefix/prompt), RLHF/alignment (Bradley-Terry reward modelling, best-of-N, constitutional AI), and the diffusion DPO/alignment and inference-acceleration families that nous-diffusion-alignment and nous-inference-acceleration re-export.

Computer vision and image generation (libs/nous/vision, package @nous/vision; layer:infra) — the broadest perception library at ~103 modules. src/index.ts re-exports recognition (action, age/gender, emotion, facial recognition/ reenactment/restoration, landmarks), analysis (anomaly detection, crowd analysis, dense video captioning, depth estimation, document layout), and a generation stack (a diffusion-model framework, ControlNet integration, DreamBooth training, depth/edge-guided generation, consistent character/style batches, animation generation, frame interpolation, FLUX support). It sits alongside nous-core as shared infra for any pipeline needing perception or image synthesis.