# Nous — Systems Deep Dive

> The `libs/nous/` area: ~21 Nx libraries that make up Oshun's in-house AI/ML
> platform layer — inference runtime, training stack, LLM orchestration, vision,
> audio and speech, a fleet of controllable generative-media pipelines, and the
> Phase-179 negotiation/privacy substrate that powers Concordia.

## What this area is

"Nous" (Greek for _mind / intellect_) is the monorepo's model-and-media compute
layer. Unlike the `libs/contracts/` area — which is wire types only — every node
here carries real TypeScript implementation: typed runtime kernels,
deterministic algorithms, and orchestration logic. The libraries split into
three families visible in their Nx `tags`:

- **`layer:infra` foundation libraries** (`nous-core`, `nous-training`,
  `nous-llm`, `nous-vision`, `nous-audio`, `nous-safety`, plus the empty
  `nous-platform` slot). These are the broad, heavily-modular platform engines —
  `nous-core` alone has ~84 source modules and `nous-training` ~138 — covering
  model serving, fine-tuning, LLM orchestration, perception, audio DSP, and
  trust & safety.
- **`layer:domain` generative-media libraries** (`nous-advanced-speech`,
  `nous-image-control`, `nous-video-control`, `nous-video-removal`,
  `nous-video-to-audio`, `nous-portrait-animation`,
  `nous-generative-relighting`, `nous-joint-av-generation`, and the two thin
  facades `nous-diffusion-alignment` / `nous-inference-acceleration`). Each is a
  curated collection of named SOTA pipelines (ControlNeXt, LivePortrait,
  IC-Light, HunyuanCustom, FoleyCrafter, …) implemented as deterministic
  TypeScript orchestration around typed model/backend seams.
- **The `phase:179` Concordia cluster** (`@nous/agreement-search`,
  `@nous/preference-inference`, `@nous/cooperative-bargaining`,
  `@nous/concordia-sealed-memory`). This is a tight, expert subsystem of
  optimization, preference-learning, and privacy primitives that the Concordia
  mediation domain builds on. These four are the only nous libraries that import
  `@concordia/contracts`.

## How the implementations are shaped

Across the generative-media libraries the recurring pattern is _deterministic
orchestration over an injectable model boundary_: the CPU-side logic (planning,
trajectory specs, quality scoring, mask compositing, preview materialization,
manifest assembly) is real and fully implemented, while the heavy neural
inference is reached through a typed provider/backend interface rather than
hardcoded. `nous-llm` is the clearest example — it defines a
`ChatCompletionProvider`/`CompletionProvider` interface (`complete(request)`), a
provider registry with fallback chains and per-provider stats, and a
validation-retry structured-output loop, but binds no specific vendor. The
Phase-179 cluster is different in character: it is dense numerical code
(Newton-Raphson MAP estimation, NSGA-II, CP-SAT, Shapley values, AES-256-GCM
envelope encryption) with deterministic seeds and known-answer behaviour.

Two nodes are honestly partial. `nous-diffusion-alignment` and
`nous-inference-acceleration` are thin **facade barrels**: each is a single
`index.ts` exposing a phase-tagged capability manifest and re-exporting a
curated subset of symbols from `@nous/training`. One node, `nous-platform`, is
an **empty scaffold** with no source at all.

## How it fits the wider system

The infra libraries (`@nous/core`, `@nous/training`, `@nous/llm`,
`@nous/vision`, `@nous/audio`, `@nous/safety`) are the shared substrate the
domain media libraries and downstream Oshun services compose against — note the
package names (`@nous/*`) differ from the Nx project names (`nous-*`). The
generative-media libraries are consumed by content/creative pipelines that need
a specific controllable capability (relighting, V2A foley, portrait dubbing,
object removal). The Phase-179 cluster is consumed by **Concordia**: it produces
and ranks candidate agreements, infers party utilities, runs the
bargaining-session state machine, and seals per-party private memory — all typed
against `@concordia/contracts` so the optimizer and the contract surface stay in
lockstep. `nous-video-removal` additionally ships a real **RunPod serverless
Python backend** (`docker/serverless/handlers/void_inference.py` +
`Dockerfile`), making it the one library here with a deployable out-of-process
inference target. Walk the dependency edges on any node below to see its exact
consumers.

## Entity reference

### nous-advanced-speech

Advanced TTS, voice, and music-generation pipelines
(`libs/nous/advanced-speech`, package `@nous/advanced-speech`; `layer:domain`).
Its ~31 modules wrap named SOTA systems: F5-TTS / CosyVoice / Fish-Audio S2-Pro
/ VoiceCraft model loaders and editing pipelines (`f5-tts-model-loader.ts`,
`voicecraft-*`), cross-TTS voice cloning, real-time voice conversion and style
transfer, emotion/prosody control, and a MusicGen / MusiConGen family
(multi-track generation, activation steering, chord/rhythm conditioning, style
transfer). The barrel `src/index.ts` re-exports each module; the implementations
are deterministic orchestration around model seams rather than the raw model
weights.

### @nous/agreement-search

The Concordia agreement-search substrate (`libs/nous/agreement-search`;
`phase:179`, `layer:domain`). `src/index.ts` describes it as the "search-kernel
registry, case-profile ranking, and cost/capability descriptors" for Phase
179.4.2. It implements a registry (`kernel-registry.ts`) plus a genuinely large
set of optimization kernels — Nash-genetic (`nash-genetic.ts`), NSGA-II with
fast-non-dominated-sort and crowding distance (`nsga-ii.ts`), CP-SAT
(`cp-sat.ts`), MCTS-LATS (`mcts-lats.ts`), Bayesian optimization with EI/UCB/PI
acquisition (`bayesian-optimization.ts`), PSRO opponent modelling
(`psro-opponent-model.ts`), MAP-Elites (`map-elites.ts`), and
coalition-stability analysis with exact and Monte-Carlo Shapley values plus
least-core allocation (`coalition-stability.ts`). Around the search it adds
candidate generation/ filtering/mutation (`clause-mutators.ts`,
`proposal-decomposition.ts`, domain playbooks), auditing
(`spec-gaming-audit.ts`, `coercive-choice-audit.ts`), and explanation rendering
(shared-neutral, party-private, Pareto). Real, expert code.

### nous-audio

The audio / speech / music DSP and analysis engine (`libs/nous/audio`, package
`@nous/audio`; `layer:infra`). With ~81 modules it spans an ASR engine
(`asr-engine.ts`), enhancement/restoration/denoising/dereverberation/declipping,
classification, tagging, fingerprinting and forensics, scene classification,
super-resolution and bandwidth extension, plus music-side analysis: beat
detection with real onset/tempo logic (`beat-detection.ts` computes onset
frames, beat candidates, and instantaneous tempo), chord recognition,
accompaniment generation, and audiobook generation. The barrel `src/index.ts`
re-exports the full surface.

### @nous/concordia-sealed-memory

Concordia privacy primitives (`libs/nous/concordia-sealed-memory`; `phase:179`,
`layer:domain`). Per its `src/index.ts` it provides §179.5.1.2 sealed memory and
§179.5.1.3 zero-retention routing. `sealed-memory-store.ts` implements real
**envelope encryption** using `node:crypto`: a per-party 256-bit DEK is wrapped
by a per-case KEK through a pluggable `KmsProvider`, every artifact is
AES-256-GCM encrypted with a fresh IV and case/party/artifact AAD binding, DEK
rotation re-encrypts and zeroes old key material, and every operation passes an
`AuthorizeFn` guard and emits an `AuditRecord`. `zero-retention-mode.ts` adds
local-model endpoint selection with HMAC retention attestations, and
`confidential-compute.ts` adds TEE-attested confidential scoring. Real
security-grade implementation.

### @nous/cooperative-bargaining

The session-scoped bargaining substrate (`libs/nous/cooperative-bargaining`;
`phase:179`, `layer:domain`). Small (two source modules) but real:
`bargaining-session.ts` defines a `BargainingSession` runtime container with a
Zod phase state machine (`intake` → `preference_inference` → `generation` →
`scoring` → `frontier_stable` → `review_pending` → `awaiting_acceptance` →
`closed`), stakeholder authority, budgets, closure reasons, and acceptance logic
(`allStakeholdersCanAccept`, `stakeholdersBlockingAcceptance`). Its docstring
states its purpose plainly: bridge the `@concordia/contracts` case model to the
§179.4 search kernels.

### nous-core

The model-inference runtime (`libs/nous/core`, package `@nous/core`;
`layer:infra`) — a large infra engine at ~84 modules. `src/index.ts` re-exports
execution providers for every backend (CPU, CUDA, ROCm, Metal, WebGPU, Vulkan,
DirectML, NPU), graph optimization, operator fusion, real quantization
(`quantization.ts` implements fp16/int8/int4 with computed scale/zero-point and
calibration ranges), memory planning, dynamic and continuous batching,
speculative decoding, KV-cache, paged/flash/radix attention, tensor/pipeline
parallelism and model sharding, plus an embeddings stack (ANN, cross-encoder
reranking, compression, binary embeddings). It is the serving foundation the
rest of the area builds on.

### nous-diffusion-alignment

A thin **facade barrel** (`libs/nous/diffusion-alignment`, package
`@nous/diffusion-alignment`; `layer:domain`). Its single `src/index.ts` declares
`NOUS_DIFFUSION_ALIGNMENT_PHASE = '47.14'` and a
`NOUS_DIFFUSION_ALIGNMENT_CAPABILITIES` manifest (diffusion-DPO, curriculum-DPO,
rich/step-aware preference optimization, Flux-DPO, LyCORIS, DoRA, LoKr, OFT,
BOFT, multi-LoRA composition, concept sliders), then re-exports the
corresponding implementation symbols (`DiffusionDPOTraining`,
`LyCORISFrameworkIntegration`, `ConceptSlidersTraining`, …) and their request
types **from `@nous/training`**. It owns no implementation of its own — it is a
curated public face over the training library's alignment family.

### nous-generative-relighting

Generative relighting and 3D-Gaussian-Splatting editing pipelines
(`libs/nous/generative-relighting`; `layer:domain`, ~23 modules). The barrel
exposes IC-Light foreground/background relighting and model loading, LumiGauss
material decomposition and relighting inference, GaussCtrl / SyncNoise / Ctrl-D
/ InterGSEdit 3DGS editing, Morpheus stylization, Text2Relight portrait
relighting, RelightMaster / GenLit video relighting, environment-map
conditioning, and multiview / temporal lighting-consistency validators
(`multiview-relighting-consistency-pipeline.ts`,
`temporal-lighting-consistency-pipeline.ts`). Deterministic orchestration plus
quality-metric scoring around the relighting model seams.

### nous-image-control

Controllable image generation and editing (`libs/nous/image-control`;
`layer:domain`, ~50 modules). It collects ControlNet++ (cycle-consistency and
training pipelines), ControlNeXt (image + video extension), Depth-Anything-v2,
Grounding-DINO text segmentation, GLIGEN, BoxDiff layout control,
Attend-and-Excite semantic binding, InstructPix2Pix, DragDiffusion,
differential/ dense-diffusion region control, concept sliders, and a
layout-to-image stack (`layout-to-image-pipeline.ts`,
`interactive-layout-editor-api.ts`, `layout-validation-system.ts`). A
`control-condition-preprocessor-registry.ts` and a
`control-backend-comparison-harness.ts` tie the conditioning backends together.

### nous-inference-acceleration

A thin **facade barrel** (`libs/nous/inference-acceleration`, package
`@nous/inference-acceleration`; `layer:domain`), the sibling of
`nous-diffusion-alignment`. Its single `src/index.ts` declares
`NOUS_INFERENCE_ACCELERATION_PHASE = '47.15'` and a
`NOUS_INFERENCE_ACCELERATION_CAPABILITIES` manifest (consistency models,
LCM-LoRA, DeepCache, pyramid-attention-broadcast, FreeU, token-merging, FORA,
SageAttention, StreamDiffusion, DemoFusion, HiDiffusion, tiled-diffusion,
flow-matching distillation), then re-exports the matching classes and `create*`
factories and request types **from `@nous/training`**. No local implementation —
a phase-scoped public surface over training's acceleration family.

### nous-joint-av-generation

Joint (simultaneous) audio-video generation (`libs/nous/joint-av-generation`;
`layer:domain`, ~34 modules). `src/index.ts` leads with a library metadata
manifest (`joint-av-generation-library.ts` returns package/project/tag
metadata), then exposes a dual-branch MMDiT architecture, unified latent-space
and multimodal-reference encoders, a joint denoising pipeline with
temporal-alignment validation, narrative decomposition, a camera-language
planner, multi-shot consistency/assembly, and a physics-awareness suite
(gravity, fluid, fabric, collision-contact, inertia/momentum) plus physics-aware
training objectives. It also ships a joint-vs-sequential A/B comparison
framework.

### nous-llm

Vendor-neutral LLM orchestration (`libs/nous/llm`, package `@nous/llm`;
`layer:infra`, ~79 modules). `chat-completion-api.ts` / `completion-api.ts`
define provider interfaces (`ChatCompletionProvider.complete(request)`), a
registry with default + fallback provider chains and per-provider stats, and
model-support filtering — it binds no specific vendor, so callers register their
own provider implementations. On top of that it implements structured output
with a validation-retry loop (`structured-output.ts`), function calling,
JSON/grammar/ regex/schema constraints, an agent stack (architecture, memory,
coordination, delegation, supervision), reasoning strategies
(chain/tree-of-thought, backtracking), a RAG stack (chunking, retrieval,
citation extraction), and deterministic NLP extraction utilities
(entity/fact/code/citation parsing).

### nous-platform

An **empty scaffold** (`libs/nous/platform`, package `@nous/platform`;
`layer:infra`). The project has a `project.json` (with build/lint/test/typecheck
targets pointing at `src/index.ts`) plus `tsconfig*` and `vitest.config.ts`, but
**no `src/` directory or source files exist** — zero implementation modules. It
is a reserved slot in the catalog, not a working library, and is shown as such
rather than overclaimed.

### nous-portrait-animation

Portrait / talking-head animation and dubbing (`libs/nous/portrait-animation`;
`layer:domain`, ~23 modules). A `portrait-animation-library.ts` manifest leads,
followed by Hallo3 (video-DiT loader + inference), LivePortrait (expression
transfer + stitching), IM-Portrait 3D-aware video diffusion, EchoMimic audio-to-
facial-motion, MuseTalk video dubbing, ChatAnyone / RAP real-time audio-driven
pipelines, multi-person and multi-style variants, expression-amplitude control,
a GPU-batching orchestrator, streaming output, and a cross-language dubbing
stack with quality validation. Orchestration plus quality metrics around the
animation model seams.

### @nous/preference-inference

Concordia preference inference (`libs/nous/preference-inference`; `phase:179`,
`layer:domain`, ~24 modules). `src/index.ts` documents five calibrated
utility-model estimators sharing a `UtilityModelFit` interface: Bradley-Terry,
Thurstone-Mosteller, Plackett-Luce, a Gaussian-process preference model
(Chu-Ghahramani 2005), and a neural utility ranker. `bradley-terry.ts` is
genuine numerical code — MAP estimation under a Gaussian prior by Newton-Raphson
with backtracking line search and a Laplace-approximation posterior covariance
for credible intervals, with Davidson-style tie handling. It adds pairwise
elicitation planning, strict-JSON comparison prompts, calibration metrics
(ECE/Brier/stability), BATNA modelling, fairness metrics/profiles, and an
abstention gate, adapting to the `@concordia/contracts` `UtilityModel`.

### nous-safety

Trust & safety (`libs/nous/safety`, package `@nous/safety`; `layer:infra`, ~64
modules). It implements abuse / harassment / hate-speech / bot / coordinated-
behaviour / influence-operation detection, a shared `content-classifier.ts`,
adversarial-input and code-injection detection, deepfake detection
(`deepfake-detection.ts` combines a weighted-signal classifier with optional
forensics inputs), hallucination and factuality checking, confidence estimation
and calibration analysis, content fingerprinting, attribution analysis, and the
operational workflow side — audit trails, approval workflows, escalation, and
human-in-the-loop. Deterministic signal-and-threshold classifiers rather than
hosted models.

### nous-training

The training and fine-tuning stack (`libs/nous/training`, package
`@nous/training`; `layer:infra`) — the largest library in the area at ~138
modules, and the implementation backing the two facade libraries. It covers
distributed training and every parallelism axis (data/model/tensor/pipeline,
ZeRO), mixed-precision and gradient checkpointing/accumulation, optimizers with
real math (`adamw-implementation.ts` computes bias-corrected moments, global
grad-norm clipping with `clipScale`, AMSGrad; plus LAMB, Adafactor, 8-bit),
fine-tuning (SFT, instruction/chat tuning, LoRA/QLoRA/adapter/prefix/prompt),
RLHF/alignment (Bradley-Terry reward modelling, best-of-N, constitutional AI),
and the diffusion DPO/alignment and inference-acceleration families that
`nous-diffusion-alignment` and `nous-inference-acceleration` re-export.

### nous-video-control

Controllable video generation (`libs/nous/video-control`; `layer:domain`, ~43
modules). It centres on camera and motion control: a Wan-VACE model loader,
camera-trajectory specification format / preset library / visualization
workbench, Motion-Prompting (object, camera, and simultaneous control),
MotionStream real-time motion control (interactive painting, motion transfer,
GPU-memory optimization, quality/latency controls), DepthDirector depth-
conditioned camera control, PostCam novel-view generation, HunyuanCustom multi-
subject generation with subject-identity preservation and temporal coherence,
and editing pipelines (AnyV2V, TokenFlow, keyframe-based) with
temporal-consistency scoring.

### nous-video-removal

Object removal / video inpainting (`libs/nous/video-removal`; `layer:domain`,
~48 modules) — and the one library here with a deployable out-of-process
backend. The TS surface implements the VOID/quadmask pipeline: SAM2 video
segmentation and mask refinement, grey-mask generation
(`grey-mask-generation.ts` materializes either a deterministic
`bbox-seeded-preview` or a `runtime-sam2` mask, driven by a Gemini
physics-consequence analyzer), quadmask encoding/compositing/validation,
DiffuEraser / CogVideoX-Fun / STTN backend integrations, edge-artifact and
perceptual-quality scoring, a fallback chain, and per-production fine-tuning. It
additionally ships a **real RunPod serverless Python backend** under `docker/`
(`serverless/handlers/void_inference.py`, `Dockerfile`,
`config/quadmask_cogvideox.py`) and a `runpod-serverless-endpoint.ts` client,
plus a Playwright human-evaluation spec.

### nous-video-to-audio

Video-to-audio (V2A) and Foley generation (`libs/nous/video-to-audio`;
`layer:domain`, ~21 modules — the smallest media library). Led by a
`video-to-audio-library.ts` manifest, it implements two model families —
Any2Audio (model loader; text-to-audio, image-to-audio, audio-editing, and
multimodal-conditioning pipelines; quality metrics) and AV-Link (loader +
inference) — plus a FoleyCrafter generation pipeline with foley material
detection, intensity control, library augmentation, and temporal-precision
scoring, a temporal-alignment scorer, a multitrack generator, and an audio-video
sync quality validator.

### nous-vision

Computer vision and image generation (`libs/nous/vision`, package
`@nous/vision`; `layer:infra`) — the broadest perception library at ~103
modules. `src/index.ts` re-exports recognition (action, age/gender, emotion,
facial recognition/ reenactment/restoration, landmarks), analysis (anomaly
detection, crowd analysis, dense video captioning, depth estimation, document
layout), and a generation stack (a diffusion-model framework, ControlNet
integration, DreamBooth training, depth/edge-guided generation, consistent
character/style batches, animation generation, frame interpolation, FLUX
support). It sits alongside `nous-core` as shared infra for any pipeline needing
perception or image synthesis.