# Nous Domain

> **Nous** - LLM Inference and Reasoning Engine

Nous is a library-only domain focused on local LLM inference, model
optimization, and high-performance serving. Its single comprehensive library,
`nous-core`, provides a complete inference runtime with 56 modules spanning
execution providers for every major hardware backend, dynamic and continuous
batching, speculative decoding, KV cache management, paged and flash attention,
tensor and pipeline parallelism, multi-LoRA serving, model format support
(SafeTensors, GGUF, GGML, PyTorch), and a full model registry with versioning
and discovery.

The library wraps ONNX Runtime as its primary inference backend (via an optional
dependency on `onnxruntime-node`) but abstracts behind provider interfaces to
support CPU, CUDA, ROCm, Metal, WebGPU, Vulkan, DirectML, and NPU execution
providers.

---

## Architecture

```
                      nous-core
                     (56 modules)
    ┌──────────────────┼──────────────────┐
    |                  |                  |
 Execution         Inference           Model
 Providers         Pipeline          Management
    |                  |                  |
 cpu, cuda,      dynamic-batching,   model-registry,
 rocm, metal,    continuous-batching, model-versioning,
 webgpu, vulkan, speculative-decoding, model-metadata,
 dml, npu        llm-serving,        model-discovery,
                 kv-cache,           model-download,
                 paged-attention,    checksum-verification,
                 flash-attention     model-conversion,
    |                  |             format-detection,
 Graph             Parallelism       safetensors,
 Optimization        |              gguf, ggml,
    |            tensor-parallelism, pytorch-loading
 memory-planning, pipeline-parallel,
 operator-fusion, model-sharding,
 quantization    moe-parallelism
    |                  |
 Generation        Caching
    |                  |
 streaming-gen,   prefix-caching,
 stop-sequences,  radix-attention,
 logit-processors, prompt-caching
 temperature-sampling,
 top-k-top-p-sampling,
 beam-search
    |
 Multi-Adapter
    |
 multi-lora-serving,
 adapter-switching,
 context-extension
```

All modules are exported from a single `nous-core` library, organized into
logical groups but delivered as one cohesive package.

---

## Library: `nous-core`

### Core Types and ONNX Abstraction

The library defines a comprehensive type system for ONNX Runtime integration:

- **`OnnxExecutionProviderName`** - Union type covering all supported backends:
  `cpu`, `cuda`, `rocm`, `dml`, `coreml`, `metal`, `webgpu`, `vulkan`,
  `xnnpack`, `nnapi`, `qnn`, `npu`
- **`OnnxTensorType`** - All numeric tensor types: `float32`, `float64`, `int8`,
  `uint8`, `int16`, `uint16`, `int32`, `uint32`, `int64`, `uint64`, `bool`
- **`OnnxTensorLike<TData>`** - Generic tensor interface with typed data arrays
- **`OnnxInferenceSessionLike`** - Session abstraction with `inputNames`,
  `outputNames`, `run()`, `release()`, `dispose()`
- **`OnnxRuntimeModule`** - Module interface for dynamic runtime loading
- **`NousOnnxSessionConfig`** - Full session configuration

### Execution Providers (8 modules)

Hardware-specific execution provider implementations:

| Module                      | Backend     | Use Case                |
| --------------------------- | ----------- | ----------------------- |
| `cpu-execution-provider`    | CPU         | Universal fallback      |
| `cuda-execution-provider`   | NVIDIA CUDA | NVIDIA GPU inference    |
| `rocm-execution-provider`   | AMD ROCm    | AMD GPU inference       |
| `metal-execution-provider`  | Apple Metal | macOS/iOS GPU           |
| `webgpu-execution-provider` | WebGPU      | Browser GPU inference   |
| `vulkan-execution-provider` | Vulkan      | Cross-platform GPU      |
| `dml-execution-provider`    | DirectML    | Windows GPU             |
| `npu-execution-provider`    | NPU         | Neural Processing Units |

### Graph Optimization (4 modules)

Model graph optimization for faster inference:

| Module               | Description                                           |
| -------------------- | ----------------------------------------------------- |
| `graph-optimization` | ONNX graph-level optimizations (basic, extended, all) |
| `memory-planning`    | Memory arena and pattern optimization                 |
| `operator-fusion`    | Fuse compatible operators to reduce overhead          |
| `quantization`       | Post-training quantization (INT8, INT4)               |

### Batching and Serving (3 modules)

High-throughput request batching for production serving:

| Module                | Description                                        |
| --------------------- | -------------------------------------------------- |
| `dynamic-batching`    | Dynamic request batching with configurable windows |
| `continuous-batching` | Continuous batching for LLM token generation       |
| `llm-serving`         | Full LLM serving pipeline with request routing     |

### Attention and KV Cache (4 modules)

Memory-efficient attention mechanisms:

| Module                 | Description                                              |
| ---------------------- | -------------------------------------------------------- |
| `kv-cache`             | Key-value cache management for autoregressive generation |
| `paged-attention`      | PagedAttention for efficient KV cache memory             |
| `flash-attention`      | FlashAttention for fused, memory-efficient attention     |
| `speculative-decoding` | Draft-model speculative decoding for faster generation   |

### Parallelism (4 modules)

Multi-GPU and distributed inference:

| Module                 | Description                         |
| ---------------------- | ----------------------------------- |
| `tensor-parallelism`   | Shard model layers across GPUs      |
| `pipeline-parallelism` | Pipeline model stages across GPUs   |
| `model-sharding`       | Automatic model sharding strategies |
| `moe-parallelism`      | Mixture-of-Experts parallelism      |

### Caching (3 modules)

Prompt and prefix caching for reduced latency:

| Module            | Description                        |
| ----------------- | ---------------------------------- |
| `prefix-caching`  | Cache common prompt prefixes       |
| `radix-attention` | Radix tree-based attention caching |
| `prompt-caching`  | Full prompt result caching         |

### Multi-Adapter Serving (3 modules)

Serve multiple LoRA adapters from a single base model:

| Module               | Description                                    |
| -------------------- | ---------------------------------------------- |
| `multi-lora-serving` | Serve multiple LoRA adapters concurrently      |
| `adapter-switching`  | Hot-swap adapters without reloading base model |
| `context-extension`  | Extend context length beyond training limits   |

### Generation Control (6 modules)

Token generation and sampling strategies:

| Module                 | Description                                 |
| ---------------------- | ------------------------------------------- |
| `streaming-generation` | Stream tokens as they are generated         |
| `stop-sequences`       | Stop generation at specific token sequences |
| `logit-processors`     | Custom logit processing pipelines           |
| `temperature-sampling` | Temperature-controlled sampling             |
| `top-k-top-p-sampling` | Top-k and nucleus (top-p) sampling          |
| `beam-search`          | Beam search decoding                        |

### Model Management (10 modules)

Model lifecycle management from discovery to deployment:

| Module                        | Description                          |
| ----------------------------- | ------------------------------------ |
| `model-registry`              | Central model registry               |
| `model-versioning`            | Model version tracking               |
| `model-metadata`              | Model metadata and capabilities      |
| `model-discovery`             | Discover available models            |
| `model-download`              | Download models from remote sources  |
| `checksum-verification`       | Verify model integrity via checksums |
| `model-conversion`            | Convert between model formats        |
| `format-detection`            | Auto-detect model file formats       |
| `model-loader`                | Unified model loading interface      |
| `model-optimization-pipeline` | End-to-end optimization pipeline     |

### Model Format Support (5 modules)

Native support for major model file formats:

| Module            | Description                     |
| ----------------- | ------------------------------- |
| `safetensors`     | Hugging Face SafeTensors format |
| `gguf`            | GGUF format (llama.cpp)         |
| `ggml`            | Legacy GGML format              |
| `pytorch-loading` | PyTorch .pt/.pth loading        |
| `tensor`          | Core tensor operations          |

### Runtime (3 modules)

Runtime management and session handling:

| Module             | Description                  |
| ------------------ | ---------------------------- |
| `runtime-loader`   | Dynamic ONNX Runtime loading |
| `session-manager`  | Inference session lifecycle  |
| `inference-engine` | High-level inference engine  |

---

## Technology Stack

| Layer             | Technology                                     |
| ----------------- | ---------------------------------------------- |
| Language          | TypeScript (ESM)                               |
| Build             | `@nx/esbuild:esbuild` (ESM format, unbundled)  |
| Testing           | Vitest                                         |
| Inference Backend | ONNX Runtime (optional dependency)             |
| Type Checking     | Explicit `typecheck` target via `tsc --noEmit` |

Unlike most domains that use `@nx/js:tsc`, Nous uses esbuild for faster builds
and ESM output. The `onnxruntime-node` package is listed as an optional
dependency, allowing the library to be used in environments where it may not be
available (e.g., browsers using WebGPU instead).

---

## Configuration

**Project tags:** `["scope:nous", "layer:infra", "type:lib"]`

**Build executor:** `@nx/esbuild:esbuild` with custom config

**Targets:**

- `build` - esbuild compilation to ESM
- `lint` - ESLint
- `test` - Vitest
- `typecheck` - TypeScript validation via `tsc --noEmit`

---

## Development

```bash
# Run tests
pnpm nx test nous-core

# Build
pnpm nx build nous-core

# Typecheck
pnpm nx typecheck nous-core

# Lint
pnpm nx lint nous-core
```

---

## Related Domains

- **Iris** - Voice and speech domain using Nous for local inference
- **Sophia** - Research and knowledge domain leveraging LLM reasoning
- All domains using AI features can consume Nous for local model serving
