Nous - LLM Inference and Reasoning Engine
Nous is a library-only domain focused on local LLM inference, model
optimization, and high-performance serving. Its single comprehensive library,
nous-core, provides a complete inference runtime with 56 modules spanning
execution providers for every major hardware backend, dynamic and continuous
batching, speculative decoding, KV cache management, paged and flash attention,
tensor and pipeline parallelism, multi-LoRA serving, model format support
(SafeTensors, GGUF, GGML, PyTorch), and a full model registry with versioning
and discovery.
The library wraps ONNX Runtime as its primary inference backend (via an optional
dependency on onnxruntime-node) but abstracts behind provider interfaces to
support CPU, CUDA, ROCm, Metal, WebGPU, Vulkan, DirectML, and NPU execution
providers.
Architecture#
nous-core
(56 modules)
┌──────────────────┼──────────────────┐
| | |
Execution Inference Model
Providers Pipeline Management
| | |
cpu, cuda, dynamic-batching, model-registry,
rocm, metal, continuous-batching, model-versioning,
webgpu, vulkan, speculative-decoding, model-metadata,
dml, npu llm-serving, model-discovery,
kv-cache, model-download,
paged-attention, checksum-verification,
flash-attention model-conversion,
| | format-detection,
Graph Parallelism safetensors,
Optimization | gguf, ggml,
| tensor-parallelism, pytorch-loading
memory-planning, pipeline-parallel,
operator-fusion, model-sharding,
quantization moe-parallelism
| |
Generation Caching
| |
streaming-gen, prefix-caching,
stop-sequences, radix-attention,
logit-processors, prompt-caching
temperature-sampling,
top-k-top-p-sampling,
beam-search
|
Multi-Adapter
|
multi-lora-serving,
adapter-switching,
context-extension
All modules are exported from a single nous-core library, organized into
logical groups but delivered as one cohesive package.
Library: nous-core#
Core Types and ONNX Abstraction#
The library defines a comprehensive type system for ONNX Runtime integration:
OnnxExecutionProviderName- Union type covering all supported backends:cpu,cuda,rocm,dml,coreml,metal,webgpu,vulkan,xnnpack,nnapi,qnn,npuOnnxTensorType- All numeric tensor types:float32,float64,int8,uint8,int16,uint16,int32,uint32,int64,uint64,boolOnnxTensorLike<TData>- Generic tensor interface with typed data arraysOnnxInferenceSessionLike- Session abstraction withinputNames,outputNames,run(),release(),dispose()OnnxRuntimeModule- Module interface for dynamic runtime loadingNousOnnxSessionConfig- Full session configuration
Execution Providers (8 modules)#
Hardware-specific execution provider implementations:
| Module | Backend | Use Case |
|---|---|---|
cpu-execution-provider |
CPU | Universal fallback |
cuda-execution-provider |
NVIDIA CUDA | NVIDIA GPU inference |
rocm-execution-provider |
AMD ROCm | AMD GPU inference |
metal-execution-provider |
Apple Metal | macOS/iOS GPU |
webgpu-execution-provider |
WebGPU | Browser GPU inference |
vulkan-execution-provider |
Vulkan | Cross-platform GPU |
dml-execution-provider |
DirectML | Windows GPU |
npu-execution-provider |
NPU | Neural Processing Units |
Graph Optimization (4 modules)#
Model graph optimization for faster inference:
| Module | Description |
|---|---|
graph-optimization |
ONNX graph-level optimizations (basic, extended, all) |
memory-planning |
Memory arena and pattern optimization |
operator-fusion |
Fuse compatible operators to reduce overhead |
quantization |
Post-training quantization (INT8, INT4) |
Batching and Serving (3 modules)#
High-throughput request batching for production serving:
| Module | Description |
|---|---|
dynamic-batching |
Dynamic request batching with configurable windows |
continuous-batching |
Continuous batching for LLM token generation |
llm-serving |
Full LLM serving pipeline with request routing |
Attention and KV Cache (4 modules)#
Memory-efficient attention mechanisms:
| Module | Description |
|---|---|
kv-cache |
Key-value cache management for autoregressive generation |
paged-attention |
PagedAttention for efficient KV cache memory |
flash-attention |
FlashAttention for fused, memory-efficient attention |
speculative-decoding |
Draft-model speculative decoding for faster generation |
Parallelism (4 modules)#
Multi-GPU and distributed inference:
| Module | Description |
|---|---|
tensor-parallelism |
Shard model layers across GPUs |
pipeline-parallelism |
Pipeline model stages across GPUs |
model-sharding |
Automatic model sharding strategies |
moe-parallelism |
Mixture-of-Experts parallelism |
Caching (3 modules)#
Prompt and prefix caching for reduced latency:
| Module | Description |
|---|---|
prefix-caching |
Cache common prompt prefixes |
radix-attention |
Radix tree-based attention caching |
prompt-caching |
Full prompt result caching |
Multi-Adapter Serving (3 modules)#
Serve multiple LoRA adapters from a single base model:
| Module | Description |
|---|---|
multi-lora-serving |
Serve multiple LoRA adapters concurrently |
adapter-switching |
Hot-swap adapters without reloading base model |
context-extension |
Extend context length beyond training limits |
Generation Control (6 modules)#
Token generation and sampling strategies:
| Module | Description |
|---|---|
streaming-generation |
Stream tokens as they are generated |
stop-sequences |
Stop generation at specific token sequences |
logit-processors |
Custom logit processing pipelines |
temperature-sampling |
Temperature-controlled sampling |
top-k-top-p-sampling |
Top-k and nucleus (top-p) sampling |
beam-search |
Beam search decoding |
Model Management (10 modules)#
Model lifecycle management from discovery to deployment:
| Module | Description |
|---|---|
model-registry |
Central model registry |
model-versioning |
Model version tracking |
model-metadata |
Model metadata and capabilities |
model-discovery |
Discover available models |
model-download |
Download models from remote sources |
checksum-verification |
Verify model integrity via checksums |
model-conversion |
Convert between model formats |
format-detection |
Auto-detect model file formats |
model-loader |
Unified model loading interface |
model-optimization-pipeline |
End-to-end optimization pipeline |
Model Format Support (5 modules)#
Native support for major model file formats:
| Module | Description |
|---|---|
safetensors |
Hugging Face SafeTensors format |
gguf |
GGUF format (llama.cpp) |
ggml |
Legacy GGML format |
pytorch-loading |
PyTorch .pt/.pth loading |
tensor |
Core tensor operations |
Runtime (3 modules)#
Runtime management and session handling:
| Module | Description |
|---|---|
runtime-loader |
Dynamic ONNX Runtime loading |
session-manager |
Inference session lifecycle |
inference-engine |
High-level inference engine |
Technology Stack#
| Layer | Technology |
|---|---|
| Language | TypeScript (ESM) |
| Build | @nx/esbuild:esbuild (ESM format, unbundled) |
| Testing | Vitest |
| Inference Backend | ONNX Runtime (optional dependency) |
| Type Checking | Explicit typecheck target via tsc --noEmit |
Unlike most domains that use @nx/js:tsc, Nous uses esbuild for faster builds
and ESM output. The onnxruntime-node package is listed as an optional
dependency, allowing the library to be used in environments where it may not be
available (e.g., browsers using WebGPU instead).
Configuration#
Project tags: ["scope:nous", "layer:infra", "type:lib"]
Build executor: @nx/esbuild:esbuild with custom config
Targets:
build- esbuild compilation to ESMlint- ESLinttest- Vitesttypecheck- TypeScript validation viatsc --noEmit
Development#
# Run tests
pnpm nx test nous-core
# Build
pnpm nx build nous-core
# Typecheck
pnpm nx typecheck nous-core
# Lint
pnpm nx lint nous-core
Related Domains#
- Iris - Voice and speech domain using Nous for local inference
- Sophia - Research and knowledge domain leveraging LLM reasoning
- All domains using AI features can consume Nous for local model serving