Eve library

Nous Domain

Guide

Revision 1

On this page

Nous - LLM Inference and Reasoning Engine

Nous is a library-only domain focused on local LLM inference, model optimization, and high-performance serving. Its single comprehensive library, nous-core, provides a complete inference runtime with 56 modules spanning execution providers for every major hardware backend, dynamic and continuous batching, speculative decoding, KV cache management, paged and flash attention, tensor and pipeline parallelism, multi-LoRA serving, model format support (SafeTensors, GGUF, GGML, PyTorch), and a full model registry with versioning and discovery.

The library wraps ONNX Runtime as its primary inference backend (via an optional dependency on onnxruntime-node) but abstracts behind provider interfaces to support CPU, CUDA, ROCm, Metal, WebGPU, Vulkan, DirectML, and NPU execution providers.


Architecture#

text
                      nous-core
                     (56 modules)
    ┌──────────────────┼──────────────────┐
    |                  |                  |
 Execution         Inference           Model
 Providers         Pipeline          Management
    |                  |                  |
 cpu, cuda,      dynamic-batching,   model-registry,
 rocm, metal,    continuous-batching, model-versioning,
 webgpu, vulkan, speculative-decoding, model-metadata,
 dml, npu        llm-serving,        model-discovery,
                 kv-cache,           model-download,
                 paged-attention,    checksum-verification,
                 flash-attention     model-conversion,
    |                  |             format-detection,
 Graph             Parallelism       safetensors,
 Optimization        |              gguf, ggml,
    |            tensor-parallelism, pytorch-loading
 memory-planning, pipeline-parallel,
 operator-fusion, model-sharding,
 quantization    moe-parallelism
    |                  |
 Generation        Caching
    |                  |
 streaming-gen,   prefix-caching,
 stop-sequences,  radix-attention,
 logit-processors, prompt-caching
 temperature-sampling,
 top-k-top-p-sampling,
 beam-search
    |
 Multi-Adapter
    |
 multi-lora-serving,
 adapter-switching,
 context-extension

All modules are exported from a single nous-core library, organized into logical groups but delivered as one cohesive package.


Library: nous-core#

Core Types and ONNX Abstraction#

The library defines a comprehensive type system for ONNX Runtime integration:

  • OnnxExecutionProviderName - Union type covering all supported backends: cpu, cuda, rocm, dml, coreml, metal, webgpu, vulkan, xnnpack, nnapi, qnn, npu
  • OnnxTensorType - All numeric tensor types: float32, float64, int8, uint8, int16, uint16, int32, uint32, int64, uint64, bool
  • OnnxTensorLike<TData> - Generic tensor interface with typed data arrays
  • OnnxInferenceSessionLike - Session abstraction with inputNames, outputNames, run(), release(), dispose()
  • OnnxRuntimeModule - Module interface for dynamic runtime loading
  • NousOnnxSessionConfig - Full session configuration

Execution Providers (8 modules)#

Hardware-specific execution provider implementations:

Module Backend Use Case
cpu-execution-provider CPU Universal fallback
cuda-execution-provider NVIDIA CUDA NVIDIA GPU inference
rocm-execution-provider AMD ROCm AMD GPU inference
metal-execution-provider Apple Metal macOS/iOS GPU
webgpu-execution-provider WebGPU Browser GPU inference
vulkan-execution-provider Vulkan Cross-platform GPU
dml-execution-provider DirectML Windows GPU
npu-execution-provider NPU Neural Processing Units

Graph Optimization (4 modules)#

Model graph optimization for faster inference:

Module Description
graph-optimization ONNX graph-level optimizations (basic, extended, all)
memory-planning Memory arena and pattern optimization
operator-fusion Fuse compatible operators to reduce overhead
quantization Post-training quantization (INT8, INT4)

Batching and Serving (3 modules)#

High-throughput request batching for production serving:

Module Description
dynamic-batching Dynamic request batching with configurable windows
continuous-batching Continuous batching for LLM token generation
llm-serving Full LLM serving pipeline with request routing

Attention and KV Cache (4 modules)#

Memory-efficient attention mechanisms:

Module Description
kv-cache Key-value cache management for autoregressive generation
paged-attention PagedAttention for efficient KV cache memory
flash-attention FlashAttention for fused, memory-efficient attention
speculative-decoding Draft-model speculative decoding for faster generation

Parallelism (4 modules)#

Multi-GPU and distributed inference:

Module Description
tensor-parallelism Shard model layers across GPUs
pipeline-parallelism Pipeline model stages across GPUs
model-sharding Automatic model sharding strategies
moe-parallelism Mixture-of-Experts parallelism

Caching (3 modules)#

Prompt and prefix caching for reduced latency:

Module Description
prefix-caching Cache common prompt prefixes
radix-attention Radix tree-based attention caching
prompt-caching Full prompt result caching

Multi-Adapter Serving (3 modules)#

Serve multiple LoRA adapters from a single base model:

Module Description
multi-lora-serving Serve multiple LoRA adapters concurrently
adapter-switching Hot-swap adapters without reloading base model
context-extension Extend context length beyond training limits

Generation Control (6 modules)#

Token generation and sampling strategies:

Module Description
streaming-generation Stream tokens as they are generated
stop-sequences Stop generation at specific token sequences
logit-processors Custom logit processing pipelines
temperature-sampling Temperature-controlled sampling
top-k-top-p-sampling Top-k and nucleus (top-p) sampling
beam-search Beam search decoding

Model Management (10 modules)#

Model lifecycle management from discovery to deployment:

Module Description
model-registry Central model registry
model-versioning Model version tracking
model-metadata Model metadata and capabilities
model-discovery Discover available models
model-download Download models from remote sources
checksum-verification Verify model integrity via checksums
model-conversion Convert between model formats
format-detection Auto-detect model file formats
model-loader Unified model loading interface
model-optimization-pipeline End-to-end optimization pipeline

Model Format Support (5 modules)#

Native support for major model file formats:

Module Description
safetensors Hugging Face SafeTensors format
gguf GGUF format (llama.cpp)
ggml Legacy GGML format
pytorch-loading PyTorch .pt/.pth loading
tensor Core tensor operations

Runtime (3 modules)#

Runtime management and session handling:

Module Description
runtime-loader Dynamic ONNX Runtime loading
session-manager Inference session lifecycle
inference-engine High-level inference engine

Technology Stack#

Layer Technology
Language TypeScript (ESM)
Build @nx/esbuild:esbuild (ESM format, unbundled)
Testing Vitest
Inference Backend ONNX Runtime (optional dependency)
Type Checking Explicit typecheck target via tsc --noEmit

Unlike most domains that use @nx/js:tsc, Nous uses esbuild for faster builds and ESM output. The onnxruntime-node package is listed as an optional dependency, allowing the library to be used in environments where it may not be available (e.g., browsers using WebGPU instead).


Configuration#

Project tags: ["scope:nous", "layer:infra", "type:lib"]

Build executor: @nx/esbuild:esbuild with custom config

Targets:

  • build - esbuild compilation to ESM
  • lint - ESLint
  • test - Vitest
  • typecheck - TypeScript validation via tsc --noEmit

Development#

bash
# Run tests
pnpm nx test nous-core

# Build
pnpm nx build nous-core

# Typecheck
pnpm nx typecheck nous-core

# Lint
pnpm nx lint nous-core

  • Iris - Voice and speech domain using Nous for local inference
  • Sophia - Research and knowledge domain leveraging LLM reasoning
  • All domains using AI features can consume Nous for local model serving