Canonical source: the canonical features document for this domain is
deep-dive/features.md, rendered in the docs
center. This file is its operational companion (service topology, APIs,
schemas, runtime detail) — reconciled by linking, not duplicated (§2, §13).
Nous provides local LLM inference, model management, embedding generation,
agent orchestration, document processing, and prompt engineering as consumable
library APIs for any application in the Oshun ecosystem.
Nous is the AI inference backbone of Oshun. It enables applications to run large
language models locally on any hardware, generate and search embeddings, build
autonomous agents, process documents into vector indexes, and engineer prompts
with full template systems -- all without depending on external cloud AI
services.
The domain ships three libraries:
| Library |
Package |
Modules |
Focus |
| nous-core |
@nous/core |
84 |
Inference runtime, hardware providers, model management, embeddings, search |
| nous-llm |
@nous/llm |
77 |
LLM APIs, agents, prompt engineering, document processing, RAG |
| nous-vision |
@nous/vision |
40 |
Computer vision, image generation, diffusion models, OCR |
1. Hardware-Accelerated Inference#
Nous abstracts the ONNX Runtime behind a unified provider interface, letting
consuming applications run inference on any available hardware without code
changes.
1.1 Execution Providers#
| Capability |
Provider |
What It Enables |
| CPU inference |
cpu |
Universal fallback; runs on any machine without GPU drivers |
| NVIDIA GPU inference |
cuda |
High-throughput inference on NVIDIA GPUs with CUDA |
| AMD GPU inference |
rocm |
Inference on AMD Radeon and Instinct GPUs |
| Apple GPU inference |
metal |
Metal-accelerated inference on macOS and iOS devices |
| Browser GPU inference |
webgpu |
In-browser GPU inference via the WebGPU API |
| Cross-platform GPU |
vulkan |
Vulkan-based GPU inference across Linux, Windows, and Android |
| Windows GPU inference |
dml |
DirectML inference on any DirectX 12-capable Windows GPU |
| Neural Processing Units |
npu |
Inference on dedicated NPU/TPU hardware accelerators |
Each provider offers device selection (e.g., GPU device ID), tuning parameters,
and graceful fallback to the next available provider when the preferred one is
unavailable.
1.2 Provider Routing and Fallback#
| Capability |
Description |
| Preferred provider ordering |
Specify a ranked list of providers; the runtime tries each in order |
| Automatic fallback |
When a provider fails or is unavailable, the next provider is used transparently |
| Provider configuration validation |
Invalid settings (e.g., negative device IDs) are rejected at configuration time |
| Runtime module loading |
ONNX Runtime is loaded dynamically; applications that do not need inference can skip the dependency entirely |
1.3 Session Management#
| Capability |
Description |
| Managed sessions |
Sessions are keyed, pooled, and released automatically |
| Session lifecycle |
Create, run, release, and dispose of inference sessions with full lifecycle control |
| Concurrent session support |
Multiple models can be loaded and run concurrently |
| Feed and fetch validation |
Input tensor names and output names are validated before execution |
2. Model Management#
A complete model lifecycle system from discovery through deployment and
optimization.
2.1 Model Registry and Discovery#
| Capability |
Description |
| Central model registry |
Register, list, look up, and remove models with metadata |
| Model discovery |
Discover available models from configured sources and repositories |
| Model versioning |
Track multiple versions of the same model with immutable version records |
| Model metadata |
Attach and query structured metadata (architecture, capabilities, license, parameter count, context length) |
2.2 Model Acquisition#
| Capability |
Description |
| Model download |
Download models from remote sources with progress tracking |
| Checksum verification |
Verify model file integrity using SHA-256 and other hash algorithms |
| Format auto-detection |
Automatically identify the format of a model file (SafeTensors, GGUF, GGML, PyTorch, TensorFlow, JAX) |
| Format |
Capability |
| SafeTensors |
Load and parse Hugging Face SafeTensors files with tensor metadata extraction |
| GGUF |
Load GGUF (llama.cpp) quantized model files with header and metadata parsing |
| GGML |
Load legacy GGML format model files |
| PyTorch |
Load PyTorch .pt and .pth checkpoint files |
| TensorFlow |
Load TensorFlow SavedModel and checkpoint formats |
| JAX |
Load JAX/Flax model parameters |
2.4 Model Conversion and Optimization#
| Capability |
Description |
| Cross-format conversion |
Convert models between supported formats |
| End-to-end optimization pipeline |
Chain format detection, loading, quantization, and optimization in a single pipeline |
| Model benchmarking |
Benchmark model inference speed, throughput, and latency across providers |
| Model profiling |
Profile model execution to identify bottlenecks and memory hotspots |
2.5 Model Caching and Lifecycle#
| Capability |
Description |
| Model caching |
Cache loaded models in memory to avoid repeated loading |
| Model eviction |
Evict least-recently-used or least-frequently-used models when memory pressure rises |
| Model preloading |
Preload models at startup for instant inference availability |
| Warm model pool |
Maintain a pool of ready-to-run model sessions for zero-latency first inference |
3. Inference Pipeline#
High-throughput, production-grade inference with batching, caching, and
parallelism.
3.1 Dynamic and Continuous Batching#
| Capability |
Description |
| Dynamic batching |
Accumulate incoming requests into batches with configurable maximum batch size and queue delay |
| Continuous batching |
Continuously batch token-generation requests for LLM serving without waiting for entire sequences to complete |
| Batch key routing |
Group requests into batches by model/session key so different models are batched independently |
| Queue management |
Configurable maximum queue sizes, backpressure, and dropped-request tracking |
| Batch statistics |
Track pending requests, batches executed, average batch size, and failure rates |
3.2 LLM Serving#
| Capability |
Description |
| Full serving pipeline |
End-to-end LLM serving with request routing, batching, and response streaming |
| Request prioritization |
Prioritize inference requests based on caller-specified priority |
| Throughput optimization |
Maximize GPU utilization through intelligent request scheduling |
3.3 Attention Mechanisms and KV Cache#
| Capability |
Description |
| KV cache management |
Allocate, grow, and evict key-value caches for autoregressive generation |
| Paged attention |
Memory-efficient KV cache using virtual-memory-style paging to reduce fragmentation |
| Flash attention |
Fused attention kernels that reduce memory reads and improve throughput |
| Speculative decoding |
Use a smaller draft model to speculatively generate tokens, then verify with the full model for faster generation |
3.4 Multi-GPU Parallelism#
| Capability |
Description |
| Tensor parallelism |
Shard model layers across multiple GPUs for larger-than-single-GPU models |
| Pipeline parallelism |
Pipeline model stages across GPUs for higher throughput |
| Model sharding |
Automatic sharding strategies that split models across available devices |
| Mixture-of-Experts parallelism |
Route different experts to different GPUs for MoE model architectures |
3.5 Prompt and Prefix Caching#
| Capability |
Description |
| Prefix caching |
Cache common prompt prefixes so repeated system prompts do not re-compute attention |
| Radix attention caching |
Radix-tree-based attention caching for efficient prefix sharing across requests |
| Prompt result caching |
Cache full prompt-to-response mappings for deterministic queries |
4. Generation Control#
Fine-grained control over how tokens are generated and when generation stops.
4.1 Sampling Strategies#
| Capability |
Description |
| Temperature sampling |
Control randomness with temperature scaling of logit distributions |
| Top-k sampling |
Restrict token selection to the top-k most probable tokens |
| Top-p (nucleus) sampling |
Restrict token selection to the smallest set of tokens whose cumulative probability exceeds p |
| Beam search |
Search for the most probable sequence using beam search with configurable beam width |
4.2 Logit Processing#
| Capability |
Description |
| Custom logit processors |
Chain multiple logit processors into a pipeline (e.g., repetition penalty, frequency penalty) |
| Repetition penalty |
Penalize already-generated tokens to reduce repetitive output |
| Logit bias |
Apply per-token biases to steer generation toward or away from specific tokens |
4.3 Streaming and Stop Conditions#
| Capability |
Description |
| Streaming generation |
Stream tokens to the consumer as they are generated, with backpressure support |
| Stop sequences |
Halt generation when specific token sequences are produced |
| Max token limits |
Cap generation at a configurable maximum token count |
5. Multi-Adapter Serving#
Serve multiple fine-tuned model variants from a single base model.
| Capability |
Description |
| Multi-LoRA serving |
Serve multiple LoRA adapters concurrently from a shared base model, reducing total memory usage |
| Hot adapter switching |
Swap LoRA adapters in and out without reloading the base model |
| Context extension |
Extend a model's effective context length beyond its original training limit using RoPE scaling and other techniques |
6. Graph Optimization and Quantization#
Optimize models for faster inference and reduced memory usage.
6.1 Graph Optimization#
| Capability |
Description |
| ONNX graph optimization |
Apply basic, extended, or all graph-level optimizations to reduce operator overhead |
| Operator fusion |
Fuse compatible operators (e.g., MatMul + Add, Conv + BN) into single operations |
| Memory planning |
Optimize memory allocation patterns with arena-based memory planning |
6.2 Quantization#
| Capability |
Description |
| FP16 quantization |
Convert float32 tensors to float16 for 2x memory reduction with minimal accuracy loss |
| INT8 quantization |
Quantize to 8-bit integers with symmetric or asymmetric modes |
| INT4 quantization |
Quantize to 4-bit integers (packed nibbles) for 8x memory reduction |
| Dynamic quantization |
Quantize weights on-the-fly using observed tensor ranges |
| Static quantization |
Quantize weights using calibration data for higher accuracy |
| Calibration collection |
Collect tensor value ranges across calibration batches for static quantization |
| Percentile calibration |
Use percentile-based range clipping to reduce outlier impact on quantization accuracy |
| Batch quantization |
Quantize entire tensor maps in one pass with compression ratio tracking |
| Dequantization |
Convert quantized tensors back to float32 for analysis or mixed-precision inference |
7. Tensor Operations#
Low-level tensor creation, manipulation, and type-safe data handling.
| Capability |
Description |
| Typed tensor creation |
Create tensors with explicit types: float32, float64, int8, uint8, int16, uint16, int32, uint32, int64, uint64, bool |
| Dimension tracking |
All tensors carry immutable dimension metadata |
| Typed array backing |
Tensors are backed by native typed arrays (Float32Array, Int8Array, etc.) for zero-copy performance |
| Tensor validation |
Validate tensor shapes, data types, and element counts before inference |
8. Embedding Generation#
Generate, manage, and search vector embeddings across text, image, audio, and
multi-modal content.
8.1 Embedding Providers#
| Capability |
Description |
| Multi-provider embedding |
Register multiple embedding providers with priority-based routing |
| Provider fallback |
Automatically fall back to the next provider when one fails |
| Provider statistics |
Track per-provider success rates, failure rates, and average latency |
| Configurable timeouts |
Set per-request timeout limits with provider-level timeout tracking |
8.2 Modality Support#
| Modality |
What It Enables |
| Text embeddings |
Generate vector representations of text passages for semantic search and similarity |
| Image embeddings |
Generate vector representations of images for visual search and classification |
| Audio embeddings |
Generate vector representations of audio for speaker identification and audio search |
| Multi-modal embeddings |
Generate combined embeddings from multiple modalities (e.g., text + image) |
8.3 Embedding Processing#
| Capability |
Description |
| Embedding normalization |
Normalize embedding vectors to unit length for cosine similarity |
| Dimensionality reduction |
Reduce embedding dimensions while preserving semantic relationships |
| Output dimension control |
Truncate or reduce embeddings to a requested dimension count |
| Batch embedding |
Generate embeddings for multiple inputs in a single batched request |
| Streaming embedding |
Generate embeddings in a streaming fashion for large input sets |
| Embedding caching |
Cache generated embeddings to avoid redundant computation |
8.4 Embedding Compression#
| Capability |
Description |
| Binary embeddings |
Compress float embeddings to binary (1-bit) vectors for ultra-fast approximate search |
| Matryoshka embeddings |
Generate variable-dimension embeddings where truncated prefixes remain semantically meaningful |
| General compression |
Apply lossy and lossless compression to embedding vectors for storage reduction |
9. Similarity Search#
Search and rank documents using vector similarity.
9.1 Nearest Neighbor Search#
| Capability |
Description |
| Approximate nearest neighbor (ANN) |
Fast approximate search using index structures (e.g., HNSW, IVF) for large-scale retrieval |
| Exact nearest neighbor |
Brute-force exact search for highest accuracy on smaller datasets |
| Hybrid search |
Combine vector similarity with keyword/BM25 search for improved relevance |
9.2 Distance Metrics#
| Capability |
Description |
| Cosine similarity |
Measure angular similarity between vectors (direction-based) |
| Euclidean distance |
Measure straight-line distance between vectors |
| Dot product |
Measure alignment between vectors (magnitude-sensitive) |
| Manhattan distance |
Measure L1 distance between vectors |
9.3 Reranking#
| Capability |
Description |
| Score-based reranking |
Rerank search results using a scoring model |
| Cross-encoder reranking |
Use cross-encoder models to deeply score query-document pairs for high-precision reranking |
10. LLM Completion APIs#
High-level interfaces for generating text, holding conversations, and following
instructions.
10.1 Completion Interfaces#
| Capability |
Description |
| Text completion |
Generate continuations of a text prompt |
| Chat completion |
Conduct multi-turn conversations with system, user, and assistant message roles |
| Instruction following |
Generate responses that follow structured instructions |
| Multi-turn conversation |
Maintain conversation history across multiple exchanges with context management |
| System prompts |
Define persistent system-level instructions that shape all responses |
| Capability |
Description |
| Function calling |
Declare callable functions with typed parameters; the LLM generates structured function call requests |
| Tool use framework |
Define tools with descriptions and schemas; the LLM selects and invokes them as needed |
| Tool selection |
Intelligent selection of appropriate tools based on the current task context |
10.3 Structured Output#
| Capability |
Description |
| JSON mode |
Constrain LLM output to valid JSON |
| Structured output |
Generate output conforming to a specified schema |
| Grammar constraints |
Constrain generation to outputs matching a formal grammar (BNF/EBNF) |
| Regex constraints |
Constrain generation to outputs matching a regular expression pattern |
| Schema validation |
Validate generated output against JSON Schema definitions |
10.4 Response Processing#
| Capability |
Description |
| Response formatting |
Format and structure raw LLM output |
| Markdown rendering |
Render LLM output as well-formed markdown |
| Code extraction |
Extract code blocks and snippets from LLM output |
| Citation extraction |
Extract citations and references from LLM output |
| Fact extraction |
Extract factual claims and assertions from LLM output |
11. Natural Language Processing#
Built-in NLP tasks that run locally.
| Capability |
Description |
| Entity extraction |
Identify and extract named entities (people, places, organizations, dates) from text |
| Sentiment analysis |
Classify the emotional tone of text (positive, negative, neutral, mixed) |
| Text classification |
Classify text into predefined or dynamic categories |
12. Prompt Engineering#
A full prompt template system with variable substitution, conditional logic,
loops, versioning, testing, and optimization.
12.1 Template System#
| Capability |
Description |
| Variable substitution |
Insert values into prompts using {{variable}} syntax with dot-notation for nested access |
| Conditional sections |
Include or exclude prompt sections based on variable values with {{#if}}/{{else}}/{{/if}} |
| Comparison operators |
Support == and != comparisons in conditions for precise branching |
| Negation |
Negate conditions with ! prefix for inverted logic |
| Loop constructs |
Iterate over arrays with {{#each items as item}}/{{/each}} with index tracking |
| Nested values |
Access deeply nested object properties via dot notation (e.g., {{user.profile.name}}) |
| Strict mode |
Throw errors on missing variables instead of leaving placeholders |
| Missing value defaults |
Provide a fallback string for unresolved variables |
| Object serialization |
Automatically stringify complex objects in template output |
12.2 Template Management#
| Capability |
Description |
| Template registration |
Register reusable prompt templates with IDs, names, descriptions, and tags |
| Template versioning |
Track template versions and revision history |
| Template listing and search |
List and filter templates by tag or search text |
| Template overwrite control |
Optionally overwrite existing templates or reject duplicates |
| Variable auto-extraction |
Automatically detect variables used in template messages |
12.3 Prompt Versioning and Testing#
| Capability |
Description |
| Prompt versioning |
Version and track prompt changes over time for reproducibility |
| Prompt testing |
Test prompts against expected outputs for quality assurance |
| Prompt optimization |
Optimize prompt wording and structure for improved model performance |
12.4 Few-Shot and Examples#
| Capability |
Description |
| Few-shot management |
Manage collections of few-shot examples for in-context learning |
| Example selection |
Select the most relevant examples for a given query |
| Dynamic examples |
Generate or retrieve examples dynamically based on the input context |
13. Advanced Reasoning#
Structured reasoning strategies that improve LLM output quality.
| Capability |
Description |
| Chain-of-thought |
Guide the LLM through step-by-step reasoning before producing a final answer |
| Tree-of-thought |
Explore multiple reasoning branches and select the most promising path |
| Graph-of-thought |
Model reasoning as a directed graph where nodes share and build on insights |
| Self-consistency |
Generate multiple reasoning chains and select the most consistent answer |
| Reflection prompts |
Prompt the LLM to reflect on and improve its own output |
| Critique prompts |
Prompt the LLM to critique its reasoning and identify weaknesses |
14. Persona and Context Management#
Control LLM behavior and inject external knowledge.
| Capability |
Description |
| Persona management |
Define and switch between named personas that shape LLM tone, expertise, and behavior |
| Context injection |
Inject external context (documents, data, user preferences) into prompts |
| Memory injection |
Inject conversation memory and historical context into prompts |
| Retrieval injection |
Inject retrieved documents and passages from vector search into prompts |
15. Agent Architecture#
Build autonomous AI agents with planning, tool use, memory, and multi-agent
coordination.
15.1 Agent Lifecycle#
| Capability |
Description |
| Agent registration |
Register agents with unique IDs, names, capabilities, and module configurations |
| Agent lifecycle states |
Manage agents through stopped, idle, running, paused, and error states |
| Concurrency control |
Configure maximum concurrent tasks per agent |
| Agent start/stop/pause/resume |
Full lifecycle control with graceful task handling on state changes |
| Agent unregistration |
Cleanly remove agents with automatic queued-task cancellation |
15.2 Task Execution#
| Capability |
Description |
| Task submission |
Submit tasks with goals, payloads, context, priority, and timeout |
| Priority-based scheduling |
Tasks are executed in priority order with FIFO tiebreaking |
| Task timeout |
Tasks that exceed their timeout are terminated with timeout state tracking |
| Task cancellation |
Cancel queued tasks before execution begins |
| Run-next and run-all |
Execute the next available task or drain all queued tasks |
| Task state tracking |
Track tasks through queued, running, completed, failed, cancelled, and timeout states |
15.3 Planning and Decomposition#
| Capability |
Description |
| Planning module |
Agents can plan before executing: generate a summary, steps, and confidence score |
| Goal decomposition |
Break complex goals into prioritized subtasks before execution |
| Multi-step reasoning |
Chain multiple reasoning steps with intermediate results |
| Backtracking |
Revert to earlier reasoning states when a path fails |
| Error recovery |
Detect and recover from errors during multi-step task execution |
15.4 Agent Memory#
| Capability |
Description |
| Agent session state |
Persist key-value state across tasks within an agent session |
| Short-term memory |
Maintain recent context for ongoing conversations |
| Long-term memory |
Persist important information across sessions |
| Episodic memory |
Store and recall specific interaction episodes |
| Semantic memory |
Store and query factual knowledge in vector form |
| Working memory |
Maintain the active reasoning context during task execution |
15.5 Multi-Agent Systems#
| Capability |
Description |
| Agent communication |
Send messages between agents for collaborative task solving |
| Multi-agent coordination |
Coordinate multiple agents working on related tasks |
| Agent delegation |
Delegate subtasks from one agent to another |
| Agent supervision |
Supervise and monitor agent execution with oversight policies |
| Action selection |
Select the best action from a set of candidates based on context and goals |
| Tool selection |
Select the most appropriate tool for a given subtask |
15.6 Runtime Events#
| Capability |
Description |
| Event history |
Track all agent and task lifecycle events with timestamps |
| Event types |
Agent registered/unregistered/started/stopped, task submitted/started/completed/failed/timeout/cancelled, module events |
| Event history limits |
Configurable maximum event history size to bound memory usage |
| Statistics |
Track agents, queued/running/completed/failed tasks, average latency, and event counts |
16. Document Processing and RAG#
Ingest, parse, chunk, and index documents for retrieval-augmented generation.
16.1 Document Ingestion#
| Capability |
Description |
| Document ingestion pipeline |
End-to-end pipeline from raw document to indexed chunks |
| PDF parsing |
Extract text, structure, and metadata from PDF documents |
| HTML parsing |
Parse and extract clean text from HTML pages |
| Markdown parsing |
Parse markdown documents preserving structure |
| Code parsing |
Parse source code files with language-aware structure extraction |
| Capability |
Description |
| Table extraction |
Extract tabular data from documents |
| Image extraction |
Extract embedded images from documents |
| Metadata extraction |
Extract document metadata (title, author, date, keywords) |
| Link extraction |
Extract hyperlinks and references from documents |
16.3 Chunking Strategies#
| Capability |
Description |
| Semantic chunking |
Split documents at semantic boundaries for coherent chunks |
| Sentence chunking |
Split documents at sentence boundaries |
| Token chunking |
Split documents based on token counts for model input limits |
| Recursive chunking |
Recursively split documents using multiple strategies for optimal chunk sizes |
| Configurable chunking |
Configure chunk sizes, overlaps, and splitting heuristics |
16.4 Vector Indexing#
| Capability |
Description |
| Vector indexing |
Index document chunks as embedding vectors for semantic retrieval |
| Hybrid indexing |
Combine vector and keyword indexes for hybrid search |
| Incremental indexing |
Add, update, and remove documents without rebuilding the entire index |
| Index optimization |
Optimize index structures for query performance |
16.5 Context Retrieval#
| Capability |
Description |
| Context retrieval |
Retrieve the most relevant document chunks for a query using vector similarity |
| Multi-source retrieval |
Retrieve from multiple indexes and merge results |
| Relevance scoring |
Score and rank retrieved chunks by relevance |
17. Computer Vision#
Local computer vision inference covering detection, segmentation, recognition,
and generative image models.
17.1 Object Detection and Classification#
| Capability |
Description |
| Object detection |
Detect and locate objects in images with bounding boxes |
| Image classification |
Classify images into categories |
| Instance segmentation |
Segment individual object instances with pixel-level masks |
| Semantic segmentation |
Assign a class label to every pixel in an image |
| Panoptic segmentation |
Combine instance and semantic segmentation for complete scene understanding |
17.2 Facial Analysis#
| Capability |
Description |
| Facial recognition |
Detect and identify faces with feature embedding extraction |
| Face landmark detection |
Detect facial landmarks (eyes, nose, mouth, jawline) |
| Face restoration |
Restore and enhance degraded or low-resolution faces |
| Emotion recognition |
Classify facial expressions into emotional categories |
| Age and gender estimation |
Estimate age range and gender from facial features |
17.3 Document and Text Recognition#
| Capability |
Description |
| OCR engine |
Extract text from images using optical character recognition |
| Handwriting recognition |
Recognize and transcribe handwritten text |
| Document layout analysis |
Analyze document structure (headers, paragraphs, columns, tables) |
| Table recognition |
Detect and extract tabular structures from document images |
| Barcode and QR recognition |
Detect and decode barcodes and QR codes |
| License plate recognition |
Detect and read vehicle license plates |
17.4 Scene Understanding#
| Capability |
Description |
| Depth estimation |
Estimate per-pixel depth from monocular images |
| Surface normal estimation |
Estimate surface normal directions from images |
| Pose estimation |
Detect human body pose keypoints |
| Visual question answering |
Answer natural language questions about image content |
| Image captioning |
Generate natural language descriptions of image content |
17.5 Image Generation (Diffusion Models)#
| Capability |
Description |
| Stable Diffusion |
Text-to-image generation using Stable Diffusion models |
| SDXL support |
Extended support for Stable Diffusion XL models |
| SD3 support |
Support for Stable Diffusion 3 architecture |
| Flux support |
Support for Flux model architecture |
| Diffusion model framework |
General framework for running diffusion model pipelines locally |
| Capability |
Description |
| Image-to-image |
Transform images using diffusion models with guidance |
| Inpainting |
Fill in masked regions of images with contextually appropriate content |
| Outpainting |
Extend images beyond their original boundaries |
| Background removal |
Remove or replace image backgrounds |
| Style transfer |
Apply artistic styles from one image to another |
| Image composition |
Composite multiple images with blending and alignment |
| Image editing |
Apply targeted edits to specific regions of images |
| Upscaling (ESRGAN) |
Enhance image resolution using Real-ESRGAN super-resolution models |
| Variation generation |
Generate variations of an existing image |
17.7 Fine-Tuning and Adaptation#
| Capability |
Description |
| LoRA management |
Load, apply, and manage LoRA adapters for diffusion models |
| DreamBooth training |
Fine-tune diffusion models on custom subjects using DreamBooth |
| Textual inversion |
Learn new concepts as text embeddings for personalized generation |
| ControlNet integration |
Guide image generation with structural inputs (edges, depth, pose) |
| IP-Adapter support |
Use reference images to guide the style and content of generated images |
18. Cross-Cutting Capabilities#
18.1 Observability#
| Capability |
Description |
| Inference statistics |
Track requests, successes, failures, latency, and throughput across all subsystems |
| Provider-level metrics |
Per-provider success rates, failure rates, and latency |
| Batch statistics |
Track batch sizes, queue depths, and utilization |
| Agent runtime statistics |
Track agent and task lifecycle metrics |
| Template system statistics |
Track render counts, failures, and latency for prompt templates |
18.2 Type Safety#
| Capability |
Description |
| Full TypeScript types |
Every API surface is fully typed with generics, unions, and readonly constraints |
| Tensor type safety |
Generic OnnxTensorLike<TData> enforces typed array backing at compile time |
| Immutable interfaces |
All public interfaces use readonly properties to prevent accidental mutation |
18.3 Runtime Flexibility#
| Capability |
Description |
| Optional ONNX Runtime |
The ONNX Runtime dependency is optional; the library can be imported without it installed |
| Server and browser support |
The same API works with onnxruntime-node (server), onnxruntime-web (browser), and mock implementations (testing) |
| ESM-first |
All libraries are ESM modules, compatible with modern bundlers and runtimes |
| Zero external cloud dependencies |
All inference runs locally; no API keys or network access required |
Capability Summary by Library#
@nous/core -- Inference Runtime (84 modules)#
| Category |
Module Count |
Key Capabilities |
| Core Types |
1 |
ONNX tensor types, session abstractions, inference request/result types |
| Execution Providers |
8 |
CPU, CUDA, ROCm, Metal, WebGPU, Vulkan, DML, NPU |
| Graph Optimization |
4 |
Graph optimization, memory planning, operator fusion, quantization |
| Batching and Serving |
3 |
Dynamic batching, continuous batching, LLM serving |
| Attention and KV Cache |
4 |
KV cache, paged attention, flash attention, speculative decoding |
| Parallelism |
4 |
Tensor parallelism, pipeline parallelism, model sharding, MoE parallelism |
| Caching |
3 |
Prefix caching, radix attention, prompt caching |
| Multi-Adapter |
3 |
Multi-LoRA serving, adapter switching, context extension |
| Generation Control |
6 |
Streaming, stop sequences, logit processors, temperature, top-k/p, beam search |
| Model Management |
14 |
Registry, versioning, metadata, discovery, download, checksums, conversion, format detection, caching, eviction, preloading, warm pool, benchmarking, profiling |
| Model Formats |
6 |
SafeTensors, GGUF, GGML, PyTorch, TensorFlow, JAX |
| Embeddings |
10 |
Generation, text, image, audio, multi-modal, normalization, dimensionality reduction, batch, streaming, caching |
| Similarity Search |
7 |
ANN, exact NN, distance metrics, hybrid search, reranking, cross-encoder reranking |
| Embedding Compression |
3 |
Binary, Matryoshka, general compression |
| Tensor and Runtime |
3 |
Core tensor, session manager, inference engine |
@nous/llm -- LLM API Layer (77 modules)#
| Category |
Module Count |
Key Capabilities |
| Completion APIs |
5 |
Completion, chat, instruction following, multi-turn, system prompts |
| Function Calling |
3 |
Function calling, tool use framework, tool selection |
| Structured Output |
5 |
JSON mode, structured output, grammar constraints, regex constraints, schema validation |
| Response Processing |
5 |
Formatting, markdown, code extraction, citation extraction, fact extraction |
| NLP Tasks |
3 |
Entity extraction, sentiment analysis, classification |
| Prompt Engineering |
7 |
Templates, versioning, testing, optimization, few-shot, example selection, dynamic examples |
| Advanced Reasoning |
6 |
Chain-of-thought, tree-of-thought, graph-of-thought, self-consistency, reflection, critique |
| Persona and Context |
5 |
Persona management, context injection, memory injection, retrieval injection |
| Agent Architecture |
6 |
Architecture, planning, goal decomposition, task execution, action selection, multi-step reasoning |
| Agent Error Handling |
2 |
Backtracking, error recovery |
| Agent Memory |
6 |
Agent memory, short-term, long-term, episodic, semantic, working memory |
| Multi-Agent |
4 |
Communication, multi-agent systems, coordination, delegation, supervision |
| Document Processing |
12 |
Ingestion, chunking (semantic, sentence, token, recursive), parsing (document, PDF, HTML, markdown, code) |
| Content Extraction |
4 |
Tables, images, metadata, links |
| Vector Indexing |
4 |
Vector indexing, hybrid indexing, incremental indexing, index optimization |
| Context Retrieval |
1 |
Context retrieval for RAG |
@nous/vision -- Computer Vision (40 modules)#
| Category |
Module Count |
Key Capabilities |
| Detection and Classification |
5 |
Object detection, image classification, instance/semantic/panoptic segmentation |
| Facial Analysis |
5 |
Recognition, landmarks, restoration, emotion, age/gender |
| Document and OCR |
6 |
OCR, handwriting, document layout, tables, barcodes, license plates |
| Scene Understanding |
5 |
Depth, surface normals, pose, VQA, captioning |
| Diffusion Models |
5 |
Stable Diffusion, SDXL, SD3, Flux, diffusion framework |
| Image Editing |
9 |
Image-to-image, inpainting, outpainting, background removal, style transfer, composition, editing, upscaling, variation |
| Fine-Tuning |
5 |
LoRA, DreamBooth, textual inversion, ControlNet, IP-Adapter |