Eve library

Nous Domain -- Product Capabilities

Guide

Revision 1

On this page

Canonical source: the canonical features document for this domain is deep-dive/features.md, rendered in the docs center. This file is its operational companion (service topology, APIs, schemas, runtime detail) — reconciled by linking, not duplicated (§2, §13).

Nous provides local LLM inference, model management, embedding generation, agent orchestration, document processing, and prompt engineering as consumable library APIs for any application in the Oshun ecosystem.

Nous is the AI inference backbone of Oshun. It enables applications to run large language models locally on any hardware, generate and search embeddings, build autonomous agents, process documents into vector indexes, and engineer prompts with full template systems -- all without depending on external cloud AI services.

The domain ships three libraries:

Library Package Modules Focus
nous-core @nous/core 84 Inference runtime, hardware providers, model management, embeddings, search
nous-llm @nous/llm 77 LLM APIs, agents, prompt engineering, document processing, RAG
nous-vision @nous/vision 40 Computer vision, image generation, diffusion models, OCR

1. Hardware-Accelerated Inference#

Nous abstracts the ONNX Runtime behind a unified provider interface, letting consuming applications run inference on any available hardware without code changes.

1.1 Execution Providers#

Capability Provider What It Enables
CPU inference cpu Universal fallback; runs on any machine without GPU drivers
NVIDIA GPU inference cuda High-throughput inference on NVIDIA GPUs with CUDA
AMD GPU inference rocm Inference on AMD Radeon and Instinct GPUs
Apple GPU inference metal Metal-accelerated inference on macOS and iOS devices
Browser GPU inference webgpu In-browser GPU inference via the WebGPU API
Cross-platform GPU vulkan Vulkan-based GPU inference across Linux, Windows, and Android
Windows GPU inference dml DirectML inference on any DirectX 12-capable Windows GPU
Neural Processing Units npu Inference on dedicated NPU/TPU hardware accelerators

Each provider offers device selection (e.g., GPU device ID), tuning parameters, and graceful fallback to the next available provider when the preferred one is unavailable.

1.2 Provider Routing and Fallback#

Capability Description
Preferred provider ordering Specify a ranked list of providers; the runtime tries each in order
Automatic fallback When a provider fails or is unavailable, the next provider is used transparently
Provider configuration validation Invalid settings (e.g., negative device IDs) are rejected at configuration time
Runtime module loading ONNX Runtime is loaded dynamically; applications that do not need inference can skip the dependency entirely

1.3 Session Management#

Capability Description
Managed sessions Sessions are keyed, pooled, and released automatically
Session lifecycle Create, run, release, and dispose of inference sessions with full lifecycle control
Concurrent session support Multiple models can be loaded and run concurrently
Feed and fetch validation Input tensor names and output names are validated before execution

2. Model Management#

A complete model lifecycle system from discovery through deployment and optimization.

2.1 Model Registry and Discovery#

Capability Description
Central model registry Register, list, look up, and remove models with metadata
Model discovery Discover available models from configured sources and repositories
Model versioning Track multiple versions of the same model with immutable version records
Model metadata Attach and query structured metadata (architecture, capabilities, license, parameter count, context length)

2.2 Model Acquisition#

Capability Description
Model download Download models from remote sources with progress tracking
Checksum verification Verify model file integrity using SHA-256 and other hash algorithms
Format auto-detection Automatically identify the format of a model file (SafeTensors, GGUF, GGML, PyTorch, TensorFlow, JAX)

2.3 Model Format Support#

Format Capability
SafeTensors Load and parse Hugging Face SafeTensors files with tensor metadata extraction
GGUF Load GGUF (llama.cpp) quantized model files with header and metadata parsing
GGML Load legacy GGML format model files
PyTorch Load PyTorch .pt and .pth checkpoint files
TensorFlow Load TensorFlow SavedModel and checkpoint formats
JAX Load JAX/Flax model parameters

2.4 Model Conversion and Optimization#

Capability Description
Cross-format conversion Convert models between supported formats
End-to-end optimization pipeline Chain format detection, loading, quantization, and optimization in a single pipeline
Model benchmarking Benchmark model inference speed, throughput, and latency across providers
Model profiling Profile model execution to identify bottlenecks and memory hotspots

2.5 Model Caching and Lifecycle#

Capability Description
Model caching Cache loaded models in memory to avoid repeated loading
Model eviction Evict least-recently-used or least-frequently-used models when memory pressure rises
Model preloading Preload models at startup for instant inference availability
Warm model pool Maintain a pool of ready-to-run model sessions for zero-latency first inference

3. Inference Pipeline#

High-throughput, production-grade inference with batching, caching, and parallelism.

3.1 Dynamic and Continuous Batching#

Capability Description
Dynamic batching Accumulate incoming requests into batches with configurable maximum batch size and queue delay
Continuous batching Continuously batch token-generation requests for LLM serving without waiting for entire sequences to complete
Batch key routing Group requests into batches by model/session key so different models are batched independently
Queue management Configurable maximum queue sizes, backpressure, and dropped-request tracking
Batch statistics Track pending requests, batches executed, average batch size, and failure rates

3.2 LLM Serving#

Capability Description
Full serving pipeline End-to-end LLM serving with request routing, batching, and response streaming
Request prioritization Prioritize inference requests based on caller-specified priority
Throughput optimization Maximize GPU utilization through intelligent request scheduling

3.3 Attention Mechanisms and KV Cache#

Capability Description
KV cache management Allocate, grow, and evict key-value caches for autoregressive generation
Paged attention Memory-efficient KV cache using virtual-memory-style paging to reduce fragmentation
Flash attention Fused attention kernels that reduce memory reads and improve throughput
Speculative decoding Use a smaller draft model to speculatively generate tokens, then verify with the full model for faster generation

3.4 Multi-GPU Parallelism#

Capability Description
Tensor parallelism Shard model layers across multiple GPUs for larger-than-single-GPU models
Pipeline parallelism Pipeline model stages across GPUs for higher throughput
Model sharding Automatic sharding strategies that split models across available devices
Mixture-of-Experts parallelism Route different experts to different GPUs for MoE model architectures

3.5 Prompt and Prefix Caching#

Capability Description
Prefix caching Cache common prompt prefixes so repeated system prompts do not re-compute attention
Radix attention caching Radix-tree-based attention caching for efficient prefix sharing across requests
Prompt result caching Cache full prompt-to-response mappings for deterministic queries

4. Generation Control#

Fine-grained control over how tokens are generated and when generation stops.

4.1 Sampling Strategies#

Capability Description
Temperature sampling Control randomness with temperature scaling of logit distributions
Top-k sampling Restrict token selection to the top-k most probable tokens
Top-p (nucleus) sampling Restrict token selection to the smallest set of tokens whose cumulative probability exceeds p
Beam search Search for the most probable sequence using beam search with configurable beam width

4.2 Logit Processing#

Capability Description
Custom logit processors Chain multiple logit processors into a pipeline (e.g., repetition penalty, frequency penalty)
Repetition penalty Penalize already-generated tokens to reduce repetitive output
Logit bias Apply per-token biases to steer generation toward or away from specific tokens

4.3 Streaming and Stop Conditions#

Capability Description
Streaming generation Stream tokens to the consumer as they are generated, with backpressure support
Stop sequences Halt generation when specific token sequences are produced
Max token limits Cap generation at a configurable maximum token count

5. Multi-Adapter Serving#

Serve multiple fine-tuned model variants from a single base model.

Capability Description
Multi-LoRA serving Serve multiple LoRA adapters concurrently from a shared base model, reducing total memory usage
Hot adapter switching Swap LoRA adapters in and out without reloading the base model
Context extension Extend a model's effective context length beyond its original training limit using RoPE scaling and other techniques

6. Graph Optimization and Quantization#

Optimize models for faster inference and reduced memory usage.

6.1 Graph Optimization#

Capability Description
ONNX graph optimization Apply basic, extended, or all graph-level optimizations to reduce operator overhead
Operator fusion Fuse compatible operators (e.g., MatMul + Add, Conv + BN) into single operations
Memory planning Optimize memory allocation patterns with arena-based memory planning

6.2 Quantization#

Capability Description
FP16 quantization Convert float32 tensors to float16 for 2x memory reduction with minimal accuracy loss
INT8 quantization Quantize to 8-bit integers with symmetric or asymmetric modes
INT4 quantization Quantize to 4-bit integers (packed nibbles) for 8x memory reduction
Dynamic quantization Quantize weights on-the-fly using observed tensor ranges
Static quantization Quantize weights using calibration data for higher accuracy
Calibration collection Collect tensor value ranges across calibration batches for static quantization
Percentile calibration Use percentile-based range clipping to reduce outlier impact on quantization accuracy
Batch quantization Quantize entire tensor maps in one pass with compression ratio tracking
Dequantization Convert quantized tensors back to float32 for analysis or mixed-precision inference

7. Tensor Operations#

Low-level tensor creation, manipulation, and type-safe data handling.

Capability Description
Typed tensor creation Create tensors with explicit types: float32, float64, int8, uint8, int16, uint16, int32, uint32, int64, uint64, bool
Dimension tracking All tensors carry immutable dimension metadata
Typed array backing Tensors are backed by native typed arrays (Float32Array, Int8Array, etc.) for zero-copy performance
Tensor validation Validate tensor shapes, data types, and element counts before inference

8. Embedding Generation#

Generate, manage, and search vector embeddings across text, image, audio, and multi-modal content.

8.1 Embedding Providers#

Capability Description
Multi-provider embedding Register multiple embedding providers with priority-based routing
Provider fallback Automatically fall back to the next provider when one fails
Provider statistics Track per-provider success rates, failure rates, and average latency
Configurable timeouts Set per-request timeout limits with provider-level timeout tracking

8.2 Modality Support#

Modality What It Enables
Text embeddings Generate vector representations of text passages for semantic search and similarity
Image embeddings Generate vector representations of images for visual search and classification
Audio embeddings Generate vector representations of audio for speaker identification and audio search
Multi-modal embeddings Generate combined embeddings from multiple modalities (e.g., text + image)

8.3 Embedding Processing#

Capability Description
Embedding normalization Normalize embedding vectors to unit length for cosine similarity
Dimensionality reduction Reduce embedding dimensions while preserving semantic relationships
Output dimension control Truncate or reduce embeddings to a requested dimension count
Batch embedding Generate embeddings for multiple inputs in a single batched request
Streaming embedding Generate embeddings in a streaming fashion for large input sets
Embedding caching Cache generated embeddings to avoid redundant computation

8.4 Embedding Compression#

Capability Description
Binary embeddings Compress float embeddings to binary (1-bit) vectors for ultra-fast approximate search
Matryoshka embeddings Generate variable-dimension embeddings where truncated prefixes remain semantically meaningful
General compression Apply lossy and lossless compression to embedding vectors for storage reduction

Search and rank documents using vector similarity.

Capability Description
Approximate nearest neighbor (ANN) Fast approximate search using index structures (e.g., HNSW, IVF) for large-scale retrieval
Exact nearest neighbor Brute-force exact search for highest accuracy on smaller datasets
Hybrid search Combine vector similarity with keyword/BM25 search for improved relevance

9.2 Distance Metrics#

Capability Description
Cosine similarity Measure angular similarity between vectors (direction-based)
Euclidean distance Measure straight-line distance between vectors
Dot product Measure alignment between vectors (magnitude-sensitive)
Manhattan distance Measure L1 distance between vectors

9.3 Reranking#

Capability Description
Score-based reranking Rerank search results using a scoring model
Cross-encoder reranking Use cross-encoder models to deeply score query-document pairs for high-precision reranking

10. LLM Completion APIs#

High-level interfaces for generating text, holding conversations, and following instructions.

10.1 Completion Interfaces#

Capability Description
Text completion Generate continuations of a text prompt
Chat completion Conduct multi-turn conversations with system, user, and assistant message roles
Instruction following Generate responses that follow structured instructions
Multi-turn conversation Maintain conversation history across multiple exchanges with context management
System prompts Define persistent system-level instructions that shape all responses

10.2 Function Calling and Tool Use#

Capability Description
Function calling Declare callable functions with typed parameters; the LLM generates structured function call requests
Tool use framework Define tools with descriptions and schemas; the LLM selects and invokes them as needed
Tool selection Intelligent selection of appropriate tools based on the current task context

10.3 Structured Output#

Capability Description
JSON mode Constrain LLM output to valid JSON
Structured output Generate output conforming to a specified schema
Grammar constraints Constrain generation to outputs matching a formal grammar (BNF/EBNF)
Regex constraints Constrain generation to outputs matching a regular expression pattern
Schema validation Validate generated output against JSON Schema definitions

10.4 Response Processing#

Capability Description
Response formatting Format and structure raw LLM output
Markdown rendering Render LLM output as well-formed markdown
Code extraction Extract code blocks and snippets from LLM output
Citation extraction Extract citations and references from LLM output
Fact extraction Extract factual claims and assertions from LLM output

11. Natural Language Processing#

Built-in NLP tasks that run locally.

Capability Description
Entity extraction Identify and extract named entities (people, places, organizations, dates) from text
Sentiment analysis Classify the emotional tone of text (positive, negative, neutral, mixed)
Text classification Classify text into predefined or dynamic categories

12. Prompt Engineering#

A full prompt template system with variable substitution, conditional logic, loops, versioning, testing, and optimization.

12.1 Template System#

Capability Description
Variable substitution Insert values into prompts using {{variable}} syntax with dot-notation for nested access
Conditional sections Include or exclude prompt sections based on variable values with {{#if}}/{{else}}/{{/if}}
Comparison operators Support == and != comparisons in conditions for precise branching
Negation Negate conditions with ! prefix for inverted logic
Loop constructs Iterate over arrays with {{#each items as item}}/{{/each}} with index tracking
Nested values Access deeply nested object properties via dot notation (e.g., {{user.profile.name}})
Strict mode Throw errors on missing variables instead of leaving placeholders
Missing value defaults Provide a fallback string for unresolved variables
Object serialization Automatically stringify complex objects in template output

12.2 Template Management#

Capability Description
Template registration Register reusable prompt templates with IDs, names, descriptions, and tags
Template versioning Track template versions and revision history
Template listing and search List and filter templates by tag or search text
Template overwrite control Optionally overwrite existing templates or reject duplicates
Variable auto-extraction Automatically detect variables used in template messages

12.3 Prompt Versioning and Testing#

Capability Description
Prompt versioning Version and track prompt changes over time for reproducibility
Prompt testing Test prompts against expected outputs for quality assurance
Prompt optimization Optimize prompt wording and structure for improved model performance

12.4 Few-Shot and Examples#

Capability Description
Few-shot management Manage collections of few-shot examples for in-context learning
Example selection Select the most relevant examples for a given query
Dynamic examples Generate or retrieve examples dynamically based on the input context

13. Advanced Reasoning#

Structured reasoning strategies that improve LLM output quality.

Capability Description
Chain-of-thought Guide the LLM through step-by-step reasoning before producing a final answer
Tree-of-thought Explore multiple reasoning branches and select the most promising path
Graph-of-thought Model reasoning as a directed graph where nodes share and build on insights
Self-consistency Generate multiple reasoning chains and select the most consistent answer
Reflection prompts Prompt the LLM to reflect on and improve its own output
Critique prompts Prompt the LLM to critique its reasoning and identify weaknesses

14. Persona and Context Management#

Control LLM behavior and inject external knowledge.

Capability Description
Persona management Define and switch between named personas that shape LLM tone, expertise, and behavior
Context injection Inject external context (documents, data, user preferences) into prompts
Memory injection Inject conversation memory and historical context into prompts
Retrieval injection Inject retrieved documents and passages from vector search into prompts

15. Agent Architecture#

Build autonomous AI agents with planning, tool use, memory, and multi-agent coordination.

15.1 Agent Lifecycle#

Capability Description
Agent registration Register agents with unique IDs, names, capabilities, and module configurations
Agent lifecycle states Manage agents through stopped, idle, running, paused, and error states
Concurrency control Configure maximum concurrent tasks per agent
Agent start/stop/pause/resume Full lifecycle control with graceful task handling on state changes
Agent unregistration Cleanly remove agents with automatic queued-task cancellation

15.2 Task Execution#

Capability Description
Task submission Submit tasks with goals, payloads, context, priority, and timeout
Priority-based scheduling Tasks are executed in priority order with FIFO tiebreaking
Task timeout Tasks that exceed their timeout are terminated with timeout state tracking
Task cancellation Cancel queued tasks before execution begins
Run-next and run-all Execute the next available task or drain all queued tasks
Task state tracking Track tasks through queued, running, completed, failed, cancelled, and timeout states

15.3 Planning and Decomposition#

Capability Description
Planning module Agents can plan before executing: generate a summary, steps, and confidence score
Goal decomposition Break complex goals into prioritized subtasks before execution
Multi-step reasoning Chain multiple reasoning steps with intermediate results
Backtracking Revert to earlier reasoning states when a path fails
Error recovery Detect and recover from errors during multi-step task execution

15.4 Agent Memory#

Capability Description
Agent session state Persist key-value state across tasks within an agent session
Short-term memory Maintain recent context for ongoing conversations
Long-term memory Persist important information across sessions
Episodic memory Store and recall specific interaction episodes
Semantic memory Store and query factual knowledge in vector form
Working memory Maintain the active reasoning context during task execution

15.5 Multi-Agent Systems#

Capability Description
Agent communication Send messages between agents for collaborative task solving
Multi-agent coordination Coordinate multiple agents working on related tasks
Agent delegation Delegate subtasks from one agent to another
Agent supervision Supervise and monitor agent execution with oversight policies
Action selection Select the best action from a set of candidates based on context and goals
Tool selection Select the most appropriate tool for a given subtask

15.6 Runtime Events#

Capability Description
Event history Track all agent and task lifecycle events with timestamps
Event types Agent registered/unregistered/started/stopped, task submitted/started/completed/failed/timeout/cancelled, module events
Event history limits Configurable maximum event history size to bound memory usage
Statistics Track agents, queued/running/completed/failed tasks, average latency, and event counts

16. Document Processing and RAG#

Ingest, parse, chunk, and index documents for retrieval-augmented generation.

16.1 Document Ingestion#

Capability Description
Document ingestion pipeline End-to-end pipeline from raw document to indexed chunks
PDF parsing Extract text, structure, and metadata from PDF documents
HTML parsing Parse and extract clean text from HTML pages
Markdown parsing Parse markdown documents preserving structure
Code parsing Parse source code files with language-aware structure extraction

16.2 Content Extraction#

Capability Description
Table extraction Extract tabular data from documents
Image extraction Extract embedded images from documents
Metadata extraction Extract document metadata (title, author, date, keywords)
Link extraction Extract hyperlinks and references from documents

16.3 Chunking Strategies#

Capability Description
Semantic chunking Split documents at semantic boundaries for coherent chunks
Sentence chunking Split documents at sentence boundaries
Token chunking Split documents based on token counts for model input limits
Recursive chunking Recursively split documents using multiple strategies for optimal chunk sizes
Configurable chunking Configure chunk sizes, overlaps, and splitting heuristics

16.4 Vector Indexing#

Capability Description
Vector indexing Index document chunks as embedding vectors for semantic retrieval
Hybrid indexing Combine vector and keyword indexes for hybrid search
Incremental indexing Add, update, and remove documents without rebuilding the entire index
Index optimization Optimize index structures for query performance

16.5 Context Retrieval#

Capability Description
Context retrieval Retrieve the most relevant document chunks for a query using vector similarity
Multi-source retrieval Retrieve from multiple indexes and merge results
Relevance scoring Score and rank retrieved chunks by relevance

17. Computer Vision#

Local computer vision inference covering detection, segmentation, recognition, and generative image models.

17.1 Object Detection and Classification#

Capability Description
Object detection Detect and locate objects in images with bounding boxes
Image classification Classify images into categories
Instance segmentation Segment individual object instances with pixel-level masks
Semantic segmentation Assign a class label to every pixel in an image
Panoptic segmentation Combine instance and semantic segmentation for complete scene understanding

17.2 Facial Analysis#

Capability Description
Facial recognition Detect and identify faces with feature embedding extraction
Face landmark detection Detect facial landmarks (eyes, nose, mouth, jawline)
Face restoration Restore and enhance degraded or low-resolution faces
Emotion recognition Classify facial expressions into emotional categories
Age and gender estimation Estimate age range and gender from facial features

17.3 Document and Text Recognition#

Capability Description
OCR engine Extract text from images using optical character recognition
Handwriting recognition Recognize and transcribe handwritten text
Document layout analysis Analyze document structure (headers, paragraphs, columns, tables)
Table recognition Detect and extract tabular structures from document images
Barcode and QR recognition Detect and decode barcodes and QR codes
License plate recognition Detect and read vehicle license plates

17.4 Scene Understanding#

Capability Description
Depth estimation Estimate per-pixel depth from monocular images
Surface normal estimation Estimate surface normal directions from images
Pose estimation Detect human body pose keypoints
Visual question answering Answer natural language questions about image content
Image captioning Generate natural language descriptions of image content

17.5 Image Generation (Diffusion Models)#

Capability Description
Stable Diffusion Text-to-image generation using Stable Diffusion models
SDXL support Extended support for Stable Diffusion XL models
SD3 support Support for Stable Diffusion 3 architecture
Flux support Support for Flux model architecture
Diffusion model framework General framework for running diffusion model pipelines locally

17.6 Image Editing and Transformation#

Capability Description
Image-to-image Transform images using diffusion models with guidance
Inpainting Fill in masked regions of images with contextually appropriate content
Outpainting Extend images beyond their original boundaries
Background removal Remove or replace image backgrounds
Style transfer Apply artistic styles from one image to another
Image composition Composite multiple images with blending and alignment
Image editing Apply targeted edits to specific regions of images
Upscaling (ESRGAN) Enhance image resolution using Real-ESRGAN super-resolution models
Variation generation Generate variations of an existing image

17.7 Fine-Tuning and Adaptation#

Capability Description
LoRA management Load, apply, and manage LoRA adapters for diffusion models
DreamBooth training Fine-tune diffusion models on custom subjects using DreamBooth
Textual inversion Learn new concepts as text embeddings for personalized generation
ControlNet integration Guide image generation with structural inputs (edges, depth, pose)
IP-Adapter support Use reference images to guide the style and content of generated images

18. Cross-Cutting Capabilities#

18.1 Observability#

Capability Description
Inference statistics Track requests, successes, failures, latency, and throughput across all subsystems
Provider-level metrics Per-provider success rates, failure rates, and latency
Batch statistics Track batch sizes, queue depths, and utilization
Agent runtime statistics Track agent and task lifecycle metrics
Template system statistics Track render counts, failures, and latency for prompt templates

18.2 Type Safety#

Capability Description
Full TypeScript types Every API surface is fully typed with generics, unions, and readonly constraints
Tensor type safety Generic OnnxTensorLike<TData> enforces typed array backing at compile time
Immutable interfaces All public interfaces use readonly properties to prevent accidental mutation

18.3 Runtime Flexibility#

Capability Description
Optional ONNX Runtime The ONNX Runtime dependency is optional; the library can be imported without it installed
Server and browser support The same API works with onnxruntime-node (server), onnxruntime-web (browser), and mock implementations (testing)
ESM-first All libraries are ESM modules, compatible with modern bundlers and runtimes
Zero external cloud dependencies All inference runs locally; no API keys or network access required

Capability Summary by Library#

@nous/core -- Inference Runtime (84 modules)#

Category Module Count Key Capabilities
Core Types 1 ONNX tensor types, session abstractions, inference request/result types
Execution Providers 8 CPU, CUDA, ROCm, Metal, WebGPU, Vulkan, DML, NPU
Graph Optimization 4 Graph optimization, memory planning, operator fusion, quantization
Batching and Serving 3 Dynamic batching, continuous batching, LLM serving
Attention and KV Cache 4 KV cache, paged attention, flash attention, speculative decoding
Parallelism 4 Tensor parallelism, pipeline parallelism, model sharding, MoE parallelism
Caching 3 Prefix caching, radix attention, prompt caching
Multi-Adapter 3 Multi-LoRA serving, adapter switching, context extension
Generation Control 6 Streaming, stop sequences, logit processors, temperature, top-k/p, beam search
Model Management 14 Registry, versioning, metadata, discovery, download, checksums, conversion, format detection, caching, eviction, preloading, warm pool, benchmarking, profiling
Model Formats 6 SafeTensors, GGUF, GGML, PyTorch, TensorFlow, JAX
Embeddings 10 Generation, text, image, audio, multi-modal, normalization, dimensionality reduction, batch, streaming, caching
Similarity Search 7 ANN, exact NN, distance metrics, hybrid search, reranking, cross-encoder reranking
Embedding Compression 3 Binary, Matryoshka, general compression
Tensor and Runtime 3 Core tensor, session manager, inference engine

@nous/llm -- LLM API Layer (77 modules)#

Category Module Count Key Capabilities
Completion APIs 5 Completion, chat, instruction following, multi-turn, system prompts
Function Calling 3 Function calling, tool use framework, tool selection
Structured Output 5 JSON mode, structured output, grammar constraints, regex constraints, schema validation
Response Processing 5 Formatting, markdown, code extraction, citation extraction, fact extraction
NLP Tasks 3 Entity extraction, sentiment analysis, classification
Prompt Engineering 7 Templates, versioning, testing, optimization, few-shot, example selection, dynamic examples
Advanced Reasoning 6 Chain-of-thought, tree-of-thought, graph-of-thought, self-consistency, reflection, critique
Persona and Context 5 Persona management, context injection, memory injection, retrieval injection
Agent Architecture 6 Architecture, planning, goal decomposition, task execution, action selection, multi-step reasoning
Agent Error Handling 2 Backtracking, error recovery
Agent Memory 6 Agent memory, short-term, long-term, episodic, semantic, working memory
Multi-Agent 4 Communication, multi-agent systems, coordination, delegation, supervision
Document Processing 12 Ingestion, chunking (semantic, sentence, token, recursive), parsing (document, PDF, HTML, markdown, code)
Content Extraction 4 Tables, images, metadata, links
Vector Indexing 4 Vector indexing, hybrid indexing, incremental indexing, index optimization
Context Retrieval 1 Context retrieval for RAG

@nous/vision -- Computer Vision (40 modules)#

Category Module Count Key Capabilities
Detection and Classification 5 Object detection, image classification, instance/semantic/panoptic segmentation
Facial Analysis 5 Recognition, landmarks, restoration, emotion, age/gender
Document and OCR 6 OCR, handwriting, document layout, tables, barcodes, license plates
Scene Understanding 5 Depth, surface normals, pose, VQA, captioning
Diffusion Models 5 Stable Diffusion, SDXL, SD3, Flux, diffusion framework
Image Editing 9 Image-to-image, inpainting, outpainting, background removal, style transfer, composition, editing, upscaling, variation
Fine-Tuning 5 LoRA, DreamBooth, textual inversion, ControlNet, IP-Adapter