Comprehensive development guide for the Nous (LLM Inference) domain libraries.
This guide covers local development setup, building, testing, debugging, and common development tasks for the 2 Nous libraries.
Prerequisites#
Required Software#
| Tool | Version | Purpose |
|---|---|---|
| Node.js | >= 20.0 | JavaScript runtime |
| pnpm | >= 10.0 | Package manager |
| Nx | (via pnpm) | Monorepo task runner |
Optional Software#
| Tool | Purpose |
|---|---|
| NVIDIA CUDA | GPU inference via CUDA execution provider |
| AMD ROCm | GPU inference via ROCm execution provider |
onnxruntime-node |
Native ONNX Runtime (optional dependency of core) |
Installation#
# From the repository root
pnpm install
No databases or external services are required. onnxruntime-node is declared
as an optional dependency and is not needed for development or testing of the
TypeScript modules.
Project Structure#
Nous has 2 libraries under libs/nous/:
libs/nous/
core/ nous-core Inference runtime (84 modules)
llm/ nous-llm LLM API layer (29 modules)
nous-core - Inference Runtime (84 modules)#
The core library is the largest single library in the Oshun monorepo. It provides a complete inference runtime organized into these logical groups:
| Group | Modules | Description |
|---|---|---|
| Core Types | 1 | ONNX Runtime type abstractions |
| Execution Providers | 8 | CPU, CUDA, ROCm, Metal, WebGPU, Vulkan, DML, NPU |
| Graph Optimization | 4 | Memory planning, operator fusion, quantization |
| Batching/Serving | 3 | Dynamic batching, continuous batching, LLM serving |
| Attention/KV Cache | 4 | KV cache, paged attention, flash attention, speculative decoding |
| Parallelism | 4 | Tensor, pipeline, model sharding, MoE |
| Caching | 3 | Prefix, radix, prompt caching |
| Multi-Adapter | 3 | Multi-LoRA serving, adapter switching, context extension |
| Generation Control | 6 | Streaming, stop sequences, sampling, beam search |
| Model Management | 10 | Registry, versioning, discovery, download, conversion |
| Model Formats | 5 | SafeTensors, GGUF, GGML, PyTorch, core tensor |
| Embeddings | 10 | Text, image, audio, multi-modal, normalization, batch, caching |
| Similarity Search | 7 | Approximate/exact nearest neighbor, distance metrics, reranking |
| Embedding Compress | 3 | Binary embeddings, Matryoshka, compression |
| Runtime | 3 | Runtime loader, session manager, inference engine |
nous-llm - LLM API Layer (29 modules)#
High-level LLM interaction APIs built on top of nous-core:
| Group | Modules | Description |
|---|---|---|
| Completion APIs | 4 | Completion, chat, instruction following, multi-turn |
| Function Calling | 2 | Tool use framework, function calling |
| Structured Output | 5 | JSON mode, grammar/regex constraints, schema validation |
| Response Processing | 5 | Formatting, markdown, code/citation/fact extraction |
| NLP Tasks | 4 | Entity extraction, sentiment, classification |
| Prompt Engineering | 7 | Templates, versioning, testing, optimization, few-shot |
| Reasoning | 2 | Chain-of-thought, dynamic examples |
Per-Library File Layout#
libs/nous/{library}/
src/
index.ts Public API barrel export
types.ts Core type definitions (nous-core only)
{module}.ts Module implementation
{module}.spec.ts Module tests
package.json
project.json
tsconfig.json
tsconfig.lib.json
tsconfig.spec.json
vitest.config.ts
Building Libraries#
Build a Single Library#
pnpm nx build nous-core
pnpm nx build nous-llm
Important: Nous uses Nx project names without the @ scope prefix (e.g.,
nous-core not @nous/core). The npm package names in package.json use the
@nous/ scope, but Nx commands reference the project.json name.
Build Configuration#
Both Nous libraries use @nx/esbuild:esbuild (not @nx/js:tsc):
| Setting | Value |
|---|---|
| Executor | @nx/esbuild:esbuild |
| Output format | ESM (format: ["esm"]) |
| Bundling | Disabled (bundle: false) |
| Output path | libs/nous/{library}/dist/ |
esbuild compiles significantly faster than tsc but does not perform type
checking. The separate typecheck target handles type validation.
Type Checking#
Both libraries include an explicit typecheck target:
pnpm nx typecheck nous-core
pnpm nx typecheck nous-llm
Since esbuild strips types without checking them, always run typecheck before committing. A successful build does NOT guarantee type safety.
Common Workflow#
# Full validation cycle
pnpm nx build nous-core
pnpm nx typecheck nous-core
pnpm nx test nous-core
pnpm nx lint nous-core
Testing#
Run Tests for a Single Library#
pnpm nx test nous-core
pnpm nx test nous-llm
Test Configuration#
import { defineConfig } from 'vitest/config';
export default defineConfig({
test: {
globals: true,
environment: 'node',
root: __dirname,
passWithNoTests: false,
},
});
Key note: passWithNoTests: false is set on nous-core, meaning the test
target will fail if no tests are found. This ensures test files exist for all
modules.
Test Count#
| Library | Test Files | Description |
|---|---|---|
nous-core |
82 | Execution providers, optimization, serving, model formats, embeddings, search |
nous-llm |
28 | Completion APIs, structured output, prompt engineering, NLP tasks |
| Total | 110 |
Running Tests with Coverage#
pnpm nx test nous-core -- --coverage
pnpm nx test nous-llm -- --coverage
Running a Specific Test File#
pnpm nx test nous-core -- --testPathPattern="dynamic-batching"
pnpm nx test nous-llm -- --testPathPattern="prompt-template"
Writing Tests#
Nous tests typically validate inference pipeline logic without requiring actual model files or GPU hardware. Use mocks for ONNX Runtime interfaces:
import { describe, it, expect, vi } from 'vitest';
import {
createDynamicBatcher,
DynamicBatcherConfig,
} from './dynamic-batching.js';
import type { OnnxInferenceSessionLike, OnnxTensorLike } from './types.js';
describe('DynamicBatching', () => {
const mockSession: OnnxInferenceSessionLike = {
inputNames: ['input'],
outputNames: ['output'],
run: vi.fn().mockResolvedValue({
output: {
type: 'float32',
data: new Float32Array([1, 2, 3]),
dims: [1, 3],
},
}),
};
it('should batch multiple requests together', async () => {
const batcher = createDynamicBatcher(mockSession, {
maxBatchSize: 4,
maxWaitTimeMs: 10,
});
const results = await Promise.all([
batcher.infer({ input: tensor }),
batcher.infer({ input: tensor }),
]);
expect(mockSession.run).toHaveBeenCalledTimes(1); // Batched into one call
expect(results).toHaveLength(2);
});
});
Testing Execution Providers#
Each execution provider has its own spec file. Tests verify configuration validation and provider-specific option handling without requiring the actual hardware backend:
describe('CUDAExecutionProvider', () => {
it('should configure device selection', () => {
const provider = createCUDAProvider({ deviceId: 0 });
expect(provider.name).toBe('cuda');
expect(provider.options.deviceId).toBe(0);
});
it('should reject invalid device IDs', () => {
expect(() => createCUDAProvider({ deviceId: -1 })).toThrow();
});
});
Linting#
# Lint a single library
pnpm nx lint nous-core
pnpm nx lint nous-llm
Common Development Tasks#
Adding a New Module to nous-core#
- Create
libs/nous/core/src/{new-module}.ts - Create
libs/nous/core/src/{new-module}.spec.ts - Export from
libs/nous/core/src/index.ts:
export * from './{new-module}.js';
- Verify:
pnpm nx build nous-core
pnpm nx typecheck nous-core
pnpm nx test nous-core
pnpm nx lint nous-core
Adding a New Execution Provider#
To add support for a new hardware backend:
- Create
libs/nous/core/src/{backend}-execution-provider.ts - Implement the
OnnxExecutionProviderConfiginterface - Add the provider name to the
OnnxExecutionProviderNameunion type intypes.ts - Create tests in
{backend}-execution-provider.spec.ts - Export from
index.ts
Adding a New Model Format#
To support a new model file format:
- Create
libs/nous/core/src/{format}.tsimplementing loading and parsing - Register with
format-detection.tsfor auto-detection - Add conversion support in
model-conversion.tsif applicable - Create tests and export
Adding a New LLM API Feature#
For high-level LLM features, add to nous-llm:
- Create
libs/nous/llm/src/{feature}.ts - Create
libs/nous/llm/src/{feature}.spec.ts - Export from
libs/nous/llm/src/index.ts - Verify all four targets (build, typecheck, test, lint)
Integrating Nous with Other Domains#
Other domains consume Nous for local inference capabilities:
{
"dependencies": {
"@nous/core": "workspace:*"
}
}
import {
createInferenceEngine,
loadModel,
createDynamicBatcher,
} from '@nous/core';
import { createChatCompletionAPI, createPromptTemplate } from '@nous/llm';
Key Patterns and Conventions#
ONNX Runtime Abstraction#
All inference operations are abstracted behind the OnnxInferenceSessionLike
interface. This interface wraps the actual ONNX Runtime session, allowing the
library to work with:
onnxruntime-node(server-side)onnxruntime-web(browser-side)- Mock implementations (testing)
When writing new inference code, always program against these interfaces rather
than importing onnxruntime-node directly.
Optional Dependency Pattern#
onnxruntime-node is listed as an optionalDependency in @nous/core's
package.json. This means:
- The library can be imported and used without ONNX Runtime installed
- Runtime loading uses dynamic imports with graceful fallbacks
- Browser environments use WebGPU instead of native ONNX Runtime
Tensor Type Safety#
The OnnxTensorLike<TData> generic interface enforces typed tensor data:
interface OnnxTensorLike<TData extends OnnxTensorData = OnnxTensorData> {
readonly type: string;
readonly data: TData;
readonly dims: readonly number[];
}
Use specific typed arrays (Float32Array, Int8Array, etc.) rather than
generic OnnxTensorData when the data type is known.
Module Flat Export#
Both libraries export all modules from a flat index.ts. There are no namespace
exports or sub-module grouping at the package level:
export * from './dynamic-batching.js';
export * from './continuous-batching.js';
export * from './speculative-decoding.js';
Consumers import directly:
import {
createDynamicBatcher,
ContinuousBatcher,
SpeculativeDecoder,
} from '@nous/core';
Execution Provider Configuration#
Each execution provider follows a consistent pattern:
export interface CUDAProviderConfig {
deviceId?: number;
cudnnConvAlgo?: string;
// ... provider-specific options
}
export function createCUDAProvider(
config?: CUDAProviderConfig
): OnnxExecutionProviderConfig {
return {
name: 'cuda',
options: {
/* validated config */
},
};
}
Debugging Tips#
esbuild Does Not Check Types#
The most common pitfall: a build succeeds but the code has type errors. Always run typecheck:
pnpm nx typecheck nous-core # Catches type errors that esbuild misses
Mock ONNX Runtime for Testing#
Since onnxruntime-node may not be installed or may not work in CI, always mock
the ONNX Runtime interfaces:
const mockSession: OnnxInferenceSessionLike = {
inputNames: ['input_ids', 'attention_mask'],
outputNames: ['logits'],
run: vi.fn().mockResolvedValue({
logits: {
type: 'float32',
data: new Float32Array(vocab_size),
dims: [1, seq_len, vocab_size],
},
}),
};
Tensor Shape Debugging#
When tensor operations produce unexpected results, log the dims:
console.log('Input dims:', tensor.dims);
console.log('Data length:', tensor.data.length);
console.log(
'Expected elements:',
tensor.dims.reduce((a, b) => a * b, 1)
);
Batch Size Calculation#
Dynamic batching tests often fail due to timing. Use vi.useFakeTimers() when
testing batch window behavior:
vi.useFakeTimers();
const batcher = createDynamicBatcher(session, { maxWaitTimeMs: 50 });
const p1 = batcher.infer(input1);
const p2 = batcher.infer(input2);
vi.advanceTimersByTime(50); // Trigger batch window close
await Promise.all([p1, p2]);
vi.useRealTimers();
Import Path Extensions#
ESM requires .js extensions:
// Correct
import { OnnxTensorLike } from './types.js';
// Incorrect
import { OnnxTensorLike } from './types';
Large Module Count#
With 84 modules in nous-core and 29 in nous-llm, build and test times can be
significant. Target specific modules during development:
# Test only the module you are working on
pnpm nx test nous-core -- --testPathPattern="paged-attention"
Project Configuration Reference#
Tags#
Both Nous libraries use: ["scope:nous", "layer:infra", "type:lib"]
Note: layer:infra (not layer:domain) reflects that Nous is infrastructure
used by other domains rather than a domain itself.
Executors#
| Target | Executor |
|---|---|
| build | @nx/esbuild:esbuild |
| lint | @nx/eslint:lint |
| test | @nx/vite:test |
| typecheck | nx:run-commands (tsc) |
Dependencies#
| Dependency | Library | Type | Purpose |
|---|---|---|---|
onnxruntime-node |
nous-core |
Optional | Native ONNX Runtime |
@types/node |
Both | Dev | Node.js type defs |
typescript |
Both | Dev | Type checking |
vitest |
Both | Dev | Test framework |
Environment Variables#
No environment variables are required for library development. For actual model inference at runtime, the following may be needed:
# GPU provider selection
NOUS_EXECUTION_PROVIDER=cuda # or cpu, metal, webgpu, etc.
# Model paths (application-level, not library-level)
NOUS_MODEL_PATH=/path/to/models
Related Documentation#
- Nous Domain README - Domain overview, architecture, and library descriptions
- CLAUDE.md - Repository-wide development guidelines