Eve library

Nous Developer Guide

Guide

Revision 1

On this page

Comprehensive development guide for the Nous (LLM Inference) domain libraries.

This guide covers local development setup, building, testing, debugging, and common development tasks for the 2 Nous libraries.


Prerequisites#

Required Software#

Tool Version Purpose
Node.js >= 20.0 JavaScript runtime
pnpm >= 10.0 Package manager
Nx (via pnpm) Monorepo task runner

Optional Software#

Tool Purpose
NVIDIA CUDA GPU inference via CUDA execution provider
AMD ROCm GPU inference via ROCm execution provider
onnxruntime-node Native ONNX Runtime (optional dependency of core)

Installation#

bash
# From the repository root
pnpm install

No databases or external services are required. onnxruntime-node is declared as an optional dependency and is not needed for development or testing of the TypeScript modules.


Project Structure#

Nous has 2 libraries under libs/nous/:

text
libs/nous/
  core/    nous-core    Inference runtime (84 modules)
  llm/     nous-llm     LLM API layer (29 modules)

nous-core - Inference Runtime (84 modules)#

The core library is the largest single library in the Oshun monorepo. It provides a complete inference runtime organized into these logical groups:

Group Modules Description
Core Types 1 ONNX Runtime type abstractions
Execution Providers 8 CPU, CUDA, ROCm, Metal, WebGPU, Vulkan, DML, NPU
Graph Optimization 4 Memory planning, operator fusion, quantization
Batching/Serving 3 Dynamic batching, continuous batching, LLM serving
Attention/KV Cache 4 KV cache, paged attention, flash attention, speculative decoding
Parallelism 4 Tensor, pipeline, model sharding, MoE
Caching 3 Prefix, radix, prompt caching
Multi-Adapter 3 Multi-LoRA serving, adapter switching, context extension
Generation Control 6 Streaming, stop sequences, sampling, beam search
Model Management 10 Registry, versioning, discovery, download, conversion
Model Formats 5 SafeTensors, GGUF, GGML, PyTorch, core tensor
Embeddings 10 Text, image, audio, multi-modal, normalization, batch, caching
Similarity Search 7 Approximate/exact nearest neighbor, distance metrics, reranking
Embedding Compress 3 Binary embeddings, Matryoshka, compression
Runtime 3 Runtime loader, session manager, inference engine

nous-llm - LLM API Layer (29 modules)#

High-level LLM interaction APIs built on top of nous-core:

Group Modules Description
Completion APIs 4 Completion, chat, instruction following, multi-turn
Function Calling 2 Tool use framework, function calling
Structured Output 5 JSON mode, grammar/regex constraints, schema validation
Response Processing 5 Formatting, markdown, code/citation/fact extraction
NLP Tasks 4 Entity extraction, sentiment, classification
Prompt Engineering 7 Templates, versioning, testing, optimization, few-shot
Reasoning 2 Chain-of-thought, dynamic examples

Per-Library File Layout#

text
libs/nous/{library}/
  src/
    index.ts                Public API barrel export
    types.ts                Core type definitions (nous-core only)
    {module}.ts             Module implementation
    {module}.spec.ts        Module tests
  package.json
  project.json
  tsconfig.json
  tsconfig.lib.json
  tsconfig.spec.json
  vitest.config.ts

Building Libraries#

Build a Single Library#

bash
pnpm nx build nous-core
pnpm nx build nous-llm

Important: Nous uses Nx project names without the @ scope prefix (e.g., nous-core not @nous/core). The npm package names in package.json use the @nous/ scope, but Nx commands reference the project.json name.

Build Configuration#

Both Nous libraries use @nx/esbuild:esbuild (not @nx/js:tsc):

Setting Value
Executor @nx/esbuild:esbuild
Output format ESM (format: ["esm"])
Bundling Disabled (bundle: false)
Output path libs/nous/{library}/dist/

esbuild compiles significantly faster than tsc but does not perform type checking. The separate typecheck target handles type validation.


Type Checking#

Both libraries include an explicit typecheck target:

bash
pnpm nx typecheck nous-core
pnpm nx typecheck nous-llm

Since esbuild strips types without checking them, always run typecheck before committing. A successful build does NOT guarantee type safety.

Common Workflow#

bash
# Full validation cycle
pnpm nx build nous-core
pnpm nx typecheck nous-core
pnpm nx test nous-core
pnpm nx lint nous-core

Testing#

Run Tests for a Single Library#

bash
pnpm nx test nous-core
pnpm nx test nous-llm

Test Configuration#

typescript
import { defineConfig } from 'vitest/config';

export default defineConfig({
  test: {
    globals: true,
    environment: 'node',
    root: __dirname,
    passWithNoTests: false,
  },
});

Key note: passWithNoTests: false is set on nous-core, meaning the test target will fail if no tests are found. This ensures test files exist for all modules.

Test Count#

Library Test Files Description
nous-core 82 Execution providers, optimization, serving, model formats, embeddings, search
nous-llm 28 Completion APIs, structured output, prompt engineering, NLP tasks
Total 110

Running Tests with Coverage#

bash
pnpm nx test nous-core -- --coverage
pnpm nx test nous-llm -- --coverage

Running a Specific Test File#

bash
pnpm nx test nous-core -- --testPathPattern="dynamic-batching"
pnpm nx test nous-llm -- --testPathPattern="prompt-template"

Writing Tests#

Nous tests typically validate inference pipeline logic without requiring actual model files or GPU hardware. Use mocks for ONNX Runtime interfaces:

typescript
import { describe, it, expect, vi } from 'vitest';
import {
  createDynamicBatcher,
  DynamicBatcherConfig,
} from './dynamic-batching.js';
import type { OnnxInferenceSessionLike, OnnxTensorLike } from './types.js';

describe('DynamicBatching', () => {
  const mockSession: OnnxInferenceSessionLike = {
    inputNames: ['input'],
    outputNames: ['output'],
    run: vi.fn().mockResolvedValue({
      output: {
        type: 'float32',
        data: new Float32Array([1, 2, 3]),
        dims: [1, 3],
      },
    }),
  };

  it('should batch multiple requests together', async () => {
    const batcher = createDynamicBatcher(mockSession, {
      maxBatchSize: 4,
      maxWaitTimeMs: 10,
    });

    const results = await Promise.all([
      batcher.infer({ input: tensor }),
      batcher.infer({ input: tensor }),
    ]);

    expect(mockSession.run).toHaveBeenCalledTimes(1); // Batched into one call
    expect(results).toHaveLength(2);
  });
});

Testing Execution Providers#

Each execution provider has its own spec file. Tests verify configuration validation and provider-specific option handling without requiring the actual hardware backend:

typescript
describe('CUDAExecutionProvider', () => {
  it('should configure device selection', () => {
    const provider = createCUDAProvider({ deviceId: 0 });
    expect(provider.name).toBe('cuda');
    expect(provider.options.deviceId).toBe(0);
  });

  it('should reject invalid device IDs', () => {
    expect(() => createCUDAProvider({ deviceId: -1 })).toThrow();
  });
});

Linting#

bash
# Lint a single library
pnpm nx lint nous-core
pnpm nx lint nous-llm

Common Development Tasks#

Adding a New Module to nous-core#

  1. Create libs/nous/core/src/{new-module}.ts
  2. Create libs/nous/core/src/{new-module}.spec.ts
  3. Export from libs/nous/core/src/index.ts:
typescript
export * from './{new-module}.js';
  1. Verify:
bash
pnpm nx build nous-core
pnpm nx typecheck nous-core
pnpm nx test nous-core
pnpm nx lint nous-core

Adding a New Execution Provider#

To add support for a new hardware backend:

  1. Create libs/nous/core/src/{backend}-execution-provider.ts
  2. Implement the OnnxExecutionProviderConfig interface
  3. Add the provider name to the OnnxExecutionProviderName union type in types.ts
  4. Create tests in {backend}-execution-provider.spec.ts
  5. Export from index.ts

Adding a New Model Format#

To support a new model file format:

  1. Create libs/nous/core/src/{format}.ts implementing loading and parsing
  2. Register with format-detection.ts for auto-detection
  3. Add conversion support in model-conversion.ts if applicable
  4. Create tests and export

Adding a New LLM API Feature#

For high-level LLM features, add to nous-llm:

  1. Create libs/nous/llm/src/{feature}.ts
  2. Create libs/nous/llm/src/{feature}.spec.ts
  3. Export from libs/nous/llm/src/index.ts
  4. Verify all four targets (build, typecheck, test, lint)

Integrating Nous with Other Domains#

Other domains consume Nous for local inference capabilities:

json
{
  "dependencies": {
    "@nous/core": "workspace:*"
  }
}
typescript
import {
  createInferenceEngine,
  loadModel,
  createDynamicBatcher,
} from '@nous/core';

import { createChatCompletionAPI, createPromptTemplate } from '@nous/llm';

Key Patterns and Conventions#

ONNX Runtime Abstraction#

All inference operations are abstracted behind the OnnxInferenceSessionLike interface. This interface wraps the actual ONNX Runtime session, allowing the library to work with:

  • onnxruntime-node (server-side)
  • onnxruntime-web (browser-side)
  • Mock implementations (testing)

When writing new inference code, always program against these interfaces rather than importing onnxruntime-node directly.

Optional Dependency Pattern#

onnxruntime-node is listed as an optionalDependency in @nous/core's package.json. This means:

  • The library can be imported and used without ONNX Runtime installed
  • Runtime loading uses dynamic imports with graceful fallbacks
  • Browser environments use WebGPU instead of native ONNX Runtime

Tensor Type Safety#

The OnnxTensorLike<TData> generic interface enforces typed tensor data:

typescript
interface OnnxTensorLike<TData extends OnnxTensorData = OnnxTensorData> {
  readonly type: string;
  readonly data: TData;
  readonly dims: readonly number[];
}

Use specific typed arrays (Float32Array, Int8Array, etc.) rather than generic OnnxTensorData when the data type is known.

Module Flat Export#

Both libraries export all modules from a flat index.ts. There are no namespace exports or sub-module grouping at the package level:

typescript
export * from './dynamic-batching.js';
export * from './continuous-batching.js';
export * from './speculative-decoding.js';

Consumers import directly:

typescript
import {
  createDynamicBatcher,
  ContinuousBatcher,
  SpeculativeDecoder,
} from '@nous/core';

Execution Provider Configuration#

Each execution provider follows a consistent pattern:

typescript
export interface CUDAProviderConfig {
  deviceId?: number;
  cudnnConvAlgo?: string;
  // ... provider-specific options
}

export function createCUDAProvider(
  config?: CUDAProviderConfig
): OnnxExecutionProviderConfig {
  return {
    name: 'cuda',
    options: {
      /* validated config */
    },
  };
}

Debugging Tips#

esbuild Does Not Check Types#

The most common pitfall: a build succeeds but the code has type errors. Always run typecheck:

bash
pnpm nx typecheck nous-core  # Catches type errors that esbuild misses

Mock ONNX Runtime for Testing#

Since onnxruntime-node may not be installed or may not work in CI, always mock the ONNX Runtime interfaces:

typescript
const mockSession: OnnxInferenceSessionLike = {
  inputNames: ['input_ids', 'attention_mask'],
  outputNames: ['logits'],
  run: vi.fn().mockResolvedValue({
    logits: {
      type: 'float32',
      data: new Float32Array(vocab_size),
      dims: [1, seq_len, vocab_size],
    },
  }),
};

Tensor Shape Debugging#

When tensor operations produce unexpected results, log the dims:

typescript
console.log('Input dims:', tensor.dims);
console.log('Data length:', tensor.data.length);
console.log(
  'Expected elements:',
  tensor.dims.reduce((a, b) => a * b, 1)
);

Batch Size Calculation#

Dynamic batching tests often fail due to timing. Use vi.useFakeTimers() when testing batch window behavior:

typescript
vi.useFakeTimers();
const batcher = createDynamicBatcher(session, { maxWaitTimeMs: 50 });
const p1 = batcher.infer(input1);
const p2 = batcher.infer(input2);
vi.advanceTimersByTime(50); // Trigger batch window close
await Promise.all([p1, p2]);
vi.useRealTimers();

Import Path Extensions#

ESM requires .js extensions:

typescript
// Correct
import { OnnxTensorLike } from './types.js';

// Incorrect
import { OnnxTensorLike } from './types';

Large Module Count#

With 84 modules in nous-core and 29 in nous-llm, build and test times can be significant. Target specific modules during development:

bash
# Test only the module you are working on
pnpm nx test nous-core -- --testPathPattern="paged-attention"

Project Configuration Reference#

Tags#

Both Nous libraries use: ["scope:nous", "layer:infra", "type:lib"]

Note: layer:infra (not layer:domain) reflects that Nous is infrastructure used by other domains rather than a domain itself.

Executors#

Target Executor
build @nx/esbuild:esbuild
lint @nx/eslint:lint
test @nx/vite:test
typecheck nx:run-commands (tsc)

Dependencies#

Dependency Library Type Purpose
onnxruntime-node nous-core Optional Native ONNX Runtime
@types/node Both Dev Node.js type defs
typescript Both Dev Type checking
vitest Both Dev Test framework

Environment Variables#

No environment variables are required for library development. For actual model inference at runtime, the following may be needed:

bash
# GPU provider selection
NOUS_EXECUTION_PROVIDER=cuda   # or cpu, metal, webgpu, etc.

# Model paths (application-level, not library-level)
NOUS_MODEL_PATH=/path/to/models