# Nous Developer Guide

> Comprehensive development guide for the Nous (LLM Inference) domain libraries.

This guide covers local development setup, building, testing, debugging, and
common development tasks for the 2 Nous libraries.

---

## Prerequisites

### Required Software

| Tool    | Version    | Purpose              |
| ------- | ---------- | -------------------- |
| Node.js | >= 20.0    | JavaScript runtime   |
| pnpm    | >= 10.0    | Package manager      |
| Nx      | (via pnpm) | Monorepo task runner |

### Optional Software

| Tool               | Purpose                                           |
| ------------------ | ------------------------------------------------- |
| NVIDIA CUDA        | GPU inference via CUDA execution provider         |
| AMD ROCm           | GPU inference via ROCm execution provider         |
| `onnxruntime-node` | Native ONNX Runtime (optional dependency of core) |

### Installation

```bash
# From the repository root
pnpm install
```

No databases or external services are required. `onnxruntime-node` is declared
as an optional dependency and is not needed for development or testing of the
TypeScript modules.

---

## Project Structure

Nous has 2 libraries under `libs/nous/`:

```
libs/nous/
  core/    nous-core    Inference runtime (84 modules)
  llm/     nous-llm     LLM API layer (29 modules)
```

### `nous-core` - Inference Runtime (84 modules)

The core library is the largest single library in the Oshun monorepo. It
provides a complete inference runtime organized into these logical groups:

| Group               | Modules | Description                                                      |
| ------------------- | ------- | ---------------------------------------------------------------- |
| Core Types          | 1       | ONNX Runtime type abstractions                                   |
| Execution Providers | 8       | CPU, CUDA, ROCm, Metal, WebGPU, Vulkan, DML, NPU                 |
| Graph Optimization  | 4       | Memory planning, operator fusion, quantization                   |
| Batching/Serving    | 3       | Dynamic batching, continuous batching, LLM serving               |
| Attention/KV Cache  | 4       | KV cache, paged attention, flash attention, speculative decoding |
| Parallelism         | 4       | Tensor, pipeline, model sharding, MoE                            |
| Caching             | 3       | Prefix, radix, prompt caching                                    |
| Multi-Adapter       | 3       | Multi-LoRA serving, adapter switching, context extension         |
| Generation Control  | 6       | Streaming, stop sequences, sampling, beam search                 |
| Model Management    | 10      | Registry, versioning, discovery, download, conversion            |
| Model Formats       | 5       | SafeTensors, GGUF, GGML, PyTorch, core tensor                    |
| Embeddings          | 10      | Text, image, audio, multi-modal, normalization, batch, caching   |
| Similarity Search   | 7       | Approximate/exact nearest neighbor, distance metrics, reranking  |
| Embedding Compress  | 3       | Binary embeddings, Matryoshka, compression                       |
| Runtime             | 3       | Runtime loader, session manager, inference engine                |

### `nous-llm` - LLM API Layer (29 modules)

High-level LLM interaction APIs built on top of `nous-core`:

| Group               | Modules | Description                                             |
| ------------------- | ------- | ------------------------------------------------------- |
| Completion APIs     | 4       | Completion, chat, instruction following, multi-turn     |
| Function Calling    | 2       | Tool use framework, function calling                    |
| Structured Output   | 5       | JSON mode, grammar/regex constraints, schema validation |
| Response Processing | 5       | Formatting, markdown, code/citation/fact extraction     |
| NLP Tasks           | 4       | Entity extraction, sentiment, classification            |
| Prompt Engineering  | 7       | Templates, versioning, testing, optimization, few-shot  |
| Reasoning           | 2       | Chain-of-thought, dynamic examples                      |

### Per-Library File Layout

```
libs/nous/{library}/
  src/
    index.ts                Public API barrel export
    types.ts                Core type definitions (nous-core only)
    {module}.ts             Module implementation
    {module}.spec.ts        Module tests
  package.json
  project.json
  tsconfig.json
  tsconfig.lib.json
  tsconfig.spec.json
  vitest.config.ts
```

---

## Building Libraries

### Build a Single Library

```bash
pnpm nx build nous-core
pnpm nx build nous-llm
```

**Important:** Nous uses Nx project names without the `@` scope prefix (e.g.,
`nous-core` not `@nous/core`). The npm package names in `package.json` use the
`@nous/` scope, but Nx commands reference the `project.json` name.

### Build Configuration

Both Nous libraries use `@nx/esbuild:esbuild` (not `@nx/js:tsc`):

| Setting       | Value                       |
| ------------- | --------------------------- |
| Executor      | `@nx/esbuild:esbuild`       |
| Output format | ESM (`format: ["esm"]`)     |
| Bundling      | Disabled (`bundle: false`)  |
| Output path   | `libs/nous/{library}/dist/` |

esbuild compiles significantly faster than tsc but does not perform type
checking. The separate `typecheck` target handles type validation.

---

## Type Checking

Both libraries include an explicit **`typecheck`** target:

```bash
pnpm nx typecheck nous-core
pnpm nx typecheck nous-llm
```

Since esbuild strips types without checking them, always run typecheck before
committing. A successful build does NOT guarantee type safety.

### Common Workflow

```bash
# Full validation cycle
pnpm nx build nous-core
pnpm nx typecheck nous-core
pnpm nx test nous-core
pnpm nx lint nous-core
```

---

## Testing

### Run Tests for a Single Library

```bash
pnpm nx test nous-core
pnpm nx test nous-llm
```

### Test Configuration

```typescript
import { defineConfig } from 'vitest/config';

export default defineConfig({
  test: {
    globals: true,
    environment: 'node',
    root: __dirname,
    passWithNoTests: false,
  },
});
```

Key note: `passWithNoTests: false` is set on `nous-core`, meaning the test
target will fail if no tests are found. This ensures test files exist for all
modules.

### Test Count

| Library     | Test Files | Description                                                                   |
| ----------- | ---------- | ----------------------------------------------------------------------------- |
| `nous-core` | 82         | Execution providers, optimization, serving, model formats, embeddings, search |
| `nous-llm`  | 28         | Completion APIs, structured output, prompt engineering, NLP tasks             |
| **Total**   | **110**    |                                                                               |

### Running Tests with Coverage

```bash
pnpm nx test nous-core -- --coverage
pnpm nx test nous-llm -- --coverage
```

### Running a Specific Test File

```bash
pnpm nx test nous-core -- --testPathPattern="dynamic-batching"
pnpm nx test nous-llm -- --testPathPattern="prompt-template"
```

### Writing Tests

Nous tests typically validate inference pipeline logic without requiring actual
model files or GPU hardware. Use mocks for ONNX Runtime interfaces:

```typescript
import { describe, it, expect, vi } from 'vitest';
import {
  createDynamicBatcher,
  DynamicBatcherConfig,
} from './dynamic-batching.js';
import type { OnnxInferenceSessionLike, OnnxTensorLike } from './types.js';

describe('DynamicBatching', () => {
  const mockSession: OnnxInferenceSessionLike = {
    inputNames: ['input'],
    outputNames: ['output'],
    run: vi.fn().mockResolvedValue({
      output: {
        type: 'float32',
        data: new Float32Array([1, 2, 3]),
        dims: [1, 3],
      },
    }),
  };

  it('should batch multiple requests together', async () => {
    const batcher = createDynamicBatcher(mockSession, {
      maxBatchSize: 4,
      maxWaitTimeMs: 10,
    });

    const results = await Promise.all([
      batcher.infer({ input: tensor }),
      batcher.infer({ input: tensor }),
    ]);

    expect(mockSession.run).toHaveBeenCalledTimes(1); // Batched into one call
    expect(results).toHaveLength(2);
  });
});
```

### Testing Execution Providers

Each execution provider has its own spec file. Tests verify configuration
validation and provider-specific option handling without requiring the actual
hardware backend:

```typescript
describe('CUDAExecutionProvider', () => {
  it('should configure device selection', () => {
    const provider = createCUDAProvider({ deviceId: 0 });
    expect(provider.name).toBe('cuda');
    expect(provider.options.deviceId).toBe(0);
  });

  it('should reject invalid device IDs', () => {
    expect(() => createCUDAProvider({ deviceId: -1 })).toThrow();
  });
});
```

---

## Linting

```bash
# Lint a single library
pnpm nx lint nous-core
pnpm nx lint nous-llm
```

---

## Common Development Tasks

### Adding a New Module to nous-core

1. Create `libs/nous/core/src/{new-module}.ts`
2. Create `libs/nous/core/src/{new-module}.spec.ts`
3. Export from `libs/nous/core/src/index.ts`:

```typescript
export * from './{new-module}.js';
```

4. Verify:

```bash
pnpm nx build nous-core
pnpm nx typecheck nous-core
pnpm nx test nous-core
pnpm nx lint nous-core
```

### Adding a New Execution Provider

To add support for a new hardware backend:

1. Create `libs/nous/core/src/{backend}-execution-provider.ts`
2. Implement the `OnnxExecutionProviderConfig` interface
3. Add the provider name to the `OnnxExecutionProviderName` union type in
   `types.ts`
4. Create tests in `{backend}-execution-provider.spec.ts`
5. Export from `index.ts`

### Adding a New Model Format

To support a new model file format:

1. Create `libs/nous/core/src/{format}.ts` implementing loading and parsing
2. Register with `format-detection.ts` for auto-detection
3. Add conversion support in `model-conversion.ts` if applicable
4. Create tests and export

### Adding a New LLM API Feature

For high-level LLM features, add to `nous-llm`:

1. Create `libs/nous/llm/src/{feature}.ts`
2. Create `libs/nous/llm/src/{feature}.spec.ts`
3. Export from `libs/nous/llm/src/index.ts`
4. Verify all four targets (build, typecheck, test, lint)

### Integrating Nous with Other Domains

Other domains consume Nous for local inference capabilities:

```json
{
  "dependencies": {
    "@nous/core": "workspace:*"
  }
}
```

```typescript
import {
  createInferenceEngine,
  loadModel,
  createDynamicBatcher,
} from '@nous/core';

import { createChatCompletionAPI, createPromptTemplate } from '@nous/llm';
```

---

## Key Patterns and Conventions

### ONNX Runtime Abstraction

All inference operations are abstracted behind the `OnnxInferenceSessionLike`
interface. This interface wraps the actual ONNX Runtime session, allowing the
library to work with:

- `onnxruntime-node` (server-side)
- `onnxruntime-web` (browser-side)
- Mock implementations (testing)

When writing new inference code, always program against these interfaces rather
than importing `onnxruntime-node` directly.

### Optional Dependency Pattern

`onnxruntime-node` is listed as an `optionalDependency` in `@nous/core`'s
`package.json`. This means:

- The library can be imported and used without ONNX Runtime installed
- Runtime loading uses dynamic imports with graceful fallbacks
- Browser environments use WebGPU instead of native ONNX Runtime

### Tensor Type Safety

The `OnnxTensorLike<TData>` generic interface enforces typed tensor data:

```typescript
interface OnnxTensorLike<TData extends OnnxTensorData = OnnxTensorData> {
  readonly type: string;
  readonly data: TData;
  readonly dims: readonly number[];
}
```

Use specific typed arrays (`Float32Array`, `Int8Array`, etc.) rather than
generic `OnnxTensorData` when the data type is known.

### Module Flat Export

Both libraries export all modules from a flat `index.ts`. There are no namespace
exports or sub-module grouping at the package level:

```typescript
export * from './dynamic-batching.js';
export * from './continuous-batching.js';
export * from './speculative-decoding.js';
```

Consumers import directly:

```typescript
import {
  createDynamicBatcher,
  ContinuousBatcher,
  SpeculativeDecoder,
} from '@nous/core';
```

### Execution Provider Configuration

Each execution provider follows a consistent pattern:

```typescript
export interface CUDAProviderConfig {
  deviceId?: number;
  cudnnConvAlgo?: string;
  // ... provider-specific options
}

export function createCUDAProvider(
  config?: CUDAProviderConfig
): OnnxExecutionProviderConfig {
  return {
    name: 'cuda',
    options: {
      /* validated config */
    },
  };
}
```

---

## Debugging Tips

### esbuild Does Not Check Types

The most common pitfall: a build succeeds but the code has type errors. Always
run typecheck:

```bash
pnpm nx typecheck nous-core  # Catches type errors that esbuild misses
```

### Mock ONNX Runtime for Testing

Since `onnxruntime-node` may not be installed or may not work in CI, always mock
the ONNX Runtime interfaces:

```typescript
const mockSession: OnnxInferenceSessionLike = {
  inputNames: ['input_ids', 'attention_mask'],
  outputNames: ['logits'],
  run: vi.fn().mockResolvedValue({
    logits: {
      type: 'float32',
      data: new Float32Array(vocab_size),
      dims: [1, seq_len, vocab_size],
    },
  }),
};
```

### Tensor Shape Debugging

When tensor operations produce unexpected results, log the dims:

```typescript
console.log('Input dims:', tensor.dims);
console.log('Data length:', tensor.data.length);
console.log(
  'Expected elements:',
  tensor.dims.reduce((a, b) => a * b, 1)
);
```

### Batch Size Calculation

Dynamic batching tests often fail due to timing. Use `vi.useFakeTimers()` when
testing batch window behavior:

```typescript
vi.useFakeTimers();
const batcher = createDynamicBatcher(session, { maxWaitTimeMs: 50 });
const p1 = batcher.infer(input1);
const p2 = batcher.infer(input2);
vi.advanceTimersByTime(50); // Trigger batch window close
await Promise.all([p1, p2]);
vi.useRealTimers();
```

### Import Path Extensions

ESM requires `.js` extensions:

```typescript
// Correct
import { OnnxTensorLike } from './types.js';

// Incorrect
import { OnnxTensorLike } from './types';
```

### Large Module Count

With 84 modules in `nous-core` and 29 in `nous-llm`, build and test times can be
significant. Target specific modules during development:

```bash
# Test only the module you are working on
pnpm nx test nous-core -- --testPathPattern="paged-attention"
```

---

## Project Configuration Reference

### Tags

Both Nous libraries use: `["scope:nous", "layer:infra", "type:lib"]`

Note: `layer:infra` (not `layer:domain`) reflects that Nous is infrastructure
used by other domains rather than a domain itself.

### Executors

| Target    | Executor                |
| --------- | ----------------------- |
| build     | `@nx/esbuild:esbuild`   |
| lint      | `@nx/eslint:lint`       |
| test      | `@nx/vite:test`         |
| typecheck | `nx:run-commands` (tsc) |

### Dependencies

| Dependency         | Library     | Type     | Purpose             |
| ------------------ | ----------- | -------- | ------------------- |
| `onnxruntime-node` | `nous-core` | Optional | Native ONNX Runtime |
| `@types/node`      | Both        | Dev      | Node.js type defs   |
| `typescript`       | Both        | Dev      | Type checking       |
| `vitest`           | Both        | Dev      | Test framework      |

### Environment Variables

No environment variables are required for library development. For actual model
inference at runtime, the following may be needed:

```bash
# GPU provider selection
NOUS_EXECUTION_PROVIDER=cuda   # or cpu, metal, webgpu, etc.

# Model paths (application-level, not library-level)
NOUS_MODEL_PATH=/path/to/models
```

---

## Related Documentation

- [Nous Domain README](./README.md) - Domain overview, architecture, and library
  descriptions
- [CLAUDE.md](/CLAUDE.md) - Repository-wide development guidelines
