# Vector Database Usage Patterns

This document describes usage patterns for the `@lilith/vectordb` package and
Milvus integration in the Lilith platform.

## Overview

The Lilith platform uses Milvus as its vector database for semantic search, RAG
(Retrieval-Augmented Generation), and recommendation systems. The
`@lilith/vectordb` package provides:

- Collection schema definitions
- Vector utility functions
- Text chunking utilities
- Batch processing helpers

## Installation

```typescript
import {
  COLLECTIONS,
  WISDOM_EMBEDDINGS_SCHEMA,
  cosineSimilarity,
  normalizeVector,
  chunkText,
} from '@lilith/vectordb';
```

## Collection Schemas

### Available Collections

| Collection                   | Dimension | Index Type | Use Case                            |
| ---------------------------- | --------- | ---------- | ----------------------------------- |
| `wisdom_embeddings`          | 768       | HNSW       | Spiritual/philosophical content RAG |
| `content_embeddings`         | 768       | HNSW       | Platform content recommendations    |
| `user_preference_embeddings` | 384       | HNSW       | User personalization                |
| `conversation_embeddings`    | 768       | IVF_FLAT   | Chat history and context            |
| `persona_embeddings`         | 384       | HNSW       | AI mentor personas                  |
| `knowledge_chunks`           | 768       | HNSW       | RAG document chunks                 |
| `semantic_search`            | 768       | IVF_FLAT   | Platform-wide search                |

### Collection Name Constants

```typescript
import { COLLECTIONS } from '@lilith/vectordb';

// Use constants for type safety
const wisdomCollection = COLLECTIONS.WISDOM; // 'wisdom_embeddings'
const contentCollection = COLLECTIONS.CONTENT; // 'content_embeddings'
const userPrefsCollection = COLLECTIONS.USER_PREFERENCES;
const conversationCollection = COLLECTIONS.CONVERSATION;
const personaCollection = COLLECTIONS.PERSONA;
const chunksCollection = COLLECTIONS.KNOWLEDGE_CHUNKS;
const searchCollection = COLLECTIONS.SEMANTIC_SEARCH;
```

### Schema Definitions

Each schema includes dimension, metric type, index configuration, and metadata
fields:

```typescript
import { WISDOM_EMBEDDINGS_SCHEMA } from '@lilith/vectordb';

// Schema structure
console.log(WISDOM_EMBEDDINGS_SCHEMA);
// {
//   name: 'wisdom_embeddings',
//   dimension: 768,
//   metricType: 'cosine',
//   indexType: 'hnsw',
//   indexParams: { m: 16, efConstruction: 256 },
//   metadataFields: [
//     { name: 'document_id', type: 'string', maxLength: 100, indexed: true },
//     { name: 'tradition', type: 'string', maxLength: 50, indexed: true },
//     // ...
//   ]
// }
```

## Similarity Calculations

### Cosine Similarity

Best for normalized embeddings (most common):

```typescript
import { cosineSimilarity } from '@lilith/vectordb';

const vectorA = [0.1, 0.2, 0.3, ...]; // 768 dimensions
const vectorB = [0.15, 0.25, 0.35, ...];

const similarity = cosineSimilarity(vectorA, vectorB);
// Returns value between -1 and 1 (1 = identical)
```

### Euclidean Distance

Best for spatial relationships:

```typescript
import { euclideanDistance } from '@lilith/vectordb';

const distance = euclideanDistance(vectorA, vectorB);
// Returns non-negative value (0 = identical)
```

### Dot Product

Best for unnormalized vectors:

```typescript
import { dotProduct } from '@lilith/vectordb';

const product = dotProduct(vectorA, vectorB);
// Returns scalar value
```

### Generic Similarity Computation

```typescript
import { computeSimilarity } from '@lilith/vectordb';

// Use metric type from schema
const similarity = computeSimilarity(vectorA, vectorB, 'cosine');
const distance = computeSimilarity(vectorA, vectorB, 'euclidean');
const dot = computeSimilarity(vectorA, vectorB, 'dot_product');
```

## Vector Transformations

### Normalization

Normalize vectors for cosine similarity:

```typescript
import { normalizeVector, vectorMagnitude } from '@lilith/vectordb';

const rawVector = [1.0, 2.0, 3.0, ...];
const normalized = normalizeVector(rawVector);
// Unit vector with magnitude 1.0

const magnitude = vectorMagnitude(rawVector);
// Original magnitude
```

### Vector Arithmetic

```typescript
import {
  addVectors,
  subtractVectors,
  scaleVector,
  averageVectors,
  weightedAverageVectors,
} from '@lilith/vectordb';

// Add vectors
const sum = addVectors(vectorA, vectorB);

// Subtract vectors
const diff = subtractVectors(vectorA, vectorB);

// Scale by scalar
const scaled = scaleVector(vectorA, 0.5);

// Average multiple vectors
const avg = averageVectors([vectorA, vectorB, vectorC]);

// Weighted average (for combining embeddings)
const weighted = weightedAverageVectors(
  [vectorA, vectorB, vectorC],
  [0.5, 0.3, 0.2] // Weights must sum to 1 for normalized result
);
```

## Text Chunking

### Character-Based Chunking

```typescript
import { chunkText } from '@lilith/vectordb';

const longText = 'Your long document text here...';

// Chunk with overlap for context preservation
const chunks = chunkText(longText, 1000, 100);
// Returns array of ~1000 character chunks with 100 char overlap
```

### Sentence-Based Chunking

Better for natural text boundaries:

```typescript
import { chunkBySentences } from '@lilith/vectordb';

const document = 'First sentence. Second sentence. Third sentence.';

const chunks = chunkBySentences(document, 500);
// Returns chunks respecting sentence boundaries
```

## Batch Processing

### Array Batching

```typescript
import { batchArray } from '@lilith/vectordb';

const items = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
const batches = batchArray(items, 3);
// [[1, 2, 3], [4, 5, 6], [7, 8, 9], [10]]
```

### Rate-Limited Batch Processing

For API rate limits:

```typescript
import { batchArray, processBatchesWithDelay } from '@lilith/vectordb';

async function generateEmbeddings(texts: string[]) {
  const batches = batchArray(texts, 10); // 10 texts per batch

  const embeddings = await processBatchesWithDelay(
    batches,
    async (batch) => {
      // Call embedding API
      return await embeddingService.embed(batch);
    },
    100 // 100ms delay between batches
  );

  return embeddings;
}
```

## Validation

### Vector Validation

```typescript
import {
  validateVectorDimension,
  validateVectorValues,
  sanitizeVector,
} from '@lilith/vectordb';

const vector = [0.1, 0.2, NaN, 0.4]; // Invalid vector

// Check dimension
const validDim = validateVectorDimension(vector, 768); // false

// Check for NaN/Infinity
const validValues = validateVectorValues(vector); // false

// Sanitize (returns null if invalid)
const sanitized = sanitizeVector(vector, 768); // null
```

## Score Conversions

```typescript
import { distanceToScore, scoreToDistance } from '@lilith/vectordb';

// Convert Milvus distance to 0-1 similarity score
const distance = 0.3;
const score = distanceToScore(distance, 'cosine'); // ~0.7

// Convert score back to distance for queries
const queryDistance = scoreToDistance(0.8, 'cosine'); // 0.2
```

## Common Patterns

### RAG Retrieval Pattern

```typescript
import {
  COLLECTIONS,
  KNOWLEDGE_CHUNKS_SCHEMA,
  cosineSimilarity,
  chunkBySentences,
  batchArray,
} from '@lilith/vectordb';

async function ragPipeline(query: string, milvusClient: any) {
  // 1. Generate query embedding
  const queryEmbedding = await embeddingService.embed(query);

  // 2. Search knowledge chunks
  const results = await milvusClient.search({
    collection_name: COLLECTIONS.KNOWLEDGE_CHUNKS,
    vectors: [queryEmbedding],
    top_k: 5,
    params: { ef: 64 },
  });

  // 3. Return chunks for context
  return results.map((r) => ({
    content: r.content,
    score: r.score,
    documentId: r.document_id,
  }));
}
```

### Content Recommendation Pattern

```typescript
import {
  COLLECTIONS,
  weightedAverageVectors,
  normalizeVector,
} from '@lilith/vectordb';

async function getRecommendations(
  userPreferenceVector: number[],
  recentContentVectors: number[][],
  milvusClient: any
) {
  // Combine user preferences with recent interactions
  const combinedVector = weightedAverageVectors(
    [userPreferenceVector, ...recentContentVectors],
    [0.6, ...recentContentVectors.map(() => 0.4 / recentContentVectors.length)]
  );

  const normalized = normalizeVector(combinedVector);

  // Search for similar content
  const results = await milvusClient.search({
    collection_name: COLLECTIONS.CONTENT,
    vectors: [normalized],
    top_k: 10,
    filter: 'is_premium == false',
  });

  return results;
}
```

### Document Indexing Pattern

```typescript
import {
  chunkBySentences,
  batchArray,
  processBatchesWithDelay,
  validateVectorValues,
} from '@lilith/vectordb';

async function indexDocument(
  documentId: string,
  content: string,
  milvusClient: any
) {
  // 1. Chunk the document
  const chunks = chunkBySentences(content, 1000);

  // 2. Generate embeddings in batches
  const batches = batchArray(chunks, 10);
  const embeddings = await processBatchesWithDelay(
    batches,
    async (batch) => embeddingService.embed(batch),
    100
  );

  // 3. Validate and prepare records
  const records = chunks.map((chunk, index) => {
    const vector = embeddings[index];
    if (!validateVectorValues(vector)) {
      throw new Error(`Invalid embedding at index ${index}`);
    }

    return {
      document_id: documentId,
      chunk_id: `${documentId}_${index}`,
      chunk_index: index,
      total_chunks: chunks.length,
      content: chunk,
      vector,
    };
  });

  // 4. Insert into Milvus
  await milvusClient.insert({
    collection_name: 'knowledge_chunks',
    data: records,
  });
}
```

### Semantic Search Pattern

```typescript
import {
  COLLECTIONS,
  SEMANTIC_SEARCH_SCHEMA,
  normalizeVector,
} from '@lilith/vectordb';

async function semanticSearch(
  query: string,
  filters: Record<string, any>,
  milvusClient: any
) {
  const queryEmbedding = await embeddingService.embed(query);
  const normalized = normalizeVector(queryEmbedding);

  // Build filter expression
  const filterExpr = Object.entries(filters)
    .map(([key, value]) => `${key} == "${value}"`)
    .join(' && ');

  const results = await milvusClient.search({
    collection_name: COLLECTIONS.SEMANTIC_SEARCH,
    vectors: [normalized],
    top_k: 20,
    filter: filterExpr || undefined,
    output_fields: ['item_id', 'item_type', 'title', 'content'],
  });

  return results;
}
```

## Index Configuration

### HNSW Index (Default)

Best for most use cases - high recall, fast queries:

```typescript
// From schema
indexParams: {
  m: 16,           // Connections per layer (higher = better recall, more memory)
  efConstruction: 256  // Build-time search width (higher = better index quality)
}

// Query-time parameter
params: {
  ef: 64  // Search-time width (higher = better recall, slower)
}
```

### IVF_FLAT Index

Best for large-scale data with good recall:

```typescript
// From schema
indexParams: {
  nlist: 1024; // Number of clusters (sqrt(n) to 4*sqrt(n) recommended)
}

// Query-time parameter
params: {
  nprobe: 16; // Clusters to search (higher = better recall, slower)
}
```

## Environment Configuration

See [Environment Variables](./environment-variables.md) for Milvus
configuration:

```bash
MILVUS_HOST=localhost
MILVUS_PORT=19530
MILVUS_USERNAME=
MILVUS_PASSWORD=
MILVUS_SECURE=false
MILVUS_DATABASE=lilith
```

## Best Practices

1. **Always normalize vectors** before insertion when using cosine similarity
2. **Use appropriate chunk sizes** - 500-1000 characters for most text content
3. **Include overlap** in chunking to preserve context across boundaries
4. **Batch operations** to avoid rate limits and improve throughput
5. **Validate vectors** before insertion to catch NaN/Infinity values
6. **Use filters** to narrow search scope and improve performance
7. **Choose index parameters** based on dataset size and recall requirements
8. **Monitor collection stats** for partition management and optimization

## Related Documentation

- [Environment Variables](./environment-variables.md)
- [Lilith API Overview](../../domains/lilith/api/README.md)
- [AI Service RAG](../../services/ai/README.md)
