# Sophia API Documentation

Sophia provides three services for knowledge management and retrieval:

1. **Search API** (port 3000) -- Hybrid semantic search, RAG pipeline, and
   citation extraction
2. **Ingestion Service** (pipeline) -- Multi-stage document ingestion with
   extraction, enrichment, chunking, and indexing
3. **Knowledge Graph Service** (port 3001) -- Entity/relationship CRUD, graph
   traversal, path finding, and entity resolution

The Search API is the primary external-facing service. The Ingestion Service and
Knowledge Graph Service are internal services accessed programmatically or via
the `@sophia/client` SDK.

---

## Search API (Port 3000)

The Search API provides hybrid search (vector + keyword), RAG
(Retrieval-Augmented Generation), cross-encoder reranking, and citation
extraction. It is implemented as a `SearchAPIService` class with request
handlers that can be mounted on any HTTP framework.

### Base URL

```
http://localhost:3000
```

### Configuration

| Environment Variable | Default     | Description      |
| -------------------- | ----------- | ---------------- |
| `PORT`               | `3000`      | HTTP server port |
| `HOST`               | `localhost` | Bind address     |

### Service Configuration

| Parameter         | Default | Description                                 |
| ----------------- | ------- | ------------------------------------------- |
| `defaultTopK`     | 10      | Default number of results returned          |
| `maxTopK`         | 100     | Maximum allowed `topK` value                |
| `enableCitations` | true    | Enable citation extraction in RAG responses |
| `enableReranking` | true    | Enable cross-encoder reranking              |

---

### Search Endpoint

#### `POST /search` (or programmatic `service.search(query)`)

Execute a hybrid search query combining dense vector similarity and sparse BM25
keyword matching.

**Request body:**

```json
{
  "text": "meditation techniques for anxiety",
  "strategy": "hybrid",
  "topK": 10,
  "threshold": 0.5,
  "filters": {
    "documentIds": ["doc_123"],
    "sources": ["sacred-texts"],
    "languages": ["en"],
    "dateRange": { "from": "2020-01-01", "to": "2026-01-01" },
    "tags": ["meditation", "mindfulness"],
    "metadata": { "tradition": "buddhism" }
  },
  "options": {
    "includeMetadata": true,
    "includeHighlights": true,
    "includeContext": true,
    "expandQuery": false,
    "rerank": true,
    "deduplicate": true,
    "hybridWeights": { "dense": 0.7, "sparse": 0.3 }
  }
}
```

| Field                       | Type     | Required | Description                                            |
| --------------------------- | -------- | -------- | ------------------------------------------------------ |
| `text`                      | string   | Yes      | Search query text                                      |
| `strategy`                  | enum     | No       | Retrieval strategy (default: `hybrid`)                 |
| `topK`                      | int      | No       | Number of results (default: 10, max: 100)              |
| `threshold`                 | number   | No       | Minimum score threshold (0-1)                          |
| `filters.documentIds`       | string[] | No       | Restrict to specific document IDs                      |
| `filters.sources`           | string[] | No       | Filter by source identifiers                           |
| `filters.languages`         | string[] | No       | Filter by language codes                               |
| `filters.dateRange`         | object   | No       | Filter by date range (`from`, `to`)                    |
| `filters.tags`              | string[] | No       | Filter by tags                                         |
| `filters.metadata`          | object   | No       | Filter by arbitrary metadata fields                    |
| `options.includeMetadata`   | boolean  | No       | Include result metadata (source, title, authors, etc.) |
| `options.includeHighlights` | boolean  | No       | Include highlighted text spans                         |
| `options.includeContext`    | boolean  | No       | Include surrounding context text                       |
| `options.expandQuery`       | boolean  | No       | Enable query expansion to multiple queries             |
| `options.rerank`            | boolean  | No       | Enable cross-encoder reranking                         |
| `options.deduplicate`       | boolean  | No       | Remove duplicate/near-duplicate results                |
| `options.hybridWeights`     | object   | No       | Custom weights for dense/sparse fusion                 |

**Retrieval strategies:**

| Strategy      | Description                                                         |
| ------------- | ------------------------------------------------------------------- |
| `dense`       | Vector similarity search only (cosine similarity)                   |
| `sparse`      | BM25 keyword search only                                            |
| `hybrid`      | Combined dense + sparse with configurable weights                   |
| `rerank`      | Two-stage: initial retrieval followed by cross-encoder reranking    |
| `multi-query` | Query expansion: generate multiple query variants and merge results |

**Response (200):**

```json
{
  "queryId": "qry_abc123",
  "query": "meditation techniques for anxiety",
  "strategy": "hybrid",
  "results": [
    {
      "resultId": "res_001",
      "chunkId": "chk_456",
      "documentId": "doc_789",
      "text": "Mindfulness meditation has been shown to reduce anxiety by...",
      "score": 0.92,
      "rank": 1,
      "metadata": {
        "source": "sacred-texts",
        "title": "The Practice of Mindfulness",
        "authors": ["Thich Nhat Hanh"],
        "date": "2015-03-15",
        "language": "en",
        "pageNumber": 42,
        "sectionTitle": "Chapter 3: Working with Anxiety"
      },
      "highlights": [
        {
          "text": "meditation",
          "startOffset": 12,
          "endOffset": 22,
          "matchType": "exact"
        },
        {
          "text": "reduce anxiety",
          "startOffset": 45,
          "endOffset": 59,
          "matchType": "semantic"
        }
      ],
      "context": {
        "before": "In this section, we explore how...",
        "after": "...leading to measurable reductions in cortisol.",
        "documentTitle": "The Practice of Mindfulness",
        "chapterTitle": "Working with Anxiety"
      }
    }
  ],
  "totalResults": 47,
  "hasMore": true,
  "processingTimeMs": 125,
  "metadata": {
    "queryExpansions": ["anxiety meditation", "mindfulness anxiety reduction"],
    "appliedFilters": { "languages": ["en"] },
    "warnings": []
  }
}
```

---

### RAG Endpoint

#### `POST /rag` (or programmatic `service.rag(request)`)

Execute a retrieval-augmented generation request: search for relevant documents,
then generate an answer with citations.

**Request body:**

```json
{
  "query": "What are the main meditation techniques for managing anxiety?",
  "topK": 10,
  "strategy": "hybrid",
  "systemPrompt": "You are a knowledgeable meditation teacher.",
  "maxTokens": 1000,
  "includeCitations": true,
  "citationOptions": {
    "minConfidence": 0.7,
    "maxCitations": 5,
    "includeGraph": false,
    "mustCiteThreshold": 0.8
  }
}
```

| Field                               | Type    | Required | Description                             |
| ----------------------------------- | ------- | -------- | --------------------------------------- |
| `query`                             | string  | Yes      | User's question                         |
| `topK`                              | int     | No       | Number of source documents to retrieve  |
| `strategy`                          | enum    | No       | Retrieval strategy (default: `hybrid`)  |
| `systemPrompt`                      | string  | No       | System prompt for the LLM               |
| `maxTokens`                         | int     | No       | Maximum tokens for the generated answer |
| `includeCitations`                  | boolean | No       | Include citation extraction             |
| `citationOptions.minConfidence`     | number  | No       | Minimum citation confidence (0-1)       |
| `citationOptions.maxCitations`      | int     | No       | Maximum number of citations             |
| `citationOptions.includeGraph`      | boolean | No       | Include citation authority graph        |
| `citationOptions.mustCiteThreshold` | number  | No       | Score above which citation is required  |

**Response (200):**

```json
{
  "queryId": "rag_abc123",
  "query": "What are the main meditation techniques for managing anxiety?",
  "answer": "Based on the search results, here's what I found about...",
  "citations": [
    {
      "citationId": "cit_001",
      "text": "Mindfulness meditation has been shown to reduce anxiety",
      "sourceChunkId": "chk_456",
      "documentId": "doc_789",
      "confidence": 0.95,
      "position": { "startOffset": 42, "endOffset": 96 },
      "metadata": {
        "source": "sacred-texts",
        "title": "The Practice of Mindfulness"
      }
    }
  ],
  "sources": [ { "resultId": "...", "chunkId": "...", "score": 0.92, ... } ],
  "processingTimeMs": 450,
  "metadata": {
    "retrievalStrategy": "hybrid"
  }
}
```

---

### Health Endpoint

#### `GET /health` (or programmatic `service.healthCheck()`)

**Response (200):**

```json
{
  "status": "healthy",
  "uptime": 3600.5
}
```

---

## Ingestion Service (Pipeline)

The Ingestion Service is a multi-stage pipeline that processes documents from
raw source to indexed, searchable chunks. It is accessed programmatically via
the `IngestionPipeline` class or the `@sophia/client` `IngestionClient`.

### Pipeline Stages

```
Fetch -> Extract -> Enrich -> Chunk -> Index
```

| Stage        | Description                                                      |
| ------------ | ---------------------------------------------------------------- |
| `fetching`   | Retrieve content from source (file, URL, API, database, stream)  |
| `extracting` | Extract text, structure, images, tables, footnotes from document |
| `enriching`  | Entity extraction, metadata inference, summary generation        |
| `chunking`   | Split document into chunks using configured strategy             |
| `indexing`   | Generate embeddings and index chunks in vector store             |

### Programmatic API

#### Submit an ingestion job

```typescript
import { createIngestionPipeline } from '@sophia/ingestion';

const pipeline = createIngestionPipeline({
  /* config */
});

const job = await pipeline.submit({
  sourceConfig: {
    sourceId: createSourceId('src_001'),
    type: 'url',
    uri: 'https://example.com/document.pdf',
    credentials: {
      type: 'bearer',
      token: 'access-token',
    },
    options: {
      timeout: 30000,
      maxSize: 52428800, // 50MB
    },
  },
  extractionConfig: {
    preserveMarkup: false,
    extractImages: true,
    extractTables: true,
    extractFootnotes: true,
    maxPages: 500,
  },
  ocrConfig: {
    enabled: true,
    language: ['en', 'sa'],
    confidence: 0.8,
    preserveLayout: true,
    deskew: true,
  },
  enrichmentConfig: {
    extractEntities: true,
    entityTypes: ['person', 'concept', 'practice', 'location', 'event'],
    generateSummary: true,
    detectLanguage: true,
    extractKeywords: true,
  },
  chunkingConfig: {
    strategy: 'semantic',
    chunkSize: 512,
    chunkOverlap: 64,
    minChunkSize: 100,
  },
  priority: 5,
  tags: ['buddhism', 'meditation'],
});
```

### Source Types

| Source Type | URI Example                        | Description        |
| ----------- | ---------------------------------- | ------------------ |
| `file`      | `/path/to/document.pdf`            | Local filesystem   |
| `url`       | `https://example.com/doc.html`     | HTTP/HTTPS URLs    |
| `api`       | `https://api.example.com/v1/texts` | REST API endpoints |
| `database`  | `postgresql://host/db?query=...`   | Database queries   |
| `stream`    | `kafka://topic/partition`          | Streaming sources  |
| `archive`   | `/path/to/archive.zip`             | ZIP/TAR archives   |

### Supported Document Formats

| Format       | Extensions              | OCR Support |
| ------------ | ----------------------- | ----------- |
| `pdf`        | `.pdf`                  | Yes         |
| `epub`       | `.epub`                 | No          |
| `html`       | `.html`, `.htm`         | No          |
| `markdown`   | `.md`, `.markdown`      | No          |
| `plain-text` | `.txt`                  | No          |
| `tei-xml`    | `.xml` (TEI schema)     | No          |
| `docx`       | `.docx`                 | No          |
| `xlsx`       | `.xlsx`                 | No          |
| `pptx`       | `.pptx`                 | No          |
| `image`      | `.png`, `.jpg`, `.tiff` | Yes         |

### Chunking Strategies

| Strategy    | Description                                             |
| ----------- | ------------------------------------------------------- |
| `fixed`     | Fixed-size chunks by token count                        |
| `paragraph` | Split on paragraph boundaries                           |
| `sentence`  | Split on sentence boundaries                            |
| `semantic`  | Split on semantic boundaries using embedding similarity |
| `recursive` | Recursive splitting with fallback strategies            |

### Job Statuses

| Status       | Description                              |
| ------------ | ---------------------------------------- |
| `pending`    | Job created, not yet started             |
| `fetching`   | Retrieving content from source           |
| `extracting` | Extracting text and structure            |
| `enriching`  | Running entity extraction, summarization |
| `chunking`   | Splitting document into chunks           |
| `indexing`   | Generating embeddings and indexing       |
| `completed`  | Pipeline finished successfully           |
| `failed`     | Pipeline failed (see error for details)  |
| `cancelled`  | Pipeline was cancelled                   |

### Error Types

| Error Class       | Code               | Stage     | Retryable | Description                       |
| ----------------- | ------------------ | --------- | --------- | --------------------------------- |
| `FetchError`      | `FETCH_ERROR`      | fetch     | Yes       | Source retrieval failed           |
| `ExtractionError` | `EXTRACTION_ERROR` | extract   | No        | Text extraction failed            |
| `OCRError`        | `OCR_ERROR`        | ocr       | Yes       | OCR processing failed             |
| `EnrichmentError` | `ENRICHMENT_ERROR` | enrich    | Yes       | Metadata/entity extraction failed |
| `IngestionError`  | (various)          | (various) | Varies    | General pipeline error            |

---

## Knowledge Graph Service (Port 3001)

The Knowledge Graph Service provides entity and relationship CRUD, graph
traversal (BFS/DFS), shortest path finding, subgraph extraction, entity
resolution, and entity merging. It is accessed programmatically via the
`KnowledgeGraphService` class or the `@sophia/client` `KnowledgeGraphClient`.

### Programmatic API

#### Create nodes

```typescript
import { createKnowledgeGraphService } from '@sophia/knowledge-graph';

const kg = createKnowledgeGraphService({
  /* config */
});

// Create a concept node
const concept = await kg.createNode({
  type: 'concept',
  labels: ['meditation', 'core-concept'],
  properties: {
    name: 'Vipassana',
    definition: 'Insight meditation focusing on the true nature of reality',
    tradition: 'Theravada Buddhism',
    category: 'meditation-technique',
    complexityLevel: 'intermediate',
    relatedTerms: ['mindfulness', 'satipatthana', 'insight'],
  },
});
```

#### Create relationships

```typescript
const relationship = await kg.createRelationship({
  type: 'BELONGS_TO',
  sourceId: vipassanaNodeId,
  targetId: theravadaNodeId,
  properties: {
    weight: 0.95,
    context: 'Vipassana is a core practice of Theravada Buddhism',
    evidence: 'Satipatthana Sutta',
  },
});
```

#### Query nodes

```typescript
const results = await kg.queryNodes({
  filters: {
    type: 'concept',
    searchText: 'meditation',
    properties: { tradition: 'buddhism' },
  },
  pagination: {
    limit: 20,
    offset: 0,
    sortBy: 'name',
    sortDirection: 'asc',
  },
  includeRelationships: true,
});
```

#### Graph traversal

```typescript
const traversal = await kg.traverse({
  startNodeId: vipassanaNodeId,
  relationshipTypes: ['BELONGS_TO', 'DERIVES_FROM', 'RELATED_TO'],
  direction: 'both',
  maxDepth: 3,
  nodeTypes: ['concept', 'practice', 'tradition'],
  includeProperties: true,
  limit: 50,
});
```

#### Path finding

```typescript
const path = await kg.findShortestPath({
  sourceId: vipassanaNodeId,
  targetId: zenNodeId,
  relationshipTypes: ['DERIVES_FROM', 'INFLUENCED_BY', 'RELATED_TO'],
  maxDepth: 6,
  weighted: true,
});
```

#### Subgraph extraction

```typescript
const subgraph = await kg.extractSubgraph({
  centerNodeId: vipassanaNodeId,
  radius: 2,
  nodeTypes: ['concept', 'practice', 'figure'],
  maxNodes: 100,
});
```

#### Entity resolution

```typescript
const resolution = await kg.resolveEntity({
  name: 'Siddhartha Gautama',
  type: 'figure',
  aliases: ['Buddha', 'Shakyamuni', 'The Blessed One'],
  maxCandidates: 5,
  minSimilarity: 0.7,
});
```

### Node Types

| Type           | Description                         | Key Properties                                                   |
| -------------- | ----------------------------------- | ---------------------------------------------------------------- |
| `tradition`    | Religious/spiritual tradition       | `originRegion`, `timePeriod`, `keyPrinciples`, `foundingFigures` |
| `text`         | Sacred or scholarly text            | `title`, `author`, `language`, `year`, `genre`, `sourceUrl`      |
| `concept`      | Philosophical/theological concept   | `definition`, `tradition`, `category`, `complexityLevel`         |
| `practice`     | Spiritual/contemplative practice    | `tradition`, `difficulty`, `prerequisites`, `duration`           |
| `figure`       | Historical or contemporary figure   | `birthYear`, `deathYear`, `traditions`, `bioSummary`, `titles`   |
| `place`        | Geographically significant location | `country`, `region`, `coordinates`, `significance`               |
| `event`        | Historical or recurring event       | `date`, `startDate`, `endDate`, `location`, `participants`       |
| `organization` | Institution or organization         | `foundedYear`, `headquarters`, `type`, `members`                 |
| `custom`       | User-defined node type              | Any `BaseNodeProperties`                                         |

### Relationship Types

| Category               | Types                                                                                   |
| ---------------------- | --------------------------------------------------------------------------------------- |
| Figure relationships   | `TAUGHT`, `WROTE`, `LIVED_IN`, `STUDIED_UNDER`, `FOUNDED`                               |
| Concept relationships  | `ALIGNS_WITH`, `CONTRADICTS`, `DERIVES_FROM`, `REFINES`, `EXTENDS`                      |
| Practice relationships | `BELONGS_TO`, `REQUIRES`, `PRECEDES`, `FOLLOWS`                                         |
| Text relationships     | `DISCUSSES`, `REFERENCES`, `CITES`, `ORIGINATES_FROM`, `TRANSLATED_FROM`                |
| Place relationships    | `ASSOCIATED_WITH`, `SACRED_TO`, `LOCATED_IN`                                            |
| Cross-domain           | `EXEMPLIFIES`, `INFLUENCED_BY`, `RELATED_TO`, `PART_OF`, `MEMBER_OF`, `PARTICIPATED_IN` |

### Traversal Options

| Option              | Type                 | Default    | Description                       |
| ------------------- | -------------------- | ---------- | --------------------------------- |
| `startNodeId`       | `NodeId`             | (required) | Starting node for traversal       |
| `relationshipTypes` | `RelationshipType[]` | all        | Filter by relationship types      |
| `direction`         | enum                 | `outgoing` | `outgoing`, `incoming`, or `both` |
| `maxDepth`          | int                  | 3          | Maximum traversal depth           |
| `nodeTypes`         | `NodeType[]`         | all        | Filter by node types              |
| `includeProperties` | boolean              | false      | Include full node properties      |
| `limit`             | int                  | 100        | Maximum nodes to return           |

### Pagination

All list/query operations support pagination:

| Parameter       | Type   | Default | Description             |
| --------------- | ------ | ------- | ----------------------- |
| `limit`         | int    | 20      | Items per page          |
| `offset`        | int    | 0       | Number of items to skip |
| `sortBy`        | string | -       | Field to sort by        |
| `sortDirection` | enum   | `asc`   | `asc` or `desc`         |

### Error Types

| Error Class                 | Code                       | Description                           |
| --------------------------- | -------------------------- | ------------------------------------- |
| `NodeNotFoundError`         | `NODE_NOT_FOUND`           | Node ID not found                     |
| `RelationshipNotFoundError` | `RELATIONSHIP_NOT_FOUND`   | Relationship ID not found             |
| `InvalidRelationshipError`  | `INVALID_RELATIONSHIP`     | Invalid relationship definition       |
| `TraversalLimitError`       | `TRAVERSAL_LIMIT_EXCEEDED` | Traversal exceeded depth/count limits |
| `GraphError`                | (various)                  | General graph error                   |

### Statistics

```typescript
const stats = await kg.getStatistics();
// {
//   totalNodes: 15420,
//   totalRelationships: 42180,
//   nodesByType: { concept: 5200, figure: 3100, text: 2800, ... },
//   relationshipsByType: { RELATED_TO: 8500, DISCUSSES: 6200, ... },
//   averageDegree: 5.47,
//   density: 0.00035,
//   connectedComponents: 12,
//   largestComponentSize: 14800
// }
```

---

## Client SDK

The `@sophia/client` package provides a unified TypeScript client:

```typescript
import {
  SophiaClient,
  SearchClient,
  IngestionClient,
  KnowledgeGraphClient,
  SearchQueryBuilder,
  GraphQueryBuilder,
} from '@sophia/client';

// Unified client
const sophia = new SophiaClient({
  searchUrl: 'http://localhost:3000',
  knowledgeGraphUrl: 'http://localhost:3001',
});

// Or individual clients
const search = new SearchClient('http://localhost:3000');
const ingestion = new IngestionClient();
const graph = new KnowledgeGraphClient('http://localhost:3001');

// Query builders
const query = new SearchQueryBuilder()
  .text('meditation anxiety')
  .strategy('hybrid')
  .topK(10)
  .filter('languages', ['en'])
  .option('rerank', true)
  .build();

const results = await search.search(query);
```

---

## Related Documentation

- [Sophia Domain Overview](../README.md) -- Full app and library inventory
- [Sophia Architecture](../architecture.md) -- System design and data flow
- [Cross-Domain Integration](../../../cross-domain-integration.md) -- Event bus
  patterns
