# Sophia Domain - Architecture

> **Canonical source:** the canonical architecture document for this domain is
> [`deep-dive/architecture.md`](./deep-dive/architecture.md), rendered in the
> docs center. This file is its operational companion (service topology, APIs,
> schemas, runtime detail) — reconciled by linking, not duplicated (§2, §13).

> Comprehensive technical architecture for the Sophia Knowledge Management and
> Research Platform

---

## System Overview

Sophia is the knowledge management and research platform within the Oshun
monorepo. It provides RAG (Retrieval-Augmented Generation) infrastructure for
document ingestion, semantic search, citation management, knowledge graph
construction, and retrieval quality evaluation. The domain is organized around
four applications and ten libraries that together form a complete research
knowledge pipeline.

### Core Responsibilities

- **Document Ingestion** - Multi-format document intake with configurable
  chunking strategies, entity extraction, and summary generation
- **Semantic Search** - Hybrid search combining vector similarity and keyword
  matching with cross-encoder reranking
- **RAG Pipeline** - Retrieval-augmented generation with citation tracking and
  groundedness scoring
- **Knowledge Graph** - Entity extraction, relationship mapping, entity
  resolution, and graph traversal
- **Evaluation** - Retrieval quality metrics, citation integrity checking, and
  benchmark management
- **Research Management** - Research project tracking, hypothesis management,
  and citation formatting

---

## Service Architecture

### Applications

#### 1. Search API (`apps/sophia/search-api`)

The primary search service providing semantic search with RAG context
generation.

| Property     | Value                                                         |
| ------------ | ------------------------------------------------------------- |
| Default Port | 3000                                                          |
| Entry Point  | `src/main.ts`                                                 |
| Service      | `SearchAPIService`                                            |
| Dependencies | `@oshun/logging`                                              |
| Internal     | `RetrievalService`, `CitationService`, `CrossEncoderReranker` |

**Key Configuration:**

```typescript
interface SearchAPIConfig {
  port?: number; // Default: 3000
  host?: string; // Default: 'localhost'
  defaultTopK?: number; // Default: 10
  maxTopK?: number; // Default: 100
  enableCitations?: boolean; // Default: true
  enableReranking?: boolean; // Default: true
}
```

**Core Operations:**

- `search(query: SearchQuery)` - Execute a search with hybrid strategy (vector +
  keyword), applies topK limits, delegates to `RetrievalService`
- `rag(request: RAGRequest)` - Full RAG pipeline: retrieves relevant documents,
  generates answer via LLM, extracts citations via `CitationService`
- `healthCheck()` - Service health and uptime reporting

**Request Handlers Factory:**

`createRequestHandlers(service)` produces framework-agnostic handlers for
`search`, `rag`, and `health` endpoints with error wrapping via `SearchError`.

#### 2. Ingestion Pipeline (`apps/sophia/ingestion`)

Document ingestion service that processes files and URLs through a multi-stage
pipeline.

| Property     | Value                                  |
| ------------ | -------------------------------------- |
| Entry Point  | `src/main.ts`                          |
| Service      | Pipeline via `createIngestionPipeline` |
| Dependencies | `@oshun/logging`                       |
| Modes        | CLI (document argument) or Service     |

**Configuration:**

```typescript
interface ServiceConfig {
  port: number; // Default: 3000 (INGESTION_PORT)
  maxConcurrentJobs: number; // Default: 5 (MAX_CONCURRENT_JOBS)
  defaultChunkSize: number; // Default: 1000 (DEFAULT_CHUNK_SIZE)
  defaultChunkOverlap: number; // Default: 200 (DEFAULT_CHUNK_OVERLAP)
  logLevel: LogLevel; // Default: 'info' (LOG_LEVEL)
}
```

**Pipeline Stages:**

1. **Source Detection** - Auto-detects URL vs file sources via
   `createURLSource()` / `createFileSource()`
2. **Document Processing** - Invokes `pipeline.processDocument()` with
   configurable enrichment
3. **Entity Extraction** - Extracts entities from document content
4. **Summary Generation** - Produces short summary and key points

**Event System:**

The pipeline emits typed events for monitoring:

- `job:created`, `job:started`, `job:completed`, `job:failed`
- `stage:started`, `stage:completed`, `stage:failed`

#### 3. Knowledge Graph (`apps/sophia/knowledge-graph`)

Standalone knowledge graph service for entity and relationship management with
graph traversal operations.

| Property     | Value                                              |
| ------------ | -------------------------------------------------- |
| Default Port | 3001                                               |
| Entry Point  | `src/main.ts`                                      |
| Service      | `KnowledgeGraphService`                            |
| Dependencies | `@oshun/logging`                                   |
| Store        | `CombinedMemoryStore` (extends `MemoryGraphStore`) |

**Configuration:**

```typescript
interface KnowledgeGraphConfig {
  port?: number; // Default: 3001
  host?: string; // Default: 'localhost'
  maxQueryLimit?: number; // Default: 100
  maxTraversalDepth?: number; // Default: 5
  enableResolution?: boolean; // Default: true
}
```

**Internal Services:**

- `EntityService` - CRUD for graph nodes with configurable max limits
- `RelationService` - CRUD for graph edges with constraint validation
- `GraphService` - Traversal, path-finding, and subgraph extraction with
  configurable limits:
  - `MAX_HOPS: 5`, `MAX_PATH_DEPTH: 10`, `MAX_RESULTS: 100`
  - `MAX_NEIGHBORS: 50`, `MAX_TRAVERSAL_NODES: 1000`, `MAX_PATHS: 50`
  - `TRAVERSAL_TIMEOUT_MS: 30000`
- `EntityResolver` - Duplicate detection and entity merging (optional,
  controlled by `enableResolution`)

**CombinedMemoryStore:**

Extends `MemoryGraphStore` with relationship storage, providing:

- In-memory relationship CRUD with auto-incrementing IDs
- Neighbor discovery by scanning relationships for source/target matches
- Filtered relationship queries with pagination
- Combined node + relationship statistics

#### 4. Workbench (`apps/sophia/workbench`)

Web-based UI for document management, search testing, and knowledge curation.

| Property    | Value                                                       |
| ----------- | ----------------------------------------------------------- |
| Entry Point | `src/`                                                      |
| Type        | Web application (React-based)                               |
| Pages       | Dashboard, Curation, Review, Annotations, Documents, Search |

**API Services Layer:**

The workbench connects to Sophia's backend via typed service clients:

| Service              | Purpose                               |
| -------------------- | ------------------------------------- |
| `AuthService`        | Authentication and session management |
| `DashboardService`   | Dashboard data and statistics         |
| `DocumentsService`   | Document CRUD and management          |
| `AnnotationsService` | Document annotation management        |
| `SearchService`      | Search queries and result display     |
| `ReviewService`      | Content review workflows              |

All services follow a singleton pattern with `getXService()` factory and
`resetXService()` for testing.

---

## Library Architecture

### Core Libraries

#### `@sophia/client` - TypeScript SDK

Unified client SDK providing type-safe access to all Sophia services.

**Exports:**

| Export                      | Purpose                                |
| --------------------------- | -------------------------------------- |
| `SophiaClient`              | Unified client for all services        |
| `createSophiaClient()`      | Factory for production client          |
| `createLocalSophiaClient()` | Factory for local development          |
| `SearchClient`              | Individual search service client       |
| `SearchQueryBuilder`        | Fluent API for building search queries |
| `IngestionClient`           | Individual ingestion service client    |
| `KnowledgeGraphClient`      | Individual knowledge graph client      |
| `GraphQueryBuilder`         | Fluent API for graph traversal queries |
| `HttpClient`                | Low-level HTTP transport               |
| `SophiaClientError`         | Typed error class with error codes     |

**Branded ID Types:**

`DocumentId`, `ChunkId`, `QueryId`, `ResultId`, `CitationId`, `JobId`,
`SourceId`, `NodeId`, `RelationshipId` - All with `createXId()` factory
functions for type-safe identifier construction.

**API Type Coverage:**

- Search: `SearchRequest`, `SearchResponse`, `RAGRequest`, `RAGResponse`,
  `Citation`, `CitationOptions`
- Ingestion: `IngestFileRequest`, `IngestUrlRequest`, `IngestionJob`,
  `DocumentMetadata`, `IngestedDocument`
- Knowledge Graph: `GraphNode`, `GraphRelationship`, `CreateNodeRequest`,
  `TraversalOptions`, `PathFindingOptions`, `SubgraphOptions`,
  `EntityResolutionRequest`

#### `@sophia/database` - Prisma Schema and Client

Database access layer providing Prisma client for Sophia's PostgreSQL schema.

**Exports:**

- `getSophiaClient()` - Singleton Prisma client instance
- `disconnectSophiaClient()` - Connection cleanup
- `createSophiaClient()` - Fresh client instance factory

**Connection:** `SOPHIA_DATABASE_URL` environment variable, schema namespace:
`sophia`

#### `@sophia/schemas` - Zod Validation Schemas

Domain validation schemas with branded ID types for all Sophia entities.

**Branded IDs with Zod Schemas:**

`DocumentId`, `CitationId`, `ClaimId`, `KnowledgePackId`, `EntityId`,
`RelationId`, `ChunkId`, `ProjectId`, `UserId`, `SourceId`, `IndexId`

Each ID type has:

- A branded TypeScript type
- A Zod schema (e.g., `DocumentIdSchema`)
- A factory function (e.g., `documentId()`)

**Domain Schemas:** Research documents, citations, claims, knowledge packs,
entities, and relations - all with Zod validation.

### Retrieval and Indexing Libraries

#### `@sophia/indexing` - Document Chunking and Embedding

Comprehensive indexing library covering chunking, embedding generation, entity
resolution, and graph visualization.

**Chunking Module:**

| Chunker                | Strategy                                            |
| ---------------------- | --------------------------------------------------- |
| `FixedSizeChunker`     | Fixed character/token window                        |
| `SlidingWindowChunker` | Overlapping sliding window                          |
| `ParagraphChunker`     | Paragraph boundary detection                        |
| `SentenceChunker`      | Sentence boundary detection                         |
| `RecursiveChunker`     | Hierarchical splitting with configurable separators |
| `MarkdownChunker`      | Markdown structure-aware splitting                  |
| `CodeChunker`          | Programming language-aware splitting                |
| `SemanticChunker`      | Embedding-based semantic boundaries                 |

Factory: `createChunker(strategy)` instantiates the appropriate chunker.
Predefined separator sets: `RECURSIVE_SEPARATORS`, `MARKDOWN_SEPARATORS`,
`CODE_SEPARATORS`.

**Embedding Module:**

| Provider                  | Backend                        |
| ------------------------- | ------------------------------ |
| `OpenAIEmbeddingProvider` | OpenAI text-embedding-3 models |
| `CohereEmbeddingProvider` | Cohere embed-v3 models         |
| `LocalEmbeddingProvider`  | Local/custom embedding models  |

`EmbeddingService` wraps providers with caching (`EmbeddingCache`) and batch
processing. Factory: `createEmbeddingServiceFromEnv()` auto-configures from
environment.

**Index Module:**

- `IndexBuilder` - Orchestrates chunking, embedding, and storage
- `InMemoryVectorStore` - Development vector store
- Vector math utilities: `cosineSimilarity`, `dotProduct`, `euclideanDistance`,
  `manhattanDistance`, `normalize`, `magnitude`, `centroid`, `reduceDimensions`
- `findTopK()` - Top-K nearest neighbor selection
- `pairwiseSimilarity()` - Matrix similarity computation

**Graph Module:**

- `EntityResolver` / `resolveEntities()` / `findDuplicates()` - Entity
  deduplication and merging
- `GraphExporter` - Export graphs in multiple formats: `exportToD3Format()`,
  `exportToDOTFormat()`, `exportToGEXFFormat()`
- `computeGraphStatistics()` / `computeNodeCentrality()` - Graph analytics

#### `@sophia/vectordb` - Vector Database Integration

Unified interface across four vector storage backends.

**Stores:**

| Store                 | Backend   | Factory                                                  |
| --------------------- | --------- | -------------------------------------------------------- |
| `MemoryVectorStore`   | In-memory | `createMemoryStore()`                                    |
| `MilvusVectorStore`   | Milvus    | `createMilvusStore()` / `createMilvusStoreFromEnv()`     |
| `QdrantVectorStore`   | Qdrant    | `createQdrantStore()` / `createQdrantStoreFromEnv()`     |
| `PineconeVectorStore` | Pinecone  | `createPineconeStore()` / `createPineconeStoreFromEnv()` |

**Predefined Collection Schemas:**

| Schema                              | Purpose                   |
| ----------------------------------- | ------------------------- |
| `KNOWLEDGE_CHUNKS_SCHEMA`           | Document chunk embeddings |
| `RESEARCH_DOCUMENTS_SCHEMA`         | Full document embeddings  |
| `CITATIONS_SCHEMA`                  | Citation embeddings       |
| `ENTITIES_SCHEMA`                   | Entity embeddings         |
| `WISDOM_EMBEDDINGS_SCHEMA`          | Wisdom/philosophy content |
| `CONTENT_EMBEDDINGS_SCHEMA`         | General content           |
| `USER_PREFERENCE_EMBEDDINGS_SCHEMA` | User preference vectors   |
| `CONVERSATION_EMBEDDINGS_SCHEMA`    | Conversation context      |
| `SEMANTIC_SEARCH_SCHEMA`            | General semantic search   |
| `PERSONA_EMBEDDINGS_SCHEMA`         | Persona/character vectors |

**Search Capabilities:**

- `search()` - Pure vector similarity search with metadata filters
- `hybridSearch()` - Combined vector + keyword search with configurable weights
  and rerank strategy (`rrf` reciprocal rank fusion or `weightedCombination`)

**Utilities:**

- Similarity: `cosineSimilarity`, `dotProduct`, `euclideanDistance`
- Fusion: `reciprocalRankFusion`, `weightedCombination`, `textSimilarity`
- Batch processing: `batchArray`, `processBatchesSequentially`,
  `processBatchesConcurrently`, `processBatchesWithProgress`
- Resilience: `withRetry` with configurable `RetryConfig`
- Health: `checkVectorStoreHealth`, `checkMultipleStoresHealth`,
  `getAggregateHealthStatus`

### Research and Analysis Libraries

#### `@sophia/evaluation` - Retrieval Quality Assessment

Quality metrics and benchmark management for search accuracy evaluation.

**Components:**

| Component            | Purpose                                              |
| -------------------- | ---------------------------------------------------- |
| `RetrievalEvaluator` | Faithfulness, relevancy, precision/recall metrics    |
| `CitationChecker`    | Quote accuracy, entailment, attribution verification |
| `GroundednessScorer` | Semantic and lexical factual accuracy scoring        |
| `BenchmarkRunner`    | Built-in and custom benchmark execution              |

#### `@sophia/theory` - Research Methodology Framework

Research project management and citation formatting.

**Components:**

| Component         | Purpose                                                                     |
| ----------------- | --------------------------------------------------------------------------- |
| `CitationManager` | Claim detection, citation formatting (Chicago, MLA, APA, Harvard, Turabian) |
| `ResearchManager` | Research project lifecycle and hypothesis tracking                          |

#### `@sophia/cultural-research` - Cultural Context Research

Cultural sensitivity and localization for research content.

**Components:**

| Component              | Purpose                                                |
| ---------------------- | ------------------------------------------------------ |
| `CulturalConsultant`   | Content sensitivity checking against cultural profiles |
| `LocalizationManager`  | Text translation and locale management                 |
| `AccessibilityManager` | WCAG accessibility checking (AA/AAA)                   |
| `CulturalDatabase`     | Cultural profile and reference storage                 |

#### `@sophia/predictions` - Predictive Analytics

Forecasting and optimization models for content platforms.

**Capabilities:**

- Time series forecasting (revenue, viewership)
- Schedule optimization (streaming time slots)
- Content performance prediction
- Seasonal factor and peak period analysis

### Infrastructure Libraries

#### `@sophia/event-publisher` - Cross-Domain Events

Typed event publisher for Sophia domain events via `@oshun/event-bus`.

**Event Types:**

| Event                             | Trigger                      |
| --------------------------------- | ---------------------------- |
| `SophiaDocumentIngestedPayload`   | Document ingestion completed |
| `SophiaDocumentUpdatedPayload`    | Document metadata updated    |
| `SophiaDocumentDeletedPayload`    | Document removed             |
| `SophiaIndexUpdatedPayload`       | Vector index updated         |
| `SophiaIndexRebuiltPayload`       | Full index rebuild completed |
| `SophiaSearchPerformedPayload`    | Search query executed        |
| `SophiaEntityExtractedPayload`    | Entity extraction completed  |
| `SophiaRelationDiscoveredPayload` | Relationship discovered      |
| `SophiaCitationCreatedPayload`    | Citation record created      |

---

## Data Flow Diagrams

### Search Pipeline

```
 User Query
     |
     v
+------------------+
| SearchAPIService |
| (search-api)     |
+--------+---------+
         |
         v
+------------------+     +--------------------+
| RetrievalService |---->| EmbeddingProvider   |
| (hybrid search)  |     | (OpenAI/Cohere/     |
+--------+---------+     |  Local)             |
         |               +--------------------+
         |
    +----+----+
    |         |
    v         v
+-------+ +----------+
|Vector | |Keyword   |
|Search | |Search    |
|(topK) | |(BM25)    |
+---+---+ +----+-----+
    |          |
    +----+-----+
         |
         v
+------------------+     +------------------+
| CrossEncoder     |     | CitationService  |
| Reranker         |     | (if RAG mode)    |
+--------+---------+     +--------+---------+
         |                        |
         v                        v
+------------------+     +------------------+
| SearchResponse   |     | RAGResponse      |
| {results, total, |     | {answer, cites,  |
|  processingTime} |     |  sources, time}  |
+------------------+     +------------------+
```

### Document Ingestion Pipeline

```
 Source (File/URL)
     |
     v
+---------------------+
| IngestionPipeline   |
| (ingestion app)     |
+----------+----------+
           |
           v
+---------------------+  Events: job:created
| Source Detection     |---------> job:started
| (File vs URL)       |
+----------+----------+
           |
     +-----+-----+
     |           |
     v           v
+----------+ +-----------+
|FileSource| |URL Source  |
|          | |(fetch &    |
|          | | extract)   |
+----+-----+ +-----+-----+
     |              |
     +------+-------+
            |
            v
+---------------------+  Events: stage:started
| Text Extraction     |---------> stage:completed
+----------+----------+
           |
           v
+---------------------+
| Chunking            |
| (8 strategies:      |
|  fixed, sliding,    |
|  paragraph, sent,   |
|  recursive, md,     |
|  code, semantic)    |
+----------+----------+
           |
           v
+---------------------+
| Enrichment          |
| - Entity Extraction |
| - Summary Generation|
| - Key Points        |
+----------+----------+
           |
           v
+---------------------+
| Embedding           |
| (OpenAI/Cohere/     |  Events: job:completed
|  Local provider)    |---------> or job:failed
+----------+----------+
           |
           v
+---------------------+
| Vector Store        |
| (Qdrant/Pinecone/   |
|  Milvus/Memory)     |
+---------------------+
```

### Knowledge Graph Operations

```
+-------------------+
| KnowledgeGraph    |
| Service           |
+--------+----------+
         |
    +----+----+----+----+
    |         |         |
    v         v         v
+--------+ +--------+ +--------+
|Entity  | |Relation| |Graph   |
|Service | |Service | |Service |
+---+----+ +---+----+ +---+----+
    |          |           |
    |          |      +----+----+----+
    |          |      |         |    |
    v          v      v         v    v
+---------------------------------------+
| CombinedMemoryStore                   |
| (extends MemoryGraphStore)            |
|                                       |
| Nodes: Map<NodeId, GraphNode>         |
| Relationships: Map<string, Rel>       |
+-------------------+-------------------+
                    |
                    v
          +------------------+
          | EntityResolver   |
          | (optional)       |
          |                  |
          | - Duplicate      |
          |   Detection      |
          | - Entity Merging |
          | - Canonical ID   |
          |   Resolution     |
          +------------------+

Graph Traversal Operations:
  - getNeighbors(nodeId, options)
  - traverse(options)           -> BFS/DFS traversal
  - findShortestPath(options)   -> Dijkstra-like path finding
  - findAllPaths(options)       -> All paths between nodes
  - extractSubgraph(options)    -> Subgraph extraction
```

### Cross-Domain Integration

```
                        +------------------+
                        | @oshun/event-bus |
                        | (Redis Streams)  |
                        +--------+---------+
                                 |
            +--------------------+--------------------+
            |                    |                    |
            v                    v                    v
+------------------+  +------------------+  +------------------+
| sophia.document  |  | sophia.entity    |  | sophia.search    |
| .ingested        |  | .extracted       |  | .performed       |
| .updated         |  | sophia.relation  |  +------------------+
| .deleted         |  | .discovered      |
+--------+---------+  +--------+---------+
         |                     |
         v                     v
+------------------+  +------------------+
| Consumers:       |  | Consumers:       |
| - Hathor (lore   |  | - Isis (context  |
|   enrichment)    |  |   for generation)|
| - Bellona (asset |  | - Hathor (world  |
|   metadata)      |  |   knowledge)     |
+------------------+  +------------------+

Events Published by Sophia:
  sophia.document.ingested   -> Document available for downstream use
  sophia.document.updated    -> Document metadata changed
  sophia.document.deleted    -> Document removed from index
  sophia.index.updated       -> Search index has new data
  sophia.index.rebuilt       -> Full index rebuild complete
  sophia.search.performed    -> Analytics tracking
  sophia.entity.extracted    -> New entity available in graph
  sophia.relation.discovered -> New relationship in graph
  sophia.citation.created    -> Citation record available
```

---

## Database Schema

### PostgreSQL (Prisma ORM)

Schema namespace: `sophia`. Connection: `SOPHIA_DATABASE_URL`. Preview features:
`fullTextSearch`, `fullTextIndex`.

#### Models

| Model                   | Table                      | Purpose                             |
| ----------------------- | -------------------------- | ----------------------------------- |
| `Document`              | `documents`                | Research document records           |
| `DocumentChunk`         | `document_chunks`          | Indexed text chunks with embeddings |
| `Citation`              | `citations`                | Citation records with verification  |
| `Entity`                | `entities`                 | Knowledge graph nodes               |
| `EntityMention`         | `entity_mentions`          | Entity occurrences in documents     |
| `Relation`              | `relations`                | Knowledge graph edges               |
| `KnowledgePack`         | `knowledge_packs`          | Curated knowledge collections       |
| `KnowledgePackDocument` | `knowledge_pack_documents` | Pack-document associations          |
| `KnowledgePackEntity`   | `knowledge_pack_entities`  | Pack-entity associations            |
| `KnowledgePackStar`     | `knowledge_pack_stars`     | User favorites                      |
| `IngestionJob`          | `ingestion_jobs`           | Ingestion pipeline tracking         |
| `AuditLog`              | `audit_log`                | Change tracking                     |

#### Document Model

The `Document` model is the central entity with comprehensive metadata:

| Field Group     | Fields                                                                                      |
| --------------- | ------------------------------------------------------------------------------------------- |
| **Identity**    | `id` (CUID), `title`, `originalTitle`, `type`, `genre`, `status`                            |
| **Authorship**  | `author`, `authors` (JSON array), `editor`, `translator`                                    |
| **Publication** | `publisher`, `publicationPlace`, `publicationYear`, `edition`, `volume`, `issue`, `pages`   |
| **Identifiers** | `isbn`, `issn`, `doi`, `url`, `externalIds` (JSON)                                          |
| **Language**    | `language`, `languages[]`, `originRegion`                                                   |
| **Content**     | `abstract`, `summary`, `keyPoints` (JSON), `keywords[]`, `subjects[]`, `tags[]`             |
| **Tradition**   | `tradition`, `traditions[]`, `category`                                                     |
| **Dates**       | `compositionYear`, `compositionEra`, `dateRange` (JSON)                                     |
| **Rights**      | `rightsStatus`, `rightsHolder`, `license`                                                   |
| **File**        | `sourceUri`, `sourceType`, `fileFormat`, `fileSize`, `checksum`, `mimeType`                 |
| **Processing**  | `pageCount`, `wordCount`, `tokenCount`, `chunkCount`, `processingTimeMs`, `processingError` |
| **Quality**     | `qualityScore`, `completeness`, `accuracy`                                                  |
| **Ownership**   | `ownerId`, `organizationId`, `projectId`                                                    |
| **Search**      | `searchVector` (tsvector for full-text search)                                              |

Unique constraint: `(ownerId, title, author)`

#### DocumentChunk Model

Indexed text units with embedding storage:

- Position tracking: `index`, `startOffset`, `endOffset`, `pageNumber`,
  `sectionTitle`, `paragraphIndex`
- Embedding: `embeddingModel`, `embeddingDimension`, `embeddingVector` (binary),
  `embeddingGeneratedAt`
- Overlap linking: `overlapBefore`, `overlapAfter`, `previousChunkId`,
  `nextChunkId`
- Context: JSON field with `{before, after, documentTitle, chapterTitle}`

#### Entity Model (Knowledge Graph Nodes)

Entities represent concepts, figures, traditions, and other knowledge units:

- 15 entity types: `TRADITION`, `TEXT`, `CONCEPT`, `PRACTICE`, `FIGURE`,
  `PLACE`, `SCHOOL`, `DOCTRINE`, `EVENT`, `TERM`, `SYMBOL`, `RITUAL`,
  `ARTIFACT`, `ORGANIZATION`, `CUSTOM`
- 5 confidence levels: `CERTAIN`, `PROBABLE`, `POSSIBLE`, `UNCERTAIN`,
  `INFERRED`
- Merge tracking: `canonicalId`, `mergedInto`, `mergedAt` for entity resolution
- Statistics: `mentionCount`, `relationCount`, `referenceCount`
- Full-text search via `searchVector` (tsvector)

#### Relation Model (Knowledge Graph Edges)

Relationships between entities with evidence tracking:

- 23 relation types across 8 categories:
  - Teaching: `TAUGHT`, `STUDIED_WITH`, `INFLUENCED`
  - Authorship: `WROTE`, `EDITED`, `TRANSLATED`
  - Membership: `FOUNDED`, `MEMBER_OF`, `BELONGS_TO`
  - Location: `LIVED_IN`, `BORN_IN`, `DIED_IN`, `LOCATED_IN`
  - Temporal: `PRECEDED`, `SUCCEEDED`, `CONTEMPORARY_WITH`
  - Reference: `REFERENCES`, `QUOTES`, `CITES`
  - Conceptual: `RELATED_TO`, `SIMILAR_TO`, `OPPOSITE_OF`, `DERIVED_FROM`,
    `PART_OF`
  - Practice: `PRACTICES`, `PRESCRIBES`
- Evidence: `evidenceDocumentId`, `evidenceChunkId`, `evidenceText`,
  `evidenceType`
- Spatial/temporal context: `temporalContext`, `spatialContext`
- Unique constraint: `(sourceId, targetId, type)`

#### Citation Model

Citation records with multi-style formatting and verification:

- 6 citation styles: `CHICAGO`, `MLA`, `APA`, `HARVARD`, `TURABIAN`, `CUSTOM`
- 5 verification statuses: `VERIFIED`, `REVIEWED`, `UNVERIFIED`, `DISPUTED`,
  `PENDING`
- Claim tracking: `claimText`, `claimType`
- Note support: `noteType`, `noteMarker`, `noteContent` (footnote/endnote)

#### KnowledgePack Model

Curated knowledge collections with versioning and community features:

- 8 categories: `TRADITION`, `PERIOD`, `TOPIC`, `FIGURE`, `TEXT`, `REGION`,
  `PRACTICE`, `CUSTOM`
- 4 visibility levels: `PRIVATE`, `TEAM`, `ORGANIZATION`, `PUBLIC`
- Statistics: `documentCount`, `entityCount`, `relationCount`, `citationCount`,
  `chunkCount`, `totalTokens`
- Usage tracking: `viewCount`, `downloadCount`, `queryCount`, `starCount`
- Quality metrics: `qualityTier`, `qualityScore`, `completeness`
- Versioning: `version`, `previousVersionId`

#### IngestionJob Model

Pipeline job tracking with multi-stage progress:

- 10 statuses: `PENDING`, `QUEUED`, `FETCHING`, `EXTRACTING`, `ENRICHING`,
  `CHUNKING`, `INDEXING`, `COMPLETED`, `FAILED`, `CANCELLED`
- 6 source types: `FILE`, `URL`, `API`, `DATABASE`, `STREAM`, `ARCHIVE`
- Configuration: `config`, `extractionConfig`, `enrichmentConfig`,
  `chunkingConfig` (all JSON)
- Results: `chunksCreated`, `tokensProcessed`, `entitiesExtracted`,
  `relationsExtracted`
- Error tracking: `errorCode`, `errorMessage`, `errorStage`, `errorDetails`
- Queue management: `priority`, `workerId`, `queueName`, `retryCount`,
  `maxRetries` (default 3)

### Enums Summary

| Enum                      | Values                                                                                                                                                                                                                                                                                            |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `DocumentType`            | BOOK, ARTICLE, THESIS, MANUSCRIPT, SCRIPTURE, COMMENTARY, ESSAY, LETTER, SPEECH, TRANSLATION, ANTHOLOGY, ENCYCLOPEDIA, DICTIONARY, OTHER                                                                                                                                                          |
| `DocumentStatus`          | DRAFT, INGESTING, EXTRACTING, ENRICHING, CHUNKING, INDEXING, PUBLISHED, ARCHIVED, FAILED, DELETED                                                                                                                                                                                                 |
| `DocumentGenre`           | RELIGIOUS, PHILOSOPHICAL, HISTORICAL, SCHOLARLY, PRACTICAL, DEVOTIONAL, COMMENTARY, NARRATIVE, POETRY, LITURGICAL, LEGAL, BIOGRAPHICAL, INSTRUCTIONAL, OTHER                                                                                                                                      |
| `EntityType`              | TRADITION, TEXT, CONCEPT, PRACTICE, FIGURE, PLACE, SCHOOL, DOCTRINE, EVENT, TERM, SYMBOL, RITUAL, ARTIFACT, ORGANIZATION, CUSTOM                                                                                                                                                                  |
| `RelationType`            | TAUGHT, STUDIED_WITH, INFLUENCED, WROTE, EDITED, TRANSLATED, FOUNDED, MEMBER_OF, BELONGS_TO, LIVED_IN, BORN_IN, DIED_IN, LOCATED_IN, PRECEDED, SUCCEEDED, CONTEMPORARY_WITH, REFERENCES, QUOTES, CITES, RELATED_TO, SIMILAR_TO, OPPOSITE_OF, DERIVED_FROM, PART_OF, PRACTICES, PRESCRIBES, CUSTOM |
| `IngestionJobStatus`      | PENDING, QUEUED, FETCHING, EXTRACTING, ENRICHING, CHUNKING, INDEXING, COMPLETED, FAILED, CANCELLED                                                                                                                                                                                                |
| `KnowledgePackCategory`   | TRADITION, PERIOD, TOPIC, FIGURE, TEXT, REGION, PRACTICE, CUSTOM                                                                                                                                                                                                                                  |
| `KnowledgePackVisibility` | PRIVATE, TEAM, ORGANIZATION, PUBLIC                                                                                                                                                                                                                                                               |

---

## API Endpoints

### Search API (Port 3000)

#### Search

| Method | Path         | Description                          |
| ------ | ------------ | ------------------------------------ |
| POST   | `/v1/search` | Semantic search with hybrid strategy |

**Request:**

```typescript
interface SearchQuery {
  text: string;
  topK?: number; // Max: 100
  strategy?: 'hybrid' | 'vector' | 'keyword';
  filters?: SearchFilters;
}
```

**Response:**

```typescript
interface SearchResponse {
  results: SearchResult[];
  totalResults: number;
  processingTimeMs: number;
}
```

#### RAG

| Method | Path      | Description                    |
| ------ | --------- | ------------------------------ |
| POST   | `/v1/rag` | Retrieval-augmented generation |

**Request:**

```typescript
interface RAGRequest {
  query: string;
  topK?: number;
  strategy?: 'hybrid' | 'vector' | 'keyword';
  includeCitations?: boolean;
  citationOptions?: CitationOptions;
}
```

**Response:**

```typescript
interface RAGResponse {
  queryId: QueryId;
  query: string;
  answer: string;
  citations: Citation[];
  sources: SearchResult[];
  processingTimeMs: number;
  metadata: { retrievalStrategy: string };
}
```

#### Health

| Method | Path      | Description    |
| ------ | --------- | -------------- |
| GET    | `/health` | Service health |

### Documents (Port 4003)

| Method | Path                | Description           |
| ------ | ------------------- | --------------------- |
| POST   | `/v1/documents`     | Ingest a document     |
| GET    | `/v1/documents/:id` | Get document metadata |
| DELETE | `/v1/documents/:id` | Delete a document     |

### Collections

| Method | Path              | Description         |
| ------ | ----------------- | ------------------- |
| GET    | `/v1/collections` | List collections    |
| POST   | `/v1/collections` | Create a collection |

### Embeddings

| Method | Path             | Description         |
| ------ | ---------------- | ------------------- |
| POST   | `/v1/embeddings` | Generate embeddings |

### Knowledge Graph (Port 3001)

#### Entities

| Operation | Description                                                    |
| --------- | -------------------------------------------------------------- |
| Create    | `createEntity(request)` - Create node with type and properties |
| Get       | `getEntity(id)` - Retrieve node by ID                          |
| Query     | `queryEntities(request)` - Filter and paginate nodes           |
| Delete    | `deleteEntity(id)` - Remove node                               |

#### Relationships

| Operation | Description                                                     |
| --------- | --------------------------------------------------------------- |
| Create    | `createRelationship(request)` - Create typed edge between nodes |
| Get       | `getRelationship(id)` - Retrieve edge by ID                     |
| ForEntity | `getRelationshipsForEntity(nodeId)` - All edges for a node      |
| Delete    | `deleteRelationship(id)` - Remove edge                          |

#### Graph Traversal

| Operation     | Description                                                |
| ------------- | ---------------------------------------------------------- |
| Neighbors     | `getNeighbors(nodeId, options)` - Direct connections       |
| Traverse      | `traverse(options)` - BFS/DFS traversal with depth limits  |
| Shortest Path | `findShortestPath(options)` - Weighted shortest path       |
| All Paths     | `findAllPaths(options)` - All paths up to max depth        |
| Subgraph      | `extractSubgraph(options)` - Connected subgraph extraction |

#### Entity Resolution

| Operation       | Description                                              |
| --------------- | -------------------------------------------------------- |
| Resolve         | `resolveEntity(request)` - Match entity against existing |
| Find Duplicates | `findDuplicates(threshold)` - Detect duplicate entities  |

---

## Technology Stack

| Layer         | Technologies                                                                            |
| ------------- | --------------------------------------------------------------------------------------- |
| Framework     | Hono (planned), currently standalone service classes                                    |
| Database      | PostgreSQL with Prisma ORM (schema: `sophia`)                                           |
| Vector Stores | Qdrant, Pinecone, Milvus, In-memory                                                     |
| Embeddings    | OpenAI text-embedding-3, Cohere embed-v3, Local                                         |
| Chunking      | 8 strategies (fixed, sliding, paragraph, sentence, recursive, markdown, code, semantic) |
| Graph Store   | In-memory (MemoryGraphStore), extensible interface                                      |
| Events        | `@oshun/event-bus` (Redis Streams)                                                      |
| Validation    | Zod schemas with branded types                                                          |
| Logging       | `@oshun/logging` (Pino-based structured logging)                                        |
| Build         | `@nx/js:tsc`                                                                            |
| Testing       | Vitest                                                                                  |
| UI            | React (Workbench app)                                                                   |

---

## Deployment Architecture

### Service Topology

```
                    +------------------+
                    |   Load Balancer  |
                    +--------+---------+
                             |
                +------------+------------+
                |            |            |
                v            v            v
         +----------+  +----------+  +----------+
         |Search API|  |Ingestion |  |Knowledge |
         |(:3000)   |  |Pipeline  |  |Graph     |
         |          |  |          |  |(:3001)   |
         +----+-----+  +----+-----+  +----+-----+
              |              |              |
         +----+--------------+--------------+----+
         |                                       |
    +----+----+                             +----+----+
    |PostgreSQL|                             |  Redis  |
    |(sophia)  |                             | (cache, |
    |          |                             |  queue) |
    +----+-----+                             +---------+
         |
    +----+----+----+----+
    |         |         |
    v         v         v
+------+ +-------+ +-------+
|Qdrant| |Milvus | |Pinecone|
|(self-| |(self- | |(cloud) |
|hosted)| |hosted)|         |
+------+ +-------+ +-------+
```

### ECS Deployment

Sophia services deploy via the shared `deploy-ecs.yml` pipeline:

- **search-api** - ECS service with auto-scaling based on request volume
- **ingestion** - ECS task for batch processing, triggered by queue depth
- **knowledge-graph** - ECS service for graph operations
- **workbench** - Static deployment (S3 + CloudFront)

### Database Migrations

Managed via the shared `database.yml` CI pipeline with Prisma migrations
targeting the `sophia` PostgreSQL database.

---

## Cross-Domain Dependencies

### Sophia Consumes From

| Source Domain | Event/Integration          | Purpose                           |
| ------------- | -------------------------- | --------------------------------- |
| Hathor        | World/lore content         | Worldbuilding knowledge ingestion |
| Isis          | Generated content metadata | AI output indexing and search     |

### Sophia Publishes To

| Target Domain | Event                      | Purpose                       |
| ------------- | -------------------------- | ----------------------------- |
| Hathor        | `sophia.document.ingested` | Lore enrichment with research |
| Bellona       | `sophia.document.updated`  | Asset metadata updates        |
| Isis          | `sophia.entity.extracted`  | Context for AI generation     |
| All           | `sophia.search.performed`  | Analytics and usage tracking  |

### Shared Infrastructure

| Dependency         | Usage                                         |
| ------------------ | --------------------------------------------- |
| `@oshun/logging`   | Structured logging across all services        |
| `@oshun/event-bus` | Cross-domain event publishing (Redis Streams) |
| PostgreSQL         | Primary data store (schema: `sophia`)         |
| Redis              | Caching, queue management, event streaming    |

---

## Environment Variables

```bash
# Database
SOPHIA_DATABASE_URL=postgresql://oshun:oshun_dev@localhost:5432/sophia

# Redis
REDIS_URL=redis://localhost:6379

# Vector Stores
QDRANT_URL=http://localhost:6333
PINECONE_API_KEY=...
MILVUS_HOST=localhost
MILVUS_PORT=19530

# Embedding Providers
OPENAI_API_KEY=...
COHERE_API_KEY=...

# Service Ports
PORT=3000                    # Search API
INGESTION_PORT=3000          # Ingestion service
HOST=localhost

# Ingestion Config
MAX_CONCURRENT_JOBS=5
DEFAULT_CHUNK_SIZE=1000
DEFAULT_CHUNK_OVERLAP=200
LOG_LEVEL=info
```

---

## Development

```bash
# Run the search API
pnpm nx dev @sophia/search-api

# Run tests
pnpm nx test @sophia/client
pnpm nx test @sophia/indexing
pnpm nx test @sophia/vectordb

# Build all Sophia libraries
pnpm nx run-many --target=build --projects=tag:scope:sophia

# Start vector database for development
docker compose -f docker/docker-compose.dev.yml --profile vectors up -d

# Run database migrations
pnpm nx run @sophia/database:prisma:migrate
pnpm nx run @sophia/database:prisma:generate
```
