# Sophia Domain

> **Sophia** - Knowledge Management and Research Platform

Sophia provides RAG (Retrieval-Augmented Generation) infrastructure for document
ingestion, semantic search with hybrid vector+keyword strategies, citation
management with multi-style formatting, knowledge graph construction with entity
resolution, and retrieval quality evaluation with benchmark support. It supports
four vector database backends and eight chunking strategies for flexible
document processing.

---

## Architecture

### Apps (4)

| App               | Port | Purpose                                                                                                   |
| ----------------- | ---- | --------------------------------------------------------------------------------------------------------- |
| `search-api`      | 3000 | `SearchAPIService` with hybrid search, RAG pipeline, cross-encoder reranking, and citation extraction     |
| `ingestion`       | -    | Multi-stage pipeline with configurable chunking (8 strategies), entity extraction, and summary generation |
| `knowledge-graph` | 3001 | `KnowledgeGraphService` with entity/relation CRUD, BFS/DFS traversal, path finding, and entity resolution |
| `workbench`       | -    | React UI with 6 pages: Dashboard, Curation, Review, Annotations, Documents, Search                        |

### Libraries (10)

| Library                     | Key Exports                                                                                                                                                                     |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `@sophia/client`            | `SophiaClient` (unified), `SearchClient`, `IngestionClient`, `KnowledgeGraphClient`, `SearchQueryBuilder`, `GraphQueryBuilder`, branded ID types                                |
| `@sophia/database`          | Prisma client (`getSophiaClient()`) for 12-model PostgreSQL schema with full-text search                                                                                        |
| `@sophia/schemas`           | Zod schemas with branded IDs for documents, citations, claims, knowledge packs, entities, relations                                                                             |
| `@sophia/indexing`          | 8 chunkers (fixed, sliding, paragraph, sentence, recursive, markdown, code, semantic), 3 embedding providers (OpenAI, Cohere, Local), `IndexBuilder`, vector math, graph export |
| `@sophia/vectordb`          | 4 vector stores (Memory, Milvus, Qdrant, Pinecone), 10 collection schemas, hybrid search with RRF, batch utilities, health checks                                               |
| `@sophia/evaluation`        | `RetrievalEvaluator`, `CitationChecker`, `GroundednessScorer`, `BenchmarkRunner`                                                                                                |
| `@sophia/theory`            | `CitationManager` (5 styles), `ResearchManager` (project/hypothesis tracking)                                                                                                   |
| `@sophia/cultural-research` | `CulturalConsultant`, `LocalizationManager`, `AccessibilityManager`, `CulturalDatabase`                                                                                         |
| `@sophia/event-publisher`   | `SophiaEventPublisher` with 9 event types for documents, search, entities, and citations                                                                                        |
| `@sophia/predictions`       | Time series forecasting, schedule optimization, content performance prediction                                                                                                  |

### Technology Stack

| Layer         | Technologies                                                        |
| ------------- | ------------------------------------------------------------------- |
| Framework     | Hono (planned), currently standalone service classes                |
| Database      | PostgreSQL with Prisma ORM (schema: `sophia`, 12 models, 966 lines) |
| Vector Stores | Qdrant, Pinecone, Milvus, In-memory                                 |
| Embeddings    | OpenAI text-embedding-3, Cohere embed-v3, Local                     |
| Validation    | Zod schemas with branded types                                      |
| Events        | `@oshun/event-bus` (Redis Streams)                                  |
| Logging       | `@oshun/logging` (Pino-based structured logging)                    |
| Build         | `@nx/js:tsc`                                                        |
| Testing       | Vitest                                                              |

---

## Database Schema

12 models in `libs/sophia/database/prisma/schema.prisma` (966 lines):

| Model                   | Table                      | Purpose                                                                                       |
| ----------------------- | -------------------------- | --------------------------------------------------------------------------------------------- |
| `Document`              | `documents`                | Research documents with 14 types, full publication metadata, quality scoring, tsvector search |
| `DocumentChunk`         | `document_chunks`          | Indexed text chunks with binary embedding storage and overlap linking                         |
| `Citation`              | `citations`                | Citation records with 6 styles, verification status, claim tracking                           |
| `Entity`                | `entities`                 | Knowledge graph nodes with 15 types, 5 confidence levels, merge tracking                      |
| `EntityMention`         | `entity_mentions`          | Entity occurrences in documents with position and context                                     |
| `Relation`              | `relations`                | Knowledge graph edges with 23 relation types and evidence tracking                            |
| `KnowledgePack`         | `knowledge_packs`          | Curated knowledge collections with versioning and community features                          |
| `KnowledgePackDocument` | `knowledge_pack_documents` | Pack-document associations                                                                    |
| `KnowledgePackEntity`   | `knowledge_pack_entities`  | Pack-entity associations with importance weighting                                            |
| `KnowledgePackStar`     | `knowledge_pack_stars`     | User favorites                                                                                |
| `IngestionJob`          | `ingestion_jobs`           | 10-stage pipeline tracking with retry support                                                 |
| `AuditLog`              | `audit_log`                | Change tracking                                                                               |

### Key Enums

- **DocumentType** (14): BOOK, ARTICLE, THESIS, MANUSCRIPT, SCRIPTURE,
  COMMENTARY, ESSAY, LETTER, SPEECH, TRANSLATION, ANTHOLOGY, ENCYCLOPEDIA,
  DICTIONARY, OTHER
- **EntityType** (15): TRADITION, TEXT, CONCEPT, PRACTICE, FIGURE, PLACE,
  SCHOOL, DOCTRINE, EVENT, TERM, SYMBOL, RITUAL, ARTIFACT, ORGANIZATION, CUSTOM
- **RelationType** (23): TAUGHT, STUDIED_WITH, INFLUENCED, WROTE, EDITED,
  TRANSLATED, FOUNDED, MEMBER_OF, BELONGS_TO, LIVED_IN, BORN_IN, DIED_IN,
  LOCATED_IN, PRECEDED, SUCCEEDED, CONTEMPORARY_WITH, REFERENCES, QUOTES, CITES,
  RELATED_TO, SIMILAR_TO, OPPOSITE_OF, DERIVED_FROM, PART_OF, PRACTICES,
  PRESCRIBES, CUSTOM

---

## API Endpoints

### Search API (Port 3000)

| Method | Path         | Description                                              |
| ------ | ------------ | -------------------------------------------------------- |
| POST   | `/v1/search` | Hybrid semantic search (vector + keyword) with reranking |
| POST   | `/v1/rag`    | RAG pipeline: retrieve, generate, cite                   |
| GET    | `/health`    | Service health check                                     |

### Documents (Port 4003)

| Method | Path                | Description           |
| ------ | ------------------- | --------------------- |
| POST   | `/v1/documents`     | Ingest a document     |
| GET    | `/v1/documents/:id` | Get document metadata |
| DELETE | `/v1/documents/:id` | Delete a document     |

### Collections

| Method | Path              | Description         |
| ------ | ----------------- | ------------------- |
| GET    | `/v1/collections` | List collections    |
| POST   | `/v1/collections` | Create a collection |

### Embeddings

| Method | Path             | Description         |
| ------ | ---------------- | ------------------- |
| POST   | `/v1/embeddings` | Generate embeddings |

### Knowledge Graph (Port 3001)

Graph operations are exposed as service methods (not REST endpoints):

| Category   | Operations                                                              |
| ---------- | ----------------------------------------------------------------------- |
| Entities   | create, get, query, delete                                              |
| Relations  | create, get, getForEntity, delete                                       |
| Traversal  | getNeighbors, traverse, findShortestPath, findAllPaths, extractSubgraph |
| Resolution | resolveEntity, findDuplicates                                           |

---

## Search Capabilities

- **Hybrid Search** - Combined keyword (BM25) and semantic vector search with
  configurable weights
- **Cross-Encoder Reranking** - Result reranking for improved relevance using
  cross-encoder models
- **Citation Tracking** - Source attribution with 6 citation styles and
  verification workflow
- **RAG Pipeline** - Full retrieval-augmented generation with LLM answer
  generation and citation extraction
- **Multi-Collection Queries** - Search across 10 predefined collection schemas
- **Context Retrieval** - Optimized context windows for LLM prompts

### Chunking Strategies

`@sophia/indexing` provides 8 chunking approaches:

| Strategy       | Class                  | Description                            |
| -------------- | ---------------------- | -------------------------------------- |
| Fixed          | `FixedSizeChunker`     | Fixed character/token window           |
| Sliding Window | `SlidingWindowChunker` | Overlapping sliding window             |
| Paragraph      | `ParagraphChunker`     | Paragraph boundary detection           |
| Sentence       | `SentenceChunker`      | Sentence boundary detection            |
| Recursive      | `RecursiveChunker`     | Hierarchical splitting with separators |
| Markdown       | `MarkdownChunker`      | Markdown structure-aware splitting     |
| Code           | `CodeChunker`          | Programming language-aware splitting   |
| Semantic       | `SemanticChunker`      | Embedding-based semantic boundaries    |

---

## Vector Store Integration

`@sophia/vectordb` provides a unified interface across four vector databases:

| Store    | SDK                           | Default Port | Factory                        |
| -------- | ----------------------------- | ------------ | ------------------------------ |
| Memory   | Built-in                      | -            | `createMemoryStore()`          |
| Qdrant   | `@qdrant/js-client-rest`      | 6333         | `createQdrantStoreFromEnv()`   |
| Pinecone | `@pinecone-database/pinecone` | Cloud        | `createPineconeStoreFromEnv()` |
| Milvus   | `@zilliz/milvus2-sdk-node`    | 19530        | `createMilvusStoreFromEnv()`   |

### Predefined Collections

10 collection schemas for different embedding use cases:
`KNOWLEDGE_CHUNKS_SCHEMA`, `RESEARCH_DOCUMENTS_SCHEMA`, `CITATIONS_SCHEMA`,
`ENTITIES_SCHEMA`, `WISDOM_EMBEDDINGS_SCHEMA`, `CONTENT_EMBEDDINGS_SCHEMA`,
`USER_PREFERENCE_EMBEDDINGS_SCHEMA`, `CONVERSATION_EMBEDDINGS_SCHEMA`,
`SEMANTIC_SEARCH_SCHEMA`, `PERSONA_EMBEDDINGS_SCHEMA`

For local development, start Qdrant:

```bash
docker compose -f docker/docker-compose.dev.yml --profile vectors up -d
```

---

## Cross-Domain Integration

### Events Published

| Event                        | Trigger                      |
| ---------------------------- | ---------------------------- |
| `sophia.document.ingested`   | Document ingestion completed |
| `sophia.document.updated`    | Document metadata changed    |
| `sophia.document.deleted`    | Document removed             |
| `sophia.index.updated`       | Vector index has new data    |
| `sophia.index.rebuilt`       | Full index rebuild completed |
| `sophia.search.performed`    | Search query executed        |
| `sophia.entity.extracted`    | New entity in graph          |
| `sophia.relation.discovered` | New relationship in graph    |
| `sophia.citation.created`    | Citation record available    |

---

## Environment Variables

```bash
SOPHIA_DATABASE_URL=postgresql://oshun:oshun_dev@localhost:5432/sophia
REDIS_URL=redis://localhost:6379

# Vector Stores
QDRANT_URL=http://localhost:6333
PINECONE_API_KEY=...
MILVUS_HOST=localhost
MILVUS_PORT=19530

# Embedding Providers
OPENAI_API_KEY=...
COHERE_API_KEY=...

# Service Configuration
PORT=3000
INGESTION_PORT=3000
MAX_CONCURRENT_JOBS=5
DEFAULT_CHUNK_SIZE=1000
DEFAULT_CHUNK_OVERLAP=200
LOG_LEVEL=info
```

---

## Development

```bash
# Start infrastructure
docker compose -f docker/docker-compose.dev.yml up -d

# Start vector database
docker compose -f docker/docker-compose.dev.yml --profile vectors up -d

# Run the search API
pnpm nx dev @sophia/search-api

# Run tests
pnpm nx test @sophia/client
pnpm nx test @sophia/indexing
pnpm nx test @sophia/vectordb
pnpm nx test @sophia/evaluation

# Build all Sophia libraries
pnpm nx run-many --target=build --projects=tag:scope:sophia

# Run database migrations
pnpm nx run @sophia/database:prisma:migrate
pnpm nx run @sophia/database:prisma:generate
```

---

## CI/CD

Sophia uses the core CI pipeline (`ci.yml`) and the general ECS deployment
pipeline (`deploy-ecs.yml`). Database migrations run via `database.yml`.

## Related Documentation

- [Architecture](./architecture.md) - Detailed Sophia architecture
- [Monorepo Architecture](../../architecture.md) - Monorepo overview
- [Cross-Domain Integration](../../cross-domain-integration.md)
