Canonical source: the canonical architecture document for this domain is
deep-dive/architecture.md, rendered in the docs center. This file is its operational companion (service topology, APIs, schemas, runtime detail) — reconciled by linking, not duplicated (§2, §13).
Comprehensive technical architecture for the Sophia Knowledge Management and Research Platform
System Overview#
Sophia is the knowledge management and research platform within the Oshun monorepo. It provides RAG (Retrieval-Augmented Generation) infrastructure for document ingestion, semantic search, citation management, knowledge graph construction, and retrieval quality evaluation. The domain is organized around four applications and ten libraries that together form a complete research knowledge pipeline.
Core Responsibilities#
- Document Ingestion - Multi-format document intake with configurable chunking strategies, entity extraction, and summary generation
- Semantic Search - Hybrid search combining vector similarity and keyword matching with cross-encoder reranking
- RAG Pipeline - Retrieval-augmented generation with citation tracking and groundedness scoring
- Knowledge Graph - Entity extraction, relationship mapping, entity resolution, and graph traversal
- Evaluation - Retrieval quality metrics, citation integrity checking, and benchmark management
- Research Management - Research project tracking, hypothesis management, and citation formatting
Service Architecture#
Applications#
1. Search API (apps/sophia/search-api)#
The primary search service providing semantic search with RAG context generation.
| Property | Value |
|---|---|
| Default Port | 3000 |
| Entry Point | src/main.ts |
| Service | SearchAPIService |
| Dependencies | @oshun/logging |
| Internal | RetrievalService, CitationService, CrossEncoderReranker |
Key Configuration:
interface SearchAPIConfig {
port?: number; // Default: 3000
host?: string; // Default: 'localhost'
defaultTopK?: number; // Default: 10
maxTopK?: number; // Default: 100
enableCitations?: boolean; // Default: true
enableReranking?: boolean; // Default: true
}
Core Operations:
search(query: SearchQuery)- Execute a search with hybrid strategy (vector + keyword), applies topK limits, delegates toRetrievalServicerag(request: RAGRequest)- Full RAG pipeline: retrieves relevant documents, generates answer via LLM, extracts citations viaCitationServicehealthCheck()- Service health and uptime reporting
Request Handlers Factory:
createRequestHandlers(service) produces framework-agnostic handlers for
search, rag, and health endpoints with error wrapping via SearchError.
2. Ingestion Pipeline (apps/sophia/ingestion)#
Document ingestion service that processes files and URLs through a multi-stage pipeline.
| Property | Value |
|---|---|
| Entry Point | src/main.ts |
| Service | Pipeline via createIngestionPipeline |
| Dependencies | @oshun/logging |
| Modes | CLI (document argument) or Service |
Configuration:
interface ServiceConfig {
port: number; // Default: 3000 (INGESTION_PORT)
maxConcurrentJobs: number; // Default: 5 (MAX_CONCURRENT_JOBS)
defaultChunkSize: number; // Default: 1000 (DEFAULT_CHUNK_SIZE)
defaultChunkOverlap: number; // Default: 200 (DEFAULT_CHUNK_OVERLAP)
logLevel: LogLevel; // Default: 'info' (LOG_LEVEL)
}
Pipeline Stages:
- Source Detection - Auto-detects URL vs file sources via
createURLSource()/createFileSource() - Document Processing - Invokes
pipeline.processDocument()with configurable enrichment - Entity Extraction - Extracts entities from document content
- Summary Generation - Produces short summary and key points
Event System:
The pipeline emits typed events for monitoring:
job:created,job:started,job:completed,job:failedstage:started,stage:completed,stage:failed
3. Knowledge Graph (apps/sophia/knowledge-graph)#
Standalone knowledge graph service for entity and relationship management with graph traversal operations.
| Property | Value |
|---|---|
| Default Port | 3001 |
| Entry Point | src/main.ts |
| Service | KnowledgeGraphService |
| Dependencies | @oshun/logging |
| Store | CombinedMemoryStore (extends MemoryGraphStore) |
Configuration:
interface KnowledgeGraphConfig {
port?: number; // Default: 3001
host?: string; // Default: 'localhost'
maxQueryLimit?: number; // Default: 100
maxTraversalDepth?: number; // Default: 5
enableResolution?: boolean; // Default: true
}
Internal Services:
EntityService- CRUD for graph nodes with configurable max limitsRelationService- CRUD for graph edges with constraint validationGraphService- Traversal, path-finding, and subgraph extraction with configurable limits:MAX_HOPS: 5,MAX_PATH_DEPTH: 10,MAX_RESULTS: 100MAX_NEIGHBORS: 50,MAX_TRAVERSAL_NODES: 1000,MAX_PATHS: 50TRAVERSAL_TIMEOUT_MS: 30000
EntityResolver- Duplicate detection and entity merging (optional, controlled byenableResolution)
CombinedMemoryStore:
Extends MemoryGraphStore with relationship storage, providing:
- In-memory relationship CRUD with auto-incrementing IDs
- Neighbor discovery by scanning relationships for source/target matches
- Filtered relationship queries with pagination
- Combined node + relationship statistics
4. Workbench (apps/sophia/workbench)#
Web-based UI for document management, search testing, and knowledge curation.
| Property | Value |
|---|---|
| Entry Point | src/ |
| Type | Web application (React-based) |
| Pages | Dashboard, Curation, Review, Annotations, Documents, Search |
API Services Layer:
The workbench connects to Sophia's backend via typed service clients:
| Service | Purpose |
|---|---|
AuthService |
Authentication and session management |
DashboardService |
Dashboard data and statistics |
DocumentsService |
Document CRUD and management |
AnnotationsService |
Document annotation management |
SearchService |
Search queries and result display |
ReviewService |
Content review workflows |
All services follow a singleton pattern with getXService() factory and
resetXService() for testing.
Library Architecture#
Core Libraries#
@sophia/client - TypeScript SDK#
Unified client SDK providing type-safe access to all Sophia services.
Exports:
| Export | Purpose |
|---|---|
SophiaClient |
Unified client for all services |
createSophiaClient() |
Factory for production client |
createLocalSophiaClient() |
Factory for local development |
SearchClient |
Individual search service client |
SearchQueryBuilder |
Fluent API for building search queries |
IngestionClient |
Individual ingestion service client |
KnowledgeGraphClient |
Individual knowledge graph client |
GraphQueryBuilder |
Fluent API for graph traversal queries |
HttpClient |
Low-level HTTP transport |
SophiaClientError |
Typed error class with error codes |
Branded ID Types:
DocumentId, ChunkId, QueryId, ResultId, CitationId, JobId,
SourceId, NodeId, RelationshipId - All with createXId() factory
functions for type-safe identifier construction.
API Type Coverage:
- Search:
SearchRequest,SearchResponse,RAGRequest,RAGResponse,Citation,CitationOptions - Ingestion:
IngestFileRequest,IngestUrlRequest,IngestionJob,DocumentMetadata,IngestedDocument - Knowledge Graph:
GraphNode,GraphRelationship,CreateNodeRequest,TraversalOptions,PathFindingOptions,SubgraphOptions,EntityResolutionRequest
@sophia/database - Prisma Schema and Client#
Database access layer providing Prisma client for Sophia's PostgreSQL schema.
Exports:
getSophiaClient()- Singleton Prisma client instancedisconnectSophiaClient()- Connection cleanupcreateSophiaClient()- Fresh client instance factory
Connection: SOPHIA_DATABASE_URL environment variable, schema namespace:
sophia
@sophia/schemas - Zod Validation Schemas#
Domain validation schemas with branded ID types for all Sophia entities.
Branded IDs with Zod Schemas:
DocumentId, CitationId, ClaimId, KnowledgePackId, EntityId,
RelationId, ChunkId, ProjectId, UserId, SourceId, IndexId
Each ID type has:
- A branded TypeScript type
- A Zod schema (e.g.,
DocumentIdSchema) - A factory function (e.g.,
documentId())
Domain Schemas: Research documents, citations, claims, knowledge packs, entities, and relations - all with Zod validation.
Retrieval and Indexing Libraries#
@sophia/indexing - Document Chunking and Embedding#
Comprehensive indexing library covering chunking, embedding generation, entity resolution, and graph visualization.
Chunking Module:
| Chunker | Strategy |
|---|---|
FixedSizeChunker |
Fixed character/token window |
SlidingWindowChunker |
Overlapping sliding window |
ParagraphChunker |
Paragraph boundary detection |
SentenceChunker |
Sentence boundary detection |
RecursiveChunker |
Hierarchical splitting with configurable separators |
MarkdownChunker |
Markdown structure-aware splitting |
CodeChunker |
Programming language-aware splitting |
SemanticChunker |
Embedding-based semantic boundaries |
Factory: createChunker(strategy) instantiates the appropriate chunker.
Predefined separator sets: RECURSIVE_SEPARATORS, MARKDOWN_SEPARATORS,
CODE_SEPARATORS.
Embedding Module:
| Provider | Backend |
|---|---|
OpenAIEmbeddingProvider |
OpenAI text-embedding-3 models |
CohereEmbeddingProvider |
Cohere embed-v3 models |
LocalEmbeddingProvider |
Local/custom embedding models |
EmbeddingService wraps providers with caching (EmbeddingCache) and batch
processing. Factory: createEmbeddingServiceFromEnv() auto-configures from
environment.
Index Module:
IndexBuilder- Orchestrates chunking, embedding, and storageInMemoryVectorStore- Development vector store- Vector math utilities:
cosineSimilarity,dotProduct,euclideanDistance,manhattanDistance,normalize,magnitude,centroid,reduceDimensions findTopK()- Top-K nearest neighbor selectionpairwiseSimilarity()- Matrix similarity computation
Graph Module:
EntityResolver/resolveEntities()/findDuplicates()- Entity deduplication and mergingGraphExporter- Export graphs in multiple formats:exportToD3Format(),exportToDOTFormat(),exportToGEXFFormat()computeGraphStatistics()/computeNodeCentrality()- Graph analytics
@sophia/vectordb - Vector Database Integration#
Unified interface across four vector storage backends.
Stores:
| Store | Backend | Factory |
|---|---|---|
MemoryVectorStore |
In-memory | createMemoryStore() |
MilvusVectorStore |
Milvus | createMilvusStore() / createMilvusStoreFromEnv() |
QdrantVectorStore |
Qdrant | createQdrantStore() / createQdrantStoreFromEnv() |
PineconeVectorStore |
Pinecone | createPineconeStore() / createPineconeStoreFromEnv() |
Predefined Collection Schemas:
| Schema | Purpose |
|---|---|
KNOWLEDGE_CHUNKS_SCHEMA |
Document chunk embeddings |
RESEARCH_DOCUMENTS_SCHEMA |
Full document embeddings |
CITATIONS_SCHEMA |
Citation embeddings |
ENTITIES_SCHEMA |
Entity embeddings |
WISDOM_EMBEDDINGS_SCHEMA |
Wisdom/philosophy content |
CONTENT_EMBEDDINGS_SCHEMA |
General content |
USER_PREFERENCE_EMBEDDINGS_SCHEMA |
User preference vectors |
CONVERSATION_EMBEDDINGS_SCHEMA |
Conversation context |
SEMANTIC_SEARCH_SCHEMA |
General semantic search |
PERSONA_EMBEDDINGS_SCHEMA |
Persona/character vectors |
Search Capabilities:
search()- Pure vector similarity search with metadata filtershybridSearch()- Combined vector + keyword search with configurable weights and rerank strategy (rrfreciprocal rank fusion orweightedCombination)
Utilities:
- Similarity:
cosineSimilarity,dotProduct,euclideanDistance - Fusion:
reciprocalRankFusion,weightedCombination,textSimilarity - Batch processing:
batchArray,processBatchesSequentially,processBatchesConcurrently,processBatchesWithProgress - Resilience:
withRetrywith configurableRetryConfig - Health:
checkVectorStoreHealth,checkMultipleStoresHealth,getAggregateHealthStatus
Research and Analysis Libraries#
@sophia/evaluation - Retrieval Quality Assessment#
Quality metrics and benchmark management for search accuracy evaluation.
Components:
| Component | Purpose |
|---|---|
RetrievalEvaluator |
Faithfulness, relevancy, precision/recall metrics |
CitationChecker |
Quote accuracy, entailment, attribution verification |
GroundednessScorer |
Semantic and lexical factual accuracy scoring |
BenchmarkRunner |
Built-in and custom benchmark execution |
@sophia/theory - Research Methodology Framework#
Research project management and citation formatting.
Components:
| Component | Purpose |
|---|---|
CitationManager |
Claim detection, citation formatting (Chicago, MLA, APA, Harvard, Turabian) |
ResearchManager |
Research project lifecycle and hypothesis tracking |
@sophia/cultural-research - Cultural Context Research#
Cultural sensitivity and localization for research content.
Components:
| Component | Purpose |
|---|---|
CulturalConsultant |
Content sensitivity checking against cultural profiles |
LocalizationManager |
Text translation and locale management |
AccessibilityManager |
WCAG accessibility checking (AA/AAA) |
CulturalDatabase |
Cultural profile and reference storage |
@sophia/predictions - Predictive Analytics#
Forecasting and optimization models for content platforms.
Capabilities:
- Time series forecasting (revenue, viewership)
- Schedule optimization (streaming time slots)
- Content performance prediction
- Seasonal factor and peak period analysis
Infrastructure Libraries#
@sophia/event-publisher - Cross-Domain Events#
Typed event publisher for Sophia domain events via @oshun/event-bus.
Event Types:
| Event | Trigger |
|---|---|
SophiaDocumentIngestedPayload |
Document ingestion completed |
SophiaDocumentUpdatedPayload |
Document metadata updated |
SophiaDocumentDeletedPayload |
Document removed |
SophiaIndexUpdatedPayload |
Vector index updated |
SophiaIndexRebuiltPayload |
Full index rebuild completed |
SophiaSearchPerformedPayload |
Search query executed |
SophiaEntityExtractedPayload |
Entity extraction completed |
SophiaRelationDiscoveredPayload |
Relationship discovered |
SophiaCitationCreatedPayload |
Citation record created |
Data Flow Diagrams#
Search Pipeline#
User Query
|
v
+------------------+
| SearchAPIService |
| (search-api) |
+--------+---------+
|
v
+------------------+ +--------------------+
| RetrievalService |---->| EmbeddingProvider |
| (hybrid search) | | (OpenAI/Cohere/ |
+--------+---------+ | Local) |
| +--------------------+
|
+----+----+
| |
v v
+-------+ +----------+
|Vector | |Keyword |
|Search | |Search |
|(topK) | |(BM25) |
+---+---+ +----+-----+
| |
+----+-----+
|
v
+------------------+ +------------------+
| CrossEncoder | | CitationService |
| Reranker | | (if RAG mode) |
+--------+---------+ +--------+---------+
| |
v v
+------------------+ +------------------+
| SearchResponse | | RAGResponse |
| {results, total, | | {answer, cites, |
| processingTime} | | sources, time} |
+------------------+ +------------------+
Document Ingestion Pipeline#
Source (File/URL)
|
v
+---------------------+
| IngestionPipeline |
| (ingestion app) |
+----------+----------+
|
v
+---------------------+ Events: job:created
| Source Detection |---------> job:started
| (File vs URL) |
+----------+----------+
|
+-----+-----+
| |
v v
+----------+ +-----------+
|FileSource| |URL Source |
| | |(fetch & |
| | | extract) |
+----+-----+ +-----+-----+
| |
+------+-------+
|
v
+---------------------+ Events: stage:started
| Text Extraction |---------> stage:completed
+----------+----------+
|
v
+---------------------+
| Chunking |
| (8 strategies: |
| fixed, sliding, |
| paragraph, sent, |
| recursive, md, |
| code, semantic) |
+----------+----------+
|
v
+---------------------+
| Enrichment |
| - Entity Extraction |
| - Summary Generation|
| - Key Points |
+----------+----------+
|
v
+---------------------+
| Embedding |
| (OpenAI/Cohere/ | Events: job:completed
| Local provider) |---------> or job:failed
+----------+----------+
|
v
+---------------------+
| Vector Store |
| (Qdrant/Pinecone/ |
| Milvus/Memory) |
+---------------------+
Knowledge Graph Operations#
+-------------------+
| KnowledgeGraph |
| Service |
+--------+----------+
|
+----+----+----+----+
| | |
v v v
+--------+ +--------+ +--------+
|Entity | |Relation| |Graph |
|Service | |Service | |Service |
+---+----+ +---+----+ +---+----+
| | |
| | +----+----+----+
| | | | |
v v v v v
+---------------------------------------+
| CombinedMemoryStore |
| (extends MemoryGraphStore) |
| |
| Nodes: Map<NodeId, GraphNode> |
| Relationships: Map<string, Rel> |
+-------------------+-------------------+
|
v
+------------------+
| EntityResolver |
| (optional) |
| |
| - Duplicate |
| Detection |
| - Entity Merging |
| - Canonical ID |
| Resolution |
+------------------+
Graph Traversal Operations:
- getNeighbors(nodeId, options)
- traverse(options) -> BFS/DFS traversal
- findShortestPath(options) -> Dijkstra-like path finding
- findAllPaths(options) -> All paths between nodes
- extractSubgraph(options) -> Subgraph extraction
Cross-Domain Integration#
+------------------+
| @oshun/event-bus |
| (Redis Streams) |
+--------+---------+
|
+--------------------+--------------------+
| | |
v v v
+------------------+ +------------------+ +------------------+
| sophia.document | | sophia.entity | | sophia.search |
| .ingested | | .extracted | | .performed |
| .updated | | sophia.relation | +------------------+
| .deleted | | .discovered |
+--------+---------+ +--------+---------+
| |
v v
+------------------+ +------------------+
| Consumers: | | Consumers: |
| - Hathor (lore | | - Isis (context |
| enrichment) | | for generation)|
| - Bellona (asset | | - Hathor (world |
| metadata) | | knowledge) |
+------------------+ +------------------+
Events Published by Sophia:
sophia.document.ingested -> Document available for downstream use
sophia.document.updated -> Document metadata changed
sophia.document.deleted -> Document removed from index
sophia.index.updated -> Search index has new data
sophia.index.rebuilt -> Full index rebuild complete
sophia.search.performed -> Analytics tracking
sophia.entity.extracted -> New entity available in graph
sophia.relation.discovered -> New relationship in graph
sophia.citation.created -> Citation record available
Database Schema#
PostgreSQL (Prisma ORM)#
Schema namespace: sophia. Connection: SOPHIA_DATABASE_URL. Preview features:
fullTextSearch, fullTextIndex.
Models#
| Model | Table | Purpose |
|---|---|---|
Document |
documents |
Research document records |
DocumentChunk |
document_chunks |
Indexed text chunks with embeddings |
Citation |
citations |
Citation records with verification |
Entity |
entities |
Knowledge graph nodes |
EntityMention |
entity_mentions |
Entity occurrences in documents |
Relation |
relations |
Knowledge graph edges |
KnowledgePack |
knowledge_packs |
Curated knowledge collections |
KnowledgePackDocument |
knowledge_pack_documents |
Pack-document associations |
KnowledgePackEntity |
knowledge_pack_entities |
Pack-entity associations |
KnowledgePackStar |
knowledge_pack_stars |
User favorites |
IngestionJob |
ingestion_jobs |
Ingestion pipeline tracking |
AuditLog |
audit_log |
Change tracking |
Document Model#
The Document model is the central entity with comprehensive metadata:
| Field Group | Fields |
|---|---|
| Identity | id (CUID), title, originalTitle, type, genre, status |
| Authorship | author, authors (JSON array), editor, translator |
| Publication | publisher, publicationPlace, publicationYear, edition, volume, issue, pages |
| Identifiers | isbn, issn, doi, url, externalIds (JSON) |
| Language | language, languages[], originRegion |
| Content | abstract, summary, keyPoints (JSON), keywords[], subjects[], tags[] |
| Tradition | tradition, traditions[], category |
| Dates | compositionYear, compositionEra, dateRange (JSON) |
| Rights | rightsStatus, rightsHolder, license |
| File | sourceUri, sourceType, fileFormat, fileSize, checksum, mimeType |
| Processing | pageCount, wordCount, tokenCount, chunkCount, processingTimeMs, processingError |
| Quality | qualityScore, completeness, accuracy |
| Ownership | ownerId, organizationId, projectId |
| Search | searchVector (tsvector for full-text search) |
Unique constraint: (ownerId, title, author)
DocumentChunk Model#
Indexed text units with embedding storage:
- Position tracking:
index,startOffset,endOffset,pageNumber,sectionTitle,paragraphIndex - Embedding:
embeddingModel,embeddingDimension,embeddingVector(binary),embeddingGeneratedAt - Overlap linking:
overlapBefore,overlapAfter,previousChunkId,nextChunkId - Context: JSON field with
{before, after, documentTitle, chapterTitle}
Entity Model (Knowledge Graph Nodes)#
Entities represent concepts, figures, traditions, and other knowledge units:
- 15 entity types:
TRADITION,TEXT,CONCEPT,PRACTICE,FIGURE,PLACE,SCHOOL,DOCTRINE,EVENT,TERM,SYMBOL,RITUAL,ARTIFACT,ORGANIZATION,CUSTOM - 5 confidence levels:
CERTAIN,PROBABLE,POSSIBLE,UNCERTAIN,INFERRED - Merge tracking:
canonicalId,mergedInto,mergedAtfor entity resolution - Statistics:
mentionCount,relationCount,referenceCount - Full-text search via
searchVector(tsvector)
Relation Model (Knowledge Graph Edges)#
Relationships between entities with evidence tracking:
- 23 relation types across 8 categories:
- Teaching:
TAUGHT,STUDIED_WITH,INFLUENCED - Authorship:
WROTE,EDITED,TRANSLATED - Membership:
FOUNDED,MEMBER_OF,BELONGS_TO - Location:
LIVED_IN,BORN_IN,DIED_IN,LOCATED_IN - Temporal:
PRECEDED,SUCCEEDED,CONTEMPORARY_WITH - Reference:
REFERENCES,QUOTES,CITES - Conceptual:
RELATED_TO,SIMILAR_TO,OPPOSITE_OF,DERIVED_FROM,PART_OF - Practice:
PRACTICES,PRESCRIBES
- Teaching:
- Evidence:
evidenceDocumentId,evidenceChunkId,evidenceText,evidenceType - Spatial/temporal context:
temporalContext,spatialContext - Unique constraint:
(sourceId, targetId, type)
Citation Model#
Citation records with multi-style formatting and verification:
- 6 citation styles:
CHICAGO,MLA,APA,HARVARD,TURABIAN,CUSTOM - 5 verification statuses:
VERIFIED,REVIEWED,UNVERIFIED,DISPUTED,PENDING - Claim tracking:
claimText,claimType - Note support:
noteType,noteMarker,noteContent(footnote/endnote)
KnowledgePack Model#
Curated knowledge collections with versioning and community features:
- 8 categories:
TRADITION,PERIOD,TOPIC,FIGURE,TEXT,REGION,PRACTICE,CUSTOM - 4 visibility levels:
PRIVATE,TEAM,ORGANIZATION,PUBLIC - Statistics:
documentCount,entityCount,relationCount,citationCount,chunkCount,totalTokens - Usage tracking:
viewCount,downloadCount,queryCount,starCount - Quality metrics:
qualityTier,qualityScore,completeness - Versioning:
version,previousVersionId
IngestionJob Model#
Pipeline job tracking with multi-stage progress:
- 10 statuses:
PENDING,QUEUED,FETCHING,EXTRACTING,ENRICHING,CHUNKING,INDEXING,COMPLETED,FAILED,CANCELLED - 6 source types:
FILE,URL,API,DATABASE,STREAM,ARCHIVE - Configuration:
config,extractionConfig,enrichmentConfig,chunkingConfig(all JSON) - Results:
chunksCreated,tokensProcessed,entitiesExtracted,relationsExtracted - Error tracking:
errorCode,errorMessage,errorStage,errorDetails - Queue management:
priority,workerId,queueName,retryCount,maxRetries(default 3)
Enums Summary#
| Enum | Values |
|---|---|
DocumentType |
BOOK, ARTICLE, THESIS, MANUSCRIPT, SCRIPTURE, COMMENTARY, ESSAY, LETTER, SPEECH, TRANSLATION, ANTHOLOGY, ENCYCLOPEDIA, DICTIONARY, OTHER |
DocumentStatus |
DRAFT, INGESTING, EXTRACTING, ENRICHING, CHUNKING, INDEXING, PUBLISHED, ARCHIVED, FAILED, DELETED |
DocumentGenre |
RELIGIOUS, PHILOSOPHICAL, HISTORICAL, SCHOLARLY, PRACTICAL, DEVOTIONAL, COMMENTARY, NARRATIVE, POETRY, LITURGICAL, LEGAL, BIOGRAPHICAL, INSTRUCTIONAL, OTHER |
EntityType |
TRADITION, TEXT, CONCEPT, PRACTICE, FIGURE, PLACE, SCHOOL, DOCTRINE, EVENT, TERM, SYMBOL, RITUAL, ARTIFACT, ORGANIZATION, CUSTOM |
RelationType |
TAUGHT, STUDIED_WITH, INFLUENCED, WROTE, EDITED, TRANSLATED, FOUNDED, MEMBER_OF, BELONGS_TO, LIVED_IN, BORN_IN, DIED_IN, LOCATED_IN, PRECEDED, SUCCEEDED, CONTEMPORARY_WITH, REFERENCES, QUOTES, CITES, RELATED_TO, SIMILAR_TO, OPPOSITE_OF, DERIVED_FROM, PART_OF, PRACTICES, PRESCRIBES, CUSTOM |
IngestionJobStatus |
PENDING, QUEUED, FETCHING, EXTRACTING, ENRICHING, CHUNKING, INDEXING, COMPLETED, FAILED, CANCELLED |
KnowledgePackCategory |
TRADITION, PERIOD, TOPIC, FIGURE, TEXT, REGION, PRACTICE, CUSTOM |
KnowledgePackVisibility |
PRIVATE, TEAM, ORGANIZATION, PUBLIC |
API Endpoints#
Search API (Port 3000)#
Search#
| Method | Path | Description |
|---|---|---|
| POST | /v1/search |
Semantic search with hybrid strategy |
Request:
interface SearchQuery {
text: string;
topK?: number; // Max: 100
strategy?: 'hybrid' | 'vector' | 'keyword';
filters?: SearchFilters;
}
Response:
interface SearchResponse {
results: SearchResult[];
totalResults: number;
processingTimeMs: number;
}
RAG#
| Method | Path | Description |
|---|---|---|
| POST | /v1/rag |
Retrieval-augmented generation |
Request:
interface RAGRequest {
query: string;
topK?: number;
strategy?: 'hybrid' | 'vector' | 'keyword';
includeCitations?: boolean;
citationOptions?: CitationOptions;
}
Response:
interface RAGResponse {
queryId: QueryId;
query: string;
answer: string;
citations: Citation[];
sources: SearchResult[];
processingTimeMs: number;
metadata: { retrievalStrategy: string };
}
Health#
| Method | Path | Description |
|---|---|---|
| GET | /health |
Service health |
Documents (Port 4003)#
| Method | Path | Description |
|---|---|---|
| POST | /v1/documents |
Ingest a document |
| GET | /v1/documents/:id |
Get document metadata |
| DELETE | /v1/documents/:id |
Delete a document |
Collections#
| Method | Path | Description |
|---|---|---|
| GET | /v1/collections |
List collections |
| POST | /v1/collections |
Create a collection |
Embeddings#
| Method | Path | Description |
|---|---|---|
| POST | /v1/embeddings |
Generate embeddings |
Knowledge Graph (Port 3001)#
Entities#
| Operation | Description |
|---|---|
| Create | createEntity(request) - Create node with type and properties |
| Get | getEntity(id) - Retrieve node by ID |
| Query | queryEntities(request) - Filter and paginate nodes |
| Delete | deleteEntity(id) - Remove node |
Relationships#
| Operation | Description |
|---|---|
| Create | createRelationship(request) - Create typed edge between nodes |
| Get | getRelationship(id) - Retrieve edge by ID |
| ForEntity | getRelationshipsForEntity(nodeId) - All edges for a node |
| Delete | deleteRelationship(id) - Remove edge |
Graph Traversal#
| Operation | Description |
|---|---|
| Neighbors | getNeighbors(nodeId, options) - Direct connections |
| Traverse | traverse(options) - BFS/DFS traversal with depth limits |
| Shortest Path | findShortestPath(options) - Weighted shortest path |
| All Paths | findAllPaths(options) - All paths up to max depth |
| Subgraph | extractSubgraph(options) - Connected subgraph extraction |
Entity Resolution#
| Operation | Description |
|---|---|
| Resolve | resolveEntity(request) - Match entity against existing |
| Find Duplicates | findDuplicates(threshold) - Detect duplicate entities |
Technology Stack#
| Layer | Technologies |
|---|---|
| Framework | Hono (planned), currently standalone service classes |
| Database | PostgreSQL with Prisma ORM (schema: sophia) |
| Vector Stores | Qdrant, Pinecone, Milvus, In-memory |
| Embeddings | OpenAI text-embedding-3, Cohere embed-v3, Local |
| Chunking | 8 strategies (fixed, sliding, paragraph, sentence, recursive, markdown, code, semantic) |
| Graph Store | In-memory (MemoryGraphStore), extensible interface |
| Events | @oshun/event-bus (Redis Streams) |
| Validation | Zod schemas with branded types |
| Logging | @oshun/logging (Pino-based structured logging) |
| Build | @nx/js:tsc |
| Testing | Vitest |
| UI | React (Workbench app) |
Deployment Architecture#
Service Topology#
+------------------+
| Load Balancer |
+--------+---------+
|
+------------+------------+
| | |
v v v
+----------+ +----------+ +----------+
|Search API| |Ingestion | |Knowledge |
|(:3000) | |Pipeline | |Graph |
| | | | |(:3001) |
+----+-----+ +----+-----+ +----+-----+
| | |
+----+--------------+--------------+----+
| |
+----+----+ +----+----+
|PostgreSQL| | Redis |
|(sophia) | | (cache, |
| | | queue) |
+----+-----+ +---------+
|
+----+----+----+----+
| | |
v v v
+------+ +-------+ +-------+
|Qdrant| |Milvus | |Pinecone|
|(self-| |(self- | |(cloud) |
|hosted)| |hosted)| |
+------+ +-------+ +-------+
ECS Deployment#
Sophia services deploy via the shared deploy-ecs.yml pipeline:
- search-api - ECS service with auto-scaling based on request volume
- ingestion - ECS task for batch processing, triggered by queue depth
- knowledge-graph - ECS service for graph operations
- workbench - Static deployment (S3 + CloudFront)
Database Migrations#
Managed via the shared database.yml CI pipeline with Prisma migrations
targeting the sophia PostgreSQL database.
Cross-Domain Dependencies#
Sophia Consumes From#
| Source Domain | Event/Integration | Purpose |
|---|---|---|
| Hathor | World/lore content | Worldbuilding knowledge ingestion |
| Isis | Generated content metadata | AI output indexing and search |
Sophia Publishes To#
| Target Domain | Event | Purpose |
|---|---|---|
| Hathor | sophia.document.ingested |
Lore enrichment with research |
| Bellona | sophia.document.updated |
Asset metadata updates |
| Isis | sophia.entity.extracted |
Context for AI generation |
| All | sophia.search.performed |
Analytics and usage tracking |
Shared Infrastructure#
| Dependency | Usage |
|---|---|
@oshun/logging |
Structured logging across all services |
@oshun/event-bus |
Cross-domain event publishing (Redis Streams) |
| PostgreSQL | Primary data store (schema: sophia) |
| Redis | Caching, queue management, event streaming |
Environment Variables#
# Database
SOPHIA_DATABASE_URL=postgresql://oshun:oshun_dev@localhost:5432/sophia
# Redis
REDIS_URL=redis://localhost:6379
# Vector Stores
QDRANT_URL=http://localhost:6333
PINECONE_API_KEY=...
MILVUS_HOST=localhost
MILVUS_PORT=19530
# Embedding Providers
OPENAI_API_KEY=...
COHERE_API_KEY=...
# Service Ports
PORT=3000 # Search API
INGESTION_PORT=3000 # Ingestion service
HOST=localhost
# Ingestion Config
MAX_CONCURRENT_JOBS=5
DEFAULT_CHUNK_SIZE=1000
DEFAULT_CHUNK_OVERLAP=200
LOG_LEVEL=info
Development#
# Run the search API
pnpm nx dev @sophia/search-api
# Run tests
pnpm nx test @sophia/client
pnpm nx test @sophia/indexing
pnpm nx test @sophia/vectordb
# Build all Sophia libraries
pnpm nx run-many --target=build --projects=tag:scope:sophia
# Start vector database for development
docker compose -f docker/docker-compose.dev.yml --profile vectors up -d
# Run database migrations
pnpm nx run @sophia/database:prisma:migrate
pnpm nx run @sophia/database:prisma:generate