Eve library

Sophia Domain - Architecture

Guide

Revision 1

On this page

Canonical source: the canonical architecture document for this domain is deep-dive/architecture.md, rendered in the docs center. This file is its operational companion (service topology, APIs, schemas, runtime detail) — reconciled by linking, not duplicated (§2, §13).

Comprehensive technical architecture for the Sophia Knowledge Management and Research Platform


System Overview#

Sophia is the knowledge management and research platform within the Oshun monorepo. It provides RAG (Retrieval-Augmented Generation) infrastructure for document ingestion, semantic search, citation management, knowledge graph construction, and retrieval quality evaluation. The domain is organized around four applications and ten libraries that together form a complete research knowledge pipeline.

Core Responsibilities#

  • Document Ingestion - Multi-format document intake with configurable chunking strategies, entity extraction, and summary generation
  • Semantic Search - Hybrid search combining vector similarity and keyword matching with cross-encoder reranking
  • RAG Pipeline - Retrieval-augmented generation with citation tracking and groundedness scoring
  • Knowledge Graph - Entity extraction, relationship mapping, entity resolution, and graph traversal
  • Evaluation - Retrieval quality metrics, citation integrity checking, and benchmark management
  • Research Management - Research project tracking, hypothesis management, and citation formatting

Service Architecture#

Applications#

1. Search API (apps/sophia/search-api)#

The primary search service providing semantic search with RAG context generation.

Property Value
Default Port 3000
Entry Point src/main.ts
Service SearchAPIService
Dependencies @oshun/logging
Internal RetrievalService, CitationService, CrossEncoderReranker

Key Configuration:

typescript
interface SearchAPIConfig {
  port?: number; // Default: 3000
  host?: string; // Default: 'localhost'
  defaultTopK?: number; // Default: 10
  maxTopK?: number; // Default: 100
  enableCitations?: boolean; // Default: true
  enableReranking?: boolean; // Default: true
}

Core Operations:

  • search(query: SearchQuery) - Execute a search with hybrid strategy (vector + keyword), applies topK limits, delegates to RetrievalService
  • rag(request: RAGRequest) - Full RAG pipeline: retrieves relevant documents, generates answer via LLM, extracts citations via CitationService
  • healthCheck() - Service health and uptime reporting

Request Handlers Factory:

createRequestHandlers(service) produces framework-agnostic handlers for search, rag, and health endpoints with error wrapping via SearchError.

2. Ingestion Pipeline (apps/sophia/ingestion)#

Document ingestion service that processes files and URLs through a multi-stage pipeline.

Property Value
Entry Point src/main.ts
Service Pipeline via createIngestionPipeline
Dependencies @oshun/logging
Modes CLI (document argument) or Service

Configuration:

typescript
interface ServiceConfig {
  port: number; // Default: 3000 (INGESTION_PORT)
  maxConcurrentJobs: number; // Default: 5 (MAX_CONCURRENT_JOBS)
  defaultChunkSize: number; // Default: 1000 (DEFAULT_CHUNK_SIZE)
  defaultChunkOverlap: number; // Default: 200 (DEFAULT_CHUNK_OVERLAP)
  logLevel: LogLevel; // Default: 'info' (LOG_LEVEL)
}

Pipeline Stages:

  1. Source Detection - Auto-detects URL vs file sources via createURLSource() / createFileSource()
  2. Document Processing - Invokes pipeline.processDocument() with configurable enrichment
  3. Entity Extraction - Extracts entities from document content
  4. Summary Generation - Produces short summary and key points

Event System:

The pipeline emits typed events for monitoring:

  • job:created, job:started, job:completed, job:failed
  • stage:started, stage:completed, stage:failed

3. Knowledge Graph (apps/sophia/knowledge-graph)#

Standalone knowledge graph service for entity and relationship management with graph traversal operations.

Property Value
Default Port 3001
Entry Point src/main.ts
Service KnowledgeGraphService
Dependencies @oshun/logging
Store CombinedMemoryStore (extends MemoryGraphStore)

Configuration:

typescript
interface KnowledgeGraphConfig {
  port?: number; // Default: 3001
  host?: string; // Default: 'localhost'
  maxQueryLimit?: number; // Default: 100
  maxTraversalDepth?: number; // Default: 5
  enableResolution?: boolean; // Default: true
}

Internal Services:

  • EntityService - CRUD for graph nodes with configurable max limits
  • RelationService - CRUD for graph edges with constraint validation
  • GraphService - Traversal, path-finding, and subgraph extraction with configurable limits:
    • MAX_HOPS: 5, MAX_PATH_DEPTH: 10, MAX_RESULTS: 100
    • MAX_NEIGHBORS: 50, MAX_TRAVERSAL_NODES: 1000, MAX_PATHS: 50
    • TRAVERSAL_TIMEOUT_MS: 30000
  • EntityResolver - Duplicate detection and entity merging (optional, controlled by enableResolution)

CombinedMemoryStore:

Extends MemoryGraphStore with relationship storage, providing:

  • In-memory relationship CRUD with auto-incrementing IDs
  • Neighbor discovery by scanning relationships for source/target matches
  • Filtered relationship queries with pagination
  • Combined node + relationship statistics

4. Workbench (apps/sophia/workbench)#

Web-based UI for document management, search testing, and knowledge curation.

Property Value
Entry Point src/
Type Web application (React-based)
Pages Dashboard, Curation, Review, Annotations, Documents, Search

API Services Layer:

The workbench connects to Sophia's backend via typed service clients:

Service Purpose
AuthService Authentication and session management
DashboardService Dashboard data and statistics
DocumentsService Document CRUD and management
AnnotationsService Document annotation management
SearchService Search queries and result display
ReviewService Content review workflows

All services follow a singleton pattern with getXService() factory and resetXService() for testing.


Library Architecture#

Core Libraries#

@sophia/client - TypeScript SDK#

Unified client SDK providing type-safe access to all Sophia services.

Exports:

Export Purpose
SophiaClient Unified client for all services
createSophiaClient() Factory for production client
createLocalSophiaClient() Factory for local development
SearchClient Individual search service client
SearchQueryBuilder Fluent API for building search queries
IngestionClient Individual ingestion service client
KnowledgeGraphClient Individual knowledge graph client
GraphQueryBuilder Fluent API for graph traversal queries
HttpClient Low-level HTTP transport
SophiaClientError Typed error class with error codes

Branded ID Types:

DocumentId, ChunkId, QueryId, ResultId, CitationId, JobId, SourceId, NodeId, RelationshipId - All with createXId() factory functions for type-safe identifier construction.

API Type Coverage:

  • Search: SearchRequest, SearchResponse, RAGRequest, RAGResponse, Citation, CitationOptions
  • Ingestion: IngestFileRequest, IngestUrlRequest, IngestionJob, DocumentMetadata, IngestedDocument
  • Knowledge Graph: GraphNode, GraphRelationship, CreateNodeRequest, TraversalOptions, PathFindingOptions, SubgraphOptions, EntityResolutionRequest

@sophia/database - Prisma Schema and Client#

Database access layer providing Prisma client for Sophia's PostgreSQL schema.

Exports:

  • getSophiaClient() - Singleton Prisma client instance
  • disconnectSophiaClient() - Connection cleanup
  • createSophiaClient() - Fresh client instance factory

Connection: SOPHIA_DATABASE_URL environment variable, schema namespace: sophia

@sophia/schemas - Zod Validation Schemas#

Domain validation schemas with branded ID types for all Sophia entities.

Branded IDs with Zod Schemas:

DocumentId, CitationId, ClaimId, KnowledgePackId, EntityId, RelationId, ChunkId, ProjectId, UserId, SourceId, IndexId

Each ID type has:

  • A branded TypeScript type
  • A Zod schema (e.g., DocumentIdSchema)
  • A factory function (e.g., documentId())

Domain Schemas: Research documents, citations, claims, knowledge packs, entities, and relations - all with Zod validation.

Retrieval and Indexing Libraries#

@sophia/indexing - Document Chunking and Embedding#

Comprehensive indexing library covering chunking, embedding generation, entity resolution, and graph visualization.

Chunking Module:

Chunker Strategy
FixedSizeChunker Fixed character/token window
SlidingWindowChunker Overlapping sliding window
ParagraphChunker Paragraph boundary detection
SentenceChunker Sentence boundary detection
RecursiveChunker Hierarchical splitting with configurable separators
MarkdownChunker Markdown structure-aware splitting
CodeChunker Programming language-aware splitting
SemanticChunker Embedding-based semantic boundaries

Factory: createChunker(strategy) instantiates the appropriate chunker. Predefined separator sets: RECURSIVE_SEPARATORS, MARKDOWN_SEPARATORS, CODE_SEPARATORS.

Embedding Module:

Provider Backend
OpenAIEmbeddingProvider OpenAI text-embedding-3 models
CohereEmbeddingProvider Cohere embed-v3 models
LocalEmbeddingProvider Local/custom embedding models

EmbeddingService wraps providers with caching (EmbeddingCache) and batch processing. Factory: createEmbeddingServiceFromEnv() auto-configures from environment.

Index Module:

  • IndexBuilder - Orchestrates chunking, embedding, and storage
  • InMemoryVectorStore - Development vector store
  • Vector math utilities: cosineSimilarity, dotProduct, euclideanDistance, manhattanDistance, normalize, magnitude, centroid, reduceDimensions
  • findTopK() - Top-K nearest neighbor selection
  • pairwiseSimilarity() - Matrix similarity computation

Graph Module:

  • EntityResolver / resolveEntities() / findDuplicates() - Entity deduplication and merging
  • GraphExporter - Export graphs in multiple formats: exportToD3Format(), exportToDOTFormat(), exportToGEXFFormat()
  • computeGraphStatistics() / computeNodeCentrality() - Graph analytics

@sophia/vectordb - Vector Database Integration#

Unified interface across four vector storage backends.

Stores:

Store Backend Factory
MemoryVectorStore In-memory createMemoryStore()
MilvusVectorStore Milvus createMilvusStore() / createMilvusStoreFromEnv()
QdrantVectorStore Qdrant createQdrantStore() / createQdrantStoreFromEnv()
PineconeVectorStore Pinecone createPineconeStore() / createPineconeStoreFromEnv()

Predefined Collection Schemas:

Schema Purpose
KNOWLEDGE_CHUNKS_SCHEMA Document chunk embeddings
RESEARCH_DOCUMENTS_SCHEMA Full document embeddings
CITATIONS_SCHEMA Citation embeddings
ENTITIES_SCHEMA Entity embeddings
WISDOM_EMBEDDINGS_SCHEMA Wisdom/philosophy content
CONTENT_EMBEDDINGS_SCHEMA General content
USER_PREFERENCE_EMBEDDINGS_SCHEMA User preference vectors
CONVERSATION_EMBEDDINGS_SCHEMA Conversation context
SEMANTIC_SEARCH_SCHEMA General semantic search
PERSONA_EMBEDDINGS_SCHEMA Persona/character vectors

Search Capabilities:

  • search() - Pure vector similarity search with metadata filters
  • hybridSearch() - Combined vector + keyword search with configurable weights and rerank strategy (rrf reciprocal rank fusion or weightedCombination)

Utilities:

  • Similarity: cosineSimilarity, dotProduct, euclideanDistance
  • Fusion: reciprocalRankFusion, weightedCombination, textSimilarity
  • Batch processing: batchArray, processBatchesSequentially, processBatchesConcurrently, processBatchesWithProgress
  • Resilience: withRetry with configurable RetryConfig
  • Health: checkVectorStoreHealth, checkMultipleStoresHealth, getAggregateHealthStatus

Research and Analysis Libraries#

@sophia/evaluation - Retrieval Quality Assessment#

Quality metrics and benchmark management for search accuracy evaluation.

Components:

Component Purpose
RetrievalEvaluator Faithfulness, relevancy, precision/recall metrics
CitationChecker Quote accuracy, entailment, attribution verification
GroundednessScorer Semantic and lexical factual accuracy scoring
BenchmarkRunner Built-in and custom benchmark execution

@sophia/theory - Research Methodology Framework#

Research project management and citation formatting.

Components:

Component Purpose
CitationManager Claim detection, citation formatting (Chicago, MLA, APA, Harvard, Turabian)
ResearchManager Research project lifecycle and hypothesis tracking

@sophia/cultural-research - Cultural Context Research#

Cultural sensitivity and localization for research content.

Components:

Component Purpose
CulturalConsultant Content sensitivity checking against cultural profiles
LocalizationManager Text translation and locale management
AccessibilityManager WCAG accessibility checking (AA/AAA)
CulturalDatabase Cultural profile and reference storage

@sophia/predictions - Predictive Analytics#

Forecasting and optimization models for content platforms.

Capabilities:

  • Time series forecasting (revenue, viewership)
  • Schedule optimization (streaming time slots)
  • Content performance prediction
  • Seasonal factor and peak period analysis

Infrastructure Libraries#

@sophia/event-publisher - Cross-Domain Events#

Typed event publisher for Sophia domain events via @oshun/event-bus.

Event Types:

Event Trigger
SophiaDocumentIngestedPayload Document ingestion completed
SophiaDocumentUpdatedPayload Document metadata updated
SophiaDocumentDeletedPayload Document removed
SophiaIndexUpdatedPayload Vector index updated
SophiaIndexRebuiltPayload Full index rebuild completed
SophiaSearchPerformedPayload Search query executed
SophiaEntityExtractedPayload Entity extraction completed
SophiaRelationDiscoveredPayload Relationship discovered
SophiaCitationCreatedPayload Citation record created

Data Flow Diagrams#

Search Pipeline#

text
 User Query
     |
     v
+------------------+
| SearchAPIService |
| (search-api)     |
+--------+---------+
         |
         v
+------------------+     +--------------------+
| RetrievalService |---->| EmbeddingProvider   |
| (hybrid search)  |     | (OpenAI/Cohere/     |
+--------+---------+     |  Local)             |
         |               +--------------------+
         |
    +----+----+
    |         |
    v         v
+-------+ +----------+
|Vector | |Keyword   |
|Search | |Search    |
|(topK) | |(BM25)    |
+---+---+ +----+-----+
    |          |
    +----+-----+
         |
         v
+------------------+     +------------------+
| CrossEncoder     |     | CitationService  |
| Reranker         |     | (if RAG mode)    |
+--------+---------+     +--------+---------+
         |                        |
         v                        v
+------------------+     +------------------+
| SearchResponse   |     | RAGResponse      |
| {results, total, |     | {answer, cites,  |
|  processingTime} |     |  sources, time}  |
+------------------+     +------------------+

Document Ingestion Pipeline#

text
 Source (File/URL)
     |
     v
+---------------------+
| IngestionPipeline   |
| (ingestion app)     |
+----------+----------+
           |
           v
+---------------------+  Events: job:created
| Source Detection     |---------> job:started
| (File vs URL)       |
+----------+----------+
           |
     +-----+-----+
     |           |
     v           v
+----------+ +-----------+
|FileSource| |URL Source  |
|          | |(fetch &    |
|          | | extract)   |
+----+-----+ +-----+-----+
     |              |
     +------+-------+
            |
            v
+---------------------+  Events: stage:started
| Text Extraction     |---------> stage:completed
+----------+----------+
           |
           v
+---------------------+
| Chunking            |
| (8 strategies:      |
|  fixed, sliding,    |
|  paragraph, sent,   |
|  recursive, md,     |
|  code, semantic)    |
+----------+----------+
           |
           v
+---------------------+
| Enrichment          |
| - Entity Extraction |
| - Summary Generation|
| - Key Points        |
+----------+----------+
           |
           v
+---------------------+
| Embedding           |
| (OpenAI/Cohere/     |  Events: job:completed
|  Local provider)    |---------> or job:failed
+----------+----------+
           |
           v
+---------------------+
| Vector Store        |
| (Qdrant/Pinecone/   |
|  Milvus/Memory)     |
+---------------------+

Knowledge Graph Operations#

text
+-------------------+
| KnowledgeGraph    |
| Service           |
+--------+----------+
         |
    +----+----+----+----+
    |         |         |
    v         v         v
+--------+ +--------+ +--------+
|Entity  | |Relation| |Graph   |
|Service | |Service | |Service |
+---+----+ +---+----+ +---+----+
    |          |           |
    |          |      +----+----+----+
    |          |      |         |    |
    v          v      v         v    v
+---------------------------------------+
| CombinedMemoryStore                   |
| (extends MemoryGraphStore)            |
|                                       |
| Nodes: Map<NodeId, GraphNode>         |
| Relationships: Map<string, Rel>       |
+-------------------+-------------------+
                    |
                    v
          +------------------+
          | EntityResolver   |
          | (optional)       |
          |                  |
          | - Duplicate      |
          |   Detection      |
          | - Entity Merging |
          | - Canonical ID   |
          |   Resolution     |
          +------------------+

Graph Traversal Operations:
  - getNeighbors(nodeId, options)
  - traverse(options)           -> BFS/DFS traversal
  - findShortestPath(options)   -> Dijkstra-like path finding
  - findAllPaths(options)       -> All paths between nodes
  - extractSubgraph(options)    -> Subgraph extraction

Cross-Domain Integration#

text
                        +------------------+
                        | @oshun/event-bus |
                        | (Redis Streams)  |
                        +--------+---------+
                                 |
            +--------------------+--------------------+
            |                    |                    |
            v                    v                    v
+------------------+  +------------------+  +------------------+
| sophia.document  |  | sophia.entity    |  | sophia.search    |
| .ingested        |  | .extracted       |  | .performed       |
| .updated         |  | sophia.relation  |  +------------------+
| .deleted         |  | .discovered      |
+--------+---------+  +--------+---------+
         |                     |
         v                     v
+------------------+  +------------------+
| Consumers:       |  | Consumers:       |
| - Hathor (lore   |  | - Isis (context  |
|   enrichment)    |  |   for generation)|
| - Bellona (asset |  | - Hathor (world  |
|   metadata)      |  |   knowledge)     |
+------------------+  +------------------+

Events Published by Sophia:
  sophia.document.ingested   -> Document available for downstream use
  sophia.document.updated    -> Document metadata changed
  sophia.document.deleted    -> Document removed from index
  sophia.index.updated       -> Search index has new data
  sophia.index.rebuilt       -> Full index rebuild complete
  sophia.search.performed    -> Analytics tracking
  sophia.entity.extracted    -> New entity available in graph
  sophia.relation.discovered -> New relationship in graph
  sophia.citation.created    -> Citation record available

Database Schema#

PostgreSQL (Prisma ORM)#

Schema namespace: sophia. Connection: SOPHIA_DATABASE_URL. Preview features: fullTextSearch, fullTextIndex.

Models#

Model Table Purpose
Document documents Research document records
DocumentChunk document_chunks Indexed text chunks with embeddings
Citation citations Citation records with verification
Entity entities Knowledge graph nodes
EntityMention entity_mentions Entity occurrences in documents
Relation relations Knowledge graph edges
KnowledgePack knowledge_packs Curated knowledge collections
KnowledgePackDocument knowledge_pack_documents Pack-document associations
KnowledgePackEntity knowledge_pack_entities Pack-entity associations
KnowledgePackStar knowledge_pack_stars User favorites
IngestionJob ingestion_jobs Ingestion pipeline tracking
AuditLog audit_log Change tracking

Document Model#

The Document model is the central entity with comprehensive metadata:

Field Group Fields
Identity id (CUID), title, originalTitle, type, genre, status
Authorship author, authors (JSON array), editor, translator
Publication publisher, publicationPlace, publicationYear, edition, volume, issue, pages
Identifiers isbn, issn, doi, url, externalIds (JSON)
Language language, languages[], originRegion
Content abstract, summary, keyPoints (JSON), keywords[], subjects[], tags[]
Tradition tradition, traditions[], category
Dates compositionYear, compositionEra, dateRange (JSON)
Rights rightsStatus, rightsHolder, license
File sourceUri, sourceType, fileFormat, fileSize, checksum, mimeType
Processing pageCount, wordCount, tokenCount, chunkCount, processingTimeMs, processingError
Quality qualityScore, completeness, accuracy
Ownership ownerId, organizationId, projectId
Search searchVector (tsvector for full-text search)

Unique constraint: (ownerId, title, author)

DocumentChunk Model#

Indexed text units with embedding storage:

  • Position tracking: index, startOffset, endOffset, pageNumber, sectionTitle, paragraphIndex
  • Embedding: embeddingModel, embeddingDimension, embeddingVector (binary), embeddingGeneratedAt
  • Overlap linking: overlapBefore, overlapAfter, previousChunkId, nextChunkId
  • Context: JSON field with {before, after, documentTitle, chapterTitle}

Entity Model (Knowledge Graph Nodes)#

Entities represent concepts, figures, traditions, and other knowledge units:

  • 15 entity types: TRADITION, TEXT, CONCEPT, PRACTICE, FIGURE, PLACE, SCHOOL, DOCTRINE, EVENT, TERM, SYMBOL, RITUAL, ARTIFACT, ORGANIZATION, CUSTOM
  • 5 confidence levels: CERTAIN, PROBABLE, POSSIBLE, UNCERTAIN, INFERRED
  • Merge tracking: canonicalId, mergedInto, mergedAt for entity resolution
  • Statistics: mentionCount, relationCount, referenceCount
  • Full-text search via searchVector (tsvector)

Relation Model (Knowledge Graph Edges)#

Relationships between entities with evidence tracking:

  • 23 relation types across 8 categories:
    • Teaching: TAUGHT, STUDIED_WITH, INFLUENCED
    • Authorship: WROTE, EDITED, TRANSLATED
    • Membership: FOUNDED, MEMBER_OF, BELONGS_TO
    • Location: LIVED_IN, BORN_IN, DIED_IN, LOCATED_IN
    • Temporal: PRECEDED, SUCCEEDED, CONTEMPORARY_WITH
    • Reference: REFERENCES, QUOTES, CITES
    • Conceptual: RELATED_TO, SIMILAR_TO, OPPOSITE_OF, DERIVED_FROM, PART_OF
    • Practice: PRACTICES, PRESCRIBES
  • Evidence: evidenceDocumentId, evidenceChunkId, evidenceText, evidenceType
  • Spatial/temporal context: temporalContext, spatialContext
  • Unique constraint: (sourceId, targetId, type)

Citation Model#

Citation records with multi-style formatting and verification:

  • 6 citation styles: CHICAGO, MLA, APA, HARVARD, TURABIAN, CUSTOM
  • 5 verification statuses: VERIFIED, REVIEWED, UNVERIFIED, DISPUTED, PENDING
  • Claim tracking: claimText, claimType
  • Note support: noteType, noteMarker, noteContent (footnote/endnote)

KnowledgePack Model#

Curated knowledge collections with versioning and community features:

  • 8 categories: TRADITION, PERIOD, TOPIC, FIGURE, TEXT, REGION, PRACTICE, CUSTOM
  • 4 visibility levels: PRIVATE, TEAM, ORGANIZATION, PUBLIC
  • Statistics: documentCount, entityCount, relationCount, citationCount, chunkCount, totalTokens
  • Usage tracking: viewCount, downloadCount, queryCount, starCount
  • Quality metrics: qualityTier, qualityScore, completeness
  • Versioning: version, previousVersionId

IngestionJob Model#

Pipeline job tracking with multi-stage progress:

  • 10 statuses: PENDING, QUEUED, FETCHING, EXTRACTING, ENRICHING, CHUNKING, INDEXING, COMPLETED, FAILED, CANCELLED
  • 6 source types: FILE, URL, API, DATABASE, STREAM, ARCHIVE
  • Configuration: config, extractionConfig, enrichmentConfig, chunkingConfig (all JSON)
  • Results: chunksCreated, tokensProcessed, entitiesExtracted, relationsExtracted
  • Error tracking: errorCode, errorMessage, errorStage, errorDetails
  • Queue management: priority, workerId, queueName, retryCount, maxRetries (default 3)

Enums Summary#

Enum Values
DocumentType BOOK, ARTICLE, THESIS, MANUSCRIPT, SCRIPTURE, COMMENTARY, ESSAY, LETTER, SPEECH, TRANSLATION, ANTHOLOGY, ENCYCLOPEDIA, DICTIONARY, OTHER
DocumentStatus DRAFT, INGESTING, EXTRACTING, ENRICHING, CHUNKING, INDEXING, PUBLISHED, ARCHIVED, FAILED, DELETED
DocumentGenre RELIGIOUS, PHILOSOPHICAL, HISTORICAL, SCHOLARLY, PRACTICAL, DEVOTIONAL, COMMENTARY, NARRATIVE, POETRY, LITURGICAL, LEGAL, BIOGRAPHICAL, INSTRUCTIONAL, OTHER
EntityType TRADITION, TEXT, CONCEPT, PRACTICE, FIGURE, PLACE, SCHOOL, DOCTRINE, EVENT, TERM, SYMBOL, RITUAL, ARTIFACT, ORGANIZATION, CUSTOM
RelationType TAUGHT, STUDIED_WITH, INFLUENCED, WROTE, EDITED, TRANSLATED, FOUNDED, MEMBER_OF, BELONGS_TO, LIVED_IN, BORN_IN, DIED_IN, LOCATED_IN, PRECEDED, SUCCEEDED, CONTEMPORARY_WITH, REFERENCES, QUOTES, CITES, RELATED_TO, SIMILAR_TO, OPPOSITE_OF, DERIVED_FROM, PART_OF, PRACTICES, PRESCRIBES, CUSTOM
IngestionJobStatus PENDING, QUEUED, FETCHING, EXTRACTING, ENRICHING, CHUNKING, INDEXING, COMPLETED, FAILED, CANCELLED
KnowledgePackCategory TRADITION, PERIOD, TOPIC, FIGURE, TEXT, REGION, PRACTICE, CUSTOM
KnowledgePackVisibility PRIVATE, TEAM, ORGANIZATION, PUBLIC

API Endpoints#

Search API (Port 3000)#

Method Path Description
POST /v1/search Semantic search with hybrid strategy

Request:

typescript
interface SearchQuery {
  text: string;
  topK?: number; // Max: 100
  strategy?: 'hybrid' | 'vector' | 'keyword';
  filters?: SearchFilters;
}

Response:

typescript
interface SearchResponse {
  results: SearchResult[];
  totalResults: number;
  processingTimeMs: number;
}

RAG#

Method Path Description
POST /v1/rag Retrieval-augmented generation

Request:

typescript
interface RAGRequest {
  query: string;
  topK?: number;
  strategy?: 'hybrid' | 'vector' | 'keyword';
  includeCitations?: boolean;
  citationOptions?: CitationOptions;
}

Response:

typescript
interface RAGResponse {
  queryId: QueryId;
  query: string;
  answer: string;
  citations: Citation[];
  sources: SearchResult[];
  processingTimeMs: number;
  metadata: { retrievalStrategy: string };
}

Health#

Method Path Description
GET /health Service health

Documents (Port 4003)#

Method Path Description
POST /v1/documents Ingest a document
GET /v1/documents/:id Get document metadata
DELETE /v1/documents/:id Delete a document

Collections#

Method Path Description
GET /v1/collections List collections
POST /v1/collections Create a collection

Embeddings#

Method Path Description
POST /v1/embeddings Generate embeddings

Knowledge Graph (Port 3001)#

Entities#

Operation Description
Create createEntity(request) - Create node with type and properties
Get getEntity(id) - Retrieve node by ID
Query queryEntities(request) - Filter and paginate nodes
Delete deleteEntity(id) - Remove node

Relationships#

Operation Description
Create createRelationship(request) - Create typed edge between nodes
Get getRelationship(id) - Retrieve edge by ID
ForEntity getRelationshipsForEntity(nodeId) - All edges for a node
Delete deleteRelationship(id) - Remove edge

Graph Traversal#

Operation Description
Neighbors getNeighbors(nodeId, options) - Direct connections
Traverse traverse(options) - BFS/DFS traversal with depth limits
Shortest Path findShortestPath(options) - Weighted shortest path
All Paths findAllPaths(options) - All paths up to max depth
Subgraph extractSubgraph(options) - Connected subgraph extraction

Entity Resolution#

Operation Description
Resolve resolveEntity(request) - Match entity against existing
Find Duplicates findDuplicates(threshold) - Detect duplicate entities

Technology Stack#

Layer Technologies
Framework Hono (planned), currently standalone service classes
Database PostgreSQL with Prisma ORM (schema: sophia)
Vector Stores Qdrant, Pinecone, Milvus, In-memory
Embeddings OpenAI text-embedding-3, Cohere embed-v3, Local
Chunking 8 strategies (fixed, sliding, paragraph, sentence, recursive, markdown, code, semantic)
Graph Store In-memory (MemoryGraphStore), extensible interface
Events @oshun/event-bus (Redis Streams)
Validation Zod schemas with branded types
Logging @oshun/logging (Pino-based structured logging)
Build @nx/js:tsc
Testing Vitest
UI React (Workbench app)

Deployment Architecture#

Service Topology#

text
                    +------------------+
                    |   Load Balancer  |
                    +--------+---------+
                             |
                +------------+------------+
                |            |            |
                v            v            v
         +----------+  +----------+  +----------+
         |Search API|  |Ingestion |  |Knowledge |
         |(:3000)   |  |Pipeline  |  |Graph     |
         |          |  |          |  |(:3001)   |
         +----+-----+  +----+-----+  +----+-----+
              |              |              |
         +----+--------------+--------------+----+
         |                                       |
    +----+----+                             +----+----+
    |PostgreSQL|                             |  Redis  |
    |(sophia)  |                             | (cache, |
    |          |                             |  queue) |
    +----+-----+                             +---------+
         |
    +----+----+----+----+
    |         |         |
    v         v         v
+------+ +-------+ +-------+
|Qdrant| |Milvus | |Pinecone|
|(self-| |(self- | |(cloud) |
|hosted)| |hosted)|         |
+------+ +-------+ +-------+

ECS Deployment#

Sophia services deploy via the shared deploy-ecs.yml pipeline:

  • search-api - ECS service with auto-scaling based on request volume
  • ingestion - ECS task for batch processing, triggered by queue depth
  • knowledge-graph - ECS service for graph operations
  • workbench - Static deployment (S3 + CloudFront)

Database Migrations#

Managed via the shared database.yml CI pipeline with Prisma migrations targeting the sophia PostgreSQL database.


Cross-Domain Dependencies#

Sophia Consumes From#

Source Domain Event/Integration Purpose
Hathor World/lore content Worldbuilding knowledge ingestion
Isis Generated content metadata AI output indexing and search

Sophia Publishes To#

Target Domain Event Purpose
Hathor sophia.document.ingested Lore enrichment with research
Bellona sophia.document.updated Asset metadata updates
Isis sophia.entity.extracted Context for AI generation
All sophia.search.performed Analytics and usage tracking

Shared Infrastructure#

Dependency Usage
@oshun/logging Structured logging across all services
@oshun/event-bus Cross-domain event publishing (Redis Streams)
PostgreSQL Primary data store (schema: sophia)
Redis Caching, queue management, event streaming

Environment Variables#

bash
# Database
SOPHIA_DATABASE_URL=postgresql://oshun:oshun_dev@localhost:5432/sophia

# Redis
REDIS_URL=redis://localhost:6379

# Vector Stores
QDRANT_URL=http://localhost:6333
PINECONE_API_KEY=...
MILVUS_HOST=localhost
MILVUS_PORT=19530

# Embedding Providers
OPENAI_API_KEY=...
COHERE_API_KEY=...

# Service Ports
PORT=3000                    # Search API
INGESTION_PORT=3000          # Ingestion service
HOST=localhost

# Ingestion Config
MAX_CONCURRENT_JOBS=5
DEFAULT_CHUNK_SIZE=1000
DEFAULT_CHUNK_OVERLAP=200
LOG_LEVEL=info

Development#

bash
# Run the search API
pnpm nx dev @sophia/search-api

# Run tests
pnpm nx test @sophia/client
pnpm nx test @sophia/indexing
pnpm nx test @sophia/vectordb

# Build all Sophia libraries
pnpm nx run-many --target=build --projects=tag:scope:sophia

# Start vector database for development
docker compose -f docker/docker-compose.dev.yml --profile vectors up -d

# Run database migrations
pnpm nx run @sophia/database:prisma:migrate
pnpm nx run @sophia/database:prisma:generate