Eve library

Sophia Domain

Guide

Revision 1

On this page

Sophia - Knowledge Management and Research Platform

Sophia provides RAG (Retrieval-Augmented Generation) infrastructure for document ingestion, semantic search with hybrid vector+keyword strategies, citation management with multi-style formatting, knowledge graph construction with entity resolution, and retrieval quality evaluation with benchmark support. It supports four vector database backends and eight chunking strategies for flexible document processing.


Architecture#

Apps (4)#

App Port Purpose
search-api 3000 SearchAPIService with hybrid search, RAG pipeline, cross-encoder reranking, and citation extraction
ingestion - Multi-stage pipeline with configurable chunking (8 strategies), entity extraction, and summary generation
knowledge-graph 3001 KnowledgeGraphService with entity/relation CRUD, BFS/DFS traversal, path finding, and entity resolution
workbench - React UI with 6 pages: Dashboard, Curation, Review, Annotations, Documents, Search

Libraries (10)#

Library Key Exports
@sophia/client SophiaClient (unified), SearchClient, IngestionClient, KnowledgeGraphClient, SearchQueryBuilder, GraphQueryBuilder, branded ID types
@sophia/database Prisma client (getSophiaClient()) for 12-model PostgreSQL schema with full-text search
@sophia/schemas Zod schemas with branded IDs for documents, citations, claims, knowledge packs, entities, relations
@sophia/indexing 8 chunkers (fixed, sliding, paragraph, sentence, recursive, markdown, code, semantic), 3 embedding providers (OpenAI, Cohere, Local), IndexBuilder, vector math, graph export
@sophia/vectordb 4 vector stores (Memory, Milvus, Qdrant, Pinecone), 10 collection schemas, hybrid search with RRF, batch utilities, health checks
@sophia/evaluation RetrievalEvaluator, CitationChecker, GroundednessScorer, BenchmarkRunner
@sophia/theory CitationManager (5 styles), ResearchManager (project/hypothesis tracking)
@sophia/cultural-research CulturalConsultant, LocalizationManager, AccessibilityManager, CulturalDatabase
@sophia/event-publisher SophiaEventPublisher with 9 event types for documents, search, entities, and citations
@sophia/predictions Time series forecasting, schedule optimization, content performance prediction

Technology Stack#

Layer Technologies
Framework Hono (planned), currently standalone service classes
Database PostgreSQL with Prisma ORM (schema: sophia, 12 models, 966 lines)
Vector Stores Qdrant, Pinecone, Milvus, In-memory
Embeddings OpenAI text-embedding-3, Cohere embed-v3, Local
Validation Zod schemas with branded types
Events @oshun/event-bus (Redis Streams)
Logging @oshun/logging (Pino-based structured logging)
Build @nx/js:tsc
Testing Vitest

Database Schema#

12 models in libs/sophia/database/prisma/schema.prisma (966 lines):

Model Table Purpose
Document documents Research documents with 14 types, full publication metadata, quality scoring, tsvector search
DocumentChunk document_chunks Indexed text chunks with binary embedding storage and overlap linking
Citation citations Citation records with 6 styles, verification status, claim tracking
Entity entities Knowledge graph nodes with 15 types, 5 confidence levels, merge tracking
EntityMention entity_mentions Entity occurrences in documents with position and context
Relation relations Knowledge graph edges with 23 relation types and evidence tracking
KnowledgePack knowledge_packs Curated knowledge collections with versioning and community features
KnowledgePackDocument knowledge_pack_documents Pack-document associations
KnowledgePackEntity knowledge_pack_entities Pack-entity associations with importance weighting
KnowledgePackStar knowledge_pack_stars User favorites
IngestionJob ingestion_jobs 10-stage pipeline tracking with retry support
AuditLog audit_log Change tracking

Key Enums#

  • DocumentType (14): BOOK, ARTICLE, THESIS, MANUSCRIPT, SCRIPTURE, COMMENTARY, ESSAY, LETTER, SPEECH, TRANSLATION, ANTHOLOGY, ENCYCLOPEDIA, DICTIONARY, OTHER
  • EntityType (15): TRADITION, TEXT, CONCEPT, PRACTICE, FIGURE, PLACE, SCHOOL, DOCTRINE, EVENT, TERM, SYMBOL, RITUAL, ARTIFACT, ORGANIZATION, CUSTOM
  • RelationType (23): TAUGHT, STUDIED_WITH, INFLUENCED, WROTE, EDITED, TRANSLATED, FOUNDED, MEMBER_OF, BELONGS_TO, LIVED_IN, BORN_IN, DIED_IN, LOCATED_IN, PRECEDED, SUCCEEDED, CONTEMPORARY_WITH, REFERENCES, QUOTES, CITES, RELATED_TO, SIMILAR_TO, OPPOSITE_OF, DERIVED_FROM, PART_OF, PRACTICES, PRESCRIBES, CUSTOM

API Endpoints#

Search API (Port 3000)#

Method Path Description
POST /v1/search Hybrid semantic search (vector + keyword) with reranking
POST /v1/rag RAG pipeline: retrieve, generate, cite
GET /health Service health check

Documents (Port 4003)#

Method Path Description
POST /v1/documents Ingest a document
GET /v1/documents/:id Get document metadata
DELETE /v1/documents/:id Delete a document

Collections#

Method Path Description
GET /v1/collections List collections
POST /v1/collections Create a collection

Embeddings#

Method Path Description
POST /v1/embeddings Generate embeddings

Knowledge Graph (Port 3001)#

Graph operations are exposed as service methods (not REST endpoints):

Category Operations
Entities create, get, query, delete
Relations create, get, getForEntity, delete
Traversal getNeighbors, traverse, findShortestPath, findAllPaths, extractSubgraph
Resolution resolveEntity, findDuplicates

Search Capabilities#

  • Hybrid Search - Combined keyword (BM25) and semantic vector search with configurable weights
  • Cross-Encoder Reranking - Result reranking for improved relevance using cross-encoder models
  • Citation Tracking - Source attribution with 6 citation styles and verification workflow
  • RAG Pipeline - Full retrieval-augmented generation with LLM answer generation and citation extraction
  • Multi-Collection Queries - Search across 10 predefined collection schemas
  • Context Retrieval - Optimized context windows for LLM prompts

Chunking Strategies#

@sophia/indexing provides 8 chunking approaches:

Strategy Class Description
Fixed FixedSizeChunker Fixed character/token window
Sliding Window SlidingWindowChunker Overlapping sliding window
Paragraph ParagraphChunker Paragraph boundary detection
Sentence SentenceChunker Sentence boundary detection
Recursive RecursiveChunker Hierarchical splitting with separators
Markdown MarkdownChunker Markdown structure-aware splitting
Code CodeChunker Programming language-aware splitting
Semantic SemanticChunker Embedding-based semantic boundaries

Vector Store Integration#

@sophia/vectordb provides a unified interface across four vector databases:

Store SDK Default Port Factory
Memory Built-in - createMemoryStore()
Qdrant @qdrant/js-client-rest 6333 createQdrantStoreFromEnv()
Pinecone @pinecone-database/pinecone Cloud createPineconeStoreFromEnv()
Milvus @zilliz/milvus2-sdk-node 19530 createMilvusStoreFromEnv()

Predefined Collections#

10 collection schemas for different embedding use cases: KNOWLEDGE_CHUNKS_SCHEMA, RESEARCH_DOCUMENTS_SCHEMA, CITATIONS_SCHEMA, ENTITIES_SCHEMA, WISDOM_EMBEDDINGS_SCHEMA, CONTENT_EMBEDDINGS_SCHEMA, USER_PREFERENCE_EMBEDDINGS_SCHEMA, CONVERSATION_EMBEDDINGS_SCHEMA, SEMANTIC_SEARCH_SCHEMA, PERSONA_EMBEDDINGS_SCHEMA

For local development, start Qdrant:

bash
docker compose -f docker/docker-compose.dev.yml --profile vectors up -d

Cross-Domain Integration#

Events Published#

Event Trigger
sophia.document.ingested Document ingestion completed
sophia.document.updated Document metadata changed
sophia.document.deleted Document removed
sophia.index.updated Vector index has new data
sophia.index.rebuilt Full index rebuild completed
sophia.search.performed Search query executed
sophia.entity.extracted New entity in graph
sophia.relation.discovered New relationship in graph
sophia.citation.created Citation record available

Environment Variables#

bash
SOPHIA_DATABASE_URL=postgresql://oshun:oshun_dev@localhost:5432/sophia
REDIS_URL=redis://localhost:6379

# Vector Stores
QDRANT_URL=http://localhost:6333
PINECONE_API_KEY=...
MILVUS_HOST=localhost
MILVUS_PORT=19530

# Embedding Providers
OPENAI_API_KEY=...
COHERE_API_KEY=...

# Service Configuration
PORT=3000
INGESTION_PORT=3000
MAX_CONCURRENT_JOBS=5
DEFAULT_CHUNK_SIZE=1000
DEFAULT_CHUNK_OVERLAP=200
LOG_LEVEL=info

Development#

bash
# Start infrastructure
docker compose -f docker/docker-compose.dev.yml up -d

# Start vector database
docker compose -f docker/docker-compose.dev.yml --profile vectors up -d

# Run the search API
pnpm nx dev @sophia/search-api

# Run tests
pnpm nx test @sophia/client
pnpm nx test @sophia/indexing
pnpm nx test @sophia/vectordb
pnpm nx test @sophia/evaluation

# Build all Sophia libraries
pnpm nx run-many --target=build --projects=tag:scope:sophia

# Run database migrations
pnpm nx run @sophia/database:prisma:migrate
pnpm nx run @sophia/database:prisma:generate

CI/CD#

Sophia uses the core CI pipeline (ci.yml) and the general ECS deployment pipeline (deploy-ecs.yml). Database migrations run via database.yml.