Sophia - Knowledge Management and Research Platform
Sophia provides RAG (Retrieval-Augmented Generation) infrastructure for document
ingestion, semantic search with hybrid vector+keyword strategies, citation
management with multi-style formatting, knowledge graph construction with entity
resolution, and retrieval quality evaluation with benchmark support. It supports
four vector database backends and eight chunking strategies for flexible
document processing.
Architecture#
Apps (4)#
App
Port
Purpose
search-api
3000
SearchAPIService with hybrid search, RAG pipeline, cross-encoder reranking, and citation extraction
ingestion
-
Multi-stage pipeline with configurable chunking (8 strategies), entity extraction, and summary generation
knowledge-graph
3001
KnowledgeGraphService with entity/relation CRUD, BFS/DFS traversal, path finding, and entity resolution
workbench
-
React UI with 6 pages: Dashboard, Curation, Review, Annotations, Documents, Search
Libraries (10)#
Library
Key Exports
@sophia/client
SophiaClient (unified), SearchClient, IngestionClient, KnowledgeGraphClient, SearchQueryBuilder, GraphQueryBuilder, branded ID types
@sophia/database
Prisma client (getSophiaClient()) for 12-model PostgreSQL schema with full-text search
@sophia/schemas
Zod schemas with branded IDs for documents, citations, claims, knowledge packs, entities, relations
@sophia/indexing
8 chunkers (fixed, sliding, paragraph, sentence, recursive, markdown, code, semantic), 3 embedding providers (OpenAI, Cohere, Local), IndexBuilder, vector math, graph export
@sophia/vectordb
4 vector stores (Memory, Milvus, Qdrant, Pinecone), 10 collection schemas, hybrid search with RRF, batch utilities, health checks
@sophia/evaluation
RetrievalEvaluator, CitationChecker, GroundednessScorer, BenchmarkRunner
@sophia/theory
CitationManager (5 styles), ResearchManager (project/hypothesis tracking)
@sophia/cultural-research
CulturalConsultant, LocalizationManager, AccessibilityManager, CulturalDatabase
@sophia/event-publisher
SophiaEventPublisher with 9 event types for documents, search, entities, and citations
@sophia/predictions
Time series forecasting, schedule optimization, content performance prediction
Technology Stack#
Layer
Technologies
Framework
Hono (planned), currently standalone service classes
Database
PostgreSQL with Prisma ORM (schema: sophia, 12 models, 966 lines)
Vector Stores
Qdrant, Pinecone, Milvus, In-memory
Embeddings
OpenAI text-embedding-3, Cohere embed-v3, Local
Validation
Zod schemas with branded types
Events
@oshun/event-bus (Redis Streams)
Logging
@oshun/logging (Pino-based structured logging)
Build
@nx/js:tsc
Testing
Vitest
Database Schema#
12 models in libs/sophia/database/prisma/schema.prisma (966 lines):
Model
Table
Purpose
Document
documents
Research documents with 14 types, full publication metadata, quality scoring, tsvector search
DocumentChunk
document_chunks
Indexed text chunks with binary embedding storage and overlap linking
Citation
citations
Citation records with 6 styles, verification status, claim tracking
Entity
entities
Knowledge graph nodes with 15 types, 5 confidence levels, merge tracking
EntityMention
entity_mentions
Entity occurrences in documents with position and context
Relation
relations
Knowledge graph edges with 23 relation types and evidence tracking
KnowledgePack
knowledge_packs
Curated knowledge collections with versioning and community features
KnowledgePackDocument
knowledge_pack_documents
Pack-document associations
KnowledgePackEntity
knowledge_pack_entities
Pack-entity associations with importance weighting
KnowledgePackStar
knowledge_pack_stars
User favorites
IngestionJob
ingestion_jobs
10-stage pipeline tracking with retry support
AuditLog
audit_log
Change tracking
Key Enums#
DocumentType (14): BOOK, ARTICLE, THESIS, MANUSCRIPT, SCRIPTURE,
COMMENTARY, ESSAY, LETTER, SPEECH, TRANSLATION, ANTHOLOGY, ENCYCLOPEDIA,
DICTIONARY, OTHER
EntityType (15): TRADITION, TEXT, CONCEPT, PRACTICE, FIGURE, PLACE,
SCHOOL, DOCTRINE, EVENT, TERM, SYMBOL, RITUAL, ARTIFACT, ORGANIZATION, CUSTOM
RelationType (23): TAUGHT, STUDIED_WITH, INFLUENCED, WROTE, EDITED,
TRANSLATED, FOUNDED, MEMBER_OF, BELONGS_TO, LIVED_IN, BORN_IN, DIED_IN,
LOCATED_IN, PRECEDED, SUCCEEDED, CONTEMPORARY_WITH, REFERENCES, QUOTES, CITES,
RELATED_TO, SIMILAR_TO, OPPOSITE_OF, DERIVED_FROM, PART_OF, PRACTICES,
PRESCRIBES, CUSTOM
API Endpoints#
Search API (Port 3000)#
Method
Path
Description
POST
/v1/search
Hybrid semantic search (vector + keyword) with reranking
POST
/v1/rag
RAG pipeline: retrieve, generate, cite
GET
/health
Service health check
Documents (Port 4003)#
Method
Path
Description
POST
/v1/documents
Ingest a document
GET
/v1/documents/:id
Get document metadata
DELETE
/v1/documents/:id
Delete a document
Collections#
Method
Path
Description
GET
/v1/collections
List collections
POST
/v1/collections
Create a collection
Embeddings#
Method
Path
Description
POST
/v1/embeddings
Generate embeddings
Knowledge Graph (Port 3001)#
Graph operations are exposed as service methods (not REST endpoints):
Category
Operations
Entities
create, get, query, delete
Relations
create, get, getForEntity, delete
Traversal
getNeighbors, traverse, findShortestPath, findAllPaths, extractSubgraph
Resolution
resolveEntity, findDuplicates
Search Capabilities#
Hybrid Search - Combined keyword (BM25) and semantic vector search with
configurable weights
Cross-Encoder Reranking - Result reranking for improved relevance using
cross-encoder models
Citation Tracking - Source attribution with 6 citation styles and
verification workflow
RAG Pipeline - Full retrieval-augmented generation with LLM answer
generation and citation extraction
Multi-Collection Queries - Search across 10 predefined collection schemas
Context Retrieval - Optimized context windows for LLM prompts
Chunking Strategies#
@sophia/indexing provides 8 chunking approaches:
Strategy
Class
Description
Fixed
FixedSizeChunker
Fixed character/token window
Sliding Window
SlidingWindowChunker
Overlapping sliding window
Paragraph
ParagraphChunker
Paragraph boundary detection
Sentence
SentenceChunker
Sentence boundary detection
Recursive
RecursiveChunker
Hierarchical splitting with separators
Markdown
MarkdownChunker
Markdown structure-aware splitting
Code
CodeChunker
Programming language-aware splitting
Semantic
SemanticChunker
Embedding-based semantic boundaries
Vector Store Integration#
@sophia/vectordb provides a unified interface across four vector databases:
Store
SDK
Default Port
Factory
Memory
Built-in
-
createMemoryStore()
Qdrant
@qdrant/js-client-rest
6333
createQdrantStoreFromEnv()
Pinecone
@pinecone-database/pinecone
Cloud
createPineconeStoreFromEnv()
Milvus
@zilliz/milvus2-sdk-node
19530
createMilvusStoreFromEnv()
Predefined Collections#
10 collection schemas for different embedding use cases:
KNOWLEDGE_CHUNKS_SCHEMA, RESEARCH_DOCUMENTS_SCHEMA, CITATIONS_SCHEMA,
ENTITIES_SCHEMA, WISDOM_EMBEDDINGS_SCHEMA, CONTENT_EMBEDDINGS_SCHEMA,
USER_PREFERENCE_EMBEDDINGS_SCHEMA, CONVERSATION_EMBEDDINGS_SCHEMA,
SEMANTIC_SEARCH_SCHEMA, PERSONA_EMBEDDINGS_SCHEMA
For local development, start Qdrant:
bash Copy
docker compose -f docker/docker-compose.dev.yml --profile vectors up -d
Cross-Domain Integration#
Events Published#
Event
Trigger
sophia.document.ingested
Document ingestion completed
sophia.document.updated
Document metadata changed
sophia.document.deleted
Document removed
sophia.index.updated
Vector index has new data
sophia.index.rebuilt
Full index rebuild completed
sophia.search.performed
Search query executed
sophia.entity.extracted
New entity in graph
sophia.relation.discovered
New relationship in graph
sophia.citation.created
Citation record available
Environment Variables#
bash Copy
SOPHIA_DATABASE_URL=postgresql://oshun:oshun_dev@localhost:5432/sophia
REDIS_URL=redis://localhost:6379
# Vector Stores
QDRANT_URL=http://localhost:6333
PINECONE_API_KEY=...
MILVUS_HOST=localhost
MILVUS_PORT=19530
# Embedding Providers
OPENAI_API_KEY=...
COHERE_API_KEY=...
# Service Configuration
PORT=3000
INGESTION_PORT=3000
MAX_CONCURRENT_JOBS=5
DEFAULT_CHUNK_SIZE=1000
DEFAULT_CHUNK_OVERLAP=200
LOG_LEVEL=info
Development#
bash Copy
# Start infrastructure
docker compose -f docker/docker-compose.dev.yml up -d
# Start vector database
docker compose -f docker/docker-compose.dev.yml --profile vectors up -d
# Run the search API
pnpm nx dev @sophia/search-api
# Run tests
pnpm nx test @sophia/client
pnpm nx test @sophia/indexing
pnpm nx test @sophia/vectordb
pnpm nx test @sophia/evaluation
# Build all Sophia libraries
pnpm nx run-many --target=build --projects=tag:scope:sophia
# Run database migrations
pnpm nx run @sophia/database:prisma:migrate
pnpm nx run @sophia/database:prisma:generate
CI/CD#
Sophia uses the core CI pipeline (ci.yml) and the general ECS deployment
pipeline (deploy-ecs.yml). Database migrations run via database.yml.