Sophia is the Research & Knowledge domain. It provides semantic search, document ingestion, knowledge graph construction, vector database management, and retrieval-augmented generation (RAG) services. Sophia powers the platform's ability to search, index, and reason over large document collections.
Prerequisites#
Required Software#
| Software | Minimum Version | Purpose |
|---|---|---|
| Node.js | >= 20.0.0 | Runtime for all Sophia services |
| pnpm | >= 9.0.0 | Package manager (monorepo workspace) |
| Docker | >= 24.0 | Infrastructure (PostgreSQL, Redis, Qdrant) |
| TypeScript | ^6.0.3 | Type checking and compilation |
Vector Database (one of the following)#
| Service | Default Port | Purpose |
|---|---|---|
| Qdrant | 6333/6334 | Primary supported vector store |
| Milvus | 19530 | Alternative vector store |
| Pinecone | (cloud) | Cloud vector store option |
For local development, Qdrant is recommended and can be started via Docker Compose profiles.
Optional#
| Software | Purpose |
|---|---|
| Neo4j | Knowledge graph backend (port 7474/7687) |
| Elasticsearch | Full-text search backend (port 9200) |
Project Structure#
apps/sophia/
search-api/ # Hybrid search, reranking, RAG, and citation API
ingestion/ # Document ingestion pipeline (connectors, extraction, enrichment)
knowledge-graph/ # Entity resolution, relation extraction, graph queries
workbench/ # React web UI for document management, search, and curation
libs/sophia/
client/ # TypeScript SDK (@sophia/client)
database/ # Prisma schema and generated client
indexing/ # Document chunking, embedding, and vector indexing pipelines
vectordb/ # Vector store abstraction (Qdrant, Milvus, Pinecone, in-memory)
schemas/ # Shared Zod schemas for entities, documents, queries
theory/ # Research management and citation tracking
cultural-research/ # Cultural consultation, localization, accessibility
predictions/ # Content prediction and scheduling optimization
Vector Store Backends#
The libs/sophia/vectordb/ library provides a unified interface with multiple
backend implementations:
| Backend | File | SDK |
|---|---|---|
| Qdrant | qdrant-store.ts |
@qdrant/js-client-rest |
| Milvus | milvus-store.ts |
@zilliz/milvus2-sdk-node |
| Pinecone | pinecone-store.ts |
@pinecone-database/pinecone |
| In-Memory | memory-store.ts |
(built-in) |
Local Development Setup#
1. Start Infrastructure#
# Core services (PostgreSQL + Redis)
docker compose -f docker/docker-compose.dev.yml up -d
# Add Qdrant vector database
docker compose -f docker/docker-compose.dev.yml --profile vectors up -d
# Optionally add Neo4j for knowledge graph
docker compose -f docker/docker-compose.dev.yml --profile graph up -d
# Optionally add Elasticsearch for full-text search
docker compose -f docker/docker-compose.dev.yml --profile search up -d
2. Set Environment Variables#
| Variable | Default | Used By |
|---|---|---|
SOPHIA_DATABASE_URL |
(from root .env) |
database lib |
PORT |
3000 |
search-api |
HOST |
localhost |
search-api |
QDRANT_URL |
http://localhost:6333 |
vectordb |
QDRANT_API_KEY |
(none for local) | vectordb |
MILVUS_ADDRESS |
localhost:19530 |
vectordb |
PINECONE_API_KEY |
(required for Pinecone) | vectordb |
PINECONE_ENVIRONMENT |
(required for Pinecone) | vectordb |
NEO4J_URL |
bolt://localhost:7687 |
knowledge-graph |
NEO4J_USER |
neo4j |
knowledge-graph |
NEO4J_PASSWORD |
password |
knowledge-graph |
OPENAI_API_KEY |
(for embedding provider) | indexing, search |
REDIS_URL |
redis://localhost:6379 |
caching |
3. Generate Prisma Client#
cd libs/sophia/database
npx prisma generate
npx prisma db push
4. Install Dependencies#
pnpm install
5. Seed the Database (optional)#
cd libs/sophia/database
npx tsx prisma/seed.ts # Standard seed data
npx tsx prisma/seed-test.ts # Test fixtures
A Lilith-to-Sophia knowledge migration script is also available:
npx tsx prisma/migrate-lilith-knowledge.ts
Running Services#
Search API#
Provides hybrid search (vector + keyword), reranking with cross-encoder models, RAG (retrieval-augmented generation), and citation extraction.
# Via Nx
pnpm nx serve @sophia/search-api
# The service starts on PORT (default 3000)
In development mode, the search API initializes with mock providers
(MockEmbeddingProvider, MockVectorStore, MockTextIndex). In production,
these are replaced with real implementations.
API endpoints:
POST /search- Execute a search query (vector, keyword, or hybrid)POST /rag- RAG request (retrieve + generate answer)GET /health- Service health check
Search strategies: vector, keyword, hybrid (default)
Ingestion Service#
Processes documents through a pipeline: connector -> extraction -> chunking -> embedding -> indexing.
pnpm nx serve @sophia/ingestion
The ingestion pipeline consists of:
- Connectors (
src/connectors/) - Fetch documents from sources - Text Extraction (
src/extraction/) - Extract text from various formats - Enrichment Pipeline (
src/enrichment/) - Metadata enrichment - Ingestion Pipeline (
src/pipeline/) - Orchestrates the full flow
Knowledge Graph Service#
Manages entity extraction, resolution, relation discovery, and graph queries.
pnpm nx serve @sophia/knowledge-graph
Services:
- Entity Service (
src/entities/) - Entity CRUD and management - Relation Service (
src/relations/) - Relationship extraction and storage - Entity Resolver (
src/resolution/) - Entity deduplication and linking - Graph Service (
src/graph/) - Graph traversal and querying
Workbench (React Web UI)#
A React SPA for document management, search, annotations, review, and curation.
# Development (Vite dev server)
pnpm nx serve @sophia/workbench
The workbench uses:
- Vite for build/dev server
- Tailwind CSS for styling
- React Router for navigation
- Pages: Dashboard, Documents, Search, Review, Curation, Annotations
Building#
Build All Sophia Projects#
pnpm nx run-many --target=build --projects=tag:scope:sophia
Build Individual Projects#
pnpm nx build @sophia/search-api
pnpm nx build @sophia/ingestion
pnpm nx build @sophia/knowledge-graph
pnpm nx build @sophia/workbench # Produces static site
pnpm nx build @sophia/indexing
pnpm nx build @sophia/vectordb
pnpm nx build @sophia/client
Testing#
Test Runner#
All Sophia projects use Vitest via the @nx/vite:test executor.
Running Tests#
# All Sophia projects
pnpm nx run-many --target=test --projects=tag:scope:sophia
# Individual projects
pnpm nx test @sophia/search-api
pnpm nx test @sophia/ingestion
pnpm nx test @sophia/knowledge-graph
pnpm nx test @sophia/workbench
pnpm nx test @sophia/indexing
pnpm nx test @sophia/vectordb
pnpm nx test @sophia/client
pnpm nx test @sophia/theory
pnpm nx test @sophia/cultural-research
Testing Without External Services#
The libs/sophia/vectordb/ library includes an in-memory vector store
(memory-store.ts) that requires no external services. Use it for tests:
import { MemoryVectorStore } from '@sophia/vectordb';
const store = new MemoryVectorStore();
// Use for tests without Qdrant/Milvus/Pinecone
The search API's MockEmbeddingProvider, MockVectorStore, and MockTextIndex
classes (in search/hybrid-search.ts) are used for development and testing
without real ML models or vector databases.
Linting#
pnpm nx run-many --target=lint --projects=tag:scope:sophia
Database Schema#
The Sophia database (libs/sophia/database/prisma/schema.prisma) contains
models for document management, knowledge entities, and search metadata. The
schema is accessed via the Prisma client generated into src/generated/client/.
Key concepts:
- Documents with chunking metadata and embedding references
- Entities with type classification and resolution status
- Relations between entities with confidence scores
- Search query logs for analytics and optimization
Common Development Tasks#
Adding a New Chunking Strategy#
The indexing library (libs/sophia/indexing/) handles document chunking.
-
Add your chunking strategy in
libs/sophia/indexing/src/chunking/:typescriptimport type { ChunkingStrategy, ChunkOptions, Chunk } from './types.js'; export class SentenceWindowChunker implements ChunkingStrategy { async chunk(text: string, options: ChunkOptions): Promise<Chunk[]> { // Implement sentence-window chunking // Each chunk includes surrounding context sentences } } -
Register the strategy in the chunking factory/index so it can be selected by name.
-
Add configuration options to the
ChunkOptionstype if your strategy introduces new parameters. -
Write tests that verify chunk boundaries, overlap behavior, and metadata preservation.
Adding a New Vector Store Backend#
-
Create a new store implementation in
libs/sophia/vectordb/src/stores/:typescriptimport type { VectorStore, VectorStoreConfig, SearchResult, } from '../types.js'; export class YourVectorStore implements VectorStore { constructor(config: VectorStoreConfig) { /* ... */ } async upsert(vectors: VectorRecord[]): Promise<void> { /* ... */ } async search(query: number[], topK: number): Promise<SearchResult[]> { /* ... */ } async delete(ids: string[]): Promise<void> { /* ... */ } async healthCheck(): Promise<boolean> { /* ... */ } } -
Export the new store from
libs/sophia/vectordb/src/stores/index.ts. -
Add the backend's SDK to the
package.jsondependencies. -
Add health check support following the pattern in
libs/sophia/vectordb/src/health.ts. -
Write tests using the standard vector store test suite. The in-memory store serves as the reference implementation.
Adding a New Document Connector#
Connectors fetch documents from external sources for ingestion.
-
Add your connector in
apps/sophia/ingestion/src/connectors/:typescriptexport class SlackConnector { async connect(config: ConnectionConfig): Promise<void> { /* ... */ } async fetchDocuments(options: FetchOptions): Promise<Document[]> { /* ... */ } async disconnect(): Promise<void> { /* ... */ } } -
Register the connector in
connectors.tsso the ingestion pipeline can use it. -
Add any required API keys to the environment variable documentation.
Adding a New Embedding Provider#
-
Implement the embedding provider interface:
typescriptexport class YourEmbeddingProvider { async embed(texts: string[]): Promise<number[][]> { /* ... */ } getDimensions(): number { /* ... */ } getModelName(): string { /* ... */ } } -
Integrate it into the indexing pipeline so documents can be embedded using your provider.
Debugging Tips#
Search Returning No Results#
- Verify the vector store is running and accessible
- Check that documents have been ingested and indexed
- For hybrid search, ensure both vector and text indices contain data
- The mock providers return empty results by default in development
Qdrant Connection Issues#
- Default URL:
http://localhost:6333 - Start Qdrant via Docker Compose vectors profile
- Check collection existence: Qdrant dashboard at
http://localhost:6333/dashboard - Verify API key is not required for local instances
Ingestion Pipeline Failures#
- Check connector configuration and credentials
- Text extraction supports multiple formats; verify the extractor for your document type
- Monitor the enrichment pipeline for metadata extraction errors
- Check Redis for queue state if using async ingestion
Knowledge Graph Issues#
- Neo4j must be running for graph operations (Docker Compose graph profile)
- Entity resolution may produce false positives; check confidence thresholds
- Graph queries use Cypher; verify query syntax
- Entity resolver deduplication depends on name similarity algorithms
Workbench Build Issues#
- The workbench uses Vite with Tailwind CSS
- Check
tailwind.config.jsandpostcss.config.jsfor CSS processing - The test setup file is at
src/test/setup.ts - Keyboard navigation hooks are in
src/hooks/useKeyboardNavigation.ts
Performance Optimization#
- The vectordb library includes batch utilities (
src/utils/batch.ts) for bulk operations - Similarity computation utilities are in
src/utils/similarity.ts - The indexing library has index builder (
src/index/builder.ts) and similarity tools (src/index/similarity.ts) - The predictions library (
libs/sophia/predictions/) provides content prediction and schedule optimization for proactive indexing