Eve library

Sophia Domain - Developer Guide

Guide

Revision 1

On this page

Sophia is the Research & Knowledge domain. It provides semantic search, document ingestion, knowledge graph construction, vector database management, and retrieval-augmented generation (RAG) services. Sophia powers the platform's ability to search, index, and reason over large document collections.


Prerequisites#

Required Software#

Software Minimum Version Purpose
Node.js >= 20.0.0 Runtime for all Sophia services
pnpm >= 9.0.0 Package manager (monorepo workspace)
Docker >= 24.0 Infrastructure (PostgreSQL, Redis, Qdrant)
TypeScript ^6.0.3 Type checking and compilation

Vector Database (one of the following)#

Service Default Port Purpose
Qdrant 6333/6334 Primary supported vector store
Milvus 19530 Alternative vector store
Pinecone (cloud) Cloud vector store option

For local development, Qdrant is recommended and can be started via Docker Compose profiles.

Optional#

Software Purpose
Neo4j Knowledge graph backend (port 7474/7687)
Elasticsearch Full-text search backend (port 9200)

Project Structure#

text
apps/sophia/
  search-api/        # Hybrid search, reranking, RAG, and citation API
  ingestion/         # Document ingestion pipeline (connectors, extraction, enrichment)
  knowledge-graph/   # Entity resolution, relation extraction, graph queries
  workbench/         # React web UI for document management, search, and curation

libs/sophia/
  client/            # TypeScript SDK (@sophia/client)
  database/          # Prisma schema and generated client
  indexing/          # Document chunking, embedding, and vector indexing pipelines
  vectordb/          # Vector store abstraction (Qdrant, Milvus, Pinecone, in-memory)
  schemas/           # Shared Zod schemas for entities, documents, queries
  theory/            # Research management and citation tracking
  cultural-research/ # Cultural consultation, localization, accessibility
  predictions/       # Content prediction and scheduling optimization

Vector Store Backends#

The libs/sophia/vectordb/ library provides a unified interface with multiple backend implementations:

Backend File SDK
Qdrant qdrant-store.ts @qdrant/js-client-rest
Milvus milvus-store.ts @zilliz/milvus2-sdk-node
Pinecone pinecone-store.ts @pinecone-database/pinecone
In-Memory memory-store.ts (built-in)

Local Development Setup#

1. Start Infrastructure#

bash
# Core services (PostgreSQL + Redis)
docker compose -f docker/docker-compose.dev.yml up -d

# Add Qdrant vector database
docker compose -f docker/docker-compose.dev.yml --profile vectors up -d

# Optionally add Neo4j for knowledge graph
docker compose -f docker/docker-compose.dev.yml --profile graph up -d

# Optionally add Elasticsearch for full-text search
docker compose -f docker/docker-compose.dev.yml --profile search up -d

2. Set Environment Variables#

Variable Default Used By
SOPHIA_DATABASE_URL (from root .env) database lib
PORT 3000 search-api
HOST localhost search-api
QDRANT_URL http://localhost:6333 vectordb
QDRANT_API_KEY (none for local) vectordb
MILVUS_ADDRESS localhost:19530 vectordb
PINECONE_API_KEY (required for Pinecone) vectordb
PINECONE_ENVIRONMENT (required for Pinecone) vectordb
NEO4J_URL bolt://localhost:7687 knowledge-graph
NEO4J_USER neo4j knowledge-graph
NEO4J_PASSWORD password knowledge-graph
OPENAI_API_KEY (for embedding provider) indexing, search
REDIS_URL redis://localhost:6379 caching

3. Generate Prisma Client#

bash
cd libs/sophia/database
npx prisma generate
npx prisma db push

4. Install Dependencies#

bash
pnpm install

5. Seed the Database (optional)#

bash
cd libs/sophia/database
npx tsx prisma/seed.ts       # Standard seed data
npx tsx prisma/seed-test.ts  # Test fixtures

A Lilith-to-Sophia knowledge migration script is also available:

bash
npx tsx prisma/migrate-lilith-knowledge.ts

Running Services#

Search API#

Provides hybrid search (vector + keyword), reranking with cross-encoder models, RAG (retrieval-augmented generation), and citation extraction.

bash
# Via Nx
pnpm nx serve @sophia/search-api

# The service starts on PORT (default 3000)

In development mode, the search API initializes with mock providers (MockEmbeddingProvider, MockVectorStore, MockTextIndex). In production, these are replaced with real implementations.

API endpoints:

  • POST /search - Execute a search query (vector, keyword, or hybrid)
  • POST /rag - RAG request (retrieve + generate answer)
  • GET /health - Service health check

Search strategies: vector, keyword, hybrid (default)

Ingestion Service#

Processes documents through a pipeline: connector -> extraction -> chunking -> embedding -> indexing.

bash
pnpm nx serve @sophia/ingestion

The ingestion pipeline consists of:

  1. Connectors (src/connectors/) - Fetch documents from sources
  2. Text Extraction (src/extraction/) - Extract text from various formats
  3. Enrichment Pipeline (src/enrichment/) - Metadata enrichment
  4. Ingestion Pipeline (src/pipeline/) - Orchestrates the full flow

Knowledge Graph Service#

Manages entity extraction, resolution, relation discovery, and graph queries.

bash
pnpm nx serve @sophia/knowledge-graph

Services:

  • Entity Service (src/entities/) - Entity CRUD and management
  • Relation Service (src/relations/) - Relationship extraction and storage
  • Entity Resolver (src/resolution/) - Entity deduplication and linking
  • Graph Service (src/graph/) - Graph traversal and querying

Workbench (React Web UI)#

A React SPA for document management, search, annotations, review, and curation.

bash
# Development (Vite dev server)
pnpm nx serve @sophia/workbench

The workbench uses:

  • Vite for build/dev server
  • Tailwind CSS for styling
  • React Router for navigation
  • Pages: Dashboard, Documents, Search, Review, Curation, Annotations

Building#

Build All Sophia Projects#

bash
pnpm nx run-many --target=build --projects=tag:scope:sophia

Build Individual Projects#

bash
pnpm nx build @sophia/search-api
pnpm nx build @sophia/ingestion
pnpm nx build @sophia/knowledge-graph
pnpm nx build @sophia/workbench      # Produces static site
pnpm nx build @sophia/indexing
pnpm nx build @sophia/vectordb
pnpm nx build @sophia/client

Testing#

Test Runner#

All Sophia projects use Vitest via the @nx/vite:test executor.

Running Tests#

bash
# All Sophia projects
pnpm nx run-many --target=test --projects=tag:scope:sophia

# Individual projects
pnpm nx test @sophia/search-api
pnpm nx test @sophia/ingestion
pnpm nx test @sophia/knowledge-graph
pnpm nx test @sophia/workbench
pnpm nx test @sophia/indexing
pnpm nx test @sophia/vectordb
pnpm nx test @sophia/client
pnpm nx test @sophia/theory
pnpm nx test @sophia/cultural-research

Testing Without External Services#

The libs/sophia/vectordb/ library includes an in-memory vector store (memory-store.ts) that requires no external services. Use it for tests:

typescript
import { MemoryVectorStore } from '@sophia/vectordb';

const store = new MemoryVectorStore();
// Use for tests without Qdrant/Milvus/Pinecone

The search API's MockEmbeddingProvider, MockVectorStore, and MockTextIndex classes (in search/hybrid-search.ts) are used for development and testing without real ML models or vector databases.

Linting#

bash
pnpm nx run-many --target=lint --projects=tag:scope:sophia

Database Schema#

The Sophia database (libs/sophia/database/prisma/schema.prisma) contains models for document management, knowledge entities, and search metadata. The schema is accessed via the Prisma client generated into src/generated/client/.

Key concepts:

  • Documents with chunking metadata and embedding references
  • Entities with type classification and resolution status
  • Relations between entities with confidence scores
  • Search query logs for analytics and optimization

Common Development Tasks#

Adding a New Chunking Strategy#

The indexing library (libs/sophia/indexing/) handles document chunking.

  1. Add your chunking strategy in libs/sophia/indexing/src/chunking/:

    typescript
    import type { ChunkingStrategy, ChunkOptions, Chunk } from './types.js';
    
    export class SentenceWindowChunker implements ChunkingStrategy {
      async chunk(text: string, options: ChunkOptions): Promise<Chunk[]> {
        // Implement sentence-window chunking
        // Each chunk includes surrounding context sentences
      }
    }
    
  2. Register the strategy in the chunking factory/index so it can be selected by name.

  3. Add configuration options to the ChunkOptions type if your strategy introduces new parameters.

  4. Write tests that verify chunk boundaries, overlap behavior, and metadata preservation.

Adding a New Vector Store Backend#

  1. Create a new store implementation in libs/sophia/vectordb/src/stores/:

    typescript
    import type {
      VectorStore,
      VectorStoreConfig,
      SearchResult,
    } from '../types.js';
    
    export class YourVectorStore implements VectorStore {
      constructor(config: VectorStoreConfig) {
        /* ... */
      }
    
      async upsert(vectors: VectorRecord[]): Promise<void> {
        /* ... */
      }
      async search(query: number[], topK: number): Promise<SearchResult[]> {
        /* ... */
      }
      async delete(ids: string[]): Promise<void> {
        /* ... */
      }
      async healthCheck(): Promise<boolean> {
        /* ... */
      }
    }
    
  2. Export the new store from libs/sophia/vectordb/src/stores/index.ts.

  3. Add the backend's SDK to the package.json dependencies.

  4. Add health check support following the pattern in libs/sophia/vectordb/src/health.ts.

  5. Write tests using the standard vector store test suite. The in-memory store serves as the reference implementation.

Adding a New Document Connector#

Connectors fetch documents from external sources for ingestion.

  1. Add your connector in apps/sophia/ingestion/src/connectors/:

    typescript
    export class SlackConnector {
      async connect(config: ConnectionConfig): Promise<void> {
        /* ... */
      }
      async fetchDocuments(options: FetchOptions): Promise<Document[]> {
        /* ... */
      }
      async disconnect(): Promise<void> {
        /* ... */
      }
    }
    
  2. Register the connector in connectors.ts so the ingestion pipeline can use it.

  3. Add any required API keys to the environment variable documentation.

Adding a New Embedding Provider#

  1. Implement the embedding provider interface:

    typescript
    export class YourEmbeddingProvider {
      async embed(texts: string[]): Promise<number[][]> {
        /* ... */
      }
      getDimensions(): number {
        /* ... */
      }
      getModelName(): string {
        /* ... */
      }
    }
    
  2. Integrate it into the indexing pipeline so documents can be embedded using your provider.


Debugging Tips#

Search Returning No Results#

  • Verify the vector store is running and accessible
  • Check that documents have been ingested and indexed
  • For hybrid search, ensure both vector and text indices contain data
  • The mock providers return empty results by default in development

Qdrant Connection Issues#

  • Default URL: http://localhost:6333
  • Start Qdrant via Docker Compose vectors profile
  • Check collection existence: Qdrant dashboard at http://localhost:6333/dashboard
  • Verify API key is not required for local instances

Ingestion Pipeline Failures#

  • Check connector configuration and credentials
  • Text extraction supports multiple formats; verify the extractor for your document type
  • Monitor the enrichment pipeline for metadata extraction errors
  • Check Redis for queue state if using async ingestion

Knowledge Graph Issues#

  • Neo4j must be running for graph operations (Docker Compose graph profile)
  • Entity resolution may produce false positives; check confidence thresholds
  • Graph queries use Cypher; verify query syntax
  • Entity resolver deduplication depends on name similarity algorithms

Workbench Build Issues#

  • The workbench uses Vite with Tailwind CSS
  • Check tailwind.config.js and postcss.config.js for CSS processing
  • The test setup file is at src/test/setup.ts
  • Keyboard navigation hooks are in src/hooks/useKeyboardNavigation.ts

Performance Optimization#

  • The vectordb library includes batch utilities (src/utils/batch.ts) for bulk operations
  • Similarity computation utilities are in src/utils/similarity.ts
  • The indexing library has index builder (src/index/builder.ts) and similarity tools (src/index/similarity.ts)
  • The predictions library (libs/sophia/predictions/) provides content prediction and schedule optimization for proactive indexing