# Sophia Domain - Developer Guide

Sophia is the Research & Knowledge domain. It provides semantic search, document
ingestion, knowledge graph construction, vector database management, and
retrieval-augmented generation (RAG) services. Sophia powers the platform's
ability to search, index, and reason over large document collections.

---

## Prerequisites

### Required Software

| Software   | Minimum Version | Purpose                                    |
| ---------- | --------------- | ------------------------------------------ |
| Node.js    | >= 20.0.0       | Runtime for all Sophia services            |
| pnpm       | >= 9.0.0        | Package manager (monorepo workspace)       |
| Docker     | >= 24.0         | Infrastructure (PostgreSQL, Redis, Qdrant) |
| TypeScript | ^6.0.3          | Type checking and compilation              |

### Vector Database (one of the following)

| Service  | Default Port | Purpose                        |
| -------- | ------------ | ------------------------------ |
| Qdrant   | 6333/6334    | Primary supported vector store |
| Milvus   | 19530        | Alternative vector store       |
| Pinecone | (cloud)      | Cloud vector store option      |

For local development, Qdrant is recommended and can be started via Docker
Compose profiles.

### Optional

| Software      | Purpose                                  |
| ------------- | ---------------------------------------- |
| Neo4j         | Knowledge graph backend (port 7474/7687) |
| Elasticsearch | Full-text search backend (port 9200)     |

---

## Project Structure

```
apps/sophia/
  search-api/        # Hybrid search, reranking, RAG, and citation API
  ingestion/         # Document ingestion pipeline (connectors, extraction, enrichment)
  knowledge-graph/   # Entity resolution, relation extraction, graph queries
  workbench/         # React web UI for document management, search, and curation

libs/sophia/
  client/            # TypeScript SDK (@sophia/client)
  database/          # Prisma schema and generated client
  indexing/          # Document chunking, embedding, and vector indexing pipelines
  vectordb/          # Vector store abstraction (Qdrant, Milvus, Pinecone, in-memory)
  schemas/           # Shared Zod schemas for entities, documents, queries
  theory/            # Research management and citation tracking
  cultural-research/ # Cultural consultation, localization, accessibility
  predictions/       # Content prediction and scheduling optimization
```

### Vector Store Backends

The `libs/sophia/vectordb/` library provides a unified interface with multiple
backend implementations:

| Backend   | File                | SDK                           |
| --------- | ------------------- | ----------------------------- |
| Qdrant    | `qdrant-store.ts`   | `@qdrant/js-client-rest`      |
| Milvus    | `milvus-store.ts`   | `@zilliz/milvus2-sdk-node`    |
| Pinecone  | `pinecone-store.ts` | `@pinecone-database/pinecone` |
| In-Memory | `memory-store.ts`   | (built-in)                    |

---

## Local Development Setup

### 1. Start Infrastructure

```bash
# Core services (PostgreSQL + Redis)
docker compose -f docker/docker-compose.dev.yml up -d

# Add Qdrant vector database
docker compose -f docker/docker-compose.dev.yml --profile vectors up -d

# Optionally add Neo4j for knowledge graph
docker compose -f docker/docker-compose.dev.yml --profile graph up -d

# Optionally add Elasticsearch for full-text search
docker compose -f docker/docker-compose.dev.yml --profile search up -d
```

### 2. Set Environment Variables

| Variable               | Default                  | Used By          |
| ---------------------- | ------------------------ | ---------------- |
| `SOPHIA_DATABASE_URL`  | (from root `.env`)       | database lib     |
| `PORT`                 | `3000`                   | search-api       |
| `HOST`                 | `localhost`              | search-api       |
| `QDRANT_URL`           | `http://localhost:6333`  | vectordb         |
| `QDRANT_API_KEY`       | (none for local)         | vectordb         |
| `MILVUS_ADDRESS`       | `localhost:19530`        | vectordb         |
| `PINECONE_API_KEY`     | (required for Pinecone)  | vectordb         |
| `PINECONE_ENVIRONMENT` | (required for Pinecone)  | vectordb         |
| `NEO4J_URL`            | `bolt://localhost:7687`  | knowledge-graph  |
| `NEO4J_USER`           | `neo4j`                  | knowledge-graph  |
| `NEO4J_PASSWORD`       | `password`               | knowledge-graph  |
| `OPENAI_API_KEY`       | (for embedding provider) | indexing, search |
| `REDIS_URL`            | `redis://localhost:6379` | caching          |

### 3. Generate Prisma Client

```bash
cd libs/sophia/database
npx prisma generate
npx prisma db push
```

### 4. Install Dependencies

```bash
pnpm install
```

### 5. Seed the Database (optional)

```bash
cd libs/sophia/database
npx tsx prisma/seed.ts       # Standard seed data
npx tsx prisma/seed-test.ts  # Test fixtures
```

A Lilith-to-Sophia knowledge migration script is also available:

```bash
npx tsx prisma/migrate-lilith-knowledge.ts
```

---

## Running Services

### Search API

Provides hybrid search (vector + keyword), reranking with cross-encoder models,
RAG (retrieval-augmented generation), and citation extraction.

```bash
# Via Nx
pnpm nx serve @sophia/search-api

# The service starts on PORT (default 3000)
```

In development mode, the search API initializes with mock providers
(`MockEmbeddingProvider`, `MockVectorStore`, `MockTextIndex`). In production,
these are replaced with real implementations.

**API endpoints:**

- `POST /search` - Execute a search query (vector, keyword, or hybrid)
- `POST /rag` - RAG request (retrieve + generate answer)
- `GET /health` - Service health check

**Search strategies:** `vector`, `keyword`, `hybrid` (default)

### Ingestion Service

Processes documents through a pipeline: connector -> extraction -> chunking ->
embedding -> indexing.

```bash
pnpm nx serve @sophia/ingestion
```

The ingestion pipeline consists of:

1. **Connectors** (`src/connectors/`) - Fetch documents from sources
2. **Text Extraction** (`src/extraction/`) - Extract text from various formats
3. **Enrichment Pipeline** (`src/enrichment/`) - Metadata enrichment
4. **Ingestion Pipeline** (`src/pipeline/`) - Orchestrates the full flow

### Knowledge Graph Service

Manages entity extraction, resolution, relation discovery, and graph queries.

```bash
pnpm nx serve @sophia/knowledge-graph
```

Services:

- **Entity Service** (`src/entities/`) - Entity CRUD and management
- **Relation Service** (`src/relations/`) - Relationship extraction and storage
- **Entity Resolver** (`src/resolution/`) - Entity deduplication and linking
- **Graph Service** (`src/graph/`) - Graph traversal and querying

### Workbench (React Web UI)

A React SPA for document management, search, annotations, review, and curation.

```bash
# Development (Vite dev server)
pnpm nx serve @sophia/workbench
```

The workbench uses:

- Vite for build/dev server
- Tailwind CSS for styling
- React Router for navigation
- Pages: Dashboard, Documents, Search, Review, Curation, Annotations

---

## Building

### Build All Sophia Projects

```bash
pnpm nx run-many --target=build --projects=tag:scope:sophia
```

### Build Individual Projects

```bash
pnpm nx build @sophia/search-api
pnpm nx build @sophia/ingestion
pnpm nx build @sophia/knowledge-graph
pnpm nx build @sophia/workbench      # Produces static site
pnpm nx build @sophia/indexing
pnpm nx build @sophia/vectordb
pnpm nx build @sophia/client
```

---

## Testing

### Test Runner

All Sophia projects use Vitest via the `@nx/vite:test` executor.

### Running Tests

```bash
# All Sophia projects
pnpm nx run-many --target=test --projects=tag:scope:sophia

# Individual projects
pnpm nx test @sophia/search-api
pnpm nx test @sophia/ingestion
pnpm nx test @sophia/knowledge-graph
pnpm nx test @sophia/workbench
pnpm nx test @sophia/indexing
pnpm nx test @sophia/vectordb
pnpm nx test @sophia/client
pnpm nx test @sophia/theory
pnpm nx test @sophia/cultural-research
```

### Testing Without External Services

The `libs/sophia/vectordb/` library includes an in-memory vector store
(`memory-store.ts`) that requires no external services. Use it for tests:

```typescript
import { MemoryVectorStore } from '@sophia/vectordb';

const store = new MemoryVectorStore();
// Use for tests without Qdrant/Milvus/Pinecone
```

The search API's `MockEmbeddingProvider`, `MockVectorStore`, and `MockTextIndex`
classes (in `search/hybrid-search.ts`) are used for development and testing
without real ML models or vector databases.

### Linting

```bash
pnpm nx run-many --target=lint --projects=tag:scope:sophia
```

---

## Database Schema

The Sophia database (`libs/sophia/database/prisma/schema.prisma`) contains
models for document management, knowledge entities, and search metadata. The
schema is accessed via the Prisma client generated into `src/generated/client/`.

Key concepts:

- Documents with chunking metadata and embedding references
- Entities with type classification and resolution status
- Relations between entities with confidence scores
- Search query logs for analytics and optimization

---

## Common Development Tasks

### Adding a New Chunking Strategy

The indexing library (`libs/sophia/indexing/`) handles document chunking.

1. Add your chunking strategy in `libs/sophia/indexing/src/chunking/`:

   ```typescript
   import type { ChunkingStrategy, ChunkOptions, Chunk } from './types.js';

   export class SentenceWindowChunker implements ChunkingStrategy {
     async chunk(text: string, options: ChunkOptions): Promise<Chunk[]> {
       // Implement sentence-window chunking
       // Each chunk includes surrounding context sentences
     }
   }
   ```

2. Register the strategy in the chunking factory/index so it can be selected by
   name.

3. Add configuration options to the `ChunkOptions` type if your strategy
   introduces new parameters.

4. Write tests that verify chunk boundaries, overlap behavior, and metadata
   preservation.

### Adding a New Vector Store Backend

1. Create a new store implementation in `libs/sophia/vectordb/src/stores/`:

   ```typescript
   import type {
     VectorStore,
     VectorStoreConfig,
     SearchResult,
   } from '../types.js';

   export class YourVectorStore implements VectorStore {
     constructor(config: VectorStoreConfig) {
       /* ... */
     }

     async upsert(vectors: VectorRecord[]): Promise<void> {
       /* ... */
     }
     async search(query: number[], topK: number): Promise<SearchResult[]> {
       /* ... */
     }
     async delete(ids: string[]): Promise<void> {
       /* ... */
     }
     async healthCheck(): Promise<boolean> {
       /* ... */
     }
   }
   ```

2. Export the new store from `libs/sophia/vectordb/src/stores/index.ts`.

3. Add the backend's SDK to the `package.json` dependencies.

4. Add health check support following the pattern in
   `libs/sophia/vectordb/src/health.ts`.

5. Write tests using the standard vector store test suite. The in-memory store
   serves as the reference implementation.

### Adding a New Document Connector

Connectors fetch documents from external sources for ingestion.

1. Add your connector in `apps/sophia/ingestion/src/connectors/`:

   ```typescript
   export class SlackConnector {
     async connect(config: ConnectionConfig): Promise<void> {
       /* ... */
     }
     async fetchDocuments(options: FetchOptions): Promise<Document[]> {
       /* ... */
     }
     async disconnect(): Promise<void> {
       /* ... */
     }
   }
   ```

2. Register the connector in `connectors.ts` so the ingestion pipeline can use
   it.

3. Add any required API keys to the environment variable documentation.

### Adding a New Embedding Provider

1. Implement the embedding provider interface:

   ```typescript
   export class YourEmbeddingProvider {
     async embed(texts: string[]): Promise<number[][]> {
       /* ... */
     }
     getDimensions(): number {
       /* ... */
     }
     getModelName(): string {
       /* ... */
     }
   }
   ```

2. Integrate it into the indexing pipeline so documents can be embedded using
   your provider.

---

## Debugging Tips

### Search Returning No Results

- Verify the vector store is running and accessible
- Check that documents have been ingested and indexed
- For hybrid search, ensure both vector and text indices contain data
- The mock providers return empty results by default in development

### Qdrant Connection Issues

- Default URL: `http://localhost:6333`
- Start Qdrant via Docker Compose vectors profile
- Check collection existence: Qdrant dashboard at
  `http://localhost:6333/dashboard`
- Verify API key is not required for local instances

### Ingestion Pipeline Failures

- Check connector configuration and credentials
- Text extraction supports multiple formats; verify the extractor for your
  document type
- Monitor the enrichment pipeline for metadata extraction errors
- Check Redis for queue state if using async ingestion

### Knowledge Graph Issues

- Neo4j must be running for graph operations (Docker Compose graph profile)
- Entity resolution may produce false positives; check confidence thresholds
- Graph queries use Cypher; verify query syntax
- Entity resolver deduplication depends on name similarity algorithms

### Workbench Build Issues

- The workbench uses Vite with Tailwind CSS
- Check `tailwind.config.js` and `postcss.config.js` for CSS processing
- The test setup file is at `src/test/setup.ts`
- Keyboard navigation hooks are in `src/hooks/useKeyboardNavigation.ts`

### Performance Optimization

- The vectordb library includes batch utilities (`src/utils/batch.ts`) for bulk
  operations
- Similarity computation utilities are in `src/utils/similarity.ts`
- The indexing library has index builder (`src/index/builder.ts`) and similarity
  tools (`src/index/similarity.ts`)
- The predictions library (`libs/sophia/predictions/`) provides content
  prediction and schedule optimization for proactive indexing
