Sophia is Oshun's knowledge management and RAG (Retrieval-Augmented Generation) infrastructure, providing document ingestion, semantic search, and citation management.
Overview#
| Aspect | Description |
|---|---|
| Domain | Knowledge Management & RAG |
| Scope Tag | scope:sophia |
| API Port | 4003 |
| Database Schema | sophia |
Capabilities#
- Document Ingestion: PDF, DOC, HTML, Markdown, plain text
- Semantic Search: Vector-based similarity search
- Knowledge Graphs: Entity extraction and relationship mapping
- Citation Management: Source tracking and attribution
- Multi-Collection: Organized document collections
- RAG Pipeline: Context retrieval for LLM prompts
Libraries#
text
libs/sophia/
├── client/ # TypeScript client for other domains
├── database/ # Prisma schema & utilities
├── schemas/ # Document & collection schemas
├── indexing/ # Vector indexing & embeddings
├── vectordb/ # Vector store integration (Qdrant/Pinecone)
├── evaluation/ # Quality metrics & retrieval accuracy
├── theory/ # Framework definitions & models
└── cultural-research/ # Cultural context & reference research
Quick Start#
Using the Client#
typescript
import { createSophiaClient } from '@sophia/client';
const sophia = createSophiaClient({
baseUrl: process.env.SOPHIA_API_URL || 'http://localhost:4003',
});
// Ingest a document
const document = await sophia.documents.ingest({
source: 'https://example.com/research.pdf',
collection: 'research',
metadata: { author: 'Dr. Smith', year: 2024 },
});
// Search with RAG
const results = await sophia.search({
query: 'What are the key findings about meditation?',
collections: ['research'],
limit: 10,
includeContext: true,
});
// Use results for RAG
for (const hit of results.hits) {
console.log(`${hit.content} [${hit.citation}]`);
}
Running Locally#
bash
# Start Sophia API
nx serve sophia-api
# Run tests
nx test @sophia/client
# Build all Sophia libraries
nx run-many --target=build --projects=tag:scope:sophia
API Endpoints#
| Method | Endpoint | Description |
|---|---|---|
| POST | /v1/documents |
Ingest document |
| GET | /v1/documents/:id |
Get document metadata |
| DELETE | /v1/documents/:id |
Delete document |
| POST | /v1/search |
Semantic search |
| GET | /v1/collections |
List collections |
| POST | /v1/collections |
Create collection |
| POST | /v1/embeddings |
Generate embeddings |
Events#
| Event | Description |
|---|---|
sophia.document.ingested |
Document fully processed |
sophia.document.failed |
Ingestion failed |
sophia.index.updated |
Collection index rebuilt |
sophia.search.completed |
Search query executed |
Configuration#
bash
# Required
SOPHIA_DATABASE_URL=postgresql://...?schema=sophia
REDIS_URL=redis://localhost:6379
# Vector Store
QDRANT_URL=http://localhost:6333
# OR
PINECONE_API_KEY=...
PINECONE_ENVIRONMENT=us-east-1
# Embeddings
OPENAI_API_KEY=sk-... # For text-embedding-3
# OR
COHERE_API_KEY=... # For embed-v3
# Storage
DOCUMENT_BUCKET=oshun-documents
Chunking Strategies#
| Strategy | Use Case |
|---|---|
fixed |
Fixed-size chunks (default 512 tokens) |
semantic |
Sentence-based with overlap |
recursive |
Hierarchical splitting |
document |
Preserve document structure |
Local Development#
Start the full development environment:
bash
# Start infrastructure (PostgreSQL, Redis, MinIO)
docker compose -f docker/docker-compose.dev.yml up -d
# Run database migrations
pnpm nx run @sophia/database:prisma:migrate:dev
# Start the Search API
pnpm nx serve sophia-search-api
See CLAUDE.md for detailed local development setup.