Eve library

@sophia - Knowledge & Research Platform

Other

Revision 1

On this page

Sophia is Oshun's knowledge management and RAG (Retrieval-Augmented Generation) infrastructure, providing document ingestion, semantic search, and citation management.

Overview#

Aspect Description
Domain Knowledge Management & RAG
Scope Tag scope:sophia
API Port 4003
Database Schema sophia

Capabilities#

  • Document Ingestion: PDF, DOC, HTML, Markdown, plain text
  • Semantic Search: Vector-based similarity search
  • Knowledge Graphs: Entity extraction and relationship mapping
  • Citation Management: Source tracking and attribution
  • Multi-Collection: Organized document collections
  • RAG Pipeline: Context retrieval for LLM prompts

Libraries#

text
libs/sophia/
├── client/              # TypeScript client for other domains
├── database/            # Prisma schema & utilities
├── schemas/             # Document & collection schemas
├── indexing/            # Vector indexing & embeddings
├── vectordb/            # Vector store integration (Qdrant/Pinecone)
├── evaluation/          # Quality metrics & retrieval accuracy
├── theory/              # Framework definitions & models
└── cultural-research/   # Cultural context & reference research

Quick Start#

Using the Client#

typescript
import { createSophiaClient } from '@sophia/client';

const sophia = createSophiaClient({
  baseUrl: process.env.SOPHIA_API_URL || 'http://localhost:4003',
});

// Ingest a document
const document = await sophia.documents.ingest({
  source: 'https://example.com/research.pdf',
  collection: 'research',
  metadata: { author: 'Dr. Smith', year: 2024 },
});

// Search with RAG
const results = await sophia.search({
  query: 'What are the key findings about meditation?',
  collections: ['research'],
  limit: 10,
  includeContext: true,
});

// Use results for RAG
for (const hit of results.hits) {
  console.log(`${hit.content} [${hit.citation}]`);
}

Running Locally#

bash
# Start Sophia API
nx serve sophia-api

# Run tests
nx test @sophia/client

# Build all Sophia libraries
nx run-many --target=build --projects=tag:scope:sophia

API Endpoints#

Method Endpoint Description
POST /v1/documents Ingest document
GET /v1/documents/:id Get document metadata
DELETE /v1/documents/:id Delete document
POST /v1/search Semantic search
GET /v1/collections List collections
POST /v1/collections Create collection
POST /v1/embeddings Generate embeddings

Events#

Event Description
sophia.document.ingested Document fully processed
sophia.document.failed Ingestion failed
sophia.index.updated Collection index rebuilt
sophia.search.completed Search query executed

Configuration#

bash
# Required
SOPHIA_DATABASE_URL=postgresql://...?schema=sophia
REDIS_URL=redis://localhost:6379

# Vector Store
QDRANT_URL=http://localhost:6333
# OR
PINECONE_API_KEY=...
PINECONE_ENVIRONMENT=us-east-1

# Embeddings
OPENAI_API_KEY=sk-...  # For text-embedding-3
# OR
COHERE_API_KEY=...     # For embed-v3

# Storage
DOCUMENT_BUCKET=oshun-documents

Chunking Strategies#

Strategy Use Case
fixed Fixed-size chunks (default 512 tokens)
semantic Sentence-based with overlap
recursive Hierarchical splitting
document Preserve document structure

Local Development#

Start the full development environment:

bash
# Start infrastructure (PostgreSQL, Redis, MinIO)
docker compose -f docker/docker-compose.dev.yml up -d

# Run database migrations
pnpm nx run @sophia/database:prisma:migrate:dev

# Start the Search API
pnpm nx serve sophia-search-api

See CLAUDE.md for detailed local development setup.