Eve library

Sophia API Documentation

Guide

Revision 1

On this page

Sophia provides three services for knowledge management and retrieval:

  1. Search API (port 3000) -- Hybrid semantic search, RAG pipeline, and citation extraction
  2. Ingestion Service (pipeline) -- Multi-stage document ingestion with extraction, enrichment, chunking, and indexing
  3. Knowledge Graph Service (port 3001) -- Entity/relationship CRUD, graph traversal, path finding, and entity resolution

The Search API is the primary external-facing service. The Ingestion Service and Knowledge Graph Service are internal services accessed programmatically or via the @sophia/client SDK.


Search API (Port 3000)#

The Search API provides hybrid search (vector + keyword), RAG (Retrieval-Augmented Generation), cross-encoder reranking, and citation extraction. It is implemented as a SearchAPIService class with request handlers that can be mounted on any HTTP framework.

Base URL#

text
http://localhost:3000

Configuration#

Environment Variable Default Description
PORT 3000 HTTP server port
HOST localhost Bind address

Service Configuration#

Parameter Default Description
defaultTopK 10 Default number of results returned
maxTopK 100 Maximum allowed topK value
enableCitations true Enable citation extraction in RAG responses
enableReranking true Enable cross-encoder reranking

Search Endpoint#

POST /search (or programmatic service.search(query))#

Execute a hybrid search query combining dense vector similarity and sparse BM25 keyword matching.

Request body:

json
{
  "text": "meditation techniques for anxiety",
  "strategy": "hybrid",
  "topK": 10,
  "threshold": 0.5,
  "filters": {
    "documentIds": ["doc_123"],
    "sources": ["sacred-texts"],
    "languages": ["en"],
    "dateRange": { "from": "2020-01-01", "to": "2026-01-01" },
    "tags": ["meditation", "mindfulness"],
    "metadata": { "tradition": "buddhism" }
  },
  "options": {
    "includeMetadata": true,
    "includeHighlights": true,
    "includeContext": true,
    "expandQuery": false,
    "rerank": true,
    "deduplicate": true,
    "hybridWeights": { "dense": 0.7, "sparse": 0.3 }
  }
}
Field Type Required Description
text string Yes Search query text
strategy enum No Retrieval strategy (default: hybrid)
topK int No Number of results (default: 10, max: 100)
threshold number No Minimum score threshold (0-1)
filters.documentIds string[] No Restrict to specific document IDs
filters.sources string[] No Filter by source identifiers
filters.languages string[] No Filter by language codes
filters.dateRange object No Filter by date range (from, to)
filters.tags string[] No Filter by tags
filters.metadata object No Filter by arbitrary metadata fields
options.includeMetadata boolean No Include result metadata (source, title, authors, etc.)
options.includeHighlights boolean No Include highlighted text spans
options.includeContext boolean No Include surrounding context text
options.expandQuery boolean No Enable query expansion to multiple queries
options.rerank boolean No Enable cross-encoder reranking
options.deduplicate boolean No Remove duplicate/near-duplicate results
options.hybridWeights object No Custom weights for dense/sparse fusion

Retrieval strategies:

Strategy Description
dense Vector similarity search only (cosine similarity)
sparse BM25 keyword search only
hybrid Combined dense + sparse with configurable weights
rerank Two-stage: initial retrieval followed by cross-encoder reranking
multi-query Query expansion: generate multiple query variants and merge results

Response (200):

json
{
  "queryId": "qry_abc123",
  "query": "meditation techniques for anxiety",
  "strategy": "hybrid",
  "results": [
    {
      "resultId": "res_001",
      "chunkId": "chk_456",
      "documentId": "doc_789",
      "text": "Mindfulness meditation has been shown to reduce anxiety by...",
      "score": 0.92,
      "rank": 1,
      "metadata": {
        "source": "sacred-texts",
        "title": "The Practice of Mindfulness",
        "authors": ["Thich Nhat Hanh"],
        "date": "2015-03-15",
        "language": "en",
        "pageNumber": 42,
        "sectionTitle": "Chapter 3: Working with Anxiety"
      },
      "highlights": [
        {
          "text": "meditation",
          "startOffset": 12,
          "endOffset": 22,
          "matchType": "exact"
        },
        {
          "text": "reduce anxiety",
          "startOffset": 45,
          "endOffset": 59,
          "matchType": "semantic"
        }
      ],
      "context": {
        "before": "In this section, we explore how...",
        "after": "...leading to measurable reductions in cortisol.",
        "documentTitle": "The Practice of Mindfulness",
        "chapterTitle": "Working with Anxiety"
      }
    }
  ],
  "totalResults": 47,
  "hasMore": true,
  "processingTimeMs": 125,
  "metadata": {
    "queryExpansions": ["anxiety meditation", "mindfulness anxiety reduction"],
    "appliedFilters": { "languages": ["en"] },
    "warnings": []
  }
}

RAG Endpoint#

POST /rag (or programmatic service.rag(request))#

Execute a retrieval-augmented generation request: search for relevant documents, then generate an answer with citations.

Request body:

json
{
  "query": "What are the main meditation techniques for managing anxiety?",
  "topK": 10,
  "strategy": "hybrid",
  "systemPrompt": "You are a knowledgeable meditation teacher.",
  "maxTokens": 1000,
  "includeCitations": true,
  "citationOptions": {
    "minConfidence": 0.7,
    "maxCitations": 5,
    "includeGraph": false,
    "mustCiteThreshold": 0.8
  }
}
Field Type Required Description
query string Yes User's question
topK int No Number of source documents to retrieve
strategy enum No Retrieval strategy (default: hybrid)
systemPrompt string No System prompt for the LLM
maxTokens int No Maximum tokens for the generated answer
includeCitations boolean No Include citation extraction
citationOptions.minConfidence number No Minimum citation confidence (0-1)
citationOptions.maxCitations int No Maximum number of citations
citationOptions.includeGraph boolean No Include citation authority graph
citationOptions.mustCiteThreshold number No Score above which citation is required

Response (200):

json
{
  "queryId": "rag_abc123",
  "query": "What are the main meditation techniques for managing anxiety?",
  "answer": "Based on the search results, here's what I found about...",
  "citations": [
    {
      "citationId": "cit_001",
      "text": "Mindfulness meditation has been shown to reduce anxiety",
      "sourceChunkId": "chk_456",
      "documentId": "doc_789",
      "confidence": 0.95,
      "position": { "startOffset": 42, "endOffset": 96 },
      "metadata": {
        "source": "sacred-texts",
        "title": "The Practice of Mindfulness"
      }
    }
  ],
  "sources": [ { "resultId": "...", "chunkId": "...", "score": 0.92, ... } ],
  "processingTimeMs": 450,
  "metadata": {
    "retrievalStrategy": "hybrid"
  }
}

Health Endpoint#

GET /health (or programmatic service.healthCheck())#

Response (200):

json
{
  "status": "healthy",
  "uptime": 3600.5
}

Ingestion Service (Pipeline)#

The Ingestion Service is a multi-stage pipeline that processes documents from raw source to indexed, searchable chunks. It is accessed programmatically via the IngestionPipeline class or the @sophia/client IngestionClient.

Pipeline Stages#

text
Fetch -> Extract -> Enrich -> Chunk -> Index
Stage Description
fetching Retrieve content from source (file, URL, API, database, stream)
extracting Extract text, structure, images, tables, footnotes from document
enriching Entity extraction, metadata inference, summary generation
chunking Split document into chunks using configured strategy
indexing Generate embeddings and index chunks in vector store

Programmatic API#

Submit an ingestion job#

typescript
import { createIngestionPipeline } from '@sophia/ingestion';

const pipeline = createIngestionPipeline({
  /* config */
});

const job = await pipeline.submit({
  sourceConfig: {
    sourceId: createSourceId('src_001'),
    type: 'url',
    uri: 'https://example.com/document.pdf',
    credentials: {
      type: 'bearer',
      token: 'access-token',
    },
    options: {
      timeout: 30000,
      maxSize: 52428800, // 50MB
    },
  },
  extractionConfig: {
    preserveMarkup: false,
    extractImages: true,
    extractTables: true,
    extractFootnotes: true,
    maxPages: 500,
  },
  ocrConfig: {
    enabled: true,
    language: ['en', 'sa'],
    confidence: 0.8,
    preserveLayout: true,
    deskew: true,
  },
  enrichmentConfig: {
    extractEntities: true,
    entityTypes: ['person', 'concept', 'practice', 'location', 'event'],
    generateSummary: true,
    detectLanguage: true,
    extractKeywords: true,
  },
  chunkingConfig: {
    strategy: 'semantic',
    chunkSize: 512,
    chunkOverlap: 64,
    minChunkSize: 100,
  },
  priority: 5,
  tags: ['buddhism', 'meditation'],
});

Source Types#

Source Type URI Example Description
file /path/to/document.pdf Local filesystem
url https://example.com/doc.html HTTP/HTTPS URLs
api https://api.example.com/v1/texts REST API endpoints
database postgresql://host/db?query=... Database queries
stream kafka://topic/partition Streaming sources
archive /path/to/archive.zip ZIP/TAR archives

Supported Document Formats#

Format Extensions OCR Support
pdf .pdf Yes
epub .epub No
html .html, .htm No
markdown .md, .markdown No
plain-text .txt No
tei-xml .xml (TEI schema) No
docx .docx No
xlsx .xlsx No
pptx .pptx No
image .png, .jpg, .tiff Yes

Chunking Strategies#

Strategy Description
fixed Fixed-size chunks by token count
paragraph Split on paragraph boundaries
sentence Split on sentence boundaries
semantic Split on semantic boundaries using embedding similarity
recursive Recursive splitting with fallback strategies

Job Statuses#

Status Description
pending Job created, not yet started
fetching Retrieving content from source
extracting Extracting text and structure
enriching Running entity extraction, summarization
chunking Splitting document into chunks
indexing Generating embeddings and indexing
completed Pipeline finished successfully
failed Pipeline failed (see error for details)
cancelled Pipeline was cancelled

Error Types#

Error Class Code Stage Retryable Description
FetchError FETCH_ERROR fetch Yes Source retrieval failed
ExtractionError EXTRACTION_ERROR extract No Text extraction failed
OCRError OCR_ERROR ocr Yes OCR processing failed
EnrichmentError ENRICHMENT_ERROR enrich Yes Metadata/entity extraction failed
IngestionError (various) (various) Varies General pipeline error

Knowledge Graph Service (Port 3001)#

The Knowledge Graph Service provides entity and relationship CRUD, graph traversal (BFS/DFS), shortest path finding, subgraph extraction, entity resolution, and entity merging. It is accessed programmatically via the KnowledgeGraphService class or the @sophia/client KnowledgeGraphClient.

Programmatic API#

Create nodes#

typescript
import { createKnowledgeGraphService } from '@sophia/knowledge-graph';

const kg = createKnowledgeGraphService({
  /* config */
});

// Create a concept node
const concept = await kg.createNode({
  type: 'concept',
  labels: ['meditation', 'core-concept'],
  properties: {
    name: 'Vipassana',
    definition: 'Insight meditation focusing on the true nature of reality',
    tradition: 'Theravada Buddhism',
    category: 'meditation-technique',
    complexityLevel: 'intermediate',
    relatedTerms: ['mindfulness', 'satipatthana', 'insight'],
  },
});

Create relationships#

typescript
const relationship = await kg.createRelationship({
  type: 'BELONGS_TO',
  sourceId: vipassanaNodeId,
  targetId: theravadaNodeId,
  properties: {
    weight: 0.95,
    context: 'Vipassana is a core practice of Theravada Buddhism',
    evidence: 'Satipatthana Sutta',
  },
});

Query nodes#

typescript
const results = await kg.queryNodes({
  filters: {
    type: 'concept',
    searchText: 'meditation',
    properties: { tradition: 'buddhism' },
  },
  pagination: {
    limit: 20,
    offset: 0,
    sortBy: 'name',
    sortDirection: 'asc',
  },
  includeRelationships: true,
});

Graph traversal#

typescript
const traversal = await kg.traverse({
  startNodeId: vipassanaNodeId,
  relationshipTypes: ['BELONGS_TO', 'DERIVES_FROM', 'RELATED_TO'],
  direction: 'both',
  maxDepth: 3,
  nodeTypes: ['concept', 'practice', 'tradition'],
  includeProperties: true,
  limit: 50,
});

Path finding#

typescript
const path = await kg.findShortestPath({
  sourceId: vipassanaNodeId,
  targetId: zenNodeId,
  relationshipTypes: ['DERIVES_FROM', 'INFLUENCED_BY', 'RELATED_TO'],
  maxDepth: 6,
  weighted: true,
});

Subgraph extraction#

typescript
const subgraph = await kg.extractSubgraph({
  centerNodeId: vipassanaNodeId,
  radius: 2,
  nodeTypes: ['concept', 'practice', 'figure'],
  maxNodes: 100,
});

Entity resolution#

typescript
const resolution = await kg.resolveEntity({
  name: 'Siddhartha Gautama',
  type: 'figure',
  aliases: ['Buddha', 'Shakyamuni', 'The Blessed One'],
  maxCandidates: 5,
  minSimilarity: 0.7,
});

Node Types#

Type Description Key Properties
tradition Religious/spiritual tradition originRegion, timePeriod, keyPrinciples, foundingFigures
text Sacred or scholarly text title, author, language, year, genre, sourceUrl
concept Philosophical/theological concept definition, tradition, category, complexityLevel
practice Spiritual/contemplative practice tradition, difficulty, prerequisites, duration
figure Historical or contemporary figure birthYear, deathYear, traditions, bioSummary, titles
place Geographically significant location country, region, coordinates, significance
event Historical or recurring event date, startDate, endDate, location, participants
organization Institution or organization foundedYear, headquarters, type, members
custom User-defined node type Any BaseNodeProperties

Relationship Types#

Category Types
Figure relationships TAUGHT, WROTE, LIVED_IN, STUDIED_UNDER, FOUNDED
Concept relationships ALIGNS_WITH, CONTRADICTS, DERIVES_FROM, REFINES, EXTENDS
Practice relationships BELONGS_TO, REQUIRES, PRECEDES, FOLLOWS
Text relationships DISCUSSES, REFERENCES, CITES, ORIGINATES_FROM, TRANSLATED_FROM
Place relationships ASSOCIATED_WITH, SACRED_TO, LOCATED_IN
Cross-domain EXEMPLIFIES, INFLUENCED_BY, RELATED_TO, PART_OF, MEMBER_OF, PARTICIPATED_IN

Traversal Options#

Option Type Default Description
startNodeId NodeId (required) Starting node for traversal
relationshipTypes RelationshipType[] all Filter by relationship types
direction enum outgoing outgoing, incoming, or both
maxDepth int 3 Maximum traversal depth
nodeTypes NodeType[] all Filter by node types
includeProperties boolean false Include full node properties
limit int 100 Maximum nodes to return

Pagination#

All list/query operations support pagination:

Parameter Type Default Description
limit int 20 Items per page
offset int 0 Number of items to skip
sortBy string - Field to sort by
sortDirection enum asc asc or desc

Error Types#

Error Class Code Description
NodeNotFoundError NODE_NOT_FOUND Node ID not found
RelationshipNotFoundError RELATIONSHIP_NOT_FOUND Relationship ID not found
InvalidRelationshipError INVALID_RELATIONSHIP Invalid relationship definition
TraversalLimitError TRAVERSAL_LIMIT_EXCEEDED Traversal exceeded depth/count limits
GraphError (various) General graph error

Statistics#

typescript
const stats = await kg.getStatistics();
// {
//   totalNodes: 15420,
//   totalRelationships: 42180,
//   nodesByType: { concept: 5200, figure: 3100, text: 2800, ... },
//   relationshipsByType: { RELATED_TO: 8500, DISCUSSES: 6200, ... },
//   averageDegree: 5.47,
//   density: 0.00035,
//   connectedComponents: 12,
//   largestComponentSize: 14800
// }

Client SDK#

The @sophia/client package provides a unified TypeScript client:

typescript
import {
  SophiaClient,
  SearchClient,
  IngestionClient,
  KnowledgeGraphClient,
  SearchQueryBuilder,
  GraphQueryBuilder,
} from '@sophia/client';

// Unified client
const sophia = new SophiaClient({
  searchUrl: 'http://localhost:3000',
  knowledgeGraphUrl: 'http://localhost:3001',
});

// Or individual clients
const search = new SearchClient('http://localhost:3000');
const ingestion = new IngestionClient();
const graph = new KnowledgeGraphClient('http://localhost:3001');

// Query builders
const query = new SearchQueryBuilder()
  .text('meditation anxiety')
  .strategy('hybrid')
  .topK(10)
  .filter('languages', ['en'])
  .option('rerank', true)
  .build();

const results = await search.search(query);