Sophia provides three services for knowledge management and retrieval:
Search API (port 3000) -- Hybrid semantic search, RAG pipeline, and
citation extraction
Ingestion Service (pipeline) -- Multi-stage document ingestion with
extraction, enrichment, chunking, and indexing
Knowledge Graph Service (port 3001) -- Entity/relationship CRUD, graph
traversal, path finding, and entity resolution
The Search API is the primary external-facing service. The Ingestion Service and
Knowledge Graph Service are internal services accessed programmatically or via
the @sophia/client SDK.
Search API (Port 3000)#
The Search API provides hybrid search (vector + keyword), RAG
(Retrieval-Augmented Generation), cross-encoder reranking, and citation
extraction. It is implemented as a SearchAPIService class with request
handlers that can be mounted on any HTTP framework.
Base URL#
text Copy
http://localhost:3000
Configuration#
Environment Variable
Default
Description
PORT
3000
HTTP server port
HOST
localhost
Bind address
Service Configuration#
Parameter
Default
Description
defaultTopK
10
Default number of results returned
maxTopK
100
Maximum allowed topK value
enableCitations
true
Enable citation extraction in RAG responses
enableReranking
true
Enable cross-encoder reranking
Search Endpoint#
POST /search (or programmatic service.search(query))#
Execute a hybrid search query combining dense vector similarity and sparse BM25
keyword matching.
Request body:
json Copy
{
"text": "meditation techniques for anxiety",
"strategy": "hybrid",
"topK": 10,
"threshold": 0.5,
"filters": {
"documentIds": ["doc_123"],
"sources": ["sacred-texts"],
"languages": ["en"],
"dateRange": { "from": "2020-01-01", "to": "2026-01-01" },
"tags": ["meditation", "mindfulness"],
"metadata": { "tradition": "buddhism" }
},
"options": {
"includeMetadata": true,
"includeHighlights": true,
"includeContext": true,
"expandQuery": false,
"rerank": true,
"deduplicate": true,
"hybridWeights": { "dense": 0.7, "sparse": 0.3 }
}
}
Field
Type
Required
Description
text
string
Yes
Search query text
strategy
enum
No
Retrieval strategy (default: hybrid)
topK
int
No
Number of results (default: 10, max: 100)
threshold
number
No
Minimum score threshold (0-1)
filters.documentIds
string[]
No
Restrict to specific document IDs
filters.sources
string[]
No
Filter by source identifiers
filters.languages
string[]
No
Filter by language codes
filters.dateRange
object
No
Filter by date range (from, to)
filters.tags
string[]
No
Filter by tags
filters.metadata
object
No
Filter by arbitrary metadata fields
options.includeMetadata
boolean
No
Include result metadata (source, title, authors, etc.)
options.includeHighlights
boolean
No
Include highlighted text spans
options.includeContext
boolean
No
Include surrounding context text
options.expandQuery
boolean
No
Enable query expansion to multiple queries
options.rerank
boolean
No
Enable cross-encoder reranking
options.deduplicate
boolean
No
Remove duplicate/near-duplicate results
options.hybridWeights
object
No
Custom weights for dense/sparse fusion
Retrieval strategies:
Strategy
Description
dense
Vector similarity search only (cosine similarity)
sparse
BM25 keyword search only
hybrid
Combined dense + sparse with configurable weights
rerank
Two-stage: initial retrieval followed by cross-encoder reranking
multi-query
Query expansion: generate multiple query variants and merge results
Response (200):
json Copy
{
"queryId": "qry_abc123",
"query": "meditation techniques for anxiety",
"strategy": "hybrid",
"results": [
{
"resultId": "res_001",
"chunkId": "chk_456",
"documentId": "doc_789",
"text": "Mindfulness meditation has been shown to reduce anxiety by...",
"score": 0.92,
"rank": 1,
"metadata": {
"source": "sacred-texts",
"title": "The Practice of Mindfulness",
"authors": ["Thich Nhat Hanh"],
"date": "2015-03-15",
"language": "en",
"pageNumber": 42,
"sectionTitle": "Chapter 3: Working with Anxiety"
},
"highlights": [
{
"text": "meditation",
"startOffset": 12,
"endOffset": 22,
"matchType": "exact"
},
{
"text": "reduce anxiety",
"startOffset": 45,
"endOffset": 59,
"matchType": "semantic"
}
],
"context": {
"before": "In this section, we explore how...",
"after": "...leading to measurable reductions in cortisol.",
"documentTitle": "The Practice of Mindfulness",
"chapterTitle": "Working with Anxiety"
}
}
],
"totalResults": 47,
"hasMore": true,
"processingTimeMs": 125,
"metadata": {
"queryExpansions": ["anxiety meditation", "mindfulness anxiety reduction"],
"appliedFilters": { "languages": ["en"] },
"warnings": []
}
}
RAG Endpoint#
POST /rag (or programmatic service.rag(request))#
Execute a retrieval-augmented generation request: search for relevant documents,
then generate an answer with citations.
Request body:
json Copy
{
"query": "What are the main meditation techniques for managing anxiety?",
"topK": 10,
"strategy": "hybrid",
"systemPrompt": "You are a knowledgeable meditation teacher.",
"maxTokens": 1000,
"includeCitations": true,
"citationOptions": {
"minConfidence": 0.7,
"maxCitations": 5,
"includeGraph": false,
"mustCiteThreshold": 0.8
}
}
Field
Type
Required
Description
query
string
Yes
User's question
topK
int
No
Number of source documents to retrieve
strategy
enum
No
Retrieval strategy (default: hybrid)
systemPrompt
string
No
System prompt for the LLM
maxTokens
int
No
Maximum tokens for the generated answer
includeCitations
boolean
No
Include citation extraction
citationOptions.minConfidence
number
No
Minimum citation confidence (0-1)
citationOptions.maxCitations
int
No
Maximum number of citations
citationOptions.includeGraph
boolean
No
Include citation authority graph
citationOptions.mustCiteThreshold
number
No
Score above which citation is required
Response (200):
json Copy
{
"queryId": "rag_abc123",
"query": "What are the main meditation techniques for managing anxiety?",
"answer": "Based on the search results, here's what I found about...",
"citations": [
{
"citationId": "cit_001",
"text": "Mindfulness meditation has been shown to reduce anxiety",
"sourceChunkId": "chk_456",
"documentId": "doc_789",
"confidence": 0.95,
"position": { "startOffset": 42, "endOffset": 96 },
"metadata": {
"source": "sacred-texts",
"title": "The Practice of Mindfulness"
}
}
],
"sources": [ { "resultId": "...", "chunkId": "...", "score": 0.92, ... } ],
"processingTimeMs": 450,
"metadata": {
"retrievalStrategy": "hybrid"
}
}
Health Endpoint#
GET /health (or programmatic service.healthCheck())#
Response (200):
json Copy
{
"status": "healthy",
"uptime": 3600.5
}
Ingestion Service (Pipeline)#
The Ingestion Service is a multi-stage pipeline that processes documents from
raw source to indexed, searchable chunks. It is accessed programmatically via
the IngestionPipeline class or the @sophia/client IngestionClient.
Pipeline Stages#
text Copy
Fetch -> Extract -> Enrich -> Chunk -> Index
Stage
Description
fetching
Retrieve content from source (file, URL, API, database, stream)
extracting
Extract text, structure, images, tables, footnotes from document
enriching
Entity extraction, metadata inference, summary generation
chunking
Split document into chunks using configured strategy
indexing
Generate embeddings and index chunks in vector store
Programmatic API#
Submit an ingestion job#
typescript Copy
import { createIngestionPipeline } from '@sophia/ingestion';
const pipeline = createIngestionPipeline({
/* config */
});
const job = await pipeline.submit({
sourceConfig: {
sourceId: createSourceId('src_001'),
type: 'url',
uri: 'https://example.com/document.pdf',
credentials: {
type: 'bearer',
token: 'access-token',
},
options: {
timeout: 30000,
maxSize: 52428800, // 50MB
},
},
extractionConfig: {
preserveMarkup: false,
extractImages: true,
extractTables: true,
extractFootnotes: true,
maxPages: 500,
},
ocrConfig: {
enabled: true,
language: ['en', 'sa'],
confidence: 0.8,
preserveLayout: true,
deskew: true,
},
enrichmentConfig: {
extractEntities: true,
entityTypes: ['person', 'concept', 'practice', 'location', 'event'],
generateSummary: true,
detectLanguage: true,
extractKeywords: true,
},
chunkingConfig: {
strategy: 'semantic',
chunkSize: 512,
chunkOverlap: 64,
minChunkSize: 100,
},
priority: 5,
tags: ['buddhism', 'meditation'],
});
Source Types#
Source Type
URI Example
Description
file
/path/to/document.pdf
Local filesystem
url
https://example.com/doc.html
HTTP/HTTPS URLs
api
https://api.example.com/v1/texts
REST API endpoints
database
postgresql://host/db?query=...
Database queries
stream
kafka://topic/partition
Streaming sources
archive
/path/to/archive.zip
ZIP/TAR archives
Format
Extensions
OCR Support
pdf
.pdf
Yes
epub
.epub
No
html
.html, .htm
No
markdown
.md, .markdown
No
plain-text
.txt
No
tei-xml
.xml (TEI schema)
No
docx
.docx
No
xlsx
.xlsx
No
pptx
.pptx
No
image
.png, .jpg, .tiff
Yes
Chunking Strategies#
Strategy
Description
fixed
Fixed-size chunks by token count
paragraph
Split on paragraph boundaries
sentence
Split on sentence boundaries
semantic
Split on semantic boundaries using embedding similarity
recursive
Recursive splitting with fallback strategies
Job Statuses#
Status
Description
pending
Job created, not yet started
fetching
Retrieving content from source
extracting
Extracting text and structure
enriching
Running entity extraction, summarization
chunking
Splitting document into chunks
indexing
Generating embeddings and indexing
completed
Pipeline finished successfully
failed
Pipeline failed (see error for details)
cancelled
Pipeline was cancelled
Error Types#
Error Class
Code
Stage
Retryable
Description
FetchError
FETCH_ERROR
fetch
Yes
Source retrieval failed
ExtractionError
EXTRACTION_ERROR
extract
No
Text extraction failed
OCRError
OCR_ERROR
ocr
Yes
OCR processing failed
EnrichmentError
ENRICHMENT_ERROR
enrich
Yes
Metadata/entity extraction failed
IngestionError
(various)
(various)
Varies
General pipeline error
Knowledge Graph Service (Port 3001)#
The Knowledge Graph Service provides entity and relationship CRUD, graph
traversal (BFS/DFS), shortest path finding, subgraph extraction, entity
resolution, and entity merging. It is accessed programmatically via the
KnowledgeGraphService class or the @sophia/client KnowledgeGraphClient.
Programmatic API#
Create nodes#
typescript Copy
import { createKnowledgeGraphService } from '@sophia/knowledge-graph';
const kg = createKnowledgeGraphService({
/* config */
});
// Create a concept node
const concept = await kg.createNode({
type: 'concept',
labels: ['meditation', 'core-concept'],
properties: {
name: 'Vipassana',
definition: 'Insight meditation focusing on the true nature of reality',
tradition: 'Theravada Buddhism',
category: 'meditation-technique',
complexityLevel: 'intermediate',
relatedTerms: ['mindfulness', 'satipatthana', 'insight'],
},
});
Create relationships#
typescript Copy
const relationship = await kg.createRelationship({
type: 'BELONGS_TO',
sourceId: vipassanaNodeId,
targetId: theravadaNodeId,
properties: {
weight: 0.95,
context: 'Vipassana is a core practice of Theravada Buddhism',
evidence: 'Satipatthana Sutta',
},
});
Query nodes#
typescript Copy
const results = await kg.queryNodes({
filters: {
type: 'concept',
searchText: 'meditation',
properties: { tradition: 'buddhism' },
},
pagination: {
limit: 20,
offset: 0,
sortBy: 'name',
sortDirection: 'asc',
},
includeRelationships: true,
});
Graph traversal#
typescript Copy
const traversal = await kg.traverse({
startNodeId: vipassanaNodeId,
relationshipTypes: ['BELONGS_TO', 'DERIVES_FROM', 'RELATED_TO'],
direction: 'both',
maxDepth: 3,
nodeTypes: ['concept', 'practice', 'tradition'],
includeProperties: true,
limit: 50,
});
Path finding#
typescript Copy
const path = await kg.findShortestPath({
sourceId: vipassanaNodeId,
targetId: zenNodeId,
relationshipTypes: ['DERIVES_FROM', 'INFLUENCED_BY', 'RELATED_TO'],
maxDepth: 6,
weighted: true,
});
typescript Copy
const subgraph = await kg.extractSubgraph({
centerNodeId: vipassanaNodeId,
radius: 2,
nodeTypes: ['concept', 'practice', 'figure'],
maxNodes: 100,
});
Entity resolution#
typescript Copy
const resolution = await kg.resolveEntity({
name: 'Siddhartha Gautama',
type: 'figure',
aliases: ['Buddha', 'Shakyamuni', 'The Blessed One'],
maxCandidates: 5,
minSimilarity: 0.7,
});
Node Types#
Type
Description
Key Properties
tradition
Religious/spiritual tradition
originRegion, timePeriod, keyPrinciples, foundingFigures
text
Sacred or scholarly text
title, author, language, year, genre, sourceUrl
concept
Philosophical/theological concept
definition, tradition, category, complexityLevel
practice
Spiritual/contemplative practice
tradition, difficulty, prerequisites, duration
figure
Historical or contemporary figure
birthYear, deathYear, traditions, bioSummary, titles
place
Geographically significant location
country, region, coordinates, significance
event
Historical or recurring event
date, startDate, endDate, location, participants
organization
Institution or organization
foundedYear, headquarters, type, members
custom
User-defined node type
Any BaseNodeProperties
Relationship Types#
Category
Types
Figure relationships
TAUGHT, WROTE, LIVED_IN, STUDIED_UNDER, FOUNDED
Concept relationships
ALIGNS_WITH, CONTRADICTS, DERIVES_FROM, REFINES, EXTENDS
Practice relationships
BELONGS_TO, REQUIRES, PRECEDES, FOLLOWS
Text relationships
DISCUSSES, REFERENCES, CITES, ORIGINATES_FROM, TRANSLATED_FROM
Place relationships
ASSOCIATED_WITH, SACRED_TO, LOCATED_IN
Cross-domain
EXEMPLIFIES, INFLUENCED_BY, RELATED_TO, PART_OF, MEMBER_OF, PARTICIPATED_IN
Traversal Options#
Option
Type
Default
Description
startNodeId
NodeId
(required)
Starting node for traversal
relationshipTypes
RelationshipType[]
all
Filter by relationship types
direction
enum
outgoing
outgoing, incoming, or both
maxDepth
int
3
Maximum traversal depth
nodeTypes
NodeType[]
all
Filter by node types
includeProperties
boolean
false
Include full node properties
limit
int
100
Maximum nodes to return
All list/query operations support pagination:
Parameter
Type
Default
Description
limit
int
20
Items per page
offset
int
0
Number of items to skip
sortBy
string
-
Field to sort by
sortDirection
enum
asc
asc or desc
Error Types#
Error Class
Code
Description
NodeNotFoundError
NODE_NOT_FOUND
Node ID not found
RelationshipNotFoundError
RELATIONSHIP_NOT_FOUND
Relationship ID not found
InvalidRelationshipError
INVALID_RELATIONSHIP
Invalid relationship definition
TraversalLimitError
TRAVERSAL_LIMIT_EXCEEDED
Traversal exceeded depth/count limits
GraphError
(various)
General graph error
Statistics#
typescript Copy
const stats = await kg.getStatistics();
// {
// totalNodes: 15420,
// totalRelationships: 42180,
// nodesByType: { concept: 5200, figure: 3100, text: 2800, ... },
// relationshipsByType: { RELATED_TO: 8500, DISCUSSES: 6200, ... },
// averageDegree: 5.47,
// density: 0.00035,
// connectedComponents: 12,
// largestComponentSize: 14800
// }
Client SDK#
The @sophia/client package provides a unified TypeScript client:
typescript Copy
import {
SophiaClient,
SearchClient,
IngestionClient,
KnowledgeGraphClient,
SearchQueryBuilder,
GraphQueryBuilder,
} from '@sophia/client';
// Unified client
const sophia = new SophiaClient({
searchUrl: 'http://localhost:3000',
knowledgeGraphUrl: 'http://localhost:3001',
});
// Or individual clients
const search = new SearchClient('http://localhost:3000');
const ingestion = new IngestionClient();
const graph = new KnowledgeGraphClient('http://localhost:3001');
// Query builders
const query = new SearchQueryBuilder()
.text('meditation anxiety')
.strategy('hybrid')
.topK(10)
.filter('languages', ['en'])
.option('rerank', true)
.build();
const results = await search.search(query);