Overview#
This document describes the content ingestion pipeline for adding wisdom texts, meditation research, philosophy texts, and other source materials to Lilith's Sophia-backed retrieval stack. The system supports crawling, parsing, chunking, embedding, and indexing content for semantic search and knowledge graph construction.
Table of Contents#
- Architecture Overview
- Source Discovery
- Document Crawling
- Document Processing
- Vector Embeddings
- Knowledge Graph Integration
- Citation Metadata
- Quality Assurance
Architecture Overview#
Pipeline Flow#
Source Discovery → Crawling → Parsing → Chunking → Embedding → Storage
↓
Entity Extraction → Graph Storage
Services Involved#
| Service | Role | Location |
|---|---|---|
| Sophia Ingestion | Source ingestion, extraction, chunking, enrichment | apps/sophia/ingestion/ |
| Sophia Search API | Search, retrieval, and document read APIs | apps/sophia/search-api/ |
| Sophia Knowledge Graph | Entity and relationship storage/querying | apps/sophia/knowledge-graph/ |
Data Stores#
| Store | Technology | Purpose |
|---|---|---|
| Document Store | PostgreSQL | Full text storage, metadata |
| Vector Store | Milvus | Semantic embeddings |
| Graph Store | Neo4j | Knowledge graph relationships |
| Time Series | TimescaleDB | Crawl and access metrics |
Source Discovery#
Source Service#
Location: apps/sophia/ingestion/src/connectors/connectors.ts
The Source Discovery Service manages registration and discovery of content sources.
Source Types#
| Type | Description | Example |
|---|---|---|
academic |
Peer-reviewed research | JSTOR, PubMed, Google Scholar |
book |
Published books | Project Gutenberg, Internet Archive |
article |
Web articles | Medium, tradition websites |
website |
Full websites | Dharma repositories, contemplative centers |
blog |
Blog content | Meditation teachers' blogs |
Source Registration#
Sources are registered with quality scoring:
interface Source {
id: string;
url: string;
name: string;
type: SourceType;
tradition?: string; // e.g., "zen", "stoicism", "vedanta"
qualityScore: number; // 0-100, affects ranking
lastCrawled?: Date;
crawlFrequency: string; // "daily", "weekly", "monthly"
status: 'active' | 'paused' | 'archived';
}
Quality Scoring Factors#
| Factor | Weight | Description |
|---|---|---|
| Authority | 30% | Academic/established source |
| Accuracy | 25% | Citation quality, fact-checking |
| Completeness | 20% | Depth of content |
| Freshness | 15% | Update frequency |
| Accessibility | 10% | Open access vs. paywalled |
Document Crawling#
Crawler Service#
Location: apps/sophia/ingestion/src/pipeline/ingestion-pipeline.ts
The Source Crawler handles fetching content from registered sources.
Crawl Configuration#
interface CrawlRequest {
sourceId: string;
maxPages?: number; // Limit pages per crawl
maxDepth?: number; // Link depth limit
respectRobotsTxt?: boolean; // Default: true
rateLimitMs?: number; // Delay between requests
extractionRules?: {
titleSelector?: string;
contentSelector?: string;
dateSelector?: string;
authorSelector?: string;
};
}
Crawl Statistics#
Each crawl tracks:
interface CrawlStats {
pagesProcessed: number;
errorsEncountered: number;
bytesDownloaded: number;
entitiesExtracted: number;
duration: number;
}
Rate Limiting#
| Source Type | Default Delay | Max Concurrent |
|---|---|---|
| Academic | 2000ms | 2 |
| Website | 1000ms | 5 |
| Blog | 500ms | 10 |
Deduplication#
The crawler maintains a deduplication cache to avoid re-processing:
- URL normalization
- Content fingerprinting
- Last-modified header checking
Document Processing#
Document Processing Service#
Location: services/ai/src/document-processing/service.ts
Supported Formats#
| Format | Parser | Extensions |
|---|---|---|
| pdf-parse | ||
| Office | officeparser | .docx, .doc, .pptx |
| Markdown | marked | .md |
| HTML | Cheerio | .html, .htm |
| Plain Text | Native | .txt |
| EPUB | epub-parser | .epub |
Document Parsing#
interface ParsedDocument {
content: string; // Extracted text
metadata: {
title?: string;
authors?: string[];
date?: Date;
language?: string;
pageCount?: number;
};
structure: {
chapters?: ChapterInfo[];
headings?: HeadingInfo[];
paragraphs?: ParagraphInfo[];
};
}
Text Chunking#
Documents are chunked for embedding:
| Strategy | Chunk Size | Overlap | Use Case |
|---|---|---|---|
| Sentence | 3-5 sentences | 1 sentence | Dense content |
| Paragraph | 1 paragraph | None | Structured texts |
| Fixed | 512 tokens | 50 tokens | General |
| Semantic | Variable | Contextual | Narrative |
Chunk Metadata#
Each chunk retains:
interface ChunkMetadata {
documentId: string;
chunkId: string;
position: number; // Order in document
content: string; // Chunk text
language: string;
tradition?: string;
genre?: string; // meditation, philosophy, commentary
title: string;
authors: string[];
license?: string;
confidence: number; // Parsing confidence
}
Vector Embeddings#
Search And Vector Storage#
Location: apps/sophia/search-api/src/main.ts
Collections#
| Collection | Dimensions | Index | Metric | Purpose |
|---|---|---|---|---|
knowledge_chunks |
768 | HNSW | COSINE | Main content |
semantic_search |
768 | IVF_FLAT | IP | Search optimization |
persona_embeddings |
384 | HNSW | L2 | Persona matching |
Embedding Pipeline#
Chunk Text → Embedding Model → Vector (768-dim) → Milvus Storage
Index Types#
| Type | Best For | Memory | Query Speed |
|---|---|---|---|
| FLAT | Small datasets | High | Fastest (exact) |
| IVF_FLAT | Medium datasets | Medium | Fast |
| IVF_PQ | Large datasets | Low | Moderate |
| HNSW | Real-time search | Medium | Very Fast |
Vector Operations#
// Insert vectors
POST /v1/collections/:name/vectors
// Similarity search
POST /v1/collections/:name/search
{
"vector": number[],
"topK": 10,
"minScore": 0.7,
"filters": { "tradition": "zen" }
}
// Hybrid search (vector + keyword)
POST /v1/collections/:name/hybrid-search
{
"vector": number[],
"keywords": ["meditation", "breath"],
"vectorWeight": 0.7,
"keywordWeight": 0.3
}
Knowledge Graph Integration#
Entity Extraction#
During ingestion, entities are extracted for the knowledge graph:
| Entity Type | Examples |
|---|---|
TRADITION |
Zen, Stoicism, Vedanta |
CONCEPT |
Mindfulness, Virtue, Dharma |
PRACTICE |
Zazen, Lectio Divina, Pranayama |
FIGURE |
Buddha, Seneca, Ramana Maharshi |
TEXT |
Dhammapada, Meditations, Yoga Sutras |
PLACE |
Bodh Gaya, Mount Athos, Zen monasteries |
Relationship Extraction#
| Relationship | Description | Example |
|---|---|---|
TAUGHT |
Teacher-student | Dogen TAUGHT Ejo |
WROTE |
Authorship | Seneca WROTE Letters |
BELONGS_TO |
Categorization | Zazen BELONGS_TO Zen |
DERIVES_FROM |
Influence | Chan DERIVES_FROM Mahayana |
ALIGNS_WITH |
Similarity | Stoic acceptance ALIGNS_WITH Buddhist equanimity |
CONTRADICTS |
Opposition | Materialist CONTRADICTS Idealist |
Graph Storage#
Location: apps/sophia/knowledge-graph/src/graph/graph-service.ts
Entities and relationships are stored in Neo4j with:
- Full-text indexing on node properties
- Constraint enforcement on unique IDs
- Connection pooling (default 50 connections)
- Query timeout: 30 seconds
Citation Metadata#
Required Citation Fields#
| Field | Required | Description |
|---|---|---|
title |
Yes | Document title |
authors |
Yes | Author names |
source |
Yes | Publication/source name |
date |
Recommended | Publication date |
url |
If available | Original URL |
license |
Recommended | Copyright/license info |
tradition |
Recommended | Associated tradition |
Citation Format#
{
"citation": {
"title": "Zen Mind, Beginner's Mind",
"authors": ["Shunryu Suzuki"],
"source": "Weatherhill",
"date": "1970",
"isbn": "978-0834800793",
"license": "copyright",
"tradition": "zen"
}
}
License Categories#
| License | Description | Usage |
|---|---|---|
public_domain |
No copyright | Full use |
creative_commons |
CC licenses | Per license terms |
fair_use |
Limited quotes | Attribution required |
copyright |
Full copyright | Metadata only |
Quality Assurance#
Ingestion Validation#
| Check | Description | Action on Fail |
|---|---|---|
| Format | Valid file format | Reject |
| Language | Supported language | Flag for review |
| Length | Minimum content length | Skip if too short |
| Duplicate | Content fingerprint check | Skip if duplicate |
| Quality | Readability score | Flag if low quality |
Content Quality Metrics#
interface QualityMetrics {
readabilityScore: number; // Flesch-Kincaid
vocabularyRichness: number; // Type-token ratio
entityDensity: number; // Entities per 1000 words
citationDensity: number; // Citations per 1000 words
languageConfidence: number; // Language detection confidence
}
Manual Review Queue#
Content flagged for review:
- Low quality scores
- Unknown language
- Missing required metadata
- Potential copyright issues
- Entity extraction uncertainty
Ingestion Workflow#
Step-by-Step Process#
-
Register Source
- Add source to discovery service
- Set crawl frequency and quality score
- Configure extraction rules
-
Schedule Crawl
- Automatic based on frequency
- Manual trigger available
-
Fetch Content
- Respect robots.txt
- Apply rate limiting
- Track crawl statistics
-
Parse Documents
- Detect format
- Extract text and metadata
- Preserve structure
-
Chunk Content
- Apply chunking strategy
- Maintain context overlap
- Attach metadata
-
Generate Embeddings
- Process through embedding model
- Store in Milvus
-
Extract Entities
- Identify traditions, concepts, figures
- Extract relationships
-
Update Graph
- Create/update Neo4j nodes
- Add relationships
-
Index for Search
- Full-text indexing (PostgreSQL)
- Vector indexing (Milvus)
-
Quality Check
- Validate metadata
- Score quality metrics
- Route to review if needed
Monitoring#
| Metric | Description | Alert Threshold |
|---|---|---|
| Crawl success rate | % successful crawls | < 90% |
| Parse success rate | % successful parses | < 95% |
| Embedding latency | Time to embed | > 5s per chunk |
| Index latency | Time to index | > 10s per document |
API Reference#
Ingestion Endpoints#
| Method | Endpoint | Description |
|---|---|---|
| POST | /v1/sources |
Register new source |
| GET | /v1/sources |
List sources |
| POST | /v1/sources/:id/crawl |
Trigger crawl |
| GET | /v1/sources/:id/status |
Crawl status |
| POST | /v1/documents/upload |
Direct upload |
| GET | /v1/documents/:id |
Get document |
| POST | /v1/documents/:id/reprocess |
Reprocess document |
Related Documents#
Last updated: January 2, 2026