Eve library

Content Ingestion Guide

Guide

Revision 1

On this page

Overview#

This document describes the content ingestion pipeline for adding wisdom texts, meditation research, philosophy texts, and other source materials to Lilith's Sophia-backed retrieval stack. The system supports crawling, parsing, chunking, embedding, and indexing content for semantic search and knowledge graph construction.

Table of Contents#

  1. Architecture Overview
  2. Source Discovery
  3. Document Crawling
  4. Document Processing
  5. Vector Embeddings
  6. Knowledge Graph Integration
  7. Citation Metadata
  8. Quality Assurance

Architecture Overview#

Pipeline Flow#

text
Source Discovery → Crawling → Parsing → Chunking → Embedding → Storage
                                          ↓
                              Entity Extraction → Graph Storage

Services Involved#

Service Role Location
Sophia Ingestion Source ingestion, extraction, chunking, enrichment apps/sophia/ingestion/
Sophia Search API Search, retrieval, and document read APIs apps/sophia/search-api/
Sophia Knowledge Graph Entity and relationship storage/querying apps/sophia/knowledge-graph/

Data Stores#

Store Technology Purpose
Document Store PostgreSQL Full text storage, metadata
Vector Store Milvus Semantic embeddings
Graph Store Neo4j Knowledge graph relationships
Time Series TimescaleDB Crawl and access metrics

Source Discovery#

Source Service#

Location: apps/sophia/ingestion/src/connectors/connectors.ts

The Source Discovery Service manages registration and discovery of content sources.

Source Types#

Type Description Example
academic Peer-reviewed research JSTOR, PubMed, Google Scholar
book Published books Project Gutenberg, Internet Archive
article Web articles Medium, tradition websites
website Full websites Dharma repositories, contemplative centers
blog Blog content Meditation teachers' blogs

Source Registration#

Sources are registered with quality scoring:

typescript
interface Source {
  id: string;
  url: string;
  name: string;
  type: SourceType;
  tradition?: string; // e.g., "zen", "stoicism", "vedanta"
  qualityScore: number; // 0-100, affects ranking
  lastCrawled?: Date;
  crawlFrequency: string; // "daily", "weekly", "monthly"
  status: 'active' | 'paused' | 'archived';
}

Quality Scoring Factors#

Factor Weight Description
Authority 30% Academic/established source
Accuracy 25% Citation quality, fact-checking
Completeness 20% Depth of content
Freshness 15% Update frequency
Accessibility 10% Open access vs. paywalled

Document Crawling#

Crawler Service#

Location: apps/sophia/ingestion/src/pipeline/ingestion-pipeline.ts

The Source Crawler handles fetching content from registered sources.

Crawl Configuration#

typescript
interface CrawlRequest {
  sourceId: string;
  maxPages?: number; // Limit pages per crawl
  maxDepth?: number; // Link depth limit
  respectRobotsTxt?: boolean; // Default: true
  rateLimitMs?: number; // Delay between requests
  extractionRules?: {
    titleSelector?: string;
    contentSelector?: string;
    dateSelector?: string;
    authorSelector?: string;
  };
}

Crawl Statistics#

Each crawl tracks:

typescript
interface CrawlStats {
  pagesProcessed: number;
  errorsEncountered: number;
  bytesDownloaded: number;
  entitiesExtracted: number;
  duration: number;
}

Rate Limiting#

Source Type Default Delay Max Concurrent
Academic 2000ms 2
Website 1000ms 5
Blog 500ms 10

Deduplication#

The crawler maintains a deduplication cache to avoid re-processing:

  • URL normalization
  • Content fingerprinting
  • Last-modified header checking

Document Processing#

Document Processing Service#

Location: services/ai/src/document-processing/service.ts

Supported Formats#

Format Parser Extensions
PDF pdf-parse .pdf
Office officeparser .docx, .doc, .pptx
Markdown marked .md
HTML Cheerio .html, .htm
Plain Text Native .txt
EPUB epub-parser .epub

Document Parsing#

typescript
interface ParsedDocument {
  content: string; // Extracted text
  metadata: {
    title?: string;
    authors?: string[];
    date?: Date;
    language?: string;
    pageCount?: number;
  };
  structure: {
    chapters?: ChapterInfo[];
    headings?: HeadingInfo[];
    paragraphs?: ParagraphInfo[];
  };
}

Text Chunking#

Documents are chunked for embedding:

Strategy Chunk Size Overlap Use Case
Sentence 3-5 sentences 1 sentence Dense content
Paragraph 1 paragraph None Structured texts
Fixed 512 tokens 50 tokens General
Semantic Variable Contextual Narrative

Chunk Metadata#

Each chunk retains:

typescript
interface ChunkMetadata {
  documentId: string;
  chunkId: string;
  position: number; // Order in document
  content: string; // Chunk text
  language: string;
  tradition?: string;
  genre?: string; // meditation, philosophy, commentary
  title: string;
  authors: string[];
  license?: string;
  confidence: number; // Parsing confidence
}

Vector Embeddings#

Search And Vector Storage#

Location: apps/sophia/search-api/src/main.ts

Collections#

Collection Dimensions Index Metric Purpose
knowledge_chunks 768 HNSW COSINE Main content
semantic_search 768 IVF_FLAT IP Search optimization
persona_embeddings 384 HNSW L2 Persona matching

Embedding Pipeline#

text
Chunk Text → Embedding Model → Vector (768-dim) → Milvus Storage

Index Types#

Type Best For Memory Query Speed
FLAT Small datasets High Fastest (exact)
IVF_FLAT Medium datasets Medium Fast
IVF_PQ Large datasets Low Moderate
HNSW Real-time search Medium Very Fast

Vector Operations#

typescript
// Insert vectors
POST /v1/collections/:name/vectors

// Similarity search
POST /v1/collections/:name/search
{
  "vector": number[],
  "topK": 10,
  "minScore": 0.7,
  "filters": { "tradition": "zen" }
}

// Hybrid search (vector + keyword)
POST /v1/collections/:name/hybrid-search
{
  "vector": number[],
  "keywords": ["meditation", "breath"],
  "vectorWeight": 0.7,
  "keywordWeight": 0.3
}

Knowledge Graph Integration#

Entity Extraction#

During ingestion, entities are extracted for the knowledge graph:

Entity Type Examples
TRADITION Zen, Stoicism, Vedanta
CONCEPT Mindfulness, Virtue, Dharma
PRACTICE Zazen, Lectio Divina, Pranayama
FIGURE Buddha, Seneca, Ramana Maharshi
TEXT Dhammapada, Meditations, Yoga Sutras
PLACE Bodh Gaya, Mount Athos, Zen monasteries

Relationship Extraction#

Relationship Description Example
TAUGHT Teacher-student Dogen TAUGHT Ejo
WROTE Authorship Seneca WROTE Letters
BELONGS_TO Categorization Zazen BELONGS_TO Zen
DERIVES_FROM Influence Chan DERIVES_FROM Mahayana
ALIGNS_WITH Similarity Stoic acceptance ALIGNS_WITH Buddhist equanimity
CONTRADICTS Opposition Materialist CONTRADICTS Idealist

Graph Storage#

Location: apps/sophia/knowledge-graph/src/graph/graph-service.ts

Entities and relationships are stored in Neo4j with:

  • Full-text indexing on node properties
  • Constraint enforcement on unique IDs
  • Connection pooling (default 50 connections)
  • Query timeout: 30 seconds

Citation Metadata#

Required Citation Fields#

Field Required Description
title Yes Document title
authors Yes Author names
source Yes Publication/source name
date Recommended Publication date
url If available Original URL
license Recommended Copyright/license info
tradition Recommended Associated tradition

Citation Format#

json
{
  "citation": {
    "title": "Zen Mind, Beginner's Mind",
    "authors": ["Shunryu Suzuki"],
    "source": "Weatherhill",
    "date": "1970",
    "isbn": "978-0834800793",
    "license": "copyright",
    "tradition": "zen"
  }
}

License Categories#

License Description Usage
public_domain No copyright Full use
creative_commons CC licenses Per license terms
fair_use Limited quotes Attribution required
copyright Full copyright Metadata only

Quality Assurance#

Ingestion Validation#

Check Description Action on Fail
Format Valid file format Reject
Language Supported language Flag for review
Length Minimum content length Skip if too short
Duplicate Content fingerprint check Skip if duplicate
Quality Readability score Flag if low quality

Content Quality Metrics#

typescript
interface QualityMetrics {
  readabilityScore: number; // Flesch-Kincaid
  vocabularyRichness: number; // Type-token ratio
  entityDensity: number; // Entities per 1000 words
  citationDensity: number; // Citations per 1000 words
  languageConfidence: number; // Language detection confidence
}

Manual Review Queue#

Content flagged for review:

  • Low quality scores
  • Unknown language
  • Missing required metadata
  • Potential copyright issues
  • Entity extraction uncertainty

Ingestion Workflow#

Step-by-Step Process#

  1. Register Source

    • Add source to discovery service
    • Set crawl frequency and quality score
    • Configure extraction rules
  2. Schedule Crawl

    • Automatic based on frequency
    • Manual trigger available
  3. Fetch Content

    • Respect robots.txt
    • Apply rate limiting
    • Track crawl statistics
  4. Parse Documents

    • Detect format
    • Extract text and metadata
    • Preserve structure
  5. Chunk Content

    • Apply chunking strategy
    • Maintain context overlap
    • Attach metadata
  6. Generate Embeddings

    • Process through embedding model
    • Store in Milvus
  7. Extract Entities

    • Identify traditions, concepts, figures
    • Extract relationships
  8. Update Graph

    • Create/update Neo4j nodes
    • Add relationships
  9. Index for Search

    • Full-text indexing (PostgreSQL)
    • Vector indexing (Milvus)
  10. Quality Check

    • Validate metadata
    • Score quality metrics
    • Route to review if needed

Monitoring#

Metric Description Alert Threshold
Crawl success rate % successful crawls < 90%
Parse success rate % successful parses < 95%
Embedding latency Time to embed > 5s per chunk
Index latency Time to index > 10s per document

API Reference#

Ingestion Endpoints#

Method Endpoint Description
POST /v1/sources Register new source
GET /v1/sources List sources
POST /v1/sources/:id/crawl Trigger crawl
GET /v1/sources/:id/status Crawl status
POST /v1/documents/upload Direct upload
GET /v1/documents/:id Get document
POST /v1/documents/:id/reprocess Reprocess document


Last updated: January 2, 2026