Eve library

Ingestion Processing Pipeline

Guide

Revision 1

On this page

The ingestion service now enforces a deterministic processing pipeline across every connector import. The pipeline is designed to eliminate duplicate payloads, remove low–value boilerplate text, detect language automatically, and persist segmentation metadata for downstream systems.

Pipeline stages#

  1. Acquisition & Extraction

    • Inline uploads are accepted verbatim; PDF sources are simulated for example.org fixtures and otherwise parsed via pdf-parse with OCR fallback when confidence < 0.6.
    • The extraction stage preserves source metadata (pages, word counts, collection hints) in processing_metadata.
  2. Boilerplate Removal

    • Headers, horizontal rules, and recurring disclaimers (e.g., Lilith Foundation notices) are stripped via cleanIngestedContent.
    • Duplicate short lines are discarded and whitespace is normalised before segmentation.
  3. Language Detection

    • detectLanguages analyses ASCII vs. accented character ratios to provide the top candidates with confidence scores; the primary language is promoted to the document record for index filtering.
  4. Deduplication

    • Every cleaned payload is hashed (sha256) to create a fingerprint.
    • The first document for a fingerprint is marked canonical; subsequent matches reuse the canonical content, mark deduplication.is_duplicate = true, and expose duplicate_of plus reused_from_document metadata while keeping the job status completed for compatibility with existing suites.
  5. Segmentation

    • segmentContent splits cleaned text into headings, paragraphs, quotes, and list items with absolute character offsets. Segments are stored in-memory for /v1/documents/:docId/segments and counted in document metadata.

API surface updates#

  • GET /v1/documents/:docId/segments returns the canonical segment list for a processed document.
  • Duplicate documents expose:
    • document.duplicate_of
    • document.deduplication (including fingerprint and first-seen timestamp)
    • job.processing_metadata.reused_from_document

Test coverage#

  • tests/ingestion.processing-pipeline.spec.ts
    • Verifies dedup metadata propagation and ensures boilerplate stripping, language detection, and segmentation output remain intact.
  • tests/ingestion.pdf-processing.spec.ts
    • Exercises the simulated PDF fixtures, OCR fallback path, chunking, document-store integration, and entity extraction.
  • tests/ingestion.document-store-layouts.spec.ts
    • Confirms checksum integrity alongside the new fingerprint bookkeeping when documents are persisted.

Operational notes#

  • The fingerprint index is kept in-memory today; replace it with a durable store before clustering the ingestion service.
  • Simulated example.org PDFs provide deterministic fixtures for CI. Real-world connectors should still stream and parse raw binaries.
  • The entity extractor now leverages the cleaned text so deduped documents surface the same concepts as their canonical source.