# Ingestion Processing Pipeline

The ingestion service now enforces a deterministic processing pipeline across
every connector import. The pipeline is designed to eliminate duplicate
payloads, remove low–value boilerplate text, detect language automatically, and
persist segmentation metadata for downstream systems.

## Pipeline stages

1. **Acquisition & Extraction**
   - Inline uploads are accepted verbatim; PDF sources are simulated for
     `example.org` fixtures and otherwise parsed via `pdf-parse` with OCR
     fallback when confidence < 0.6.
   - The extraction stage preserves source metadata (pages, word counts,
     collection hints) in `processing_metadata`.

2. **Boilerplate Removal**
   - Headers, horizontal rules, and recurring disclaimers (e.g., Lilith
     Foundation notices) are stripped via `cleanIngestedContent`.
   - Duplicate short lines are discarded and whitespace is normalised before
     segmentation.

3. **Language Detection**
   - `detectLanguages` analyses ASCII vs. accented character ratios to provide
     the top candidates with confidence scores; the primary language is promoted
     to the document record for index filtering.

4. **Deduplication**
   - Every cleaned payload is hashed (`sha256`) to create a fingerprint.
   - The first document for a fingerprint is marked canonical; subsequent
     matches reuse the canonical content, mark
     `deduplication.is_duplicate = true`, and expose `duplicate_of` plus
     `reused_from_document` metadata while keeping the job status `completed`
     for compatibility with existing suites.

5. **Segmentation**
   - `segmentContent` splits cleaned text into headings, paragraphs, quotes, and
     list items with absolute character offsets. Segments are stored in-memory
     for `/v1/documents/:docId/segments` and counted in document metadata.

## API surface updates

- `GET /v1/documents/:docId/segments` returns the canonical segment list for a
  processed document.
- Duplicate documents expose:
  - `document.duplicate_of`
  - `document.deduplication` (including fingerprint and first-seen timestamp)
  - `job.processing_metadata.reused_from_document`

## Test coverage

- `tests/ingestion.processing-pipeline.spec.ts`
  - Verifies dedup metadata propagation and ensures boilerplate stripping,
    language detection, and segmentation output remain intact.
- `tests/ingestion.pdf-processing.spec.ts`
  - Exercises the simulated PDF fixtures, OCR fallback path, chunking,
    document-store integration, and entity extraction.
- `tests/ingestion.document-store-layouts.spec.ts`
  - Confirms checksum integrity alongside the new fingerprint bookkeeping when
    documents are persisted.

## Operational notes

- The fingerprint index is kept in-memory today; replace it with a durable store
  before clustering the ingestion service.
- Simulated `example.org` PDFs provide deterministic fixtures for CI. Real-world
  connectors should still stream and parse raw binaries.
- The entity extractor now leverages the cleaned text so deduped documents
  surface the same concepts as their canonical source.
