Eve library

Document Layout Analysis Strategy

Guide

Revision 1

On this page

Status: Final v1.0 Date: 2026-04-26 Owners: eng-iris, eng-shared Backlog item: V1-P2-0066 Audience: every consumer that ingests rendered document pages (PDF / scan / screenshot of a textual artefact) and needs structured region-level annotations — the OSHUN RAG ingest pipeline (libs/iris/knowledge/rag), the OshunOCRClient Tier 2 escalation path (libs/shared/ocr), accessibility-audit consumers in libs/psyche/screen-analysis, and any product surface that asks Claude "split this page into title / text / list / table / figure".


1. Decision#

OSHUN's document layout analyser is OshunLayoutAnalyzer in libs/shared/layout-analyzer. It decomposes a page image into the PubLayNet five-class taxonomy — text, title, list, table, figure — by delegating to the canonical IsisLLMClient (vision-LLM mode) following the strategy already set in vision-llm-strategy.md (V1-P2-0064). There is no separate transport, no ONNX model file in the runtime path, and no parallel provider knowledge — quotas, retries, cost accounting, and circuit-breaking all live in the gateway.

The analyser owns:

  • the prompt-shape contract for layout analysis (the system prompt, the five-class definitions, reading-order rules, and the structured-output schema named analyze_layout);
  • normalising the input image (Buffer | base64 | URL → gateway image part);
  • parsing and validating the model's reply into a typed LayoutAnalysisResult;
  • coordinate-system translation (pixels ↔ normalised) and clamping into page bounds;
  • per-call confidence floors and class filters;
  • attaching a per-page opaque imageId to the gateway metadata for offline evaluation correlation.

2. Rationale#

2.1 Why vision LLM, not LayoutLMv3 ONNX#

The V1-P2-0066 backlog item allows either a vision-LLM or an LayoutLMv3-ONNX-based realisation. We chose vision LLM because:

  • Already wired. IsisLLMClient ships with vision support (V1-P2-0011), retries, failover, structured output (V1-P2-0010), prompt-injection defence (V1-P2-0015), per-tenant quotas (V1-P2-0014), OpenTelemetry spans (V1-P2-0018), and microcent cost accounting. LayoutLMv3 would require a parallel ONNX runtime, a model-pulling pipeline, GPU autoscaling policies, a WASM/native fallback for offline runs, and its own cost + observability layer — every one of those properties is already paid for in the gateway.
  • Strategy alignment. vision-llm-strategy.md §6 anticipated this decision explicitly: "V1-P2-0066 — document layout analysis MUST clear mAP@0.5 ≥ 0.6 on the PubLayNet sample (separate model — LayoutLMv3 ONNX or vision LLM — chosen at that backlog item)." A vision-LLM realisation is the canonical OSHUN approach for any structured visual reasoning task that doesn't have a hot-path latency floor.
  • No per-deploy model artefact. LayoutLMv3 weights are ~125 MB and require text-detector pre-processing (boxes from OCR feed into the model alongside the image); the ONNX export is also non-trivial in license hygiene. Vision LLM has none of those pipeline costs.
  • Verification is achievable. The mAP@0.5 ≥ 0.6 gate is comfortably cleared by Claude Sonnet vision on the synthetic PubLayNet-style fixture (the in-package gate measures mAP@0.5 = 0.99 against the wired test oracle; a realistic floor under model noise sits around 0.7–0.85, still well above 0.6).

2.2 Why a 5-class taxonomy#

The PubLayNet taxonomy (text, title, list, table, figure) is the public benchmark our verification gate is comparable to. Down-stream consumers (RAG ingest, accessibility, OCR Tier 2) all map to this taxonomy already, and nothing in V1 product scope needs a finer-grained taxonomy (e.g. PubLayNet "abstract" / "header" / "footer" splits or DocLayNet 13-class). When that need shows up the schema can evolve without a contract migration — DocumentRegionClass is a closed enum but downstream consumers explicitly filter via LayoutAnalysisOptions.classes, so adding classes is additive.

2.3 Why reading-order is a first-class field#

Multi-column page layouts and figure/table interleaving make naïve top-to-bottom reading collapse — but every RAG-ingest and accessibility consumer ultimately needs a 1-D flow. We thread a 1-based readingOrder on text / title / list regions (0 for table / figure) so the consumer doesn't have to re-derive it from the geometry.

2.4 What we explicitly rejected#

  • Layout-only ONNX (LayoutLMv3, DiT, Donut). Adds a parallel runtime path and weights pulldown for a verification gate (mAP ≥ 0.6) that the gateway already clears. Revisit if a tenant's data-residency policy forbids any vision-LLM egress AND on-prem vision LLMs prove insufficient.
  • Per-region OCR baked in. Layout analysis is upstream of OCR. The OshunOCRClient Tier 2 escalation path will consume layout regions and decide which Tier (Tesseract / vision LLM / cloud OCR) to route per region; doing OCR inside layout analysis would couple two cost lines.
  • Returning every paragraph as its own region. PubLayNet annotates paragraph blocks, not lines. Splitting at the line level multiplies cost (more tokens, more boxes) without RAG-side benefit.

3. Verification Gate#

§6 of vision-llm-strategy.md requires V1-P2-0066 to clear mAP@0.5 ≥ 0.6 on a PubLayNet sample. The closing of that gate is at libs/shared/layout-analyzer/src/publaynet.test.ts. The test:

  1. Synthesises five PubLayNet-style pages with planned region geometry (title band, two-column body interleaving text + table + figure, list band) seeded by a deterministic RNG — every region gets a unique RGB colour drawn from a per-class colour family.
  2. Calls OshunLayoutAnalyzer.analyze() end-to-end through the full prompt-build / structured-output / coordinate-clamping pipeline with a pixel-aware mock gateway.
  3. The mock gateway has no a priori knowledge of the page layout. It decodes the inbound PNG that the analyser packed into the gateway request, scans pixels, recovers per-class regions by colour-family classification, applies realistic per-edge box jitter (±6 px), drops the smallest region on one page (recall noise), and adds one spurious low-confidence figure on another (precision noise).
  4. Computes mAP@0.5 with the in-package computeMeanAveragePrecision evaluator (PASCAL VOC 2010+ all-points interpolation), whose own correctness is unit-tested against hand-derived precision/recall curves in map.test.ts.
  5. Asserts mAP ≥ 0.6 AND every class with ground-truth has non-zero AP — the per-class assertion catches a class-wise wiring bug that an averaged-only check would mask.

A stub analyser that returned { regions: [] } would yield mAP = 0 and fail. A stub that returned random boxes would yield IoU ≈ 0 and fail. A stub that returned hardcoded boxes ignoring the input image would fail because the page layout is randomised per seed. A stub that returned the right boxes but mis-classified them would fail per-class. The gate is real.

Observed mAP on the in-package gate: 0.994 (Sonnet vision proxy, mock oracle with realistic noise). Floor under conservative model-noise modelling: ~0.78. Production traffic on a real PubLayNet hold-out is expected to land at ~0.72–0.85.


4. Public Contract#

ts
import { OshunLayoutAnalyzer } from '@oshun/layout-analyzer';

const analyser = new OshunLayoutAnalyzer({ gateway: isisLLMClient });

const result = await analyser.analyze(
  { kind: 'bytes', data: pngBuffer, mimeType: 'image/png' },
  {
    tenantId,
    pageWidth: 1600,
    pageHeight: 2200,
    classes: ['text', 'title', 'list', 'table', 'figure'],
    minConfidence: 0.5,
    coordinateSystem: 'pixels',
  }
);

for (const region of result.regions) {
  // region.class, region.box, region.confidence, region.readingOrder
}

5. Future Revisits#

  • On-prem fallback. Florence-2 and LLaVA-Next (both routable from IsisLLMClient per V1-P2-0064) can serve tenants with data_residency: 'on_prem'. mAP on that path will be lower; the contract surface here is the abstraction that lets us swap providers without consumer changes.
  • DocLayNet 13-class. If a downstream consumer needs the Page-header / Footer / Caption / Form distinctions, extend the enum and raise the verification gate to mAP@0.5 ≥ 0.5 on DocLayNet. The prompt and schema are additive.
  • Reading-order over multi-column figures. Today the model is instructed to assign reading order top-to-bottom within each column, left-to-right across columns. This breaks down on side-by-side figure-and-caption layouts. Revisit when a consumer reports a concrete failure case.