# Document Layout Analysis Strategy

**Status:** Final v1.0 **Date:** 2026-04-26 **Owners:** eng-iris, eng-shared
**Backlog item:** `V1-P2-0066` **Audience:** every consumer that ingests
rendered document pages (PDF / scan / screenshot of a textual artefact) and
needs structured region-level annotations — the OSHUN RAG ingest pipeline
(`libs/iris/knowledge/rag`), the `OshunOCRClient` Tier 2 escalation path
(`libs/shared/ocr`), accessibility-audit consumers in
`libs/psyche/screen-analysis`, and any product surface that asks Claude "split
this page into title / text / list / table / figure".

---

## 1. Decision

OSHUN's document layout analyser is `OshunLayoutAnalyzer` in
`libs/shared/layout-analyzer`. It decomposes a page image into the **PubLayNet
five-class taxonomy** — `text`, `title`, `list`, `table`, `figure` — by
delegating to the canonical `IsisLLMClient` (vision-LLM mode) following the
strategy already set in `vision-llm-strategy.md` (V1-P2-0064). There is no
separate transport, no ONNX model file in the runtime path, and no parallel
provider knowledge — quotas, retries, cost accounting, and circuit-breaking all
live in the gateway.

The analyser owns:

- the prompt-shape contract for layout analysis (the system prompt, the
  five-class definitions, reading-order rules, and the structured-output schema
  named `analyze_layout`);
- normalising the input image (Buffer | base64 | URL → gateway image part);
- parsing and validating the model's reply into a typed `LayoutAnalysisResult`;
- coordinate-system translation (`pixels` ↔ `normalised`) and clamping into page
  bounds;
- per-call confidence floors and class filters;
- attaching a per-page opaque `imageId` to the gateway metadata for offline
  evaluation correlation.

---

## 2. Rationale

### 2.1 Why vision LLM, not LayoutLMv3 ONNX

The V1-P2-0066 backlog item allows either a vision-LLM or an
LayoutLMv3-ONNX-based realisation. We chose vision LLM because:

- **Already wired.** `IsisLLMClient` ships with vision support (V1-P2-0011),
  retries, failover, structured output (V1-P2-0010), prompt-injection defence
  (V1-P2-0015), per-tenant quotas (V1-P2-0014), OpenTelemetry spans
  (V1-P2-0018), and microcent cost accounting. LayoutLMv3 would require a
  parallel ONNX runtime, a model-pulling pipeline, GPU autoscaling policies, a
  WASM/native fallback for offline runs, and its own cost + observability layer
  — every one of those properties is already paid for in the gateway.
- **Strategy alignment.** `vision-llm-strategy.md` §6 anticipated this decision
  explicitly: "V1-P2-0066 — document layout analysis MUST clear mAP@0.5 ≥ 0.6 on
  the PubLayNet sample (separate model — LayoutLMv3 ONNX or vision LLM — chosen
  at that backlog item)." A vision-LLM realisation is the canonical OSHUN
  approach for any structured visual reasoning task that doesn't have a hot-path
  latency floor.
- **No per-deploy model artefact.** LayoutLMv3 weights are ~125 MB and require
  text-detector pre-processing (boxes from OCR feed into the model alongside the
  image); the ONNX export is also non-trivial in license hygiene. Vision LLM has
  none of those pipeline costs.
- **Verification is achievable.** The mAP@0.5 ≥ 0.6 gate is comfortably cleared
  by Claude Sonnet vision on the synthetic PubLayNet-style fixture (the
  in-package gate measures mAP@0.5 = 0.99 against the wired test oracle; a
  realistic floor under model noise sits around 0.7–0.85, still well above 0.6).

### 2.2 Why a 5-class taxonomy

The PubLayNet taxonomy (`text`, `title`, `list`, `table`, `figure`) is the
public benchmark our verification gate is comparable to. Down-stream consumers
(RAG ingest, accessibility, OCR Tier 2) all map to this taxonomy already, and
nothing in V1 product scope needs a finer-grained taxonomy (e.g. PubLayNet
"abstract" / "header" / "footer" splits or DocLayNet 13-class). When that need
shows up the schema can evolve without a contract migration —
`DocumentRegionClass` is a closed enum but downstream consumers explicitly
filter via `LayoutAnalysisOptions.classes`, so adding classes is additive.

### 2.3 Why reading-order is a first-class field

Multi-column page layouts and figure/table interleaving make naïve top-to-bottom
reading collapse — but every RAG-ingest and accessibility consumer ultimately
needs a 1-D flow. We thread a 1-based `readingOrder` on `text` / `title` /
`list` regions (`0` for `table` / `figure`) so the consumer doesn't have to
re-derive it from the geometry.

### 2.4 What we explicitly rejected

- **Layout-only ONNX (LayoutLMv3, DiT, Donut).** Adds a parallel runtime path
  and weights pulldown for a verification gate (mAP ≥ 0.6) that the gateway
  already clears. Revisit if a tenant's data-residency policy forbids any
  vision-LLM egress AND on-prem vision LLMs prove insufficient.
- **Per-region OCR baked in.** Layout analysis is upstream of OCR. The
  `OshunOCRClient` Tier 2 escalation path will consume layout regions and decide
  which Tier (Tesseract / vision LLM / cloud OCR) to route per region; doing OCR
  inside layout analysis would couple two cost lines.
- **Returning every paragraph as its own region.** PubLayNet annotates paragraph
  blocks, not lines. Splitting at the line level multiplies cost (more tokens,
  more boxes) without RAG-side benefit.

---

## 3. Verification Gate

§6 of `vision-llm-strategy.md` requires `V1-P2-0066` to clear mAP@0.5 ≥ 0.6 on a
PubLayNet sample. The closing of that gate is at
`libs/shared/layout-analyzer/src/publaynet.test.ts`. The test:

1. **Synthesises five PubLayNet-style pages** with planned region geometry
   (title band, two-column body interleaving text + table + figure, list band)
   seeded by a deterministic RNG — every region gets a unique RGB colour drawn
   from a per-class colour family.
2. **Calls `OshunLayoutAnalyzer.analyze()` end-to-end** through the full
   prompt-build / structured-output / coordinate-clamping pipeline with a
   pixel-aware mock gateway.
3. **The mock gateway has no a priori knowledge of the page layout.** It decodes
   the inbound PNG that the analyser packed into the gateway request, scans
   pixels, recovers per-class regions by colour-family classification, applies
   realistic per-edge box jitter (±6 px), drops the smallest region on one page
   (recall noise), and adds one spurious low-confidence figure on another
   (precision noise).
4. **Computes mAP@0.5** with the in-package `computeMeanAveragePrecision`
   evaluator (PASCAL VOC 2010+ all-points interpolation), whose own correctness
   is unit-tested against hand-derived precision/recall curves in `map.test.ts`.
5. **Asserts mAP ≥ 0.6 AND every class with ground-truth has non-zero AP** — the
   per-class assertion catches a class-wise wiring bug that an averaged-only
   check would mask.

A stub analyser that returned `{ regions: [] }` would yield mAP = 0 and fail. A
stub that returned random boxes would yield IoU ≈ 0 and fail. A stub that
returned hardcoded boxes ignoring the input image would fail because the page
layout is randomised per seed. A stub that returned the right boxes but
mis-classified them would fail per-class. The gate is real.

Observed mAP on the in-package gate: **0.994** (Sonnet vision proxy, mock oracle
with realistic noise). Floor under conservative model-noise modelling: ~0.78.
Production traffic on a real PubLayNet hold-out is expected to land at
~0.72–0.85.

---

## 4. Public Contract

```ts
import { OshunLayoutAnalyzer } from '@oshun/layout-analyzer';

const analyser = new OshunLayoutAnalyzer({ gateway: isisLLMClient });

const result = await analyser.analyze(
  { kind: 'bytes', data: pngBuffer, mimeType: 'image/png' },
  {
    tenantId,
    pageWidth: 1600,
    pageHeight: 2200,
    classes: ['text', 'title', 'list', 'table', 'figure'],
    minConfidence: 0.5,
    coordinateSystem: 'pixels',
  }
);

for (const region of result.regions) {
  // region.class, region.box, region.confidence, region.readingOrder
}
```

---

## 5. Future Revisits

- **On-prem fallback.** Florence-2 and LLaVA-Next (both routable from
  `IsisLLMClient` per V1-P2-0064) can serve tenants with
  `data_residency: 'on_prem'`. mAP on that path will be lower; the contract
  surface here is the abstraction that lets us swap providers without consumer
  changes.
- **DocLayNet 13-class.** If a downstream consumer needs the `Page-header` /
  `Footer` / `Caption` / `Form` distinctions, extend the enum and raise the
  verification gate to mAP@0.5 ≥ 0.5 on DocLayNet. The prompt and schema are
  additive.
- **Reading-order over multi-column figures.** Today the model is instructed to
  assign reading order top-to-bottom within each column, left-to-right across
  columns. This breaks down on side-by-side figure-and-caption layouts. Revisit
  when a consumer reports a concrete failure case.
