Status: Final v1.0 Date: 2026-04-26 Owners: eng-iris, eng-shared
Backlog item: V1-P2-0066 Audience: every consumer that ingests
rendered document pages (PDF / scan / screenshot of a textual artefact) and
needs structured region-level annotations — the OSHUN RAG ingest pipeline
(libs/iris/knowledge/rag), the OshunOCRClient Tier 2 escalation path
(libs/shared/ocr), accessibility-audit consumers in
libs/psyche/screen-analysis, and any product surface that asks Claude "split
this page into title / text / list / table / figure".
1. Decision#
OSHUN's document layout analyser is OshunLayoutAnalyzer in
libs/shared/layout-analyzer. It decomposes a page image into the PubLayNet
five-class taxonomy — text, title, list, table, figure — by
delegating to the canonical IsisLLMClient (vision-LLM mode) following the
strategy already set in vision-llm-strategy.md (V1-P2-0064). There is no
separate transport, no ONNX model file in the runtime path, and no parallel
provider knowledge — quotas, retries, cost accounting, and circuit-breaking all
live in the gateway.
The analyser owns:
- the prompt-shape contract for layout analysis (the system prompt, the
five-class definitions, reading-order rules, and the structured-output schema
named
analyze_layout); - normalising the input image (Buffer | base64 | URL → gateway image part);
- parsing and validating the model's reply into a typed
LayoutAnalysisResult; - coordinate-system translation (
pixels↔normalised) and clamping into page bounds; - per-call confidence floors and class filters;
- attaching a per-page opaque
imageIdto the gateway metadata for offline evaluation correlation.
2. Rationale#
2.1 Why vision LLM, not LayoutLMv3 ONNX#
The V1-P2-0066 backlog item allows either a vision-LLM or an LayoutLMv3-ONNX-based realisation. We chose vision LLM because:
- Already wired.
IsisLLMClientships with vision support (V1-P2-0011), retries, failover, structured output (V1-P2-0010), prompt-injection defence (V1-P2-0015), per-tenant quotas (V1-P2-0014), OpenTelemetry spans (V1-P2-0018), and microcent cost accounting. LayoutLMv3 would require a parallel ONNX runtime, a model-pulling pipeline, GPU autoscaling policies, a WASM/native fallback for offline runs, and its own cost + observability layer — every one of those properties is already paid for in the gateway. - Strategy alignment.
vision-llm-strategy.md§6 anticipated this decision explicitly: "V1-P2-0066 — document layout analysis MUST clear mAP@0.5 ≥ 0.6 on the PubLayNet sample (separate model — LayoutLMv3 ONNX or vision LLM — chosen at that backlog item)." A vision-LLM realisation is the canonical OSHUN approach for any structured visual reasoning task that doesn't have a hot-path latency floor. - No per-deploy model artefact. LayoutLMv3 weights are ~125 MB and require text-detector pre-processing (boxes from OCR feed into the model alongside the image); the ONNX export is also non-trivial in license hygiene. Vision LLM has none of those pipeline costs.
- Verification is achievable. The mAP@0.5 ≥ 0.6 gate is comfortably cleared by Claude Sonnet vision on the synthetic PubLayNet-style fixture (the in-package gate measures mAP@0.5 = 0.99 against the wired test oracle; a realistic floor under model noise sits around 0.7–0.85, still well above 0.6).
2.2 Why a 5-class taxonomy#
The PubLayNet taxonomy (text, title, list, table, figure) is the
public benchmark our verification gate is comparable to. Down-stream consumers
(RAG ingest, accessibility, OCR Tier 2) all map to this taxonomy already, and
nothing in V1 product scope needs a finer-grained taxonomy (e.g. PubLayNet
"abstract" / "header" / "footer" splits or DocLayNet 13-class). When that need
shows up the schema can evolve without a contract migration —
DocumentRegionClass is a closed enum but downstream consumers explicitly
filter via LayoutAnalysisOptions.classes, so adding classes is additive.
2.3 Why reading-order is a first-class field#
Multi-column page layouts and figure/table interleaving make naïve top-to-bottom
reading collapse — but every RAG-ingest and accessibility consumer ultimately
needs a 1-D flow. We thread a 1-based readingOrder on text / title /
list regions (0 for table / figure) so the consumer doesn't have to
re-derive it from the geometry.
2.4 What we explicitly rejected#
- Layout-only ONNX (LayoutLMv3, DiT, Donut). Adds a parallel runtime path and weights pulldown for a verification gate (mAP ≥ 0.6) that the gateway already clears. Revisit if a tenant's data-residency policy forbids any vision-LLM egress AND on-prem vision LLMs prove insufficient.
- Per-region OCR baked in. Layout analysis is upstream of OCR. The
OshunOCRClientTier 2 escalation path will consume layout regions and decide which Tier (Tesseract / vision LLM / cloud OCR) to route per region; doing OCR inside layout analysis would couple two cost lines. - Returning every paragraph as its own region. PubLayNet annotates paragraph blocks, not lines. Splitting at the line level multiplies cost (more tokens, more boxes) without RAG-side benefit.
3. Verification Gate#
§6 of vision-llm-strategy.md requires V1-P2-0066 to clear mAP@0.5 ≥ 0.6 on a
PubLayNet sample. The closing of that gate is at
libs/shared/layout-analyzer/src/publaynet.test.ts. The test:
- Synthesises five PubLayNet-style pages with planned region geometry (title band, two-column body interleaving text + table + figure, list band) seeded by a deterministic RNG — every region gets a unique RGB colour drawn from a per-class colour family.
- Calls
OshunLayoutAnalyzer.analyze()end-to-end through the full prompt-build / structured-output / coordinate-clamping pipeline with a pixel-aware mock gateway. - The mock gateway has no a priori knowledge of the page layout. It decodes the inbound PNG that the analyser packed into the gateway request, scans pixels, recovers per-class regions by colour-family classification, applies realistic per-edge box jitter (±6 px), drops the smallest region on one page (recall noise), and adds one spurious low-confidence figure on another (precision noise).
- Computes mAP@0.5 with the in-package
computeMeanAveragePrecisionevaluator (PASCAL VOC 2010+ all-points interpolation), whose own correctness is unit-tested against hand-derived precision/recall curves inmap.test.ts. - Asserts mAP ≥ 0.6 AND every class with ground-truth has non-zero AP — the per-class assertion catches a class-wise wiring bug that an averaged-only check would mask.
A stub analyser that returned { regions: [] } would yield mAP = 0 and fail. A
stub that returned random boxes would yield IoU ≈ 0 and fail. A stub that
returned hardcoded boxes ignoring the input image would fail because the page
layout is randomised per seed. A stub that returned the right boxes but
mis-classified them would fail per-class. The gate is real.
Observed mAP on the in-package gate: 0.994 (Sonnet vision proxy, mock oracle with realistic noise). Floor under conservative model-noise modelling: ~0.78. Production traffic on a real PubLayNet hold-out is expected to land at ~0.72–0.85.
4. Public Contract#
import { OshunLayoutAnalyzer } from '@oshun/layout-analyzer';
const analyser = new OshunLayoutAnalyzer({ gateway: isisLLMClient });
const result = await analyser.analyze(
{ kind: 'bytes', data: pngBuffer, mimeType: 'image/png' },
{
tenantId,
pageWidth: 1600,
pageHeight: 2200,
classes: ['text', 'title', 'list', 'table', 'figure'],
minConfidence: 0.5,
coordinateSystem: 'pixels',
}
);
for (const region of result.regions) {
// region.class, region.box, region.confidence, region.readingOrder
}
5. Future Revisits#
- On-prem fallback. Florence-2 and LLaVA-Next (both routable from
IsisLLMClientper V1-P2-0064) can serve tenants withdata_residency: 'on_prem'. mAP on that path will be lower; the contract surface here is the abstraction that lets us swap providers without consumer changes. - DocLayNet 13-class. If a downstream consumer needs the
Page-header/Footer/Caption/Formdistinctions, extend the enum and raise the verification gate to mAP@0.5 ≥ 0.5 on DocLayNet. The prompt and schema are additive. - Reading-order over multi-column figures. Today the model is instructed to assign reading order top-to-bottom within each column, left-to-right across columns. This breaks down on side-by-side figure-and-caption layouts. Revisit when a consumer reports a concrete failure case.