# 02 — Gap Analysis

The gaps that block a thorough scene-from-image composer, ranked by severity and
dependency. "Severity" = how much of the composer's value is lost without this.
"Blocks" = which downstream blocks cannot start until this is done.

## Headline gaps (must-fix)

### G1 · Operator-checkpoint primitive missing

**Severity:** critical. **Blocks:** every other block (A is the foundation).
**Owner block:** [A](./block-plans/A-orchestrator-checkpoint-primitive.md)

`PhaseType.CHECKPOINT` exists as an enum value
(`operation-orchestrator/src/types.ts:134`) and `injectCheckpointPhases`
synthetically inserts CHECKPOINT phases at a configured frequency
(`execution-orchestrator.ts:1362-1389`), but the runtime treats them as ordinary
phases. There is no:

- `ChainStatus.AWAITING_APPROVAL` state (only `BLOCKED` exists, and it has no
  approval semantics)
- `approveCheckpoint(pipelineId, phaseId, decision)` method
- `CheckpointDecision` envelope (`approved | rejected | modified`, `reviewer`,
  `reviewedAt`, `notes`, `modifications`)
- `pipeline_awaiting_approval` event with the intermediate artifact
- Per-checkpoint timeout policy with `auto-approve | auto-reject | escalate`
- Persistence so checkpoints survive process restart

Without this, the composer can't pause for grounding-gate review, can't loop
back if novelty is too high, can't ask for explicit operator approval before
paying for object-generation calls. Every multi-phase Isis operation (composer,
lipsync, cosmetic, venue capture) hits the same wall.

### G2 · Splat ↔ mesh frame bridging missing

**Severity:** critical for composed output. **Blocks:**
[E](./block-plans/E-mesh-to-gaussians-bake-pipeline.md) baseline, all of
[B](./block-plans/B-scene-from-image-composer-library.md). **Owner block:**
[B](./block-plans/B-scene-from-image-composer-library.md) (initial helper);
[F](./block-plans/F-lighting-consistency.md) (lighting-aware upgrade).

Marble emits `bounds: { min, max }` in metres world-space
(`marble-provider.ts:77-81`). Hunyuan/Tripo/Meshy emit GLBs in their own local
frame, usually unit-normalised. Existing `scene-unit-normalization.ts:62-84` is
for **importing external assets** and takes user-declared
`sourceUnit`/`sourceUpAxis` — it doesn't bridge two co-generated assets where
neither side has declared a frame. Existing `placement-rules-engine.ts:18-83`
validates placements assuming everything is already in metres.

Without bridging, a "composed scene" is a bag of files. The composer must:

1. Detect the mesh's local-frame bbox via `parseGLB` + `getDocumentStats`
   (already used in `hunyuan-provider.ts:370-402`)
2. Compute a 4×4 transform: scale = (target dimension / source dimension),
   rotation = axis fix if needed, translation = (anchor − scaled bbox centre)
3. Record the transform in the manifest under
   `metadata.sceneFromImage.placements`

This is ~40 LoC of math plus tests. The lighting-aware upgrade in block F
extends it to estimate the splat's local light field at the placement point and
propagate that to the bake step.

### G3 · Mesh-to-Gaussians rasteriser missing

**Severity:** critical for "single composed `.spz`" output. **Blocks:**
[F](./block-plans/F-lighting-consistency.md) (depends on baked Gaussians being
available to score coherence). **Owner block:**
[E](./block-plans/E-mesh-to-gaussians-bake-pipeline.md)

To produce a single composed `.spz` rather than "splat + meshes side by side",
we need to convert each foreground mesh into Gaussians and merge them into the
Marble splat. There is no mesh-to-Gaussians rasteriser anywhere in
`libs/isis/gaussian-splatting/`.

Three implementation paths exist; block E ships path 1 + path 3:

1. **Surface sampling (baseline).** Poisson-disc sample N points across the mesh
   surface, fit one Gaussian per sample with scale ∝ local surface area, color
   from the diffuse texture, opacity = 1, rotation aligned to the surface
   normal.
2. **Differentiable splatting fit (high fidelity).** Render the mesh from M
   viewpoints, run gradient-descent Gaussian fitting against the renders for K
   steps. Requires a differentiable renderer; we'd integrate
   `gsplat`/`nerfstudio` Rust bindings via `libs/isis/3d-inference-local`.
3. **Hybrid.** Surface sampling for initialization, then 100–500 steps of
   differentiable refinement against PBR renders.

Block E baselines on path 1 and adds path 3 as an opt-in `quality: 'high'` flag.
Path 2 alone is rejected (no warm start = slow convergence).

### G4 · Lighting consistency missing

**Severity:** high — the single highest-leverage step for composed-scene quality
perception. **Blocks:** none structurally, but without it the composed scene
looks visibly wrong (mesh-baked Gaussians don't match the environment's
lighting). **Owner block:** [F](./block-plans/F-lighting-consistency.md)

Marble bakes radiance (lighting + albedo combined) into per-Gaussian
spherical-harmonic coefficients. Hunyuan emits albedo + normal + PBR factors
that have never been lit by the Marble environment. Naively rasterising the mesh
into Gaussians using its own albedo produces meshes that are visibly "flat-lit"
against an env that has directional light, ambient occlusion, and color cast.

Block F adds:

1. Environment SH light field estimation — sample the Marble splat's SH at the
   mesh's placement point, derive an irradiance environment map.
2. Apply that lighting to the mesh's PBR shading at sample time, before fitting
   Gaussians.
3. Composition coherence gate — render the composed scene from 6–8 views and
   score perceptual consistency between baked-mesh regions and env regions via a
   perceptual-loss VLM.

### G5 · `.spz` exporter missing

**Severity:** critical for any external interop. **Blocks:**
[E](./block-plans/E-mesh-to-gaussians-bake-pipeline.md) output,
[I](./block-plans/I-observability-policy-and-consensus.md) gallery. **Owner
block:** [D](./block-plans/D-spz-exporter.md)

This was task #3 in the original adoption work and was deferred because
implementing a bit-exact Niantic `.spz` codec without their reference loader to
verify against is risky. Block D picks it up properly: download the Niantic
reference, write the encoder against the public spec
(`https://github.com/nianticlabs/spz`), round-trip a corpus of reference splats
and assert byte-level equality.

The current `IsisSplatExporter` (`isis-splat-exporter.ts`) writes a proprietary
`.isplat` format for Unreal/Unity and is not Niantic-compatible. It stays — it's
a different surface — but block D adds `.spz` alongside it.

## Secondary gaps (must-fix for thoroughness)

### G6 · Foreground detection subphase missing

**Severity:** medium — needed for VLM-detected and cropped-regions modes.
**Blocks:** the `vlm-detected` and `cropped-regions` `ObjectSpec` source types
in [C](./block-plans/C-foreground-detection-and-provider-routing.md). **Owner
block:** [C](./block-plans/C-foreground-detection-and-provider-routing.md)

The composer needs to know what to mesh-generate. Three sources:

1. **Explicit prompts** — caller supplies
   `[{ prompt: 'cypress tree', anchor: 'centre-ground' }, …]`
2. **Cropped regions** — caller supplies bboxes; composer crops the source image
   and sends each crop to the mesh provider's image-to-3D path
3. **VLM-detected** — composer calls a VLM adapter (reuse the grounding gate's
   `ThreeDQualityGroundingVlmAdapter`) to detect salient foreground

Block C adds a `ForegroundDetectionSubphase` with IoU-based overlap merge,
background-vs-foreground classifier, and configurable detection policy (`topK`,
`minConfidence`, `maxObjects`, `excludeLabels`).

### G7 · Anchor-constraint DSL missing

**Severity:** medium — needed for non-trivial scenes. **Blocks:** multi-object
placement scenarios. **Owner block:**
[C](./block-plans/C-foreground-detection-and-provider-routing.md)

A single `anchor: 'centre-ground'` string is fine for one-object scenes. Real
scenes have spatial constraints between objects ("the lantern is 1.5m forward of
the cypress tree along its facing"; "the chair is on the floor nearest the
camera"). Block C ships a small constraint-solver-friendly DSL covering:
absolute world position, bbox-relative ("on floor inside this bbox"),
anchor-relative ("forward of {object-id} by N metres"), surface-snapped ("on
nearest surface below this point", requires splat depth queries).

### G8 · Provider routing per object missing

**Severity:** low for v1; medium long-term. **Blocks:** none initially. **Owner
block:** [C](./block-plans/C-foreground-detection-and-provider-routing.md)

The composer should route each object to a different mesh provider based on
characteristics (high-detail character → Hunyuan; simple prop → Tripo;
hard-surface mechanical → Meshy). Block C adds a `MeshProviderRouter` that
reuses `unified-generation-pipeline.ts`'s scoring infrastructure.

### G9 · State persistence missing

**Severity:** high for production. **Blocks:** resume-after-crash;
resume-after-checkpoint-rejection. **Owner block:**
[H](./block-plans/H-state-persistence-and-resume.md)

`PipelineOrchestrator` keeps chains in an in-memory `Map`
(`pipeline-orchestrator.ts:1088`). If the process restarts, all in-flight
pipelines are lost. The composer needs to survive restarts (a typical run takes
5–30 minutes; deploys happen).

Block H adds Prisma models for `Pipeline`, `Chain`, `Phase`, `Checkpoint`,
`Artifact`, with a `resumePipeline(pipelineId, fromPhaseId, modifications)` API.

### G10 · Job-envelope dispatch not wired

**Severity:** high for production load. **Blocks:** running composer at scale
(BFF process gets pinned by GPU work). **Owner block:**
[G](./block-plans/G-job-envelope-and-gpu-dispatch.md)

Heavy phases (Marble generation, mesh generation, splat bake) currently call
providers in-process. A production composer dispatches them via
`@isis/job-envelope` to the GPU worker pool through Redis.

### G11 · Observability and policy not wired

**Severity:** medium. **Blocks:** running composer in a regulated/multi-tenant
environment. **Owner block:**
[I](./block-plans/I-observability-policy-and-consensus.md)

The composer needs to register every run as an `OutputRecord` for the gallery,
write `composited-from` lineage edges, enforce per-run `token-budget` ceilings,
run `model-governance-3d` license checks against the destination license, and
feed `anomaly-detection` with object-distribution samples so we notice drift.

### G12 · Agent-consensus checkpoint policy not adopted

**Severity:** low — opt-in feature. **Blocks:** unattended batch runs with
quality gating. **Owner block:**
[I](./block-plans/I-observability-policy-and-consensus.md)

`@isis/agent-consensus` provides multi-LLM debate and convergence. The
composer's checkpoints can use it as an "automated reviewer" — three agents
evaluate the intermediate artifact and vote on whether to proceed. Lets the
composer run unattended in batch with stricter-than-grounding-gate quality
criteria.

## Gap dependency graph

```
A (checkpoint primitive)
├─ B (composer library)
│  ├─ C (detection/routing)
│  └─ E (bake pipeline)
│     ├─ D (.spz exporter)
│     └─ F (lighting consistency)
├─ G (job-envelope dispatch)
├─ H (state persistence)
└─ I (observability/policy/consensus)
```

A is the only hard gate. B depends on A. C, E, F can run in parallel once B is
far enough along that its phase interfaces are stable. D is independent but
feeds E. G, H, I can start any time after A.

## Gaps explicitly out of scope for this plan

- **Marble provider improvements** beyond what shipped. Future work like passing
  alternative encoder models, supporting `marble-2.x` when it ships, etc.
  Tracked separately.
- **Grounding-gate adapter implementations** beyond the test fixture. Production
  needs a real Claude-vision adapter and/or local VLM adapter; those are tracked
  under the existing 3d-quality-gates roadmap, not here.
- **Splat-to-mesh extraction.** The reverse direction (extract a clean mesh from
  a splat scene) is interesting but outside the composer's surface.
- **Animation in composed scenes.** Static-pose only for v1–v3. `marble-1.1`'s
  `environment-motion` mode is supported as a passthrough but the composer does
  not generate per-object animation.
