The gaps that block a thorough scene-from-image composer, ranked by severity and dependency. "Severity" = how much of the composer's value is lost without this. "Blocks" = which downstream blocks cannot start until this is done.
Headline gaps (must-fix)#
G1 · Operator-checkpoint primitive missing#
Severity: critical. Blocks: every other block (A is the foundation). Owner block: A
PhaseType.CHECKPOINT exists as an enum value
(operation-orchestrator/src/types.ts:134) and injectCheckpointPhases
synthetically inserts CHECKPOINT phases at a configured frequency
(execution-orchestrator.ts:1362-1389), but the runtime treats them as ordinary
phases. There is no:
ChainStatus.AWAITING_APPROVALstate (onlyBLOCKEDexists, and it has no approval semantics)approveCheckpoint(pipelineId, phaseId, decision)methodCheckpointDecisionenvelope (approved | rejected | modified,reviewer,reviewedAt,notes,modifications)pipeline_awaiting_approvalevent with the intermediate artifact- Per-checkpoint timeout policy with
auto-approve | auto-reject | escalate - Persistence so checkpoints survive process restart
Without this, the composer can't pause for grounding-gate review, can't loop back if novelty is too high, can't ask for explicit operator approval before paying for object-generation calls. Every multi-phase Isis operation (composer, lipsync, cosmetic, venue capture) hits the same wall.
G2 · Splat ↔ mesh frame bridging missing#
Severity: critical for composed output. Blocks: E baseline, all of B. Owner block: B (initial helper); F (lighting-aware upgrade).
Marble emits bounds: { min, max } in metres world-space
(marble-provider.ts:77-81). Hunyuan/Tripo/Meshy emit GLBs in their own local
frame, usually unit-normalised. Existing scene-unit-normalization.ts:62-84 is
for importing external assets and takes user-declared
sourceUnit/sourceUpAxis — it doesn't bridge two co-generated assets where
neither side has declared a frame. Existing placement-rules-engine.ts:18-83
validates placements assuming everything is already in metres.
Without bridging, a "composed scene" is a bag of files. The composer must:
- Detect the mesh's local-frame bbox via
parseGLB+getDocumentStats(already used inhunyuan-provider.ts:370-402) - Compute a 4×4 transform: scale = (target dimension / source dimension), rotation = axis fix if needed, translation = (anchor − scaled bbox centre)
- Record the transform in the manifest under
metadata.sceneFromImage.placements
This is ~40 LoC of math plus tests. The lighting-aware upgrade in block F extends it to estimate the splat's local light field at the placement point and propagate that to the bake step.
G3 · Mesh-to-Gaussians rasteriser missing#
Severity: critical for "single composed .spz" output. Blocks:
F (depends on baked Gaussians being
available to score coherence). Owner block:
E
To produce a single composed .spz rather than "splat + meshes side by side",
we need to convert each foreground mesh into Gaussians and merge them into the
Marble splat. There is no mesh-to-Gaussians rasteriser anywhere in
libs/isis/gaussian-splatting/.
Three implementation paths exist; block E ships path 1 + path 3:
- Surface sampling (baseline). Poisson-disc sample N points across the mesh surface, fit one Gaussian per sample with scale ∝ local surface area, color from the diffuse texture, opacity = 1, rotation aligned to the surface normal.
- Differentiable splatting fit (high fidelity). Render the mesh from M
viewpoints, run gradient-descent Gaussian fitting against the renders for K
steps. Requires a differentiable renderer; we'd integrate
gsplat/nerfstudioRust bindings vialibs/isis/3d-inference-local. - Hybrid. Surface sampling for initialization, then 100–500 steps of differentiable refinement against PBR renders.
Block E baselines on path 1 and adds path 3 as an opt-in quality: 'high' flag.
Path 2 alone is rejected (no warm start = slow convergence).
G4 · Lighting consistency missing#
Severity: high — the single highest-leverage step for composed-scene quality perception. Blocks: none structurally, but without it the composed scene looks visibly wrong (mesh-baked Gaussians don't match the environment's lighting). Owner block: F
Marble bakes radiance (lighting + albedo combined) into per-Gaussian spherical-harmonic coefficients. Hunyuan emits albedo + normal + PBR factors that have never been lit by the Marble environment. Naively rasterising the mesh into Gaussians using its own albedo produces meshes that are visibly "flat-lit" against an env that has directional light, ambient occlusion, and color cast.
Block F adds:
- Environment SH light field estimation — sample the Marble splat's SH at the mesh's placement point, derive an irradiance environment map.
- Apply that lighting to the mesh's PBR shading at sample time, before fitting Gaussians.
- Composition coherence gate — render the composed scene from 6–8 views and score perceptual consistency between baked-mesh regions and env regions via a perceptual-loss VLM.
G5 · .spz exporter missing#
Severity: critical for any external interop. Blocks: E output, I gallery. Owner block: D
This was task #3 in the original adoption work and was deferred because
implementing a bit-exact Niantic .spz codec without their reference loader to
verify against is risky. Block D picks it up properly: download the Niantic
reference, write the encoder against the public spec
(https://github.com/nianticlabs/spz), round-trip a corpus of reference splats
and assert byte-level equality.
The current IsisSplatExporter (isis-splat-exporter.ts) writes a proprietary
.isplat format for Unreal/Unity and is not Niantic-compatible. It stays — it's
a different surface — but block D adds .spz alongside it.
Secondary gaps (must-fix for thoroughness)#
G6 · Foreground detection subphase missing#
Severity: medium — needed for VLM-detected and cropped-regions modes.
Blocks: the vlm-detected and cropped-regions ObjectSpec source types
in C. Owner
block: C
The composer needs to know what to mesh-generate. Three sources:
- Explicit prompts — caller supplies
[{ prompt: 'cypress tree', anchor: 'centre-ground' }, …] - Cropped regions — caller supplies bboxes; composer crops the source image and sends each crop to the mesh provider's image-to-3D path
- VLM-detected — composer calls a VLM adapter (reuse the grounding gate's
ThreeDQualityGroundingVlmAdapter) to detect salient foreground
Block C adds a ForegroundDetectionSubphase with IoU-based overlap merge,
background-vs-foreground classifier, and configurable detection policy (topK,
minConfidence, maxObjects, excludeLabels).
G7 · Anchor-constraint DSL missing#
Severity: medium — needed for non-trivial scenes. Blocks: multi-object placement scenarios. Owner block: C
A single anchor: 'centre-ground' string is fine for one-object scenes. Real
scenes have spatial constraints between objects ("the lantern is 1.5m forward of
the cypress tree along its facing"; "the chair is on the floor nearest the
camera"). Block C ships a small constraint-solver-friendly DSL covering:
absolute world position, bbox-relative ("on floor inside this bbox"),
anchor-relative ("forward of {object-id} by N metres"), surface-snapped ("on
nearest surface below this point", requires splat depth queries).
G8 · Provider routing per object missing#
Severity: low for v1; medium long-term. Blocks: none initially. Owner block: C
The composer should route each object to a different mesh provider based on
characteristics (high-detail character → Hunyuan; simple prop → Tripo;
hard-surface mechanical → Meshy). Block C adds a MeshProviderRouter that
reuses unified-generation-pipeline.ts's scoring infrastructure.
G9 · State persistence missing#
Severity: high for production. Blocks: resume-after-crash; resume-after-checkpoint-rejection. Owner block: H
PipelineOrchestrator keeps chains in an in-memory Map
(pipeline-orchestrator.ts:1088). If the process restarts, all in-flight
pipelines are lost. The composer needs to survive restarts (a typical run takes
5–30 minutes; deploys happen).
Block H adds Prisma models for Pipeline, Chain, Phase, Checkpoint,
Artifact, with a resumePipeline(pipelineId, fromPhaseId, modifications) API.
G10 · Job-envelope dispatch not wired#
Severity: high for production load. Blocks: running composer at scale (BFF process gets pinned by GPU work). Owner block: G
Heavy phases (Marble generation, mesh generation, splat bake) currently call
providers in-process. A production composer dispatches them via
@isis/job-envelope to the GPU worker pool through Redis.
G11 · Observability and policy not wired#
Severity: medium. Blocks: running composer in a regulated/multi-tenant environment. Owner block: I
The composer needs to register every run as an OutputRecord for the gallery,
write composited-from lineage edges, enforce per-run token-budget ceilings,
run model-governance-3d license checks against the destination license, and
feed anomaly-detection with object-distribution samples so we notice drift.
G12 · Agent-consensus checkpoint policy not adopted#
Severity: low — opt-in feature. Blocks: unattended batch runs with quality gating. Owner block: I
@isis/agent-consensus provides multi-LLM debate and convergence. The
composer's checkpoints can use it as an "automated reviewer" — three agents
evaluate the intermediate artifact and vote on whether to proceed. Lets the
composer run unattended in batch with stricter-than-grounding-gate quality
criteria.
Gap dependency graph#
A (checkpoint primitive)
├─ B (composer library)
│ ├─ C (detection/routing)
│ └─ E (bake pipeline)
│ ├─ D (.spz exporter)
│ └─ F (lighting consistency)
├─ G (job-envelope dispatch)
├─ H (state persistence)
└─ I (observability/policy/consensus)
A is the only hard gate. B depends on A. C, E, F can run in parallel once B is far enough along that its phase interfaces are stable. D is independent but feeds E. G, H, I can start any time after A.
Gaps explicitly out of scope for this plan#
- Marble provider improvements beyond what shipped. Future work like passing
alternative encoder models, supporting
marble-2.xwhen it ships, etc. Tracked separately. - Grounding-gate adapter implementations beyond the test fixture. Production needs a real Claude-vision adapter and/or local VLM adapter; those are tracked under the existing 3d-quality-gates roadmap, not here.
- Splat-to-mesh extraction. The reverse direction (extract a clean mesh from a splat scene) is interesting but outside the composer's surface.
- Animation in composed scenes. Static-pose only for v1–v3.
marble-1.1'senvironment-motionmode is supported as a passthrough but the composer does not generate per-object animation.