Eve library

02 — Gap Analysis

Other

Revision 1

On this page

The gaps that block a thorough scene-from-image composer, ranked by severity and dependency. "Severity" = how much of the composer's value is lost without this. "Blocks" = which downstream blocks cannot start until this is done.

Headline gaps (must-fix)#

G1 · Operator-checkpoint primitive missing#

Severity: critical. Blocks: every other block (A is the foundation). Owner block: A

PhaseType.CHECKPOINT exists as an enum value (operation-orchestrator/src/types.ts:134) and injectCheckpointPhases synthetically inserts CHECKPOINT phases at a configured frequency (execution-orchestrator.ts:1362-1389), but the runtime treats them as ordinary phases. There is no:

  • ChainStatus.AWAITING_APPROVAL state (only BLOCKED exists, and it has no approval semantics)
  • approveCheckpoint(pipelineId, phaseId, decision) method
  • CheckpointDecision envelope (approved | rejected | modified, reviewer, reviewedAt, notes, modifications)
  • pipeline_awaiting_approval event with the intermediate artifact
  • Per-checkpoint timeout policy with auto-approve | auto-reject | escalate
  • Persistence so checkpoints survive process restart

Without this, the composer can't pause for grounding-gate review, can't loop back if novelty is too high, can't ask for explicit operator approval before paying for object-generation calls. Every multi-phase Isis operation (composer, lipsync, cosmetic, venue capture) hits the same wall.

G2 · Splat ↔ mesh frame bridging missing#

Severity: critical for composed output. Blocks: E baseline, all of B. Owner block: B (initial helper); F (lighting-aware upgrade).

Marble emits bounds: { min, max } in metres world-space (marble-provider.ts:77-81). Hunyuan/Tripo/Meshy emit GLBs in their own local frame, usually unit-normalised. Existing scene-unit-normalization.ts:62-84 is for importing external assets and takes user-declared sourceUnit/sourceUpAxis — it doesn't bridge two co-generated assets where neither side has declared a frame. Existing placement-rules-engine.ts:18-83 validates placements assuming everything is already in metres.

Without bridging, a "composed scene" is a bag of files. The composer must:

  1. Detect the mesh's local-frame bbox via parseGLB + getDocumentStats (already used in hunyuan-provider.ts:370-402)
  2. Compute a 4×4 transform: scale = (target dimension / source dimension), rotation = axis fix if needed, translation = (anchor − scaled bbox centre)
  3. Record the transform in the manifest under metadata.sceneFromImage.placements

This is ~40 LoC of math plus tests. The lighting-aware upgrade in block F extends it to estimate the splat's local light field at the placement point and propagate that to the bake step.

G3 · Mesh-to-Gaussians rasteriser missing#

Severity: critical for "single composed .spz" output. Blocks: F (depends on baked Gaussians being available to score coherence). Owner block: E

To produce a single composed .spz rather than "splat + meshes side by side", we need to convert each foreground mesh into Gaussians and merge them into the Marble splat. There is no mesh-to-Gaussians rasteriser anywhere in libs/isis/gaussian-splatting/.

Three implementation paths exist; block E ships path 1 + path 3:

  1. Surface sampling (baseline). Poisson-disc sample N points across the mesh surface, fit one Gaussian per sample with scale ∝ local surface area, color from the diffuse texture, opacity = 1, rotation aligned to the surface normal.
  2. Differentiable splatting fit (high fidelity). Render the mesh from M viewpoints, run gradient-descent Gaussian fitting against the renders for K steps. Requires a differentiable renderer; we'd integrate gsplat/nerfstudio Rust bindings via libs/isis/3d-inference-local.
  3. Hybrid. Surface sampling for initialization, then 100–500 steps of differentiable refinement against PBR renders.

Block E baselines on path 1 and adds path 3 as an opt-in quality: 'high' flag. Path 2 alone is rejected (no warm start = slow convergence).

G4 · Lighting consistency missing#

Severity: high — the single highest-leverage step for composed-scene quality perception. Blocks: none structurally, but without it the composed scene looks visibly wrong (mesh-baked Gaussians don't match the environment's lighting). Owner block: F

Marble bakes radiance (lighting + albedo combined) into per-Gaussian spherical-harmonic coefficients. Hunyuan emits albedo + normal + PBR factors that have never been lit by the Marble environment. Naively rasterising the mesh into Gaussians using its own albedo produces meshes that are visibly "flat-lit" against an env that has directional light, ambient occlusion, and color cast.

Block F adds:

  1. Environment SH light field estimation — sample the Marble splat's SH at the mesh's placement point, derive an irradiance environment map.
  2. Apply that lighting to the mesh's PBR shading at sample time, before fitting Gaussians.
  3. Composition coherence gate — render the composed scene from 6–8 views and score perceptual consistency between baked-mesh regions and env regions via a perceptual-loss VLM.

G5 · .spz exporter missing#

Severity: critical for any external interop. Blocks: E output, I gallery. Owner block: D

This was task #3 in the original adoption work and was deferred because implementing a bit-exact Niantic .spz codec without their reference loader to verify against is risky. Block D picks it up properly: download the Niantic reference, write the encoder against the public spec (https://github.com/nianticlabs/spz), round-trip a corpus of reference splats and assert byte-level equality.

The current IsisSplatExporter (isis-splat-exporter.ts) writes a proprietary .isplat format for Unreal/Unity and is not Niantic-compatible. It stays — it's a different surface — but block D adds .spz alongside it.

Secondary gaps (must-fix for thoroughness)#

G6 · Foreground detection subphase missing#

Severity: medium — needed for VLM-detected and cropped-regions modes. Blocks: the vlm-detected and cropped-regions ObjectSpec source types in C. Owner block: C

The composer needs to know what to mesh-generate. Three sources:

  1. Explicit prompts — caller supplies [{ prompt: 'cypress tree', anchor: 'centre-ground' }, …]
  2. Cropped regions — caller supplies bboxes; composer crops the source image and sends each crop to the mesh provider's image-to-3D path
  3. VLM-detected — composer calls a VLM adapter (reuse the grounding gate's ThreeDQualityGroundingVlmAdapter) to detect salient foreground

Block C adds a ForegroundDetectionSubphase with IoU-based overlap merge, background-vs-foreground classifier, and configurable detection policy (topK, minConfidence, maxObjects, excludeLabels).

G7 · Anchor-constraint DSL missing#

Severity: medium — needed for non-trivial scenes. Blocks: multi-object placement scenarios. Owner block: C

A single anchor: 'centre-ground' string is fine for one-object scenes. Real scenes have spatial constraints between objects ("the lantern is 1.5m forward of the cypress tree along its facing"; "the chair is on the floor nearest the camera"). Block C ships a small constraint-solver-friendly DSL covering: absolute world position, bbox-relative ("on floor inside this bbox"), anchor-relative ("forward of {object-id} by N metres"), surface-snapped ("on nearest surface below this point", requires splat depth queries).

G8 · Provider routing per object missing#

Severity: low for v1; medium long-term. Blocks: none initially. Owner block: C

The composer should route each object to a different mesh provider based on characteristics (high-detail character → Hunyuan; simple prop → Tripo; hard-surface mechanical → Meshy). Block C adds a MeshProviderRouter that reuses unified-generation-pipeline.ts's scoring infrastructure.

G9 · State persistence missing#

Severity: high for production. Blocks: resume-after-crash; resume-after-checkpoint-rejection. Owner block: H

PipelineOrchestrator keeps chains in an in-memory Map (pipeline-orchestrator.ts:1088). If the process restarts, all in-flight pipelines are lost. The composer needs to survive restarts (a typical run takes 5–30 minutes; deploys happen).

Block H adds Prisma models for Pipeline, Chain, Phase, Checkpoint, Artifact, with a resumePipeline(pipelineId, fromPhaseId, modifications) API.

G10 · Job-envelope dispatch not wired#

Severity: high for production load. Blocks: running composer at scale (BFF process gets pinned by GPU work). Owner block: G

Heavy phases (Marble generation, mesh generation, splat bake) currently call providers in-process. A production composer dispatches them via @isis/job-envelope to the GPU worker pool through Redis.

G11 · Observability and policy not wired#

Severity: medium. Blocks: running composer in a regulated/multi-tenant environment. Owner block: I

The composer needs to register every run as an OutputRecord for the gallery, write composited-from lineage edges, enforce per-run token-budget ceilings, run model-governance-3d license checks against the destination license, and feed anomaly-detection with object-distribution samples so we notice drift.

G12 · Agent-consensus checkpoint policy not adopted#

Severity: low — opt-in feature. Blocks: unattended batch runs with quality gating. Owner block: I

@isis/agent-consensus provides multi-LLM debate and convergence. The composer's checkpoints can use it as an "automated reviewer" — three agents evaluate the intermediate artifact and vote on whether to proceed. Lets the composer run unattended in batch with stricter-than-grounding-gate quality criteria.

Gap dependency graph#

text
A (checkpoint primitive)
├─ B (composer library)
│  ├─ C (detection/routing)
│  └─ E (bake pipeline)
│     ├─ D (.spz exporter)
│     └─ F (lighting consistency)
├─ G (job-envelope dispatch)
├─ H (state persistence)
└─ I (observability/policy/consensus)

A is the only hard gate. B depends on A. C, E, F can run in parallel once B is far enough along that its phase interfaces are stable. D is independent but feeds E. G, H, I can start any time after A.

Gaps explicitly out of scope for this plan#

  • Marble provider improvements beyond what shipped. Future work like passing alternative encoder models, supporting marble-2.x when it ships, etc. Tracked separately.
  • Grounding-gate adapter implementations beyond the test fixture. Production needs a real Claude-vision adapter and/or local VLM adapter; those are tracked under the existing 3d-quality-gates roadmap, not here.
  • Splat-to-mesh extraction. The reverse direction (extract a clean mesh from a splat scene) is interesting but outside the composer's surface.
  • Animation in composed scenes. Static-pose only for v1–v3. marble-1.1's environment-motion mode is supported as a passthrough but the composer does not generate per-object animation.