Architectural decisions for the scene-from-image composer. One section per decision: Context, Options considered, Decision, Consequences, Status.
ADR-001 · Composer library placement#
Status: Accepted (user-confirmed in plan-mode iteration).
Context. The composer touches multiple existing libraries: 3D providers, audio providers, grounding gate, scene-assembly placement validators, output manifest. It needs a home that doesn't pollute libraries that have a single-purpose identity.
Options considered.
- New library
@isis/scene-from-image-composer— dedicated lib. - Inside
libs/isis/3d-scene-assembly/— adds an orchestrator to a currently-purely-declarative library. - Inside
libs/isis/3d-generation/src/workflow/— forces3d-generationto depend onaudio-generationand3d-quality-gates(currently it doesn't).
Decision. Option 1 — new lib libs/isis/scene-from-image-composer/.
Consequences.
- One more entry in
libs/isis/(already 35+; cost is trivial). 3d-scene-assemblystays purely declarative — its orchestration-free identity is preserved for future readers.- Composer owns its README, types, public surface, tests. Easier to evolve independently.
- Path mapping in
tsconfig.base.jsonneeds an entry;pnpm-workspace.yamlpicks it up automatically.
ADR-002 · Orchestration substrate#
Status: Accepted (recommended; user asked for the thorough version).
Context. The composer is a 5–7 phase linear-with-checkpoints pipeline. Two
existing substrates could host it: UnifiedGenerationPipeline (single-
modality, multi-provider router) or PipelineOrchestrator (multi-phase,
multi-chain). A custom thin state machine is also viable.
Options considered.
- Custom thin async state machine — ~150 LoC, simple, easy to test, but
reinvents what
operation-orchestratorprovides and forks the eventual operations dashboard. - PipelineOrchestrator with a real checkpoint primitive — uses the existing
substrate; requires extending operation-orchestrator first to add
AWAITING_APPROVAL,approveCheckpoint, etc. (block A). - Hybrid: thin state machine that emits orchestrator-shaped events for forward compatibility — half-measure that doesn't actually integrate.
Decision. Option 2 — extend operation-orchestrator (block A) and build the
composer on PipelineOrchestrator.
Reasoning. The user asked for the long-term option. The composer is one of N
future multi-phase Isis operations; building bespoke state machines for each is
the predictable two-years-in regret. The half-implemented checkpoint machinery
(PhaseType.CHECKPOINT, injectCheckpointPhases) is a clear invitation to
finish the primitive. The ~2 weeks to extend the orchestrator returns value
across composer + lipsync pipeline + cosmetic pipeline + venue capture
pipeline + every future multi-phase op.
Consequences.
- Block A is a hard gate for block B and downstream.
- The composer's state machine becomes a configuration of chains and phases, not a procedural function. Slightly heavier to define, much easier to observe and persist.
- Other multi-phase operations can adopt the same primitive immediately.
- The composer benefits from
retry-orchestrator,failure-orchestrator, batch composition, dependency graphs — none of which we'd have built into the thin state machine.
ADR-003 · Object source modes#
Status: Accepted (user asked for the thorough version; all four modes ship).
Context. "Where do the foreground objects come from?" has at least three distinct production use cases, each with different ergonomics.
Options considered.
- Explicit prompts only — caller supplies an array of
{ prompt, anchor }objects. Deterministic, scriptable, simple. - Cropped image regions — caller supplies bboxes; composer crops the source image and feeds each crop to the provider's image-to-3D path. Preserves visual identity ("that specific tree", not "a tree").
- VLM-detected salient objects — composer auto-detects via the grounding-gate's VLM adapter. Best for batch unattended runs.
- Skip object generation — environment-only mode. For panoramas with no foreground.
- Mixed — caller supplies a heterogeneous list spanning the above modes.
Decision. All five modes ship. They are unified through a single
ObjectSpec discriminated union in
block C:
type ObjectSpec =
| { source: 'explicit'; prompt: string; anchor: AnchorSpec }
| {
source: 'cropped';
bbox: BBox;
prompt?: string;
anchor: AnchorSpec;
removeBackground?: boolean;
}
| {
source: 'vlm-detected';
bbox: BBox;
label: string;
confidence: number;
anchor: AnchorSpec;
}
| { source: 'mixed'; specs: ObjectSpec[] };
The "skip object generation" path is just objects: [].
Reasoning. The three primary modes cover non-overlapping use cases: explicit for CI/golden tests, cropped for designer-driven workflows, VLM- detected for batch. Investing in all three up front avoids retrofitting the phase contract later.
Consequences.
- Block C is larger (~5w) than a single-mode version.
- The
ForegroundDetectionSubphasebecomes a first-class citizen in the composer, not a v1.1 retrofit. - Mode selection happens at composer config; downstream phases stay
mode-agnostic — they all consume
PlacedObjectafter detection + provider call.
ADR-004 · Splat ↔ mesh frame bridging strategy#
Status: Accepted (user asked for the thorough version).
Context. Marble emits a Gaussian-splat scene with bounds in metres. Foreground meshes are unit-normalised local-frame GLBs. Producing a "composed scene" requires choosing how to bridge them.
Options considered.
- Don't bridge — manifest lists splat and meshes as independent files. Consumer assembles at runtime. Forever an asterisk on "composed".
- Compute a transform per mesh, write to manifest — each mesh gets a 4×4 transform; consumer applies at load time. Lean, ~40 LoC.
- Bake meshes into the splat as Gaussians, merge into single
.spz— true single-file composed scene; works in any splat viewer. Requires.spzcodec + mesh-to-Gaussians rasteriser + splat merging + lighting consistency. ~6–10 weeks.
Decision. Option 3 — bake meshes into the splat. Block E does the
rasterisation + merging; block F does the lighting consistency; block D does the
.spz codec.
Reasoning. The user asked for the long-term option. Option 2 produces artefacts that every consumer (web viewer, Unreal plugin, mobile AR app, Blender importer) has to specifically support. Option 3 produces artefacts that work in any conformant splat viewer with no per-consumer code. Lighting mismatch (Marble's baked radiance vs Hunyuan's PBR) is also fundamentally unfixable in the consumer layer — it has to happen at bake time.
Consequences.
- Hardest block to ship; ~6–10 weeks for E + F + the dependency on D.
- The composer also keeps the source meshes + transforms + un-baked env splat in
a
sources/subdirectory of the manifest. Bake is destructive; sources let you re-bake later with different mesh placements without re-running Marble. - The "transform-only" intermediate output (option 2) ships in Phase 1 as a
stop-gap so block B is testable before block E lands. Phase 2 deprecates it in
favour of the baked
.spz.
ADR-005 · Job dispatch#
Status: Accepted.
Context. Heavy phases (Marble call, mesh generation, splat bake) are GPU workloads. Calling them in-process from the BFF blocks the BFF event loop and prevents horizontal scaling.
Options considered.
- In-process direct calls — simplest; appropriate for a v1 demo.
- Dispatch via
@isis/job-envelopeto Redis-backed GPU workers — the established Isis pattern for GPU work. - Dispatch via Kubernetes Job objects — full K8s native; over-engineered for the use case.
Decision. Option 2 — every heavy phase dispatches via @isis/job-envelope.
The composer waits on job IDs, not in-process Promises. Block G wires this.
Consequences.
- Composer can run on a small BFF instance; GPU workers scale independently.
- Phase-completion events come back through the job-envelope event bus, not directly from the provider's polling loop.
- Block H (state persistence) becomes essential — if the BFF restarts, the composer needs to resume against still-running GPU jobs.
ADR-006 · Pipeline state persistence layer#
Status: Accepted.
Context. PipelineOrchestrator is in-memory only. Composer runs are 5–30
minutes; deploys happen during that window. State must survive process restart
and worker restart.
Options considered.
- Postgres via
@isis/database's Prisma layer. - Redis (already used for the GPU job queue).
- Custom file-backed storage.
- None — restart loses all in-flight pipelines.
Decision. Option 1 — Postgres via Prisma. New tables: Pipeline, Chain,
Phase, PhaseArtifact, Checkpoint, CheckpointDecision.
Reasoning. Postgres is already the system of record for manifests
(manifest-storage.ts) and gives us atomicity, indexes, and a stable backup
story. Redis is fine for ephemeral GPU job state but not for multi-day audit
trails. File-backed storage is a footgun in a multi-replica environment.
Consequences.
- Block H ships a Prisma migration + a
PipelineRepositoryclass + changes toPipelineOrchestratorto read/write state through it. - The
resumePipeline(pipelineId, fromPhaseId, modifications)API becomes the primary tool for handling checkpoint rejections. - Backwards compat: pipelines started in-memory before block H lands are abandoned on restart. Acceptable since composer isn't yet in production.
ADR-007 · License/budget/anomaly integration point#
Status: Accepted.
Context. The composer pulls together outputs from multiple providers, each with its own license. It also accumulates real cost across multiple provider calls. Drift in object distributions over time signals model or input changes.
Options considered.
- Per-block ad hoc integration — every block wires its own license check, budget check, anomaly sample. Inconsistent, duplicated.
- Composer-level cross-cutting integration — the composer hooks into
model-governance-3d,token-budget,anomaly-detectiononce, and the integration is shared across all blocks.
Decision. Option 2 — single integration point in the composer. Block I owns the wiring.
Consequences.
- The composer's
composecall accepts (optionally) apolicyContextcarrying budget, license-policy ID, anomaly-detection scope. - Each phase reports back cost + license-tags; composer aggregates and enforces.
- A single hard-fail rule: if any artefact's license is incompatible with the destination license, the whole pipeline halts at the next checkpoint.
ADR-008 · Agent-consensus checkpoint policy#
Status: Provisional — opt-in for v1, default for v2.
Context. Checkpoints (block A) can be approved by a human, by an auto-policy
timeout, or by automated agents (@isis/agent-consensus). Whether to default to
human approval, auto-approve, or agent-consensus is a product decision.
Options considered.
- Human-only. Always wait for an explicit
approveCheckpointcall. - Auto-approve with timeout. If no decision in N minutes, proceed.
- Agent-consensus. Three LLM agents debate the intermediate artifact and vote.
Decision. v1 default = auto-approve with auto-approve-after: 5min
(per-checkpoint configurable). Agent-consensus available as opt-in via
policy: 'consensus-3' per checkpoint. Human-only configurable per checkpoint.
v2 may flip the default.
Consequences.
- v1 composer is usable unattended without writing operator-UI code.
- Agent-consensus integration is real but not on the critical path.
- Human-approval UI is its own product surface, deferred outside this plan.
ADR-009 · Foreground detection adapter sourcing#
Status: Accepted.
Context. Block C's VLM-detected mode needs an object-detection VLM. The
grounding gate already defines ThreeDQualityGroundingVlmAdapter with a similar
contract.
Options considered.
- Reuse
ThreeDQualityGroundingVlmAdapter— single adapter contract across QA gate and detection. - New
SceneFromImageDetectionAdapter— distinct contract, possibly richer (segmentation masks, depth estimation).
Decision. Option 1 — reuse ThreeDQualityGroundingVlmAdapter. Detection
needs label + confidence + boundingBox, all of which the grounding adapter
already returns.
Consequences.
- Adapter authors implement one contract, satisfy two consumers.
- If detection ever needs richer outputs (masks, depth), we extend the shared contract rather than fork it.
ADR-010 · Where placement transforms are stored in the manifest#
Status: Accepted.
Context. OutputManifest.metadata is a free-form Record<string, unknown>
(outputs/src/types.ts:366). The composer needs a typed extension point.
Decision. Composer-specific data lives under metadata.sceneFromImage.*
with a typed schema declared in the composer library and re-exported for
downstream consumers. Specifically:
metadata: {
sceneFromImage: {
schemaVersion: '1.0.0';
placements: Array<{ fileId: OutputFileId; transform: number[16]; anchor: AnchorSpec }>;
sceneBoundsMeters: { min: [n,n,n]; max: [n,n,n] };
bakeMetadata?: { method: 'surface-sampling' | 'differentiable' | 'hybrid'; gaussianCount: number };
lightingMetadata?: { shCoefficients: number[]; estimationMethod: string };
checkpointHistory: Array<{ phase: string; decision: 'approved' | 'rejected' | 'modified'; reviewer: string; reviewedAt: string }>;
policyContext?: { licensePolicyId: string; budgetCents: number; spentCents: number };
}
}
Consequences.
- Manifest validator (
manifest-validator.ts) gets an optional rule set for the composer extension; non-composer manifests are unaffected. - Downstream consumers (Three.js viewer, Unreal plugin, gallery UI) read this typed shape via the composer's exported types.
ADR-011 · Composer public API shape#
Status: Accepted.
Context. Composer entry point should be (a) callable from a single function, (b) compatible with operation-orchestrator's Pipeline-based model, (c) extensible to streaming / long-running scenarios.
Decision. Two-tier API.
-
Convenience:
tsasync function composeSceneFromImage( input: SceneFromImageRequest ): Promise<SceneFromImageResult>;Synchronous from the caller's perspective; awaits the pipeline.
-
Advanced:
tsfunction createSceneFromImagePipeline( input: SceneFromImageRequest ): PipelineDefinition;Returns a pipeline definition that the caller can hand to a configured
PipelineOrchestrator, listen to events on, persist via block H, etc.
The convenience API is implemented as a thin wrapper around the advanced API plus an in-process orchestrator with auto-approving checkpoints.
Consequences.
- Test code uses the convenience API.
- Production code uses the advanced API to get persistence, observability, resume, agent-consensus checkpoints.
- Both surfaces are stable and exported from the lib's public index.