End-to-end scene composition from a single input image. Fuses environment generation (World Labs Marble), foreground meshes (Hunyuan / Tripo / Meshy), ambient SFX (ElevenLabs SFX), and an input-grounding QA gate behind a single async call.
Implements all nine blocks (A through I) of the
ISIS_GAPS plan. Block A's operator-checkpoint
primitive in @isis/operation-orchestrator is the foundation; subsequent blocks
B-I layer additional capabilities on top while preserving the lib's two-tier
public surface.
Status#
| Phase | Scope | Status |
|---|---|---|
| 1 — MVP slice | explicit-prompt object mode; transform-only frame bridging (no bake); in-process providers; auto-approving checkpoints | shipped — block B |
| 2a — Thorough | full ObjectSpec union (explicit / cropped / vlm-detected / mixed); foreground detection subphase; anchor DSL + solver; mesh-provider router | shipped — block C |
| 2b — Thorough | baked single .spz output (surface-sampling + differentiable hybrid); lighting consistency; persistence + resume |
shipped — D/E/F/H |
| 3 — Production | GPU dispatch via Redis JobChannel; gallery; lineage; budget; license; agent-consensus | shipped — G/I |
Install#
Workspace package — referenced via pnpm catalog:
{
"dependencies": {
"@isis/scene-from-image-composer": "workspace:*"
}
}
Public API — two tiers#
Convenience: composeSceneFromImage#
Runs end-to-end with ephemeral in-process orchestrators. The right call for tests, scripts, and one-shot CLI use.
import { composeSceneFromImage } from '@isis/scene-from-image-composer';
import {
createMarbleProvider,
createHunyuanProvider,
} from '@isis/3d-generation/providers';
import { createSFXProvider } from '@isis/audio-generation/generation';
const result = await composeSceneFromImage({
assetId: 'cypress-garden-001',
sourceImage: imageBytes, // Buffer of the input image (PNG/JPEG)
providers: {
marble: createMarbleProvider({ apiKey: process.env.WORLD_LABS_API_KEY }),
mesh: createHunyuanProvider({ apiKey: process.env.HUNYUAN_API_KEY }),
sfx: createSFXProvider({
provider: 'elevenlabs',
apiKey: process.env.ELEVENLABS_API_KEY,
}),
},
objects: [
{
source: 'explicit',
id: 'tree-1',
prompt: 'cypress tree',
anchor: { type: 'centre-ground' },
},
{
source: 'explicit',
id: 'lantern-1',
prompt: 'stone lantern',
anchor: { type: 'on-floor-front-centre' },
},
],
ambient: { prompt: 'gentle wind through cypress, distant chimes' },
});
// result.manifest — OutputManifest with .spz scene + .glb meshes + .mp3 ambient
// result.phaseOutputs — typed per-phase outputs (env, grounding?, objects?, align?, audio?)
// result.checkpointHistory — empty for the convenience path (auto-approve)
// result.metrics — totalDurationMs + perPhaseMs
Advanced: createSceneFromImagePipeline#
Returns a SceneFromImagePipeline whose .run(opts?) accepts caller-supplied
orchestrators. Necessary for block H (persistence), block I (gallery + lineage),
and any consumer that needs the full block-A event stream on their own
orchestrator instance.
import { createSceneFromImagePipeline } from '@isis/scene-from-image-composer';
import {
createCheckpointCoordinator,
createExecutionOrchestrator,
createPipelineOrchestrator,
} from '@isis/operation-orchestrator';
// One shared coordinator across both orchestrators (per A-003 finding).
const coordinator = createCheckpointCoordinator();
const executionOrchestrator = createExecutionOrchestrator(
undefined,
coordinator
);
const pipelineOrchestrator = createPipelineOrchestrator(undefined, coordinator);
// Subscribe to whatever events the consumer cares about.
pipelineOrchestrator.on('pipeline_started', (e) =>
console.log('started', e.pipelineId)
);
pipelineOrchestrator.on('pipeline_completed', (e) =>
console.log('done', e.durationMs, 'ms')
);
executionOrchestrator.on('phase_started', (e) =>
console.log('phase', e.phaseId)
);
executionOrchestrator.on('phase_completed', (e) =>
console.log('phase done', e.phaseId, e.status)
);
const pipeline = createSceneFromImagePipeline(request);
const result = await pipeline.run({
executionOrchestrator,
pipelineOrchestrator,
coordinator,
});
The advanced API also exposes the static definition (pipeline.request,
pipeline.phaseOrder) so callers can persist or inspect the planned run before
executing.
Object source modes (block C)#
ObjectSpec is a discriminated union with three caller-supplied source kinds
plus an auto-derived mixed strategy. The composer auto-derives an
ObjectStrategy from the request shape via
deriveObjectStrategy(request); the detection-phase runs only when the strategy
is 'vlm-detected' or 'mixed'.
explicit — text-prompt object#
The block-B path. Each spec carries a prompt; the mesh provider's
generateFromText produces the mesh.
objects: [
{
source: 'explicit',
id: 'tree-1',
prompt: 'cypress tree',
anchor: { type: 'centre-ground' },
},
];
cropped — caller-supplied bbox + optional prompt#
The composer crops the source image to bbox, then calls the mesh provider's
generateFromImage. Useful when the caller already knows which region holds the
subject (e.g. from a UI selection).
objects: [
{
source: 'cropped',
id: 'subject-1',
bbox: { x: 0.2, y: 0.3, w: 0.4, h: 0.5 }, // normalised image coordinates
prompt: 'antique brass lamp', // optional guidance
removeBackground: true,
anchor: { type: 'bbox-floor', bbox: { x: 0.2, y: 0.3, w: 0.4, h: 0.5 } },
},
];
vlm-detected — auto-detected from the source image#
Empty objects, plus a groundingAdapter, plus an optional
DetectionPolicy. The detection-subphase calls the
VLM, merges overlapping detections by IoU, drops background labels (sky, wall,
floor, ...) via the curated
foreground classifier, and synthesises
a VlmDetectedObjectSpec per surviving detection. The objects-phase then
image→3Ds each crop.
import { ClaudeGroundingAdapter } from '@isis/3d-quality-gates';
await composeSceneFromImage({
...request,
groundingAdapter: new ClaudeGroundingAdapter({
apiKey: process.env.ANTHROPIC_API_KEY,
}),
// No `objects` → strategy='vlm-detected'.
detection: {
topK: 8, // keep at most 8 detections (default)
minConfidence: 0.6, // override the 0.5 default
iouMergeThreshold: 0.5, // pairs over this IoU collapse to one
excludeLabels: ['shadow'], // caller-supplied drops (post-canonicalisation)
requireLabels: ['chair'], // these survive even when classify(label) === 'background'
costCapCents: 200, // greedy clamp under this cap (paired with a CostEstimator)
},
});
mixed — caller-supplied + detected#
When objects is non-empty AND groundingAdapter is set, the strategy is
'mixed': the detection-phase augments the caller's list. Both lists feed the
objects-phase; the constraint-solver places everything in one batch (supporting
forward-of / right-of relative anchors that reference other specs by id).
MeshProviderRouter — per-spec routing#
Block C's router picks per-spec providers from a pool with weighted scoring
(quality + cost + latency), a soft providerHint boost, and built-in rules
(character → Hunyuan, mechanical → Meshy, simple-prop → Tripo). Inject via the
advanced API's runtime.meshProviderRouter:
import {
MeshProviderRouter,
flatRateCostEstimator,
} from '@isis/scene-from-image-composer';
const router = new MeshProviderRouter({
providers: [
{
provider: hunyuan,
capabilities: hunyuan.getCapabilities(),
costPerCallCents: 80,
qualityScore: 0.85,
},
{
provider: tripo,
capabilities: tripo.getCapabilities(),
costPerCallCents: 30,
qualityScore: 0.7,
},
{
provider: meshy,
capabilities: meshy.getCapabilities(),
costPerCallCents: 60,
qualityScore: 0.8,
},
],
});
await createSceneFromImagePipeline(request).run({
runtime: {
meshProviderRouter: router,
detectionCostEstimator: flatRateCostEstimator(60), // 60¢ per detected spec
},
});
Falls back to request.providers.mesh for any spec the router can't satisfy
(e.g. image→3D needed but only text-capable providers in the pool).
Phases#
Seven phases, run sequentially in PHASE_ORDER:
| Phase | Owner module | What it does |
|---|---|---|
environment |
phases/environment-phase.ts |
Calls Marble; emits the env .spz + bounds + preview URLs |
grounding |
phases/grounding-phase.ts |
Optional — input-grounding QA gate (skip if no adapter) |
detection |
phases/detection-phase.ts |
Block C — VLM detect + IoU merge + foreground classify + topK/maxObjects/cost-cap clamp |
objects |
phases/objects-phase.ts |
Iterates caller-supplied + detected specs; routes per spec; explicit→text, cropped/vlm-detected→image |
align |
phases/align-phase.ts |
Block C — constraint solver resolves all anchors, then per-mesh 4×4 column-major transforms |
audio |
phases/audio-phase.ts |
Optional — ambient SFX (skip if no sfx provider) |
manifest |
phases/manifest-phase.ts |
Assembles OutputManifest with files + provenance + metadata.sceneFromImage |
Each phase wraps provider errors as
ComposerError({ code: 'phase-failed', phase: '…', cause }).
Manifest extension#
The composer writes typed metadata under manifest.metadata.sceneFromImage.
Schema declared in src/manifest-extension.ts:
{
schemaVersion: '1.0.0',
assetId: '…',
sceneBoundsMeters: { min: [-5, 0, -5], max: [5, 6, 5] },
placements: [
{ fileId: 'mesh-000-tree.glb', transform: [/* 16 floats */], anchor: { type: 'centre-ground' } },
// …
],
}
Downstream consumers (Three.js viewer, Unreal plugin, block I gallery) read
placements[].transform and apply at load time. Block H's persistence layer
validates round-trips with validateScenecomposerManifestMetadata(unknown).
Test utilities#
Fixture provider factories live under
@isis/scene-from-image-composer/test-utils. Reusable across the composer's own
tests and downstream consumers' tests.
import {
createFixtureMarbleProvider,
createFixtureMeshProvider,
createFixtureSfxProvider,
createFixtureGroundingAdapter,
} from '@isis/scene-from-image-composer/test-utils';
const marble = createFixtureMarbleProvider({
bounds: { min: [-5, 0, -5], max: [5, 6, 5] },
ambientPrompt: 'forest at dawn',
});
const mesh = createFixtureMeshProvider({ throwOnCallNumber: 2 });
Each fixture exposes its vi.fn() mock for caller-side
expect(...).toHaveBeenCalled() assertions and supports the failure modes the
composer's adversarial tests rely on.
Test conventions#
- Per CLAUDE.md /
oshun_nx_brokenmemory: run tests withnpx vitest run --no-isolatefrom this directory (bypassing Nx). - Stub-indicator scan is part of pre-commit; phrases like
would,placeholder,todooutside test files are blocked.
Capabilities shipped across blocks D-I#
- Bake meshes into the splat — block E ships the mesh-to-Gaussians
rasteriser. Set
outputMode: 'baked'withbakeOptions.method: 'surface-sampling'(Poisson area-weighted oriented disks) or'hybrid'(surface-sampling warm start + CPU-rendered multi-view supervision- gradient-descent refinement of positions/colours/opacities).
- Lighting consistency — block F estimates the env's SH light field from the Marble splat and applies PBR shading (sh9 + env-map encodings) to mesh Gaussians at bake time, plus a composition coherence gate with pluggable feature adapter.
- GPU dispatch — block G's pluggable
JobChanneltransport (RedisJobChannelfor production,InMemoryJobChannelfor tests) routes heavy phases (env / objects / bake) through@isis/job-envelopeto a worker pool. SetdispatchMode: 'job-envelope'+dispatchConfig: { redisUrl }. - Persistence — block H ships 6 Prisma models + a
PipelineRepositoryinterface (in-memory + Postgres impls). SetrepositoryConfig: { mode: 'postgres' }for durable state; SIGKILL-resume golden incomposer.persistence.test.ts. - Operator policies + agent consensus — block I threads six external isis
libraries (output-gallery, lineage, token-budget, model-governance-3d,
anomaly-detection, agent-consensus) into the composer's runner via the
policyContextfield. Consensus checkpoint handler decides 3-agent fixture cases across all-SUPPORT / all-OPPOSE / CONDITIONAL / 2-vs-2 / throw / timeout paths.
The advanced API surface is the seam each of those blocks plugs into; the public
types stayed stable across blocks D-I. See
ISIS_GAPS/runbooks/production-runbook.md
for the deployment recipe and
ISIS_GAPS/runbooks/manual-qa.md
for the external-tool acceptance recipes.