Owner library: libs/isis/scene-from-image-composer (extends block B)
Estimate: ~5 weeks Blocks: none Dependencies: block B Status:
not started
Overview#
Extends the composer's object-sourcing surface from "explicit prompts only" to
all four modes: explicit, cropped regions, VLM-detected, and mixed. Adds a
foreground-detection subphase, an IoU-based overlap merge, a
background-vs-foreground classifier, an anchor DSL with a small constraint
solver, and a MeshProviderRouter that picks per-object providers based on
characteristics.
This is the largest block by LoC and the one most likely to surface UX questions. Blocks E and F do not depend on it; this can land in parallel.
Files to create#
Foreground detection subphase#
libs/isis/scene-from-image-composer/src/phases/detection-phase.ts— the subphase betweenenvironmentandobjects(only inserted when anobjectStrategyrequiring detection is configured).libs/isis/scene-from-image-composer/src/phases/detection-phase.test.tslibs/isis/scene-from-image-composer/src/detection/overlap-merge.ts— IoU-based overlap merge; given a list of detections, merge any pair with IoU > threshold.libs/isis/scene-from-image-composer/src/detection/overlap-merge.test.tslibs/isis/scene-from-image-composer/src/detection/foreground-classifier.ts— given a label, return'foreground' | 'background' | 'unknown'. Uses a configurable label-set with sensible defaults (sky, wall, floor, ceiling, road, water → background).libs/isis/scene-from-image-composer/src/detection/foreground-classifier.test.tslibs/isis/scene-from-image-composer/src/detection/cost-estimator.ts— given detected objects + provider config, estimate generation cost.
ObjectSpec union and image-cropping#
libs/isis/scene-from-image-composer/src/objects/object-spec.ts— the discriminated union (replaces the placeholder from block B).libs/isis/scene-from-image-composer/src/objects/crop-image.ts— usessharp(already a dep of3d-generation) to crop bbox regions; honoursremoveBackgroundflag.libs/isis/scene-from-image-composer/src/objects/crop-image.test.ts
Anchor DSL + constraint solver#
libs/isis/scene-from-image-composer/src/align/anchor-spec.ts(extends block B's stub) — full discriminated union covering absolute, bbox-relative, anchor-relative, and surface-snapped.libs/isis/scene-from-image-composer/src/align/constraint-solver.ts— small constraint propagator. Inputs: an array ofAnchorSpec(some relative to others, some absolute); output: resolved world positions.libs/isis/scene-from-image-composer/src/align/constraint-solver.test.ts
Provider routing#
libs/isis/scene-from-image-composer/src/routing/mesh-provider-router.ts— selects a provider per object spec via weighted scoring.libs/isis/scene-from-image-composer/src/routing/mesh-provider-router.test.tslibs/isis/scene-from-image-composer/src/routing/scoring-rules.ts— built-in rules (high-detail-character → Hunyuan, hard-surface-mechanical → Meshy, etc.).
Adapter contract reuse#
No new adapter contract — reuses ThreeDQualityGroundingVlmAdapter from
libs/isis/3d-quality-gates/src/delivery/input-grounding-gate.ts. This is
ADR-009.
Files to modify#
libs/isis/scene-from-image-composer/src/types.ts:- Replace placeholder
ObjectSpecwith the real discriminated union. - Add
objectStrategy: 'explicit' | 'cropped' | 'vlm-detected' | 'mixed'toSceneFromImageRequest(defaults to'explicit'if every spec is explicit, else'mixed'). - Add
detectionPolicy?: DetectionPolicyfield for VLM-detected mode.
- Replace placeholder
libs/isis/scene-from-image-composer/src/composer.ts:- Insert
detection-phasebetweenenvironmentandobjectswhen the strategy requires it.
- Insert
libs/isis/scene-from-image-composer/src/phases/objects-phase.ts:- Use
MeshProviderRouterto pick a provider per object. - Handle
croppedandvlm-detectedspecs by callingcrop-imagefirst, thenprovider.generateFromImage(crop).
- Use
libs/isis/scene-from-image-composer/src/phases/align-phase.ts:- Use
constraint-solverfor multi-object placement instead of independentresolveAnchorcalls.
- Use
Public API surface#
// objects/object-spec.ts
export type BBox = {
readonly x: number; // 0..1 normalised
readonly y: number;
readonly w: number;
readonly h: number;
};
export type ObjectSpec =
| {
readonly source: 'explicit';
readonly id: string;
readonly prompt: string;
readonly anchor: AnchorSpec;
readonly providerHint?: string;
}
| {
readonly source: 'cropped';
readonly id: string;
readonly bbox: BBox;
readonly prompt?: string;
readonly anchor: AnchorSpec;
readonly removeBackground?: boolean;
readonly providerHint?: string;
}
| {
readonly source: 'vlm-detected';
readonly id: string;
readonly bbox: BBox;
readonly label: string;
readonly confidence: number;
readonly anchor: AnchorSpec;
readonly providerHint?: string;
};
// detection/types.ts
export interface DetectionPolicy {
readonly topK?: number; // default 8
readonly minConfidence?: number; // default 0.5
readonly maxObjects?: number; // default 12 — hard upper bound
readonly minBboxArea?: number; // default 0.005 (0.5% of image)
readonly excludeLabels?: readonly string[];
readonly requireLabels?: readonly string[];
readonly iouMergeThreshold?: number; // default 0.5
readonly costCapCents?: number; // halt detection if estimated cost exceeds
}
// align/anchor-spec.ts
export type AnchorSpec =
| {
readonly type: 'absolute';
readonly position: readonly [number, number, number];
}
| { readonly type: 'bbox-floor'; readonly bbox: BBox }
| { readonly type: 'centre-ground' }
| { readonly type: 'on-floor-front-centre' }
| {
readonly type: 'forward-of';
readonly otherId: string;
readonly distanceMeters: number;
}
| {
readonly type: 'right-of';
readonly otherId: string;
readonly distanceMeters: number;
}
| {
readonly type: 'surface-snapped';
readonly hint: readonly [number, number, number];
};
// routing/mesh-provider-router.ts
export interface MeshProviderRouterConfig {
readonly providers: ReadonlyArray<{
readonly provider: I3DProvider;
readonly capabilities: Provider3DCapabilities;
readonly costPerCallCents: number;
}>;
readonly weights?: {
readonly cost: number;
readonly quality: number;
readonly latency: number;
};
}
export class MeshProviderRouter {
constructor(config: MeshProviderRouterConfig);
route(spec: ObjectSpec): I3DProvider;
}
Algorithms#
IoU-based overlap merge#
for each unordered pair (a, b) in detections:
iou = area(intersect(a.bbox, b.bbox)) / area(union(a.bbox, b.bbox))
if iou > threshold:
merged.label = (a.confidence > b.confidence ? a : b).label
merged.bbox = union(a.bbox, b.bbox)
merged.confidence = max(a.confidence, b.confidence)
replace a, b with merged
repeat until no merges occur
Foreground classifier#
classify(label):
normalised = canonicaliseGroundingLabel(label) // reuse from 3d-quality-gates
if BACKGROUND_LABELS.has(normalised): return 'background'
if FOREGROUND_LABELS.has(normalised): return 'foreground'
return 'unknown'
BACKGROUND_LABELS: sky, wall, floor, ceiling, ground, grass, road, pavement,
water, sea, river, lake, sand, snow, mountain (distant),
horizon, cloud, fog, mist
FOREGROUND_LABELS: tree, chair, table, lamp, person, animal, vase, book,
bottle, plant, statue, etc. (curated; configurable)
unknown defaults to foreground to avoid silently dropping objects; caller
can override via DetectionPolicy.excludeLabels.
Constraint solver#
Two-pass algorithm:
- Pass 1 — absolute resolution. Resolve every anchor whose type doesn't
depend on another object:
absolute,bbox-floor,centre-ground,on-floor-front-centre,surface-snapped. - Pass 2 — relative resolution. Iterate. For each unresolved anchor that
depends on a now-resolved anchor (
forward-of,right-of), resolve it. Continue until either all are resolved or no progress is made in a pass. - If unresolved anchors remain after pass 2 stops making progress, throw
UnresolvableAnchorsErrorlisting the cycle/missing references.
forward-of direction: by convention "forward" is +Z in the splat's world frame
(Marble uses Y-up Z-forward by default per World Labs docs).
Cost estimator#
estimateCost(detections, providerRouter):
for each detection:
provider = providerRouter.route(detection)
totalCents += provider.costPerCallCents
return totalCents
MeshProviderRouter scoring#
score(spec, candidate):
base = weights.quality * candidate.capabilities.qualityScore +
weights.cost * (1 / candidate.costPerCallCents) +
weights.latency * (1 / candidate.capabilities.averageGenerationTime)
if spec.providerHint && candidate.provider.type === spec.providerHint:
return base * 1.5 // strong hint preference
if scoringRules.matches(spec, candidate):
return base * 1.2 // soft rule match
return base
route(spec):
return argmax(providers, candidate => score(spec, candidate)).provider
Test plan#
Unit (per file):
overlap-merge.test.ts: synthetic detections — non-overlapping pass through; perfectly overlapping merge into one; partial overlap above/below threshold; chain of three (A∩B and B∩C, A∩C below) merge correctly.foreground-classifier.test.ts: every default background label → background; every default foreground label → foreground; unknown → foreground; canonicalisation applied (plurals).crop-image.test.ts: 4×4 white image cropped to bbox(0.25, 0.25, 0.5, 0.5)produces 2×2 image; withremoveBackground: trueplus stub bg-remove, alpha channel set; coordinate clamping for out-of-range bboxes.constraint-solver.test.ts: 3 absolutes → resolves immediately; chain of 4 with eachforward-ofthe previous → resolves in 4 passes; cycle of 2 (Aforward-ofB, Bforward-ofA) → throws; missing reference (forward-of: 'nonexistent') → throws.mesh-provider-router.test.ts: weighted scoring picks expected provider;providerHintoverrides scoring; ties resolved by provider order.
Integration:
composer.test.ts(extend from block B): 3-object scene with mixed modes (1 explicit, 1 cropped, 1 vlm-detected) produces 3 placed meshes; routing picks expected providers per mode.- VLM-detected with
costCapCents: 100: detection halts at the cap; emitsdetection-cost-cap-hitevent; composer treats as soft-fail and proceeds with detected-so-far. - VLM-detected end-to-end with fixture adapter: detected 5 objects → 4 pass detection policy → 4 placed meshes.
- Cropped mode: input image
100×100PNG, bbox(0.1, 0.1, 0.3, 0.3)→ cropped buffer dimensions30×30(asserted viasharp.metadata()).
Adversarial:
- Detection returns 0 objects →
ObjectsPhaseskipped silently, manifest hasplacements: []. - Detection returns 100 objects →
maxObjectsclamps to 12. - Two
vlm-detectedobjects with identical bbox → IoU merge collapses to one. MeshProviderRouterwith 0 providers → throwsNoProvidersConfiguredErroron firstroute()call.
Acceptance criteria#
- All new tests pass:
npx vitest runinlibs/isis/scene-from-image-composer(block B + block C tests together). npx tsc --noEmitclean.- Stub-indicator scan clean.
- The composer can run in any of the four modes via a single config flag.
- Detection-phase emits a
detection-previewevent before object generation so a checkpoint can review detected boxes (block A integration). - README updated with one snippet per mode.
Open questions#
- Background labels. The default list is curated from common detector vocabularies (COCO + ADE20K). Should it be configurable per-tenant or global? Recommended: global default, per-request override.
surface-snappedanchor. Requires depth queries against the splat, which we don't have until block E. Until then,surface-snappedfalls back tocentre-groundwith a warning. v1.5 wires the real depth query.- Cost cap behaviour. Halt or proceed-with-subset? Recommended: proceed-with-subset (the user's already paid for detection); emit a soft warning event.
providerHintenforcement. Strict ("must use this provider") or soft ("prefer if capable")? Recommended: soft. If hinted provider lacks required capability, fall back with warning.