Eve library

Block C · Foreground Detection and Provider Routing

Other

Revision 1

On this page

Owner library: libs/isis/scene-from-image-composer (extends block B) Estimate: ~5 weeks Blocks: none Dependencies: block B Status: not started

Overview#

Extends the composer's object-sourcing surface from "explicit prompts only" to all four modes: explicit, cropped regions, VLM-detected, and mixed. Adds a foreground-detection subphase, an IoU-based overlap merge, a background-vs-foreground classifier, an anchor DSL with a small constraint solver, and a MeshProviderRouter that picks per-object providers based on characteristics.

This is the largest block by LoC and the one most likely to surface UX questions. Blocks E and F do not depend on it; this can land in parallel.

Files to create#

Foreground detection subphase#

  • libs/isis/scene-from-image-composer/src/phases/detection-phase.ts — the subphase between environment and objects (only inserted when an objectStrategy requiring detection is configured).
  • libs/isis/scene-from-image-composer/src/phases/detection-phase.test.ts
  • libs/isis/scene-from-image-composer/src/detection/overlap-merge.ts — IoU-based overlap merge; given a list of detections, merge any pair with IoU > threshold.
  • libs/isis/scene-from-image-composer/src/detection/overlap-merge.test.ts
  • libs/isis/scene-from-image-composer/src/detection/foreground-classifier.ts — given a label, return 'foreground' | 'background' | 'unknown'. Uses a configurable label-set with sensible defaults (sky, wall, floor, ceiling, road, water → background).
  • libs/isis/scene-from-image-composer/src/detection/foreground-classifier.test.ts
  • libs/isis/scene-from-image-composer/src/detection/cost-estimator.ts — given detected objects + provider config, estimate generation cost.

ObjectSpec union and image-cropping#

  • libs/isis/scene-from-image-composer/src/objects/object-spec.ts — the discriminated union (replaces the placeholder from block B).
  • libs/isis/scene-from-image-composer/src/objects/crop-image.ts — uses sharp (already a dep of 3d-generation) to crop bbox regions; honours removeBackground flag.
  • libs/isis/scene-from-image-composer/src/objects/crop-image.test.ts

Anchor DSL + constraint solver#

  • libs/isis/scene-from-image-composer/src/align/anchor-spec.ts (extends block B's stub) — full discriminated union covering absolute, bbox-relative, anchor-relative, and surface-snapped.
  • libs/isis/scene-from-image-composer/src/align/constraint-solver.ts — small constraint propagator. Inputs: an array of AnchorSpec (some relative to others, some absolute); output: resolved world positions.
  • libs/isis/scene-from-image-composer/src/align/constraint-solver.test.ts

Provider routing#

  • libs/isis/scene-from-image-composer/src/routing/mesh-provider-router.ts — selects a provider per object spec via weighted scoring.
  • libs/isis/scene-from-image-composer/src/routing/mesh-provider-router.test.ts
  • libs/isis/scene-from-image-composer/src/routing/scoring-rules.ts — built-in rules (high-detail-character → Hunyuan, hard-surface-mechanical → Meshy, etc.).

Adapter contract reuse#

No new adapter contract — reuses ThreeDQualityGroundingVlmAdapter from libs/isis/3d-quality-gates/src/delivery/input-grounding-gate.ts. This is ADR-009.

Files to modify#

  • libs/isis/scene-from-image-composer/src/types.ts:
    • Replace placeholder ObjectSpec with the real discriminated union.
    • Add objectStrategy: 'explicit' | 'cropped' | 'vlm-detected' | 'mixed' to SceneFromImageRequest (defaults to 'explicit' if every spec is explicit, else 'mixed').
    • Add detectionPolicy?: DetectionPolicy field for VLM-detected mode.
  • libs/isis/scene-from-image-composer/src/composer.ts:
    • Insert detection-phase between environment and objects when the strategy requires it.
  • libs/isis/scene-from-image-composer/src/phases/objects-phase.ts:
    • Use MeshProviderRouter to pick a provider per object.
    • Handle cropped and vlm-detected specs by calling crop-image first, then provider.generateFromImage(crop).
  • libs/isis/scene-from-image-composer/src/phases/align-phase.ts:
    • Use constraint-solver for multi-object placement instead of independent resolveAnchor calls.

Public API surface#

ts
// objects/object-spec.ts

export type BBox = {
  readonly x: number; // 0..1 normalised
  readonly y: number;
  readonly w: number;
  readonly h: number;
};

export type ObjectSpec =
  | {
      readonly source: 'explicit';
      readonly id: string;
      readonly prompt: string;
      readonly anchor: AnchorSpec;
      readonly providerHint?: string;
    }
  | {
      readonly source: 'cropped';
      readonly id: string;
      readonly bbox: BBox;
      readonly prompt?: string;
      readonly anchor: AnchorSpec;
      readonly removeBackground?: boolean;
      readonly providerHint?: string;
    }
  | {
      readonly source: 'vlm-detected';
      readonly id: string;
      readonly bbox: BBox;
      readonly label: string;
      readonly confidence: number;
      readonly anchor: AnchorSpec;
      readonly providerHint?: string;
    };

// detection/types.ts

export interface DetectionPolicy {
  readonly topK?: number; // default 8
  readonly minConfidence?: number; // default 0.5
  readonly maxObjects?: number; // default 12 — hard upper bound
  readonly minBboxArea?: number; // default 0.005 (0.5% of image)
  readonly excludeLabels?: readonly string[];
  readonly requireLabels?: readonly string[];
  readonly iouMergeThreshold?: number; // default 0.5
  readonly costCapCents?: number; // halt detection if estimated cost exceeds
}

// align/anchor-spec.ts

export type AnchorSpec =
  | {
      readonly type: 'absolute';
      readonly position: readonly [number, number, number];
    }
  | { readonly type: 'bbox-floor'; readonly bbox: BBox }
  | { readonly type: 'centre-ground' }
  | { readonly type: 'on-floor-front-centre' }
  | {
      readonly type: 'forward-of';
      readonly otherId: string;
      readonly distanceMeters: number;
    }
  | {
      readonly type: 'right-of';
      readonly otherId: string;
      readonly distanceMeters: number;
    }
  | {
      readonly type: 'surface-snapped';
      readonly hint: readonly [number, number, number];
    };

// routing/mesh-provider-router.ts

export interface MeshProviderRouterConfig {
  readonly providers: ReadonlyArray<{
    readonly provider: I3DProvider;
    readonly capabilities: Provider3DCapabilities;
    readonly costPerCallCents: number;
  }>;
  readonly weights?: {
    readonly cost: number;
    readonly quality: number;
    readonly latency: number;
  };
}

export class MeshProviderRouter {
  constructor(config: MeshProviderRouterConfig);
  route(spec: ObjectSpec): I3DProvider;
}

Algorithms#

IoU-based overlap merge#

text
for each unordered pair (a, b) in detections:
  iou = area(intersect(a.bbox, b.bbox)) / area(union(a.bbox, b.bbox))
  if iou > threshold:
    merged.label   = (a.confidence > b.confidence ? a : b).label
    merged.bbox    = union(a.bbox, b.bbox)
    merged.confidence = max(a.confidence, b.confidence)
    replace a, b with merged
repeat until no merges occur

Foreground classifier#

text
classify(label):
  normalised = canonicaliseGroundingLabel(label)  // reuse from 3d-quality-gates
  if BACKGROUND_LABELS.has(normalised): return 'background'
  if FOREGROUND_LABELS.has(normalised): return 'foreground'
  return 'unknown'

BACKGROUND_LABELS: sky, wall, floor, ceiling, ground, grass, road, pavement,
                   water, sea, river, lake, sand, snow, mountain (distant),
                   horizon, cloud, fog, mist
FOREGROUND_LABELS: tree, chair, table, lamp, person, animal, vase, book,
                   bottle, plant, statue, etc. (curated; configurable)

unknown defaults to foreground to avoid silently dropping objects; caller can override via DetectionPolicy.excludeLabels.

Constraint solver#

Two-pass algorithm:

  1. Pass 1 — absolute resolution. Resolve every anchor whose type doesn't depend on another object: absolute, bbox-floor, centre-ground, on-floor-front-centre, surface-snapped.
  2. Pass 2 — relative resolution. Iterate. For each unresolved anchor that depends on a now-resolved anchor (forward-of, right-of), resolve it. Continue until either all are resolved or no progress is made in a pass.
  3. If unresolved anchors remain after pass 2 stops making progress, throw UnresolvableAnchorsError listing the cycle/missing references.

forward-of direction: by convention "forward" is +Z in the splat's world frame (Marble uses Y-up Z-forward by default per World Labs docs).

Cost estimator#

text
estimateCost(detections, providerRouter):
  for each detection:
    provider = providerRouter.route(detection)
    totalCents += provider.costPerCallCents
  return totalCents

MeshProviderRouter scoring#

text
score(spec, candidate):
  base = weights.quality * candidate.capabilities.qualityScore +
         weights.cost * (1 / candidate.costPerCallCents) +
         weights.latency * (1 / candidate.capabilities.averageGenerationTime)
  if spec.providerHint && candidate.provider.type === spec.providerHint:
    return base * 1.5  // strong hint preference
  if scoringRules.matches(spec, candidate):
    return base * 1.2  // soft rule match
  return base

route(spec):
  return argmax(providers, candidate => score(spec, candidate)).provider

Test plan#

Unit (per file):

  • overlap-merge.test.ts: synthetic detections — non-overlapping pass through; perfectly overlapping merge into one; partial overlap above/below threshold; chain of three (A∩B and B∩C, A∩C below) merge correctly.
  • foreground-classifier.test.ts: every default background label → background; every default foreground label → foreground; unknown → foreground; canonicalisation applied (plurals).
  • crop-image.test.ts: 4×4 white image cropped to bbox (0.25, 0.25, 0.5, 0.5) produces 2×2 image; with removeBackground: true plus stub bg-remove, alpha channel set; coordinate clamping for out-of-range bboxes.
  • constraint-solver.test.ts: 3 absolutes → resolves immediately; chain of 4 with each forward-of the previous → resolves in 4 passes; cycle of 2 (A forward-of B, B forward-of A) → throws; missing reference (forward-of: 'nonexistent') → throws.
  • mesh-provider-router.test.ts: weighted scoring picks expected provider; providerHint overrides scoring; ties resolved by provider order.

Integration:

  • composer.test.ts (extend from block B): 3-object scene with mixed modes (1 explicit, 1 cropped, 1 vlm-detected) produces 3 placed meshes; routing picks expected providers per mode.
  • VLM-detected with costCapCents: 100: detection halts at the cap; emits detection-cost-cap-hit event; composer treats as soft-fail and proceeds with detected-so-far.
  • VLM-detected end-to-end with fixture adapter: detected 5 objects → 4 pass detection policy → 4 placed meshes.
  • Cropped mode: input image 100×100 PNG, bbox (0.1, 0.1, 0.3, 0.3) → cropped buffer dimensions 30×30 (asserted via sharp.metadata()).

Adversarial:

  • Detection returns 0 objects → ObjectsPhase skipped silently, manifest has placements: [].
  • Detection returns 100 objects → maxObjects clamps to 12.
  • Two vlm-detected objects with identical bbox → IoU merge collapses to one.
  • MeshProviderRouter with 0 providers → throws NoProvidersConfiguredError on first route() call.

Acceptance criteria#

  • All new tests pass: npx vitest run in libs/isis/scene-from-image-composer (block B + block C tests together).
  • npx tsc --noEmit clean.
  • Stub-indicator scan clean.
  • The composer can run in any of the four modes via a single config flag.
  • Detection-phase emits a detection-preview event before object generation so a checkpoint can review detected boxes (block A integration).
  • README updated with one snippet per mode.

Open questions#

  • Background labels. The default list is curated from common detector vocabularies (COCO + ADE20K). Should it be configurable per-tenant or global? Recommended: global default, per-request override.
  • surface-snapped anchor. Requires depth queries against the splat, which we don't have until block E. Until then, surface-snapped falls back to centre-ground with a warning. v1.5 wires the real depth query.
  • Cost cap behaviour. Halt or proceed-with-subset? Recommended: proceed-with-subset (the user's already paid for detection); emit a soft warning event.
  • providerHint enforcement. Strict ("must use this provider") or soft ("prefer if capable")? Recommended: soft. If hinted provider lacks required capability, fall back with warning.