# Block C · Foreground Detection and Provider Routing

**Owner library:** `libs/isis/scene-from-image-composer` (extends block B)
**Estimate:** ~5 weeks **Blocks:** none **Dependencies:** block B **Status:**
not started

## Overview

Extends the composer's object-sourcing surface from "explicit prompts only" to
all four modes: explicit, cropped regions, VLM-detected, and mixed. Adds a
foreground-detection subphase, an IoU-based overlap merge, a
background-vs-foreground classifier, an anchor DSL with a small constraint
solver, and a `MeshProviderRouter` that picks per-object providers based on
characteristics.

This is the largest block by LoC and the one most likely to surface UX
questions. Blocks E and F do not depend on it; this can land in parallel.

## Files to create

### Foreground detection subphase

- `libs/isis/scene-from-image-composer/src/phases/detection-phase.ts` — the
  subphase between `environment` and `objects` (only inserted when an
  `objectStrategy` requiring detection is configured).
- `libs/isis/scene-from-image-composer/src/phases/detection-phase.test.ts`
- `libs/isis/scene-from-image-composer/src/detection/overlap-merge.ts` —
  IoU-based overlap merge; given a list of detections, merge any pair with IoU >
  threshold.
- `libs/isis/scene-from-image-composer/src/detection/overlap-merge.test.ts`
- `libs/isis/scene-from-image-composer/src/detection/foreground-classifier.ts` —
  given a label, return `'foreground' | 'background' | 'unknown'`. Uses a
  configurable label-set with sensible defaults (sky, wall, floor, ceiling,
  road, water → background).
- `libs/isis/scene-from-image-composer/src/detection/foreground-classifier.test.ts`
- `libs/isis/scene-from-image-composer/src/detection/cost-estimator.ts` — given
  detected objects + provider config, estimate generation cost.

### ObjectSpec union and image-cropping

- `libs/isis/scene-from-image-composer/src/objects/object-spec.ts` — the
  discriminated union (replaces the placeholder from block B).
- `libs/isis/scene-from-image-composer/src/objects/crop-image.ts` — uses `sharp`
  (already a dep of `3d-generation`) to crop bbox regions; honours
  `removeBackground` flag.
- `libs/isis/scene-from-image-composer/src/objects/crop-image.test.ts`

### Anchor DSL + constraint solver

- `libs/isis/scene-from-image-composer/src/align/anchor-spec.ts` (extends block
  B's stub) — full discriminated union covering absolute, bbox-relative,
  anchor-relative, and surface-snapped.
- `libs/isis/scene-from-image-composer/src/align/constraint-solver.ts` — small
  constraint propagator. Inputs: an array of `AnchorSpec` (some relative to
  others, some absolute); output: resolved world positions.
- `libs/isis/scene-from-image-composer/src/align/constraint-solver.test.ts`

### Provider routing

- `libs/isis/scene-from-image-composer/src/routing/mesh-provider-router.ts` —
  selects a provider per object spec via weighted scoring.
- `libs/isis/scene-from-image-composer/src/routing/mesh-provider-router.test.ts`
- `libs/isis/scene-from-image-composer/src/routing/scoring-rules.ts` — built-in
  rules (high-detail-character → Hunyuan, hard-surface-mechanical → Meshy,
  etc.).

### Adapter contract reuse

No new adapter contract — reuses `ThreeDQualityGroundingVlmAdapter` from
`libs/isis/3d-quality-gates/src/delivery/input-grounding-gate.ts`. This is
ADR-009.

## Files to modify

- `libs/isis/scene-from-image-composer/src/types.ts`:
  - Replace placeholder `ObjectSpec` with the real discriminated union.
  - Add `objectStrategy: 'explicit' | 'cropped' | 'vlm-detected' | 'mixed'` to
    `SceneFromImageRequest` (defaults to `'explicit'` if every spec is explicit,
    else `'mixed'`).
  - Add `detectionPolicy?: DetectionPolicy` field for VLM-detected mode.
- `libs/isis/scene-from-image-composer/src/composer.ts`:
  - Insert `detection-phase` between `environment` and `objects` when the
    strategy requires it.
- `libs/isis/scene-from-image-composer/src/phases/objects-phase.ts`:
  - Use `MeshProviderRouter` to pick a provider per object.
  - Handle `cropped` and `vlm-detected` specs by calling `crop-image` first,
    then `provider.generateFromImage(crop)`.
- `libs/isis/scene-from-image-composer/src/phases/align-phase.ts`:
  - Use `constraint-solver` for multi-object placement instead of independent
    `resolveAnchor` calls.

## Public API surface

```ts
// objects/object-spec.ts

export type BBox = {
  readonly x: number; // 0..1 normalised
  readonly y: number;
  readonly w: number;
  readonly h: number;
};

export type ObjectSpec =
  | {
      readonly source: 'explicit';
      readonly id: string;
      readonly prompt: string;
      readonly anchor: AnchorSpec;
      readonly providerHint?: string;
    }
  | {
      readonly source: 'cropped';
      readonly id: string;
      readonly bbox: BBox;
      readonly prompt?: string;
      readonly anchor: AnchorSpec;
      readonly removeBackground?: boolean;
      readonly providerHint?: string;
    }
  | {
      readonly source: 'vlm-detected';
      readonly id: string;
      readonly bbox: BBox;
      readonly label: string;
      readonly confidence: number;
      readonly anchor: AnchorSpec;
      readonly providerHint?: string;
    };

// detection/types.ts

export interface DetectionPolicy {
  readonly topK?: number; // default 8
  readonly minConfidence?: number; // default 0.5
  readonly maxObjects?: number; // default 12 — hard upper bound
  readonly minBboxArea?: number; // default 0.005 (0.5% of image)
  readonly excludeLabels?: readonly string[];
  readonly requireLabels?: readonly string[];
  readonly iouMergeThreshold?: number; // default 0.5
  readonly costCapCents?: number; // halt detection if estimated cost exceeds
}

// align/anchor-spec.ts

export type AnchorSpec =
  | {
      readonly type: 'absolute';
      readonly position: readonly [number, number, number];
    }
  | { readonly type: 'bbox-floor'; readonly bbox: BBox }
  | { readonly type: 'centre-ground' }
  | { readonly type: 'on-floor-front-centre' }
  | {
      readonly type: 'forward-of';
      readonly otherId: string;
      readonly distanceMeters: number;
    }
  | {
      readonly type: 'right-of';
      readonly otherId: string;
      readonly distanceMeters: number;
    }
  | {
      readonly type: 'surface-snapped';
      readonly hint: readonly [number, number, number];
    };

// routing/mesh-provider-router.ts

export interface MeshProviderRouterConfig {
  readonly providers: ReadonlyArray<{
    readonly provider: I3DProvider;
    readonly capabilities: Provider3DCapabilities;
    readonly costPerCallCents: number;
  }>;
  readonly weights?: {
    readonly cost: number;
    readonly quality: number;
    readonly latency: number;
  };
}

export class MeshProviderRouter {
  constructor(config: MeshProviderRouterConfig);
  route(spec: ObjectSpec): I3DProvider;
}
```

## Algorithms

### IoU-based overlap merge

```
for each unordered pair (a, b) in detections:
  iou = area(intersect(a.bbox, b.bbox)) / area(union(a.bbox, b.bbox))
  if iou > threshold:
    merged.label   = (a.confidence > b.confidence ? a : b).label
    merged.bbox    = union(a.bbox, b.bbox)
    merged.confidence = max(a.confidence, b.confidence)
    replace a, b with merged
repeat until no merges occur
```

### Foreground classifier

```
classify(label):
  normalised = canonicaliseGroundingLabel(label)  // reuse from 3d-quality-gates
  if BACKGROUND_LABELS.has(normalised): return 'background'
  if FOREGROUND_LABELS.has(normalised): return 'foreground'
  return 'unknown'

BACKGROUND_LABELS: sky, wall, floor, ceiling, ground, grass, road, pavement,
                   water, sea, river, lake, sand, snow, mountain (distant),
                   horizon, cloud, fog, mist
FOREGROUND_LABELS: tree, chair, table, lamp, person, animal, vase, book,
                   bottle, plant, statue, etc. (curated; configurable)
```

`unknown` defaults to `foreground` to avoid silently dropping objects; caller
can override via `DetectionPolicy.excludeLabels`.

### Constraint solver

Two-pass algorithm:

1. **Pass 1 — absolute resolution.** Resolve every anchor whose type doesn't
   depend on another object: `absolute`, `bbox-floor`, `centre-ground`,
   `on-floor-front-centre`, `surface-snapped`.
2. **Pass 2 — relative resolution.** Iterate. For each unresolved anchor that
   depends on a now-resolved anchor (`forward-of`, `right-of`), resolve it.
   Continue until either all are resolved or no progress is made in a pass.
3. If unresolved anchors remain after pass 2 stops making progress, throw
   `UnresolvableAnchorsError` listing the cycle/missing references.

`forward-of` direction: by convention "forward" is +Z in the splat's world frame
(Marble uses Y-up Z-forward by default per World Labs docs).

### Cost estimator

```
estimateCost(detections, providerRouter):
  for each detection:
    provider = providerRouter.route(detection)
    totalCents += provider.costPerCallCents
  return totalCents
```

### MeshProviderRouter scoring

```
score(spec, candidate):
  base = weights.quality * candidate.capabilities.qualityScore +
         weights.cost * (1 / candidate.costPerCallCents) +
         weights.latency * (1 / candidate.capabilities.averageGenerationTime)
  if spec.providerHint && candidate.provider.type === spec.providerHint:
    return base * 1.5  // strong hint preference
  if scoringRules.matches(spec, candidate):
    return base * 1.2  // soft rule match
  return base

route(spec):
  return argmax(providers, candidate => score(spec, candidate)).provider
```

## Test plan

**Unit (per file):**

- `overlap-merge.test.ts`: synthetic detections — non-overlapping pass through;
  perfectly overlapping merge into one; partial overlap above/below threshold;
  chain of three (A∩B and B∩C, A∩C below) merge correctly.
- `foreground-classifier.test.ts`: every default background label → background;
  every default foreground label → foreground; unknown → foreground;
  canonicalisation applied (plurals).
- `crop-image.test.ts`: 4×4 white image cropped to bbox `(0.25, 0.25, 0.5, 0.5)`
  produces 2×2 image; with `removeBackground: true` plus stub bg-remove, alpha
  channel set; coordinate clamping for out-of-range bboxes.
- `constraint-solver.test.ts`: 3 absolutes → resolves immediately; chain of 4
  with each `forward-of` the previous → resolves in 4 passes; cycle of 2 (A
  `forward-of` B, B `forward-of` A) → throws; missing reference
  (`forward-of: 'nonexistent'`) → throws.
- `mesh-provider-router.test.ts`: weighted scoring picks expected provider;
  `providerHint` overrides scoring; ties resolved by provider order.

**Integration:**

- `composer.test.ts` (extend from block B): 3-object scene with mixed modes (1
  explicit, 1 cropped, 1 vlm-detected) produces 3 placed meshes; routing picks
  expected providers per mode.
- VLM-detected with `costCapCents: 100`: detection halts at the cap; emits
  `detection-cost-cap-hit` event; composer treats as soft-fail and proceeds with
  detected-so-far.
- VLM-detected end-to-end with fixture adapter: detected 5 objects → 4 pass
  detection policy → 4 placed meshes.
- Cropped mode: input image `100×100` PNG, bbox `(0.1, 0.1, 0.3, 0.3)` → cropped
  buffer dimensions `30×30` (asserted via `sharp.metadata()`).

**Adversarial:**

- Detection returns 0 objects → `ObjectsPhase` skipped silently, manifest has
  `placements: []`.
- Detection returns 100 objects → `maxObjects` clamps to 12.
- Two `vlm-detected` objects with identical bbox → IoU merge collapses to one.
- `MeshProviderRouter` with 0 providers → throws `NoProvidersConfiguredError` on
  first `route()` call.

## Acceptance criteria

- All new tests pass: `npx vitest run` in `libs/isis/scene-from-image-composer`
  (block B + block C tests together).
- `npx tsc --noEmit` clean.
- Stub-indicator scan clean.
- The composer can run in any of the four modes via a single config flag.
- Detection-phase emits a `detection-preview` event before object generation so
  a checkpoint can review detected boxes (block A integration).
- README updated with one snippet per mode.

## Open questions

- **Background labels.** The default list is curated from common detector
  vocabularies (COCO + ADE20K). Should it be configurable per-tenant or global?
  Recommended: global default, per-request override.
- **`surface-snapped` anchor.** Requires depth queries against the splat, which
  we don't have until block E. Until then, `surface-snapped` falls back to
  `centre-ground` with a warning. v1.5 wires the real depth query.
- **Cost cap behaviour.** Halt or proceed-with-subset? Recommended:
  proceed-with-subset (the user's already paid for detection); emit a soft
  warning event.
- **`providerHint` enforcement.** Strict ("must use this provider") or soft
  ("prefer if capable")? Recommended: soft. If hinted provider lacks required
  capability, fall back with warning.
