# `@isis/scene-from-image-composer`

End-to-end scene composition from a single input image. Fuses environment
generation (World Labs Marble), foreground meshes (Hunyuan / Tripo / Meshy),
ambient SFX (ElevenLabs SFX), and an input-grounding QA gate behind a single
async call.

Implements **all nine blocks (A through I)** of the
[`ISIS_GAPS`](../../../ISIS_GAPS/README.md) plan. Block A's operator-checkpoint
primitive in `@isis/operation-orchestrator` is the foundation; subsequent blocks
B-I layer additional capabilities on top while preserving the lib's two-tier
public surface.

## Status

| Phase          | Scope                                                                                                                                       | Status                |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | --------------------- |
| 1 — MVP slice  | explicit-prompt object mode; transform-only frame bridging (no bake); in-process providers; auto-approving checkpoints                      | **shipped — block B** |
| 2a — Thorough  | full ObjectSpec union (explicit / cropped / vlm-detected / mixed); foreground detection subphase; anchor DSL + solver; mesh-provider router | **shipped — block C** |
| 2b — Thorough  | baked single `.spz` output (surface-sampling + differentiable hybrid); lighting consistency; persistence + resume                           | **shipped — D/E/F/H** |
| 3 — Production | GPU dispatch via Redis JobChannel; gallery; lineage; budget; license; agent-consensus                                                       | **shipped — G/I**     |

## Install

Workspace package — referenced via pnpm catalog:

```json
{
  "dependencies": {
    "@isis/scene-from-image-composer": "workspace:*"
  }
}
```

## Public API — two tiers

### Convenience: `composeSceneFromImage`

Runs end-to-end with ephemeral in-process orchestrators. The right call for
tests, scripts, and one-shot CLI use.

```ts
import { composeSceneFromImage } from '@isis/scene-from-image-composer';
import {
  createMarbleProvider,
  createHunyuanProvider,
} from '@isis/3d-generation/providers';
import { createSFXProvider } from '@isis/audio-generation/generation';

const result = await composeSceneFromImage({
  assetId: 'cypress-garden-001',
  sourceImage: imageBytes, // Buffer of the input image (PNG/JPEG)
  providers: {
    marble: createMarbleProvider({ apiKey: process.env.WORLD_LABS_API_KEY }),
    mesh: createHunyuanProvider({ apiKey: process.env.HUNYUAN_API_KEY }),
    sfx: createSFXProvider({
      provider: 'elevenlabs',
      apiKey: process.env.ELEVENLABS_API_KEY,
    }),
  },
  objects: [
    {
      source: 'explicit',
      id: 'tree-1',
      prompt: 'cypress tree',
      anchor: { type: 'centre-ground' },
    },
    {
      source: 'explicit',
      id: 'lantern-1',
      prompt: 'stone lantern',
      anchor: { type: 'on-floor-front-centre' },
    },
  ],
  ambient: { prompt: 'gentle wind through cypress, distant chimes' },
});

// result.manifest         — OutputManifest with .spz scene + .glb meshes + .mp3 ambient
// result.phaseOutputs     — typed per-phase outputs (env, grounding?, objects?, align?, audio?)
// result.checkpointHistory — empty for the convenience path (auto-approve)
// result.metrics          — totalDurationMs + perPhaseMs
```

### Advanced: `createSceneFromImagePipeline`

Returns a `SceneFromImagePipeline` whose `.run(opts?)` accepts caller-supplied
orchestrators. Necessary for block H (persistence), block I (gallery + lineage),
and any consumer that needs the full block-A event stream on their own
orchestrator instance.

```ts
import { createSceneFromImagePipeline } from '@isis/scene-from-image-composer';
import {
  createCheckpointCoordinator,
  createExecutionOrchestrator,
  createPipelineOrchestrator,
} from '@isis/operation-orchestrator';

// One shared coordinator across both orchestrators (per A-003 finding).
const coordinator = createCheckpointCoordinator();
const executionOrchestrator = createExecutionOrchestrator(
  undefined,
  coordinator
);
const pipelineOrchestrator = createPipelineOrchestrator(undefined, coordinator);

// Subscribe to whatever events the consumer cares about.
pipelineOrchestrator.on('pipeline_started', (e) =>
  console.log('started', e.pipelineId)
);
pipelineOrchestrator.on('pipeline_completed', (e) =>
  console.log('done', e.durationMs, 'ms')
);
executionOrchestrator.on('phase_started', (e) =>
  console.log('phase', e.phaseId)
);
executionOrchestrator.on('phase_completed', (e) =>
  console.log('phase done', e.phaseId, e.status)
);

const pipeline = createSceneFromImagePipeline(request);
const result = await pipeline.run({
  executionOrchestrator,
  pipelineOrchestrator,
  coordinator,
});
```

The advanced API also exposes the static definition (`pipeline.request`,
`pipeline.phaseOrder`) so callers can persist or inspect the planned run before
executing.

## Object source modes (block C)

`ObjectSpec` is a discriminated union with three caller-supplied source kinds
plus an auto-derived `mixed` strategy. The composer auto-derives an
[`ObjectStrategy`](src/object-strategy.ts) from the request shape via
`deriveObjectStrategy(request)`; the detection-phase runs only when the strategy
is `'vlm-detected'` or `'mixed'`.

### `explicit` — text-prompt object

The block-B path. Each spec carries a prompt; the mesh provider's
`generateFromText` produces the mesh.

```ts
objects: [
  {
    source: 'explicit',
    id: 'tree-1',
    prompt: 'cypress tree',
    anchor: { type: 'centre-ground' },
  },
];
```

### `cropped` — caller-supplied bbox + optional prompt

The composer crops the source image to `bbox`, then calls the mesh provider's
`generateFromImage`. Useful when the caller already knows which region holds the
subject (e.g. from a UI selection).

```ts
objects: [
  {
    source: 'cropped',
    id: 'subject-1',
    bbox: { x: 0.2, y: 0.3, w: 0.4, h: 0.5 }, // normalised image coordinates
    prompt: 'antique brass lamp', // optional guidance
    removeBackground: true,
    anchor: { type: 'bbox-floor', bbox: { x: 0.2, y: 0.3, w: 0.4, h: 0.5 } },
  },
];
```

### `vlm-detected` — auto-detected from the source image

Empty `objects`, plus a `groundingAdapter`, plus an optional
[`DetectionPolicy`](src/detection/types.ts). The detection-subphase calls the
VLM, merges overlapping detections by IoU, drops background labels (sky, wall,
floor, ...) via the curated
[foreground classifier](src/detection/foreground-classifier.ts), and synthesises
a `VlmDetectedObjectSpec` per surviving detection. The objects-phase then
image→3Ds each crop.

```ts
import { ClaudeGroundingAdapter } from '@isis/3d-quality-gates';

await composeSceneFromImage({
  ...request,
  groundingAdapter: new ClaudeGroundingAdapter({
    apiKey: process.env.ANTHROPIC_API_KEY,
  }),
  // No `objects` → strategy='vlm-detected'.
  detection: {
    topK: 8, // keep at most 8 detections (default)
    minConfidence: 0.6, // override the 0.5 default
    iouMergeThreshold: 0.5, // pairs over this IoU collapse to one
    excludeLabels: ['shadow'], // caller-supplied drops (post-canonicalisation)
    requireLabels: ['chair'], // these survive even when classify(label) === 'background'
    costCapCents: 200, // greedy clamp under this cap (paired with a CostEstimator)
  },
});
```

### `mixed` — caller-supplied + detected

When `objects` is non-empty AND `groundingAdapter` is set, the strategy is
`'mixed'`: the detection-phase augments the caller's list. Both lists feed the
objects-phase; the constraint-solver places everything in one batch (supporting
`forward-of` / `right-of` relative anchors that reference other specs by id).

### `MeshProviderRouter` — per-spec routing

Block C's router picks per-spec providers from a pool with weighted scoring
(quality + cost + latency), a soft `providerHint` boost, and built-in rules
(character → Hunyuan, mechanical → Meshy, simple-prop → Tripo). Inject via the
advanced API's `runtime.meshProviderRouter`:

```ts
import {
  MeshProviderRouter,
  flatRateCostEstimator,
} from '@isis/scene-from-image-composer';

const router = new MeshProviderRouter({
  providers: [
    {
      provider: hunyuan,
      capabilities: hunyuan.getCapabilities(),
      costPerCallCents: 80,
      qualityScore: 0.85,
    },
    {
      provider: tripo,
      capabilities: tripo.getCapabilities(),
      costPerCallCents: 30,
      qualityScore: 0.7,
    },
    {
      provider: meshy,
      capabilities: meshy.getCapabilities(),
      costPerCallCents: 60,
      qualityScore: 0.8,
    },
  ],
});

await createSceneFromImagePipeline(request).run({
  runtime: {
    meshProviderRouter: router,
    detectionCostEstimator: flatRateCostEstimator(60), // 60¢ per detected spec
  },
});
```

Falls back to `request.providers.mesh` for any spec the router can't satisfy
(e.g. image→3D needed but only text-capable providers in the pool).

## Phases

Seven phases, run sequentially in `PHASE_ORDER`:

| Phase         | Owner module                  | What it does                                                                                          |
| ------------- | ----------------------------- | ----------------------------------------------------------------------------------------------------- |
| `environment` | `phases/environment-phase.ts` | Calls Marble; emits the env `.spz` + bounds + preview URLs                                            |
| `grounding`   | `phases/grounding-phase.ts`   | Optional — input-grounding QA gate (skip if no adapter)                                               |
| `detection`   | `phases/detection-phase.ts`   | Block C — VLM detect + IoU merge + foreground classify + topK/maxObjects/cost-cap clamp               |
| `objects`     | `phases/objects-phase.ts`     | Iterates caller-supplied + detected specs; routes per spec; explicit→text, cropped/vlm-detected→image |
| `align`       | `phases/align-phase.ts`       | Block C — constraint solver resolves all anchors, then per-mesh 4×4 column-major transforms           |
| `audio`       | `phases/audio-phase.ts`       | Optional — ambient SFX (skip if no sfx provider)                                                      |
| `manifest`    | `phases/manifest-phase.ts`    | Assembles `OutputManifest` with files + provenance + metadata.sceneFromImage                          |

Each phase wraps provider errors as
`ComposerError({ code: 'phase-failed', phase: '…', cause })`.

## Manifest extension

The composer writes typed metadata under `manifest.metadata.sceneFromImage`.
Schema declared in `src/manifest-extension.ts`:

```ts
{
  schemaVersion: '1.0.0',
  assetId: '…',
  sceneBoundsMeters: { min: [-5, 0, -5], max: [5, 6, 5] },
  placements: [
    { fileId: 'mesh-000-tree.glb', transform: [/* 16 floats */], anchor: { type: 'centre-ground' } },
    // …
  ],
}
```

Downstream consumers (Three.js viewer, Unreal plugin, block I gallery) read
`placements[].transform` and apply at load time. Block H's persistence layer
validates round-trips with `validateScenecomposerManifestMetadata(unknown)`.

## Test utilities

Fixture provider factories live under
`@isis/scene-from-image-composer/test-utils`. Reusable across the composer's own
tests and downstream consumers' tests.

```ts
import {
  createFixtureMarbleProvider,
  createFixtureMeshProvider,
  createFixtureSfxProvider,
  createFixtureGroundingAdapter,
} from '@isis/scene-from-image-composer/test-utils';

const marble = createFixtureMarbleProvider({
  bounds: { min: [-5, 0, -5], max: [5, 6, 5] },
  ambientPrompt: 'forest at dawn',
});
const mesh = createFixtureMeshProvider({ throwOnCallNumber: 2 });
```

Each fixture exposes its `vi.fn()` mock for caller-side
`expect(...).toHaveBeenCalled()` assertions and supports the failure modes the
composer's adversarial tests rely on.

## Test conventions

- Per CLAUDE.md / `oshun_nx_broken` memory: run tests with
  `npx vitest run --no-isolate` from this directory (bypassing Nx).
- Stub-indicator scan is part of pre-commit; phrases like `would`,
  `placeholder`, `todo` outside test files are blocked.

## Capabilities shipped across blocks D-I

- **Bake meshes into the splat** — block E ships the mesh-to-Gaussians
  rasteriser. Set `outputMode: 'baked'` with
  `bakeOptions.method: 'surface-sampling'` (Poisson area-weighted oriented
  disks) or `'hybrid'` (surface-sampling warm start + CPU-rendered multi-view
  supervision
  - gradient-descent refinement of positions/colours/opacities).
- **Lighting consistency** — block F estimates the env's SH light field from the
  Marble splat and applies PBR shading (sh9 + env-map encodings) to mesh
  Gaussians at bake time, plus a composition coherence gate with pluggable
  feature adapter.
- **GPU dispatch** — block G's pluggable `JobChannel` transport
  (`RedisJobChannel` for production, `InMemoryJobChannel` for tests) routes
  heavy phases (env / objects / bake) through `@isis/job-envelope` to a worker
  pool. Set `dispatchMode: 'job-envelope'` + `dispatchConfig: { redisUrl }`.
- **Persistence** — block H ships 6 Prisma models + a `PipelineRepository`
  interface (in-memory + Postgres impls). Set
  `repositoryConfig: { mode: 'postgres' }` for durable state; SIGKILL-resume
  golden in `composer.persistence.test.ts`.
- **Operator policies + agent consensus** — block I threads six external isis
  libraries (output-gallery, lineage, token-budget, model-governance-3d,
  anomaly-detection, agent-consensus) into the composer's runner via the
  `policyContext` field. Consensus checkpoint handler decides 3-agent fixture
  cases across all-SUPPORT / all-OPPOSE / CONDITIONAL / 2-vs-2 / throw / timeout
  paths.

The advanced API surface is the seam each of those blocks plugs into; the public
types stayed stable across blocks D-I. See
[`ISIS_GAPS/runbooks/production-runbook.md`](../../../ISIS_GAPS/runbooks/production-runbook.md)
for the deployment recipe and
[`ISIS_GAPS/runbooks/manual-qa.md`](../../../ISIS_GAPS/runbooks/manual-qa.md)
for the external-tool acceptance recipes.