Eve library

`@isis/scene-from-image-composer`

Other

Revision 1

On this page

End-to-end scene composition from a single input image. Fuses environment generation (World Labs Marble), foreground meshes (Hunyuan / Tripo / Meshy), ambient SFX (ElevenLabs SFX), and an input-grounding QA gate behind a single async call.

Implements all nine blocks (A through I) of the ISIS_GAPS plan. Block A's operator-checkpoint primitive in @isis/operation-orchestrator is the foundation; subsequent blocks B-I layer additional capabilities on top while preserving the lib's two-tier public surface.

Status#

Phase Scope Status
1 — MVP slice explicit-prompt object mode; transform-only frame bridging (no bake); in-process providers; auto-approving checkpoints shipped — block B
2a — Thorough full ObjectSpec union (explicit / cropped / vlm-detected / mixed); foreground detection subphase; anchor DSL + solver; mesh-provider router shipped — block C
2b — Thorough baked single .spz output (surface-sampling + differentiable hybrid); lighting consistency; persistence + resume shipped — D/E/F/H
3 — Production GPU dispatch via Redis JobChannel; gallery; lineage; budget; license; agent-consensus shipped — G/I

Install#

Workspace package — referenced via pnpm catalog:

json
{
  "dependencies": {
    "@isis/scene-from-image-composer": "workspace:*"
  }
}

Public API — two tiers#

Convenience: composeSceneFromImage#

Runs end-to-end with ephemeral in-process orchestrators. The right call for tests, scripts, and one-shot CLI use.

ts
import { composeSceneFromImage } from '@isis/scene-from-image-composer';
import {
  createMarbleProvider,
  createHunyuanProvider,
} from '@isis/3d-generation/providers';
import { createSFXProvider } from '@isis/audio-generation/generation';

const result = await composeSceneFromImage({
  assetId: 'cypress-garden-001',
  sourceImage: imageBytes, // Buffer of the input image (PNG/JPEG)
  providers: {
    marble: createMarbleProvider({ apiKey: process.env.WORLD_LABS_API_KEY }),
    mesh: createHunyuanProvider({ apiKey: process.env.HUNYUAN_API_KEY }),
    sfx: createSFXProvider({
      provider: 'elevenlabs',
      apiKey: process.env.ELEVENLABS_API_KEY,
    }),
  },
  objects: [
    {
      source: 'explicit',
      id: 'tree-1',
      prompt: 'cypress tree',
      anchor: { type: 'centre-ground' },
    },
    {
      source: 'explicit',
      id: 'lantern-1',
      prompt: 'stone lantern',
      anchor: { type: 'on-floor-front-centre' },
    },
  ],
  ambient: { prompt: 'gentle wind through cypress, distant chimes' },
});

// result.manifest         — OutputManifest with .spz scene + .glb meshes + .mp3 ambient
// result.phaseOutputs     — typed per-phase outputs (env, grounding?, objects?, align?, audio?)
// result.checkpointHistory — empty for the convenience path (auto-approve)
// result.metrics          — totalDurationMs + perPhaseMs

Advanced: createSceneFromImagePipeline#

Returns a SceneFromImagePipeline whose .run(opts?) accepts caller-supplied orchestrators. Necessary for block H (persistence), block I (gallery + lineage), and any consumer that needs the full block-A event stream on their own orchestrator instance.

ts
import { createSceneFromImagePipeline } from '@isis/scene-from-image-composer';
import {
  createCheckpointCoordinator,
  createExecutionOrchestrator,
  createPipelineOrchestrator,
} from '@isis/operation-orchestrator';

// One shared coordinator across both orchestrators (per A-003 finding).
const coordinator = createCheckpointCoordinator();
const executionOrchestrator = createExecutionOrchestrator(
  undefined,
  coordinator
);
const pipelineOrchestrator = createPipelineOrchestrator(undefined, coordinator);

// Subscribe to whatever events the consumer cares about.
pipelineOrchestrator.on('pipeline_started', (e) =>
  console.log('started', e.pipelineId)
);
pipelineOrchestrator.on('pipeline_completed', (e) =>
  console.log('done', e.durationMs, 'ms')
);
executionOrchestrator.on('phase_started', (e) =>
  console.log('phase', e.phaseId)
);
executionOrchestrator.on('phase_completed', (e) =>
  console.log('phase done', e.phaseId, e.status)
);

const pipeline = createSceneFromImagePipeline(request);
const result = await pipeline.run({
  executionOrchestrator,
  pipelineOrchestrator,
  coordinator,
});

The advanced API also exposes the static definition (pipeline.request, pipeline.phaseOrder) so callers can persist or inspect the planned run before executing.

Object source modes (block C)#

ObjectSpec is a discriminated union with three caller-supplied source kinds plus an auto-derived mixed strategy. The composer auto-derives an ObjectStrategy from the request shape via deriveObjectStrategy(request); the detection-phase runs only when the strategy is 'vlm-detected' or 'mixed'.

explicit — text-prompt object#

The block-B path. Each spec carries a prompt; the mesh provider's generateFromText produces the mesh.

ts
objects: [
  {
    source: 'explicit',
    id: 'tree-1',
    prompt: 'cypress tree',
    anchor: { type: 'centre-ground' },
  },
];

cropped — caller-supplied bbox + optional prompt#

The composer crops the source image to bbox, then calls the mesh provider's generateFromImage. Useful when the caller already knows which region holds the subject (e.g. from a UI selection).

ts
objects: [
  {
    source: 'cropped',
    id: 'subject-1',
    bbox: { x: 0.2, y: 0.3, w: 0.4, h: 0.5 }, // normalised image coordinates
    prompt: 'antique brass lamp', // optional guidance
    removeBackground: true,
    anchor: { type: 'bbox-floor', bbox: { x: 0.2, y: 0.3, w: 0.4, h: 0.5 } },
  },
];

vlm-detected — auto-detected from the source image#

Empty objects, plus a groundingAdapter, plus an optional DetectionPolicy. The detection-subphase calls the VLM, merges overlapping detections by IoU, drops background labels (sky, wall, floor, ...) via the curated foreground classifier, and synthesises a VlmDetectedObjectSpec per surviving detection. The objects-phase then image→3Ds each crop.

ts
import { ClaudeGroundingAdapter } from '@isis/3d-quality-gates';

await composeSceneFromImage({
  ...request,
  groundingAdapter: new ClaudeGroundingAdapter({
    apiKey: process.env.ANTHROPIC_API_KEY,
  }),
  // No `objects` → strategy='vlm-detected'.
  detection: {
    topK: 8, // keep at most 8 detections (default)
    minConfidence: 0.6, // override the 0.5 default
    iouMergeThreshold: 0.5, // pairs over this IoU collapse to one
    excludeLabels: ['shadow'], // caller-supplied drops (post-canonicalisation)
    requireLabels: ['chair'], // these survive even when classify(label) === 'background'
    costCapCents: 200, // greedy clamp under this cap (paired with a CostEstimator)
  },
});

mixed — caller-supplied + detected#

When objects is non-empty AND groundingAdapter is set, the strategy is 'mixed': the detection-phase augments the caller's list. Both lists feed the objects-phase; the constraint-solver places everything in one batch (supporting forward-of / right-of relative anchors that reference other specs by id).

MeshProviderRouter — per-spec routing#

Block C's router picks per-spec providers from a pool with weighted scoring (quality + cost + latency), a soft providerHint boost, and built-in rules (character → Hunyuan, mechanical → Meshy, simple-prop → Tripo). Inject via the advanced API's runtime.meshProviderRouter:

ts
import {
  MeshProviderRouter,
  flatRateCostEstimator,
} from '@isis/scene-from-image-composer';

const router = new MeshProviderRouter({
  providers: [
    {
      provider: hunyuan,
      capabilities: hunyuan.getCapabilities(),
      costPerCallCents: 80,
      qualityScore: 0.85,
    },
    {
      provider: tripo,
      capabilities: tripo.getCapabilities(),
      costPerCallCents: 30,
      qualityScore: 0.7,
    },
    {
      provider: meshy,
      capabilities: meshy.getCapabilities(),
      costPerCallCents: 60,
      qualityScore: 0.8,
    },
  ],
});

await createSceneFromImagePipeline(request).run({
  runtime: {
    meshProviderRouter: router,
    detectionCostEstimator: flatRateCostEstimator(60), // 60¢ per detected spec
  },
});

Falls back to request.providers.mesh for any spec the router can't satisfy (e.g. image→3D needed but only text-capable providers in the pool).

Phases#

Seven phases, run sequentially in PHASE_ORDER:

Phase Owner module What it does
environment phases/environment-phase.ts Calls Marble; emits the env .spz + bounds + preview URLs
grounding phases/grounding-phase.ts Optional — input-grounding QA gate (skip if no adapter)
detection phases/detection-phase.ts Block C — VLM detect + IoU merge + foreground classify + topK/maxObjects/cost-cap clamp
objects phases/objects-phase.ts Iterates caller-supplied + detected specs; routes per spec; explicit→text, cropped/vlm-detected→image
align phases/align-phase.ts Block C — constraint solver resolves all anchors, then per-mesh 4×4 column-major transforms
audio phases/audio-phase.ts Optional — ambient SFX (skip if no sfx provider)
manifest phases/manifest-phase.ts Assembles OutputManifest with files + provenance + metadata.sceneFromImage

Each phase wraps provider errors as ComposerError({ code: 'phase-failed', phase: '…', cause }).

Manifest extension#

The composer writes typed metadata under manifest.metadata.sceneFromImage. Schema declared in src/manifest-extension.ts:

ts
{
  schemaVersion: '1.0.0',
  assetId: '…',
  sceneBoundsMeters: { min: [-5, 0, -5], max: [5, 6, 5] },
  placements: [
    { fileId: 'mesh-000-tree.glb', transform: [/* 16 floats */], anchor: { type: 'centre-ground' } },
    // …
  ],
}

Downstream consumers (Three.js viewer, Unreal plugin, block I gallery) read placements[].transform and apply at load time. Block H's persistence layer validates round-trips with validateScenecomposerManifestMetadata(unknown).

Test utilities#

Fixture provider factories live under @isis/scene-from-image-composer/test-utils. Reusable across the composer's own tests and downstream consumers' tests.

ts
import {
  createFixtureMarbleProvider,
  createFixtureMeshProvider,
  createFixtureSfxProvider,
  createFixtureGroundingAdapter,
} from '@isis/scene-from-image-composer/test-utils';

const marble = createFixtureMarbleProvider({
  bounds: { min: [-5, 0, -5], max: [5, 6, 5] },
  ambientPrompt: 'forest at dawn',
});
const mesh = createFixtureMeshProvider({ throwOnCallNumber: 2 });

Each fixture exposes its vi.fn() mock for caller-side expect(...).toHaveBeenCalled() assertions and supports the failure modes the composer's adversarial tests rely on.

Test conventions#

  • Per CLAUDE.md / oshun_nx_broken memory: run tests with npx vitest run --no-isolate from this directory (bypassing Nx).
  • Stub-indicator scan is part of pre-commit; phrases like would, placeholder, todo outside test files are blocked.

Capabilities shipped across blocks D-I#

  • Bake meshes into the splat — block E ships the mesh-to-Gaussians rasteriser. Set outputMode: 'baked' with bakeOptions.method: 'surface-sampling' (Poisson area-weighted oriented disks) or 'hybrid' (surface-sampling warm start + CPU-rendered multi-view supervision
    • gradient-descent refinement of positions/colours/opacities).
  • Lighting consistency — block F estimates the env's SH light field from the Marble splat and applies PBR shading (sh9 + env-map encodings) to mesh Gaussians at bake time, plus a composition coherence gate with pluggable feature adapter.
  • GPU dispatch — block G's pluggable JobChannel transport (RedisJobChannel for production, InMemoryJobChannel for tests) routes heavy phases (env / objects / bake) through @isis/job-envelope to a worker pool. Set dispatchMode: 'job-envelope' + dispatchConfig: { redisUrl }.
  • Persistence — block H ships 6 Prisma models + a PipelineRepository interface (in-memory + Postgres impls). Set repositoryConfig: { mode: 'postgres' } for durable state; SIGKILL-resume golden in composer.persistence.test.ts.
  • Operator policies + agent consensus — block I threads six external isis libraries (output-gallery, lineage, token-budget, model-governance-3d, anomaly-detection, agent-consensus) into the composer's runner via the policyContext field. Consensus checkpoint handler decides 3-agent fixture cases across all-SUPPORT / all-OPPOSE / CONDITIONAL / 2-vs-2 / throw / timeout paths.

The advanced API surface is the seam each of those blocks plugs into; the public types stayed stable across blocks D-I. See ISIS_GAPS/runbooks/production-runbook.md for the deployment recipe and ISIS_GAPS/runbooks/manual-qa.md for the external-tool acceptance recipes.