Eve library

Block B · Scene-From-Image Composer Library

Other

Revision 1

On this page

Owner library: libs/isis/scene-from-image-composer (new) Estimate: ~1 week on top of block A Blocks: none directly; downstream blocks (C, E, F, G, H, I) extend the composer's surface. Dependencies: block A. Status: not started

Overview#

The composer library itself. Defines the public API (composeSceneFromImage, createSceneFromImagePipeline), the typed phase contracts, the artifact types (SceneFromImageRequest, SceneFromImageResult), the splat-mesh frame-bridging helper, the manifest-extension shape, and the in-process default orchestrator configuration with auto-approving checkpoints.

This block targets the composer's MVP slice: explicit-prompt object mode only, transform-only frame bridging (no bake), in-process providers, no GPU dispatch. Later blocks layer the additional capabilities on top without changing the public API contract.

Files to create#

Library scaffolding#

  • libs/isis/scene-from-image-composer/package.json
  • libs/isis/scene-from-image-composer/tsconfig.json
  • libs/isis/scene-from-image-composer/project.json
  • libs/isis/scene-from-image-composer/README.md
  • libs/isis/scene-from-image-composer/src/index.ts

Public API#

  • libs/isis/scene-from-image-composer/src/types.ts — SceneFromImageRequest, SceneFromImageResult, ComposerError discriminated union, PhaseId enum, ObjectSpec (placeholder; real one lands in block C).
  • libs/isis/scene-from-image-composer/src/composer.ts — implements composeSceneFromImage (convenience) and createSceneFromImagePipeline (advanced).
  • libs/isis/scene-from-image-composer/src/composer.test.ts

Phase implementations#

  • libs/isis/scene-from-image-composer/src/phases/environment-phase.ts — calls MarbleProvider.generateFromImage, returns EnvironmentPhaseOutput.
  • libs/isis/scene-from-image-composer/src/phases/grounding-phase.ts — calls evaluate3DQualityInputGroundingGate, returns GroundingPhaseOutput.
  • libs/isis/scene-from-image-composer/src/phases/objects-phase.ts — iterates over ObjectSpec[], calls the configured mesh provider for each, returns ObjectsPhaseOutput.
  • libs/isis/scene-from-image-composer/src/phases/align-phase.ts — computes per-object 4×4 transforms; returns AlignPhaseOutput.
  • libs/isis/scene-from-image-composer/src/phases/audio-phase.ts — generates ambient + per-object SFX via SFXProvider, returns AudioPhaseOutput.
  • libs/isis/scene-from-image-composer/src/phases/manifest-phase.ts — builds the OutputManifest via @isis/outputs's manifest-builder.ts, returns the final SceneFromImageResult.
  • One *.test.ts per phase file.

Frame-bridging helper#

  • libs/isis/scene-from-image-composer/src/align/align-mesh-to-splat-frame.ts
  • libs/isis/scene-from-image-composer/src/align/align-mesh-to-splat-frame.test.ts
  • libs/isis/scene-from-image-composer/src/align/anchor-spec.ts — AnchorSpec discriminated union and resolveAnchor helper (block B ships only the simple variants; block C extends with constraint-solver variants).

Pipeline definition + orchestrator integration#

  • libs/isis/scene-from-image-composer/src/pipeline/build-pipeline.ts — produces a PipelineDefinition (consumable by block A's PipelineOrchestrator) from a SceneFromImageRequest.
  • libs/isis/scene-from-image-composer/src/pipeline/in-process-orchestrator.ts — thin convenience wrapper for the composeSceneFromImage API; spins up an ephemeral PipelineOrchestrator with auto-approving checkpoints and awaits completion.
  • libs/isis/scene-from-image-composer/src/pipeline/build-pipeline.test.ts

Manifest extension types#

  • libs/isis/scene-from-image-composer/src/manifest-extension.ts — ScenecomposerManifestMetadata (typed shape under OutputManifest.metadata.sceneFromImage), validator, exported re-exports.

Files to modify#

  • tsconfig.base.json — add path mapping "@isis/scene-from-image-composer": ["libs/isis/scene-from-image-composer/src/index.ts"].
  • pnpm-workspace.yaml — auto-picks up the new lib; verify.

Public API surface#

ts
// types.ts (public)

export interface SceneFromImageRequest {
  readonly sourceImage: Buffer;
  readonly mimeType?: string; // 'image/png' default

  // Provider injection — composer doesn't construct providers itself.
  readonly providers: {
    readonly marble: MarbleProvider;
    readonly mesh?: I3DProvider; // Hunyuan/Tripo/Meshy/etc.
    readonly sfx?: ISFXProvider;
  };

  // QA gate adapter. If omitted, grounding phase is skipped.
  readonly groundingAdapter?: ThreeDQualityGroundingVlmAdapter;

  // What objects to add. Defaults to []. Block C extends the union.
  readonly objects?: readonly ObjectSpec[];

  // Ambient-SFX prompt. If omitted, derived from Marble's ambient_prompt.
  readonly ambient?: { readonly prompt: string; readonly durationSec?: number };

  // Per-phase opt-out.
  readonly phases?: {
    readonly skipGrounding?: boolean;
    readonly skipObjects?: boolean;
    readonly skipAudio?: boolean;
  };

  // Orchestrator hooks.
  readonly checkpoints?: Partial<Record<PhaseId, CheckpointPolicy>>;

  // Policy context (block I).
  readonly policyContext?: PolicyContext;

  // Identity/tenancy.
  readonly assetId: string;
  readonly tenantId?: string;
  readonly userId?: string;
}

export type PhaseId =
  | 'environment'
  | 'grounding'
  | 'objects'
  | 'align'
  | 'audio'
  | 'manifest';

export interface SceneFromImageResult {
  readonly manifest: OutputManifest;
  readonly phaseOutputs: {
    readonly environment: EnvironmentPhaseOutput;
    readonly grounding?: GroundingPhaseOutput;
    readonly objects?: ObjectsPhaseOutput;
    readonly align?: AlignPhaseOutput;
    readonly audio?: AudioPhaseOutput;
  };
  readonly checkpointHistory: readonly CheckpointDecision[];
  readonly metrics: {
    readonly totalDurationMs: number;
    readonly perPhaseMs: Record<PhaseId, number>;
  };
}

// composer.ts (public)

export async function composeSceneFromImage(
  input: SceneFromImageRequest
): Promise<SceneFromImageResult>;

export function createSceneFromImagePipeline(
  input: SceneFromImageRequest
): PipelineDefinition;

Phase contracts#

Each phase is an async function (ctx: PhaseContext) => Promise<PhaseOutput> where PhaseContext carries the request + outputs of all preceding phases

  • a logger + (block I) a policy hook.
ts
// phases/environment-phase.ts
export interface EnvironmentPhaseOutput {
  readonly splatScene: Generated3DModel; // format: 'spz'
  readonly bounds: {
    readonly min: [number, number, number];
    readonly max: [number, number, number];
  };
  readonly previewUrls: readonly string[];
  readonly ambientPromptSuggestion?: string;
}

// phases/grounding-phase.ts
export interface GroundingPhaseOutput {
  readonly evaluation: ThreeDQualityInputGroundingEvaluation;
  readonly decision: ThreeDQualityInputGroundingDecision;
}

// phases/objects-phase.ts
export interface ObjectsPhaseOutput {
  readonly meshes: readonly GeneratedMesh[]; // one per ObjectSpec
}
export interface GeneratedMesh {
  readonly objectSpec: ObjectSpec;
  readonly model: Generated3DModel; // format: 'glb' typically
  readonly localBboxMeters: {
    readonly width: number;
    readonly height: number;
    readonly depth: number;
  };
}

// phases/align-phase.ts
export interface AlignPhaseOutput {
  readonly placements: readonly Placement[];
}
export interface Placement {
  readonly meshIndex: number; // index into ObjectsPhaseOutput.meshes
  readonly transform: Float32Array; // 16-element column-major 4×4
  readonly anchor: AnchorSpec;
}

// phases/audio-phase.ts
export interface AudioPhaseOutput {
  readonly ambient?: GeneratedSFX;
  readonly perObject: ReadonlyArray<{
    readonly meshIndex: number;
    readonly sfx: GeneratedSFX;
  }>;
}

Frame-bridging algorithm (align-mesh-to-splat-frame.ts)#

Inputs:

  • splatBoundsMeters: { min: [n,n,n]; max: [n,n,n] } — Marble world bounds.
  • meshLocalBbox: { width, height, depth } — from getDocumentStats(parseGLB(modelData)).
  • anchor: AnchorSpec — where in the splat to put the mesh.
  • desiredHeightMeters?: number — optional. If omitted, derived from anchor type (e.g. "centre-ground" defaults to 1m for visibility).

Output:

  • Float32Array(16) — column-major 4×4 transform M such that for any local vertex v, Mv is the world-space placement.

Steps:

  1. Scale. s = desiredHeightMeters / meshLocalBbox.height. (If desiredHeightMeters is omitted, default to the anchor's defaultHeightMeters.)
  2. Translation. Compute worldAnchor: [x, y, z] from anchor and splatBoundsMeters:
    • centre-ground: [(min.x+max.x)/2, min.y, (min.z+max.z)/2]
    • on-floor-front-centre: [(min.x+max.x)/2, min.y, max.z - 0.5]
    • { x, y, z }: as supplied.
  3. Mesh anchor offset. Default mesh anchor is bottom-centre (y = 0, x = z = bbox centre in mesh-local frame). Compute scaled mesh-local bottom-centre: [(scaled bbox centre x), 0, (scaled bbox centre z)].
  4. Translation matrix. t = worldAnchor - meshLocalBottomCentre * s.
  5. Rotation. v1 assumes Y-up for both Marble and GLB (Hunyuan, Tripo, Meshy, Rodin all default to Y-up GLB). For now: identity rotation. Future: detect axis convention from GLB asset.extras if present, fall back to identity.
  6. Compose. M = T * R * S written in column-major.

Validation:

  • Throws AlignmentError if meshLocalBbox.height <= 0 (degenerate mesh).
  • Throws if splatBoundsMeters is inverted (max < min) on any axis.

In-process orchestrator#

ts
// pipeline/in-process-orchestrator.ts (sketch)

export async function runInProcess(
  input: SceneFromImageRequest
): Promise<SceneFromImageResult> {
  const orchestrator = createPipelineOrchestrator({
    enableCheckpoints: true,
    checkpointPolicy: input.checkpoints
      ? mergeWithDefaultPolicy(input.checkpoints)
      : { timeoutMs: 5 * 60_000, onTimeout: 'approve' },
  });
  const definition = createSceneFromImagePipeline(input);
  const pipeline = orchestrator.runPipeline(definition);
  return await collectResult(orchestrator, pipeline.id);
}

Test plan#

Unit per phase:

  • Each phase test mocks the providers/adapter it depends on and asserts the output shape against fixture data. No network calls.
  • align-phase.test.ts: deterministic transforms — synthetic 10×10×10 splat bounds + unit-cube mesh + centre-ground anchor produces scale=1, translation=[0, -5, 0], rotation=I to within FP epsilon. Multiple anchor variants tested (10+ cases).

Integration (composer.test.ts):

  • Full happy path with all stub providers: produces a SceneFromImageResult with manifest.files.length === 4 (splat + 1 mesh + ambient + 1 preview), metadata.sceneFromImage.placements.length === 1, and exactly the expected provenance records.
  • Skip-objects mode: manifest.files.length === 2 (splat + ambient).
  • Skip-grounding mode: phaseOutputs.grounding === undefined.
  • Skip-audio mode: phaseOutputs.audio === undefined.
  • Failure: Marble provider throws → composer throws ComposerError with phase: 'environment'.
  • Checkpoint rejection: handler returns rejected after environment phase → composer throws ComposerError with phase: 'environment', cause: 'checkpoint-rejected', no downstream phases run.
  • Modification flow: handler returns modified with overlay { ambient: { prompt: 'override' } } → audio phase receives the override.

Adversarial:

  • Empty objects: [] does not break align phase.
  • Empty preview URLs from Marble: thumbnail unset, no crash.
  • groundingAdapter omitted → grounding phase skipped silently.
  • Manifest extension validator rejects malformed placements arrays.

Acceptance criteria#

  • All new tests pass: npx vitest run in libs/isis/scene-from-image-composer.
  • npx tsc --noEmit clean.
  • Stub-indicator scan clean on all new files.
  • The composer library has zero direct fetch/network calls outside provider injection (verified by adversarial grep).
  • README documents the convenience and advanced APIs with one runnable snippet each.
  • tsconfig.base.json path-mapping verified by importing @isis/scene-from-image-composer from a sibling lib.

Open questions#

  • Default mesh anchor. Bottom-centre seems right but Hunyuan in practice produces some meshes that are not bbox-aligned (fur poking out, etc.). Should v1 use bbox-bottom-centre or geometric-bottom-centre? Start with bbox-bottom-centre; revisit if visual QA fails.
  • Checkpoint default per phase. v1 ships auto-approve-after: 5min for every phase. Block I changes the default per environment (production vs CI vs local).
  • MarbleProvider lives in 3d-generation — composer needs to import the provider type. Adds a circular-import risk. Resolve by importing only I3DProvider from 3d-generation/types in the composer, and MarbleProvider only in the test fixtures and the documentation. The caller injects a constructed provider.