Owner library: libs/isis/scene-from-image-composer (new) Estimate: ~1
week on top of block A Blocks: none directly; downstream blocks (C, E, F, G,
H, I) extend the composer's surface. Dependencies: block A. Status: not
started
Overview#
The composer library itself. Defines the public API (composeSceneFromImage,
createSceneFromImagePipeline), the typed phase contracts, the artifact types
(SceneFromImageRequest, SceneFromImageResult), the splat-mesh frame-bridging
helper, the manifest-extension shape, and the in-process default orchestrator
configuration with auto-approving checkpoints.
This block targets the composer's MVP slice: explicit-prompt object mode only, transform-only frame bridging (no bake), in-process providers, no GPU dispatch. Later blocks layer the additional capabilities on top without changing the public API contract.
Files to create#
Library scaffolding#
libs/isis/scene-from-image-composer/package.jsonlibs/isis/scene-from-image-composer/tsconfig.jsonlibs/isis/scene-from-image-composer/project.jsonlibs/isis/scene-from-image-composer/README.mdlibs/isis/scene-from-image-composer/src/index.ts
Public API#
libs/isis/scene-from-image-composer/src/types.ts—SceneFromImageRequest,SceneFromImageResult,ComposerErrordiscriminated union,PhaseIdenum,ObjectSpec(placeholder; real one lands in block C).libs/isis/scene-from-image-composer/src/composer.ts— implementscomposeSceneFromImage(convenience) andcreateSceneFromImagePipeline(advanced).libs/isis/scene-from-image-composer/src/composer.test.ts
Phase implementations#
libs/isis/scene-from-image-composer/src/phases/environment-phase.ts— callsMarbleProvider.generateFromImage, returnsEnvironmentPhaseOutput.libs/isis/scene-from-image-composer/src/phases/grounding-phase.ts— callsevaluate3DQualityInputGroundingGate, returnsGroundingPhaseOutput.libs/isis/scene-from-image-composer/src/phases/objects-phase.ts— iterates overObjectSpec[], calls the configured mesh provider for each, returnsObjectsPhaseOutput.libs/isis/scene-from-image-composer/src/phases/align-phase.ts— computes per-object 4×4 transforms; returnsAlignPhaseOutput.libs/isis/scene-from-image-composer/src/phases/audio-phase.ts— generates ambient + per-object SFX viaSFXProvider, returnsAudioPhaseOutput.libs/isis/scene-from-image-composer/src/phases/manifest-phase.ts— builds theOutputManifestvia@isis/outputs'smanifest-builder.ts, returns the finalSceneFromImageResult.- One
*.test.tsper phase file.
Frame-bridging helper#
libs/isis/scene-from-image-composer/src/align/align-mesh-to-splat-frame.tslibs/isis/scene-from-image-composer/src/align/align-mesh-to-splat-frame.test.tslibs/isis/scene-from-image-composer/src/align/anchor-spec.ts—AnchorSpecdiscriminated union andresolveAnchorhelper (block B ships only the simple variants; block C extends with constraint-solver variants).
Pipeline definition + orchestrator integration#
libs/isis/scene-from-image-composer/src/pipeline/build-pipeline.ts— produces aPipelineDefinition(consumable by block A'sPipelineOrchestrator) from aSceneFromImageRequest.libs/isis/scene-from-image-composer/src/pipeline/in-process-orchestrator.ts— thin convenience wrapper for thecomposeSceneFromImageAPI; spins up an ephemeralPipelineOrchestratorwith auto-approving checkpoints and awaits completion.libs/isis/scene-from-image-composer/src/pipeline/build-pipeline.test.ts
Manifest extension types#
libs/isis/scene-from-image-composer/src/manifest-extension.ts—ScenecomposerManifestMetadata(typed shape underOutputManifest.metadata.sceneFromImage), validator, exported re-exports.
Files to modify#
tsconfig.base.json— add path mapping"@isis/scene-from-image-composer": ["libs/isis/scene-from-image-composer/src/index.ts"].pnpm-workspace.yaml— auto-picks up the new lib; verify.
Public API surface#
// types.ts (public)
export interface SceneFromImageRequest {
readonly sourceImage: Buffer;
readonly mimeType?: string; // 'image/png' default
// Provider injection — composer doesn't construct providers itself.
readonly providers: {
readonly marble: MarbleProvider;
readonly mesh?: I3DProvider; // Hunyuan/Tripo/Meshy/etc.
readonly sfx?: ISFXProvider;
};
// QA gate adapter. If omitted, grounding phase is skipped.
readonly groundingAdapter?: ThreeDQualityGroundingVlmAdapter;
// What objects to add. Defaults to []. Block C extends the union.
readonly objects?: readonly ObjectSpec[];
// Ambient-SFX prompt. If omitted, derived from Marble's ambient_prompt.
readonly ambient?: { readonly prompt: string; readonly durationSec?: number };
// Per-phase opt-out.
readonly phases?: {
readonly skipGrounding?: boolean;
readonly skipObjects?: boolean;
readonly skipAudio?: boolean;
};
// Orchestrator hooks.
readonly checkpoints?: Partial<Record<PhaseId, CheckpointPolicy>>;
// Policy context (block I).
readonly policyContext?: PolicyContext;
// Identity/tenancy.
readonly assetId: string;
readonly tenantId?: string;
readonly userId?: string;
}
export type PhaseId =
| 'environment'
| 'grounding'
| 'objects'
| 'align'
| 'audio'
| 'manifest';
export interface SceneFromImageResult {
readonly manifest: OutputManifest;
readonly phaseOutputs: {
readonly environment: EnvironmentPhaseOutput;
readonly grounding?: GroundingPhaseOutput;
readonly objects?: ObjectsPhaseOutput;
readonly align?: AlignPhaseOutput;
readonly audio?: AudioPhaseOutput;
};
readonly checkpointHistory: readonly CheckpointDecision[];
readonly metrics: {
readonly totalDurationMs: number;
readonly perPhaseMs: Record<PhaseId, number>;
};
}
// composer.ts (public)
export async function composeSceneFromImage(
input: SceneFromImageRequest
): Promise<SceneFromImageResult>;
export function createSceneFromImagePipeline(
input: SceneFromImageRequest
): PipelineDefinition;
Phase contracts#
Each phase is an async function (ctx: PhaseContext) => Promise<PhaseOutput>
where PhaseContext carries the request + outputs of all preceding phases
- a logger + (block I) a policy hook.
// phases/environment-phase.ts
export interface EnvironmentPhaseOutput {
readonly splatScene: Generated3DModel; // format: 'spz'
readonly bounds: {
readonly min: [number, number, number];
readonly max: [number, number, number];
};
readonly previewUrls: readonly string[];
readonly ambientPromptSuggestion?: string;
}
// phases/grounding-phase.ts
export interface GroundingPhaseOutput {
readonly evaluation: ThreeDQualityInputGroundingEvaluation;
readonly decision: ThreeDQualityInputGroundingDecision;
}
// phases/objects-phase.ts
export interface ObjectsPhaseOutput {
readonly meshes: readonly GeneratedMesh[]; // one per ObjectSpec
}
export interface GeneratedMesh {
readonly objectSpec: ObjectSpec;
readonly model: Generated3DModel; // format: 'glb' typically
readonly localBboxMeters: {
readonly width: number;
readonly height: number;
readonly depth: number;
};
}
// phases/align-phase.ts
export interface AlignPhaseOutput {
readonly placements: readonly Placement[];
}
export interface Placement {
readonly meshIndex: number; // index into ObjectsPhaseOutput.meshes
readonly transform: Float32Array; // 16-element column-major 4×4
readonly anchor: AnchorSpec;
}
// phases/audio-phase.ts
export interface AudioPhaseOutput {
readonly ambient?: GeneratedSFX;
readonly perObject: ReadonlyArray<{
readonly meshIndex: number;
readonly sfx: GeneratedSFX;
}>;
}
Frame-bridging algorithm (align-mesh-to-splat-frame.ts)#
Inputs:
splatBoundsMeters: { min: [n,n,n]; max: [n,n,n] }— Marble world bounds.meshLocalBbox: { width, height, depth }— fromgetDocumentStats(parseGLB(modelData)).anchor: AnchorSpec— where in the splat to put the mesh.desiredHeightMeters?: number— optional. If omitted, derived from anchor type (e.g. "centre-ground" defaults to 1m for visibility).
Output:
Float32Array(16)— column-major 4×4 transform M such that for any local vertexv,Mvis the world-space placement.
Steps:
- Scale.
s = desiredHeightMeters / meshLocalBbox.height. (IfdesiredHeightMetersis omitted, default to the anchor'sdefaultHeightMeters.) - Translation. Compute
worldAnchor: [x, y, z]fromanchorandsplatBoundsMeters:centre-ground:[(min.x+max.x)/2, min.y, (min.z+max.z)/2]on-floor-front-centre:[(min.x+max.x)/2, min.y, max.z - 0.5]{ x, y, z }: as supplied.
- Mesh anchor offset. Default mesh anchor is
bottom-centre(y = 0, x = z = bbox centre in mesh-local frame). Compute scaled mesh-local bottom-centre:[(scaled bbox centre x), 0, (scaled bbox centre z)]. - Translation matrix.
t = worldAnchor - meshLocalBottomCentre * s. - Rotation. v1 assumes Y-up for both Marble and GLB (Hunyuan, Tripo, Meshy,
Rodin all default to Y-up GLB). For now: identity rotation. Future: detect
axis convention from GLB
asset.extrasif present, fall back to identity. - Compose.
M = T * R * Swritten in column-major.
Validation:
- Throws
AlignmentErrorifmeshLocalBbox.height <= 0(degenerate mesh). - Throws if
splatBoundsMetersis inverted (max < min) on any axis.
In-process orchestrator#
// pipeline/in-process-orchestrator.ts (sketch)
export async function runInProcess(
input: SceneFromImageRequest
): Promise<SceneFromImageResult> {
const orchestrator = createPipelineOrchestrator({
enableCheckpoints: true,
checkpointPolicy: input.checkpoints
? mergeWithDefaultPolicy(input.checkpoints)
: { timeoutMs: 5 * 60_000, onTimeout: 'approve' },
});
const definition = createSceneFromImagePipeline(input);
const pipeline = orchestrator.runPipeline(definition);
return await collectResult(orchestrator, pipeline.id);
}
Test plan#
Unit per phase:
- Each phase test mocks the providers/adapter it depends on and asserts the output shape against fixture data. No network calls.
align-phase.test.ts: deterministic transforms — synthetic 10×10×10 splat bounds + unit-cube mesh +centre-groundanchor producesscale=1, translation=[0, -5, 0], rotation=Ito within FP epsilon. Multiple anchor variants tested (10+ cases).
Integration (composer.test.ts):
- Full happy path with all stub providers: produces a
SceneFromImageResultwithmanifest.files.length === 4(splat + 1 mesh + ambient + 1 preview),metadata.sceneFromImage.placements.length === 1, and exactly the expected provenance records. - Skip-objects mode:
manifest.files.length === 2(splat + ambient). - Skip-grounding mode:
phaseOutputs.grounding === undefined. - Skip-audio mode:
phaseOutputs.audio === undefined. - Failure: Marble provider throws → composer throws
ComposerErrorwithphase: 'environment'. - Checkpoint rejection: handler returns
rejectedafterenvironmentphase → composer throwsComposerErrorwithphase: 'environment',cause: 'checkpoint-rejected', no downstream phases run. - Modification flow: handler returns
modifiedwith overlay{ ambient: { prompt: 'override' } }→ audio phase receives the override.
Adversarial:
- Empty
objects: []does not break align phase. - Empty preview URLs from Marble: thumbnail unset, no crash.
groundingAdapteromitted → grounding phase skipped silently.- Manifest extension validator rejects malformed
placementsarrays.
Acceptance criteria#
- All new tests pass:
npx vitest runinlibs/isis/scene-from-image-composer. npx tsc --noEmitclean.- Stub-indicator scan clean on all new files.
- The composer library has zero direct fetch/network calls outside provider injection (verified by adversarial grep).
- README documents the convenience and advanced APIs with one runnable snippet each.
tsconfig.base.jsonpath-mapping verified by importing@isis/scene-from-image-composerfrom a sibling lib.
Open questions#
- Default mesh anchor. Bottom-centre seems right but Hunyuan in practice produces some meshes that are not bbox-aligned (fur poking out, etc.). Should v1 use bbox-bottom-centre or geometric-bottom-centre? Start with bbox-bottom-centre; revisit if visual QA fails.
- Checkpoint default per phase. v1 ships
auto-approve-after: 5minfor every phase. Block I changes the default per environment (production vs CI vs local). MarbleProviderlives in3d-generation— composer needs to import the provider type. Adds a circular-import risk. Resolve by importing onlyI3DProviderfrom3d-generation/typesin the composer, andMarbleProvideronly in the test fixtures and the documentation. The caller injects a constructed provider.