Inventory of existing Isis code that the scene-from-image composer depends on or extends. Each entry: what it does, what it exposes, what it doesn't (the gaps). All file:line refs verified at the time of writing.
3D generation providers#
libs/isis/3d-generation/src/providers/
| Provider | File | Type | Output |
|---|---|---|---|
| Marble | marble-provider.ts (newly shipped) |
env splat | .spz + previews |
| Hunyuan | hunyuan-provider.ts |
object mesh | .glb/.obj/.ply |
| Tripo | tripo-provider.ts |
object mesh | .glb |
| Meshy | meshy-provider.ts |
object mesh | .glb |
| Rodin | rodin-provider.ts |
object mesh | .glb |
| Trellis | trellis-provider.ts |
object mesh | .glb |
| ThreeDFY | threedfy-provider.ts |
object mesh | .glb |
All extend Base3DProvider (base-provider.ts) which gives them: typed event
emitter, polling helper (pollForCompletion), API-key auth, file-download
helper, and a job-status registry. The Marble provider in particular reports
world-space bounds in metres and emits a Generated3DModel with format: 'spz'
and a marble: MarbleSceneExtras field carrying the preview-URL list,
ambient-prompt suggestion, and timing metrics (marble-provider.ts:96-103).
Hunyuan parses GLB output via parseGLB and getDocumentStats from
processing/gltf-utils.ts and produces a
boundingBox: { width, height, depth } in the GLB's local frame — usually
unit-normalised (hunyuan-provider.ts:370-402).
Gap relevant to composer: there's no shared "scene-aware" placement metadata — each provider returns a single mesh in its own local frame with no coordinate-system declaration.
Multi-provider routing#
libs/isis/ai-providers/src/orchestration/unified-generation-pipeline.ts
UnifiedGenerationPipeline extends EventEmitter (line 774). Public surface:
createUnifiedPipeline(config)(line 1906)createUnifiedPipelineFromEnv()(line 1915)createGenerationRequest(modality, input, options)(line 1941)interface GenerationRequest(line 128) — modality, type, input, requirements, preferences, context
Capabilities: registers providers with capabilities/health/routing metadata;
selects providers via weighted scoring across cost/quality/latency/reliability;
fallback chains; generateBatch; generateParallel; in-flight request
deduplication; circuit breakers.
Gap relevant to composer: the pipeline is single-modality per request (give
me one image, give me one model, give me one audio). It is not a sequencer of
multiple modalities — there's no built-in "first do an env generation, then for
each detected object do a mesh generation, then do ambient SFX" composition. The
composer either wraps UnifiedGenerationPipeline (calling it once per phase) or
uses the providers directly. We choose the latter — see 03-decisions.md
ADR-002.
Operation orchestration#
libs/isis/operation-orchestrator/src/
Five sub-orchestrators in their own subdirectories:
pipeline-orchestrator/— sequencing of chainsexecution-orchestrator/— phases within a chainbatch-orchestrator/— batch compositionfailure-orchestrator/— dead-letter queue, failure-pattern analysisretry-orchestrator/— exponential/linear/Fibonacci backoff
Type model (types.ts):
ChainStatusenum (line 118):PENDING | RUNNING | COMPLETED | FAILED | CANCELLED | BLOCKEDPhaseTypeenum (line 129):SETUP | EXECUTION | VALIDATION | TEARDOWN | CHECKPOINTPipelineMode(line 139):SEQUENTIAL | PARALLEL | ADAPTIVEPriorityLevel(line 147):CRITICAL | HIGH | NORMAL | LOW | BACKGROUND
Public factory: createPipelineOrchestrator(config?) at
pipeline-orchestrator.ts:1037. Class: PipelineOrchestrator at
pipeline-orchestrator.ts:618.
Critical gap: the operator-checkpoint primitive is half-implemented.
enableCheckpoints: falseis the default config flag (execution-orchestrator.ts:41)- When
enableCheckpointsis true,injectCheckpointPhases(...)synthetically insertsPhaseType.CHECKPOINTphases between regular phases at a configured frequency (execution-orchestrator.ts:1056-1057, function defined at1362-1389) - BUT the injected CHECKPOINT phases have no special runtime semantics — they
execute like any other phase. There's no
approveCheckpoint()API, noAWAITING_APPROVALchain status, no event indicating "operator decision required", no per-checkpoint timeout policy grep -r approveCheckpoint awaitApproval AWAITING_APPROVAL libs/isis/operation-orchestrator/src/returns zero hits
This is the single biggest dependency for the composer. Fixing it is block A.
Audio generation#
libs/isis/audio-generation/src/generation/sfx-provider.ts
Class SFXProvider (line 266) implements ISFXProvider. Public methods:
| Method | Line | Purpose |
|---|---|---|
generate(request: SFXGenerationRequest) |
315 | Generic prompt-driven SFX |
generateFoley(request: FoleyRequest) |
361 | Surface-aware foley |
generateAmbient(request: AmbientRequest) |
369 | Loopable env ambience |
generateImpact(request: ImpactRequest) |
382 | Material-aware impacts |
generateAmbient auto-sets seamlessLoop: true and duration: 30s. Returns
GeneratedSFX with audioData: Buffer and optional downloadUrl. Provider
factories cover ElevenLabs, AudioLDM, AudioCraft, Freesound, and a local runner.
Gap relevant to composer: none functionally; the composer can call these
as-is. The composer adds the scene-context layer (passing an "ambient
prompt" derived from the Marble scene's metadata to generateAmbient, then
emitting per-object foley from the placed-meshes list).
Output management#
libs/isis/outputs/src/
| File | Purpose |
|---|---|
types.ts |
OutputManifest interface at line 366 |
manifest/manifest-builder.ts |
Fluent builder for manifests |
manifest/manifest-storage.ts |
Persistence for manifests |
manifest/manifest-validator.ts |
Schema validation |
lineage/lineage-graph.ts |
Cross-asset lineage edges |
provenance/provenance-tracker.ts |
Per-asset provenance records |
OutputManifest shape (relevant fields):
files: OutputFile[]— each withcategory∈image | video | audio | model-3d | texture | animation | document | data | archiveprovenance: ProvenanceRecord[]— generator, model, inputs, parameters, duration, compute infolineage?: LineageGraph— cross-asset edges includingcomposited-from,derived-from,variant-ofmetadata?: Record<string, unknown>— free-form extension point for composer-specific data (placement transforms, SH light fields, etc.)
Gap relevant to composer: none structurally — the manifest can hold the full
composed output. Composer adds typed metadata under metadata.sceneFromImage.*
(placements, SH coefficients, scene bounds, checkpoint history). See
04-api-surfaces.md for the typed extension shape.
Output gallery#
libs/isis/output-gallery/src/
UI surface for browsing/filtering generated outputs. OutputRecord:
outputId, tenantId, userId, domain, kind, model, assetUrl, thumbnailUrl, costCents, tags.
Gap relevant to composer: the composer needs to register every run as an
OutputRecord so it shows up in the gallery. Wired in
block I.
Scene assembly#
libs/isis/3d-scene-assembly/src/
24+ files; all currently purely declarative — recipes, scene graphs, validators, taxonomies. There is no orchestration code in this lib.
Most relevant for composer:
scene-unit-normalization.ts:62-84—createThreeDImportedAssetNormalizationResulttakessourceUnit(mm/cm/m/inch/foot),sourceUpAxis,sourceForwardAxis,measuredScale; returnsscaleMultiplierToMetersandrotationFixEulerDeg. Designed for importing external assets, not for bridging two co-generated assets in different frames.placement-rules-engine.ts:18-83—createThreeDPlacementRulesResulttakessceneBoundsMeters: { width, height, depth }and validates placements against bounds + clearance. Assumes all inputs in the same metres frame.procedural-assembly-recipes.ts— static catalog of assembly recipes (mech, furniture, weapon) with stage descriptions but no execution logic.semantic-scene-graph.ts— graph definition (nodes, connectors, constraints).bom-material-summary.ts,kitbash-library-builder.ts,connector-taxonomy.ts,donor-part-suggestions.ts,style-harmonization.ts,symmetry-placement.ts— all declarative validators/transformers.
Gap relevant to composer: no splat-aware placement helper. The composer
needs a new utility that takes (Marble bounds in metres, GLB local bbox) and
emits a 4×4 transform. Lives in
block B initially;
refactored into gaussian-splatting/lighting/ once block F lands.
Gaussian splatting#
libs/isis/gaussian-splatting/src/
Subdirs: capture/, exporters/, four-dgs/, nerf/, optimization/,
preprocessing/, rendering/, training/. Comprehensive on the training side;
sparse on the export side.
exporters/:
isis-splat-exporter.ts— internal.isplatbinary format for Unreal/Unity (8-byte magic0x4c505349"ISPL"). Real implementation, not Niantic-compatible.index.ts— only exportsIsisSplatExporter; no.spz,.ply, or.ksplatexporter despite those formats appearing in theExportFormatunion (types.ts:700-708).
rendering/splat-renderer.ts defines LoadedScene (positions, scales,
rotations, opacities, colors, optional spherical harmonics) — the canonical
in-memory splat representation.
Gaps relevant to composer:
- No
.spzexporter — block D - No mesh-to-Gaussians rasteriser — block E
- No splat-merging algorithm — block E
- No environment SH lighting estimator — block F
Quality gates#
libs/isis/3d-quality-gates/src/
Heavy structure: taxonomy/, corpus/, delivery/, operations/,
manifest.ts, blueprint-types.ts. Recently added:
delivery/input-grounding-gate.ts(newly shipped) — input-image vs output-renders set comparison via pluggable VLM adapter; computes recall and novelty; returnspass | soft-fail | hard-failwith default thresholds tuned to Marble's reported failure rates.
Gap relevant to composer: the composer needs a composition-coherence gate that scores style consistency between baked-mesh regions and environment regions in the final composed renders — distinct from input-grounding. Added in block F.
Job envelope and dispatch#
libs/isis/job-envelope/src/job-envelope.ts
Zod-schema'd typed envelope for GPU-worker jobs. IsisJobPriority
(low|normal|high|urgent); IsisGenerationType includes gaussian-splatting,
mesh-processing, image-to-3d, text-to-3d, plus inpainting, upscaling, etc.
Envelope is the canonical wire format consumed by the GPU worker pool through
Redis.
Gap relevant to composer: the heavy phases (Marble generation, mesh generation, splat bake) currently call providers in-process. For a production composer they should dispatch via job-envelope so the BFF process isn't blocked on GPU work. Wired in block G.
Database#
libs/isis/database/
Prisma schema (prisma/schema.prisma), migrations, generated client. Used by
manifest-storage.ts and other persistence layers.
Gap relevant to composer: no pipeline-state persistence. The orchestrator
keeps chains in an in-memory Map (pipeline-orchestrator.ts:1088) and has no
Prisma model for Pipeline, Chain, Phase, or Checkpoint. Block H adds the
schema and the resumePipeline API.
Other libs touched by the thorough plan#
| Lib | Role in composer | Block |
|---|---|---|
agent-consensus |
Multi-agent debate at checkpoints | I |
token-budget |
Cost enforcement per provider call | I |
model-governance-3d |
License/policy compliance on outputs | I |
anomaly-detection |
Drift watch on object distributions | I |
Each is briefly inventoried in its respective block plan; survey-level coverage in this doc would be redundant.
Cross-cutting observations#
-
No existing image-to-anything orchestrator in
libs/isis/. Grep forimage-to-scene | compose | composer | from-imagereturns only the new Marble provider (image-to-splat) and a few style-transfer adapters (ip-adapter-provider.ts,instantid-provider.ts) which are image-conditioned generation, not composition. The composer is genuinely new ground. -
Three V2 services exist with similar shape at
apps/v2/isis-commentary-lipsync-pipeline/,isis-cosmetic-pipeline/,isis-venue-capture-pipeline/. These would benefit from the operator-checkpoint primitive added in block A, but they're outside the composer's direct scope and tracked elsewhere. -
The
oshun_iris_native_bindingmemory notes that V1-P2-07xx tasks wire OS input/capture via@oshun/iris-computer-use-native. Unrelated to the composer, but the task style (atomic ops only; loadController + fixture-binding test pattern) is worth mirroring inISIS_GAPS_TODOS.mdfor consistency.