Eve library

01 — Current State Survey

Other

Revision 1

On this page

Inventory of existing Isis code that the scene-from-image composer depends on or extends. Each entry: what it does, what it exposes, what it doesn't (the gaps). All file:line refs verified at the time of writing.

3D generation providers#

libs/isis/3d-generation/src/providers/

Provider File Type Output
Marble marble-provider.ts (newly shipped) env splat .spz + previews
Hunyuan hunyuan-provider.ts object mesh .glb/.obj/.ply
Tripo tripo-provider.ts object mesh .glb
Meshy meshy-provider.ts object mesh .glb
Rodin rodin-provider.ts object mesh .glb
Trellis trellis-provider.ts object mesh .glb
ThreeDFY threedfy-provider.ts object mesh .glb

All extend Base3DProvider (base-provider.ts) which gives them: typed event emitter, polling helper (pollForCompletion), API-key auth, file-download helper, and a job-status registry. The Marble provider in particular reports world-space bounds in metres and emits a Generated3DModel with format: 'spz' and a marble: MarbleSceneExtras field carrying the preview-URL list, ambient-prompt suggestion, and timing metrics (marble-provider.ts:96-103).

Hunyuan parses GLB output via parseGLB and getDocumentStats from processing/gltf-utils.ts and produces a boundingBox: { width, height, depth } in the GLB's local frame — usually unit-normalised (hunyuan-provider.ts:370-402).

Gap relevant to composer: there's no shared "scene-aware" placement metadata — each provider returns a single mesh in its own local frame with no coordinate-system declaration.

Multi-provider routing#

libs/isis/ai-providers/src/orchestration/unified-generation-pipeline.ts

UnifiedGenerationPipeline extends EventEmitter (line 774). Public surface:

  • createUnifiedPipeline(config) (line 1906)
  • createUnifiedPipelineFromEnv() (line 1915)
  • createGenerationRequest(modality, input, options) (line 1941)
  • interface GenerationRequest (line 128) — modality, type, input, requirements, preferences, context

Capabilities: registers providers with capabilities/health/routing metadata; selects providers via weighted scoring across cost/quality/latency/reliability; fallback chains; generateBatch; generateParallel; in-flight request deduplication; circuit breakers.

Gap relevant to composer: the pipeline is single-modality per request (give me one image, give me one model, give me one audio). It is not a sequencer of multiple modalities — there's no built-in "first do an env generation, then for each detected object do a mesh generation, then do ambient SFX" composition. The composer either wraps UnifiedGenerationPipeline (calling it once per phase) or uses the providers directly. We choose the latter — see 03-decisions.md ADR-002.

Operation orchestration#

libs/isis/operation-orchestrator/src/

Five sub-orchestrators in their own subdirectories:

  • pipeline-orchestrator/ — sequencing of chains
  • execution-orchestrator/ — phases within a chain
  • batch-orchestrator/ — batch composition
  • failure-orchestrator/ — dead-letter queue, failure-pattern analysis
  • retry-orchestrator/ — exponential/linear/Fibonacci backoff

Type model (types.ts):

  • ChainStatus enum (line 118): PENDING | RUNNING | COMPLETED | FAILED | CANCELLED | BLOCKED
  • PhaseType enum (line 129): SETUP | EXECUTION | VALIDATION | TEARDOWN | CHECKPOINT
  • PipelineMode (line 139): SEQUENTIAL | PARALLEL | ADAPTIVE
  • PriorityLevel (line 147): CRITICAL | HIGH | NORMAL | LOW | BACKGROUND

Public factory: createPipelineOrchestrator(config?) at pipeline-orchestrator.ts:1037. Class: PipelineOrchestrator at pipeline-orchestrator.ts:618.

Critical gap: the operator-checkpoint primitive is half-implemented.

  • enableCheckpoints: false is the default config flag (execution-orchestrator.ts:41)
  • When enableCheckpoints is true, injectCheckpointPhases(...) synthetically inserts PhaseType.CHECKPOINT phases between regular phases at a configured frequency (execution-orchestrator.ts:1056-1057, function defined at 1362-1389)
  • BUT the injected CHECKPOINT phases have no special runtime semantics — they execute like any other phase. There's no approveCheckpoint() API, no AWAITING_APPROVAL chain status, no event indicating "operator decision required", no per-checkpoint timeout policy
  • grep -r approveCheckpoint awaitApproval AWAITING_APPROVAL libs/isis/operation-orchestrator/src/ returns zero hits

This is the single biggest dependency for the composer. Fixing it is block A.

Audio generation#

libs/isis/audio-generation/src/generation/sfx-provider.ts

Class SFXProvider (line 266) implements ISFXProvider. Public methods:

Method Line Purpose
generate(request: SFXGenerationRequest) 315 Generic prompt-driven SFX
generateFoley(request: FoleyRequest) 361 Surface-aware foley
generateAmbient(request: AmbientRequest) 369 Loopable env ambience
generateImpact(request: ImpactRequest) 382 Material-aware impacts

generateAmbient auto-sets seamlessLoop: true and duration: 30s. Returns GeneratedSFX with audioData: Buffer and optional downloadUrl. Provider factories cover ElevenLabs, AudioLDM, AudioCraft, Freesound, and a local runner.

Gap relevant to composer: none functionally; the composer can call these as-is. The composer adds the scene-context layer (passing an "ambient prompt" derived from the Marble scene's metadata to generateAmbient, then emitting per-object foley from the placed-meshes list).

Output management#

libs/isis/outputs/src/

File Purpose
types.ts OutputManifest interface at line 366
manifest/manifest-builder.ts Fluent builder for manifests
manifest/manifest-storage.ts Persistence for manifests
manifest/manifest-validator.ts Schema validation
lineage/lineage-graph.ts Cross-asset lineage edges
provenance/provenance-tracker.ts Per-asset provenance records

OutputManifest shape (relevant fields):

  • files: OutputFile[] — each with category ∈ image | video | audio | model-3d | texture | animation | document | data | archive
  • provenance: ProvenanceRecord[] — generator, model, inputs, parameters, duration, compute info
  • lineage?: LineageGraph — cross-asset edges including composited-from, derived-from, variant-of
  • metadata?: Record<string, unknown> — free-form extension point for composer-specific data (placement transforms, SH light fields, etc.)

Gap relevant to composer: none structurally — the manifest can hold the full composed output. Composer adds typed metadata under metadata.sceneFromImage.* (placements, SH coefficients, scene bounds, checkpoint history). See 04-api-surfaces.md for the typed extension shape.

libs/isis/output-gallery/src/

UI surface for browsing/filtering generated outputs. OutputRecord: outputId, tenantId, userId, domain, kind, model, assetUrl, thumbnailUrl, costCents, tags.

Gap relevant to composer: the composer needs to register every run as an OutputRecord so it shows up in the gallery. Wired in block I.

Scene assembly#

libs/isis/3d-scene-assembly/src/

24+ files; all currently purely declarative — recipes, scene graphs, validators, taxonomies. There is no orchestration code in this lib.

Most relevant for composer:

  • scene-unit-normalization.ts:62-84 — createThreeDImportedAssetNormalizationResult takes sourceUnit (mm/cm/m/inch/foot), sourceUpAxis, sourceForwardAxis, measuredScale; returns scaleMultiplierToMeters and rotationFixEulerDeg. Designed for importing external assets, not for bridging two co-generated assets in different frames.
  • placement-rules-engine.ts:18-83 — createThreeDPlacementRulesResult takes sceneBoundsMeters: { width, height, depth } and validates placements against bounds + clearance. Assumes all inputs in the same metres frame.
  • procedural-assembly-recipes.ts — static catalog of assembly recipes (mech, furniture, weapon) with stage descriptions but no execution logic.
  • semantic-scene-graph.ts — graph definition (nodes, connectors, constraints).
  • bom-material-summary.ts, kitbash-library-builder.ts, connector-taxonomy.ts, donor-part-suggestions.ts, style-harmonization.ts, symmetry-placement.ts — all declarative validators/transformers.

Gap relevant to composer: no splat-aware placement helper. The composer needs a new utility that takes (Marble bounds in metres, GLB local bbox) and emits a 4×4 transform. Lives in block B initially; refactored into gaussian-splatting/lighting/ once block F lands.

Gaussian splatting#

libs/isis/gaussian-splatting/src/

Subdirs: capture/, exporters/, four-dgs/, nerf/, optimization/, preprocessing/, rendering/, training/. Comprehensive on the training side; sparse on the export side.

exporters/:

  • isis-splat-exporter.ts — internal .isplat binary format for Unreal/Unity (8-byte magic 0x4c505349 "ISPL"). Real implementation, not Niantic-compatible.
  • index.ts — only exports IsisSplatExporter; no .spz, .ply, or .ksplat exporter despite those formats appearing in the ExportFormat union (types.ts:700-708).

rendering/splat-renderer.ts defines LoadedScene (positions, scales, rotations, opacities, colors, optional spherical harmonics) — the canonical in-memory splat representation.

Gaps relevant to composer:

  1. No .spz exporter — block D
  2. No mesh-to-Gaussians rasteriser — block E
  3. No splat-merging algorithm — block E
  4. No environment SH lighting estimator — block F

Quality gates#

libs/isis/3d-quality-gates/src/

Heavy structure: taxonomy/, corpus/, delivery/, operations/, manifest.ts, blueprint-types.ts. Recently added:

  • delivery/input-grounding-gate.ts (newly shipped) — input-image vs output-renders set comparison via pluggable VLM adapter; computes recall and novelty; returns pass | soft-fail | hard-fail with default thresholds tuned to Marble's reported failure rates.

Gap relevant to composer: the composer needs a composition-coherence gate that scores style consistency between baked-mesh regions and environment regions in the final composed renders — distinct from input-grounding. Added in block F.

Job envelope and dispatch#

libs/isis/job-envelope/src/job-envelope.ts

Zod-schema'd typed envelope for GPU-worker jobs. IsisJobPriority (low|normal|high|urgent); IsisGenerationType includes gaussian-splatting, mesh-processing, image-to-3d, text-to-3d, plus inpainting, upscaling, etc. Envelope is the canonical wire format consumed by the GPU worker pool through Redis.

Gap relevant to composer: the heavy phases (Marble generation, mesh generation, splat bake) currently call providers in-process. For a production composer they should dispatch via job-envelope so the BFF process isn't blocked on GPU work. Wired in block G.

Database#

libs/isis/database/

Prisma schema (prisma/schema.prisma), migrations, generated client. Used by manifest-storage.ts and other persistence layers.

Gap relevant to composer: no pipeline-state persistence. The orchestrator keeps chains in an in-memory Map (pipeline-orchestrator.ts:1088) and has no Prisma model for Pipeline, Chain, Phase, or Checkpoint. Block H adds the schema and the resumePipeline API.

Other libs touched by the thorough plan#

Lib Role in composer Block
agent-consensus Multi-agent debate at checkpoints I
token-budget Cost enforcement per provider call I
model-governance-3d License/policy compliance on outputs I
anomaly-detection Drift watch on object distributions I

Each is briefly inventoried in its respective block plan; survey-level coverage in this doc would be redundant.

Cross-cutting observations#

  1. No existing image-to-anything orchestrator in libs/isis/. Grep for image-to-scene | compose | composer | from-image returns only the new Marble provider (image-to-splat) and a few style-transfer adapters (ip-adapter-provider.ts, instantid-provider.ts) which are image-conditioned generation, not composition. The composer is genuinely new ground.

  2. Three V2 services exist with similar shape at apps/v2/isis-commentary-lipsync-pipeline/, isis-cosmetic-pipeline/, isis-venue-capture-pipeline/. These would benefit from the operator-checkpoint primitive added in block A, but they're outside the composer's direct scope and tracked elsewhere.

  3. The oshun_iris_native_binding memory notes that V1-P2-07xx tasks wire OS input/capture via @oshun/iris-computer-use-native. Unrelated to the composer, but the task style (atomic ops only; loadController + fixture-binding test pattern) is worth mirroring in ISIS_GAPS_TODOS.md for consistency.