# 01 — Current State Survey

Inventory of existing Isis code that the scene-from-image composer depends on or
extends. Each entry: what it does, what it exposes, what it doesn't (the gaps).
All file:line refs verified at the time of writing.

## 3D generation providers

`libs/isis/3d-generation/src/providers/`

| Provider | File                                 | Type        | Output               |
| -------- | ------------------------------------ | ----------- | -------------------- |
| Marble   | `marble-provider.ts` (newly shipped) | env splat   | `.spz` + previews    |
| Hunyuan  | `hunyuan-provider.ts`                | object mesh | `.glb`/`.obj`/`.ply` |
| Tripo    | `tripo-provider.ts`                  | object mesh | `.glb`               |
| Meshy    | `meshy-provider.ts`                  | object mesh | `.glb`               |
| Rodin    | `rodin-provider.ts`                  | object mesh | `.glb`               |
| Trellis  | `trellis-provider.ts`                | object mesh | `.glb`               |
| ThreeDFY | `threedfy-provider.ts`               | object mesh | `.glb`               |

All extend `Base3DProvider` (`base-provider.ts`) which gives them: typed event
emitter, polling helper (`pollForCompletion`), API-key auth, file-download
helper, and a job-status registry. The Marble provider in particular reports
world-space bounds in metres and emits a `Generated3DModel` with `format: 'spz'`
and a `marble: MarbleSceneExtras` field carrying the preview-URL list,
ambient-prompt suggestion, and timing metrics (`marble-provider.ts:96-103`).

Hunyuan parses GLB output via `parseGLB` and `getDocumentStats` from
`processing/gltf-utils.ts` and produces a
`boundingBox: { width, height, depth }` in the GLB's local frame — usually
unit-normalised (`hunyuan-provider.ts:370-402`).

**Gap relevant to composer:** there's no shared "scene-aware" placement metadata
— each provider returns a single mesh in its own local frame with no
coordinate-system declaration.

## Multi-provider routing

`libs/isis/ai-providers/src/orchestration/unified-generation-pipeline.ts`

`UnifiedGenerationPipeline extends EventEmitter` (line 774). Public surface:

- `createUnifiedPipeline(config)` (line 1906)
- `createUnifiedPipelineFromEnv()` (line 1915)
- `createGenerationRequest(modality, input, options)` (line 1941)
- `interface GenerationRequest` (line 128) — modality, type, input,
  requirements, preferences, context

Capabilities: registers providers with capabilities/health/routing metadata;
selects providers via weighted scoring across cost/quality/latency/reliability;
fallback chains; `generateBatch`; `generateParallel`; in-flight request
deduplication; circuit breakers.

**Gap relevant to composer:** the pipeline is single-modality per request (give
me one image, give me one model, give me one audio). It is not a sequencer of
multiple modalities — there's no built-in "first do an env generation, then for
each detected object do a mesh generation, then do ambient SFX" composition. The
composer either wraps `UnifiedGenerationPipeline` (calling it once per phase) or
uses the providers directly. We choose the latter — see `03-decisions.md`
ADR-002.

## Operation orchestration

`libs/isis/operation-orchestrator/src/`

Five sub-orchestrators in their own subdirectories:

- `pipeline-orchestrator/` — sequencing of chains
- `execution-orchestrator/` — phases within a chain
- `batch-orchestrator/` — batch composition
- `failure-orchestrator/` — dead-letter queue, failure-pattern analysis
- `retry-orchestrator/` — exponential/linear/Fibonacci backoff

Type model (`types.ts`):

- `ChainStatus` enum (line 118):
  `PENDING | RUNNING | COMPLETED | FAILED | CANCELLED | BLOCKED`
- `PhaseType` enum (line 129):
  `SETUP | EXECUTION | VALIDATION | TEARDOWN | CHECKPOINT`
- `PipelineMode` (line 139): `SEQUENTIAL | PARALLEL | ADAPTIVE`
- `PriorityLevel` (line 147): `CRITICAL | HIGH | NORMAL | LOW | BACKGROUND`

Public factory: `createPipelineOrchestrator(config?)` at
`pipeline-orchestrator.ts:1037`. Class: `PipelineOrchestrator` at
`pipeline-orchestrator.ts:618`.

**Critical gap:** the operator-checkpoint primitive is **half-implemented**.

- `enableCheckpoints: false` is the default config flag
  (`execution-orchestrator.ts:41`)
- When `enableCheckpoints` is true, `injectCheckpointPhases(...)` synthetically
  inserts `PhaseType.CHECKPOINT` phases between regular phases at a configured
  frequency (`execution-orchestrator.ts:1056-1057`, function defined at
  `1362-1389`)
- BUT the injected CHECKPOINT phases have no special runtime semantics — they
  execute like any other phase. There's no `approveCheckpoint()` API, no
  `AWAITING_APPROVAL` chain status, no event indicating "operator decision
  required", no per-checkpoint timeout policy
- `grep -r approveCheckpoint awaitApproval AWAITING_APPROVAL libs/isis/operation-orchestrator/src/`
  returns zero hits

This is the single biggest dependency for the composer. Fixing it is
[block A](./block-plans/A-orchestrator-checkpoint-primitive.md).

## Audio generation

`libs/isis/audio-generation/src/generation/sfx-provider.ts`

Class `SFXProvider` (line 266) implements `ISFXProvider`. Public methods:

| Method                                     | Line | Purpose                   |
| ------------------------------------------ | ---- | ------------------------- |
| `generate(request: SFXGenerationRequest)`  | 315  | Generic prompt-driven SFX |
| `generateFoley(request: FoleyRequest)`     | 361  | Surface-aware foley       |
| `generateAmbient(request: AmbientRequest)` | 369  | Loopable env ambience     |
| `generateImpact(request: ImpactRequest)`   | 382  | Material-aware impacts    |

`generateAmbient` auto-sets `seamlessLoop: true` and `duration: 30s`. Returns
`GeneratedSFX` with `audioData: Buffer` and optional `downloadUrl`. Provider
factories cover ElevenLabs, AudioLDM, AudioCraft, Freesound, and a local runner.

**Gap relevant to composer:** none functionally; the composer can call these
as-is. The composer adds the **scene-context layer** (passing an "ambient
prompt" derived from the Marble scene's metadata to `generateAmbient`, then
emitting per-object foley from the placed-meshes list).

## Output management

`libs/isis/outputs/src/`

| File                               | Purpose                                |
| ---------------------------------- | -------------------------------------- |
| `types.ts`                         | `OutputManifest` interface at line 366 |
| `manifest/manifest-builder.ts`     | Fluent builder for manifests           |
| `manifest/manifest-storage.ts`     | Persistence for manifests              |
| `manifest/manifest-validator.ts`   | Schema validation                      |
| `lineage/lineage-graph.ts`         | Cross-asset lineage edges              |
| `provenance/provenance-tracker.ts` | Per-asset provenance records           |

`OutputManifest` shape (relevant fields):

- `files: OutputFile[]` — each with `category` ∈
  `image | video | audio | model-3d | texture | animation | document | data | archive`
- `provenance: ProvenanceRecord[]` — generator, model, inputs, parameters,
  duration, compute info
- `lineage?: LineageGraph` — cross-asset edges including `composited-from`,
  `derived-from`, `variant-of`
- `metadata?: Record<string, unknown>` — free-form extension point for
  composer-specific data (placement transforms, SH light fields, etc.)

**Gap relevant to composer:** none structurally — the manifest can hold the full
composed output. Composer adds typed metadata under `metadata.sceneFromImage.*`
(placements, SH coefficients, scene bounds, checkpoint history). See
`04-api-surfaces.md` for the typed extension shape.

## Output gallery

`libs/isis/output-gallery/src/`

UI surface for browsing/filtering generated outputs. `OutputRecord`:
`outputId, tenantId, userId, domain, kind, model, assetUrl, thumbnailUrl, costCents, tags`.

**Gap relevant to composer:** the composer needs to register every run as an
`OutputRecord` so it shows up in the gallery. Wired in
[block I](./block-plans/I-observability-policy-and-consensus.md).

## Scene assembly

`libs/isis/3d-scene-assembly/src/`

24+ files; all currently **purely declarative** — recipes, scene graphs,
validators, taxonomies. There is no orchestration code in this lib.

Most relevant for composer:

- `scene-unit-normalization.ts:62-84` —
  `createThreeDImportedAssetNormalizationResult` takes `sourceUnit`
  (mm/cm/m/inch/foot), `sourceUpAxis`, `sourceForwardAxis`, `measuredScale`;
  returns `scaleMultiplierToMeters` and `rotationFixEulerDeg`. Designed for
  **importing external assets**, not for bridging two co-generated assets in
  different frames.
- `placement-rules-engine.ts:18-83` — `createThreeDPlacementRulesResult` takes
  `sceneBoundsMeters: { width, height, depth }` and validates placements against
  bounds + clearance. Assumes all inputs in the same metres frame.
- `procedural-assembly-recipes.ts` — static catalog of assembly recipes (mech,
  furniture, weapon) with stage descriptions but no execution logic.
- `semantic-scene-graph.ts` — graph definition (nodes, connectors, constraints).
- `bom-material-summary.ts`, `kitbash-library-builder.ts`,
  `connector-taxonomy.ts`, `donor-part-suggestions.ts`,
  `style-harmonization.ts`, `symmetry-placement.ts` — all declarative
  validators/transformers.

**Gap relevant to composer:** no splat-aware placement helper. The composer
needs a new utility that takes (Marble bounds in metres, GLB local bbox) and
emits a 4×4 transform. Lives in
[block B](./block-plans/B-scene-from-image-composer-library.md) initially;
refactored into `gaussian-splatting/lighting/` once block F lands.

## Gaussian splatting

`libs/isis/gaussian-splatting/src/`

Subdirs: `capture/`, `exporters/`, `four-dgs/`, `nerf/`, `optimization/`,
`preprocessing/`, `rendering/`, `training/`. Comprehensive on the training side;
sparse on the export side.

`exporters/`:

- `isis-splat-exporter.ts` — internal `.isplat` binary format for Unreal/Unity
  (8-byte magic `0x4c505349` "ISPL"). Real implementation, not
  Niantic-compatible.
- `index.ts` — only exports `IsisSplatExporter`; **no `.spz`, `.ply`, or
  `.ksplat` exporter** despite those formats appearing in the `ExportFormat`
  union (`types.ts:700-708`).

`rendering/splat-renderer.ts` defines `LoadedScene` (positions, scales,
rotations, opacities, colors, optional spherical harmonics) — the canonical
in-memory splat representation.

**Gaps relevant to composer:**

1. No `.spz` exporter — [block D](./block-plans/D-spz-exporter.md)
2. No mesh-to-Gaussians rasteriser —
   [block E](./block-plans/E-mesh-to-gaussians-bake-pipeline.md)
3. No splat-merging algorithm —
   [block E](./block-plans/E-mesh-to-gaussians-bake-pipeline.md)
4. No environment SH lighting estimator —
   [block F](./block-plans/F-lighting-consistency.md)

## Quality gates

`libs/isis/3d-quality-gates/src/`

Heavy structure: `taxonomy/`, `corpus/`, `delivery/`, `operations/`,
`manifest.ts`, `blueprint-types.ts`. Recently added:

- `delivery/input-grounding-gate.ts` (newly shipped) — input-image vs
  output-renders set comparison via pluggable VLM adapter; computes recall and
  novelty; returns `pass | soft-fail | hard-fail` with default thresholds tuned
  to Marble's reported failure rates.

**Gap relevant to composer:** the composer needs a **composition-coherence
gate** that scores style consistency between baked-mesh regions and environment
regions in the final composed renders — distinct from input-grounding. Added in
[block F](./block-plans/F-lighting-consistency.md).

## Job envelope and dispatch

`libs/isis/job-envelope/src/job-envelope.ts`

Zod-schema'd typed envelope for GPU-worker jobs. `IsisJobPriority`
(`low|normal|high|urgent`); `IsisGenerationType` includes `gaussian-splatting`,
`mesh-processing`, `image-to-3d`, `text-to-3d`, plus inpainting, upscaling, etc.
Envelope is the canonical wire format consumed by the GPU worker pool through
Redis.

**Gap relevant to composer:** the heavy phases (Marble generation, mesh
generation, splat bake) currently call providers in-process. For a production
composer they should dispatch via job-envelope so the BFF process isn't blocked
on GPU work. Wired in
[block G](./block-plans/G-job-envelope-and-gpu-dispatch.md).

## Database

`libs/isis/database/`

Prisma schema (`prisma/schema.prisma`), migrations, generated client. Used by
`manifest-storage.ts` and other persistence layers.

**Gap relevant to composer:** no pipeline-state persistence. The orchestrator
keeps chains in an in-memory `Map` (`pipeline-orchestrator.ts:1088`) and has no
Prisma model for `Pipeline`, `Chain`, `Phase`, or `Checkpoint`. Block H adds the
schema and the `resumePipeline` API.

## Other libs touched by the thorough plan

| Lib                   | Role in composer                     | Block |
| --------------------- | ------------------------------------ | ----- |
| `agent-consensus`     | Multi-agent debate at checkpoints    | I     |
| `token-budget`        | Cost enforcement per provider call   | I     |
| `model-governance-3d` | License/policy compliance on outputs | I     |
| `anomaly-detection`   | Drift watch on object distributions  | I     |

Each is briefly inventoried in its respective block plan; survey-level coverage
in this doc would be redundant.

## Cross-cutting observations

1. **No existing image-to-anything orchestrator** in `libs/isis/`. Grep for
   `image-to-scene | compose | composer | from-image` returns only the new
   Marble provider (image-to-splat) and a few style-transfer adapters
   (`ip-adapter-provider.ts`, `instantid-provider.ts`) which are
   image-conditioned generation, not composition. The composer is genuinely new
   ground.

2. **Three V2 services exist with similar shape** at
   `apps/v2/isis-commentary-lipsync-pipeline/`, `isis-cosmetic-pipeline/`,
   `isis-venue-capture-pipeline/`. These would benefit from the
   operator-checkpoint primitive added in block A, but they're outside the
   composer's direct scope and tracked elsewhere.

3. **The `oshun_iris_native_binding` memory** notes that V1-P2-07xx tasks wire
   OS input/capture via `@oshun/iris-computer-use-native`. Unrelated to the
   composer, but the **task style** (atomic ops only; loadController +
   fixture-binding test pattern) is worth mirroring in `ISIS_GAPS_TODOS.md` for
   consistency.
