# 03 — Decisions (ADR-style)

Architectural decisions for the scene-from-image composer. One section per
decision: Context, Options considered, Decision, Consequences, Status.

---

## ADR-001 · Composer library placement

**Status:** Accepted (user-confirmed in plan-mode iteration).

**Context.** The composer touches multiple existing libraries: 3D providers,
audio providers, grounding gate, scene-assembly placement validators, output
manifest. It needs a home that doesn't pollute libraries that have a
single-purpose identity.

**Options considered.**

1. **New library `@isis/scene-from-image-composer`** — dedicated lib.
2. Inside `libs/isis/3d-scene-assembly/` — adds an orchestrator to a
   currently-purely-declarative library.
3. Inside `libs/isis/3d-generation/src/workflow/` — forces `3d-generation` to
   depend on `audio-generation` and `3d-quality-gates` (currently it doesn't).

**Decision.** Option 1 — new lib `libs/isis/scene-from-image-composer/`.

**Consequences.**

- One more entry in `libs/isis/` (already 35+; cost is trivial).
- `3d-scene-assembly` stays purely declarative — its orchestration-free identity
  is preserved for future readers.
- Composer owns its README, types, public surface, tests. Easier to evolve
  independently.
- Path mapping in `tsconfig.base.json` needs an entry; `pnpm-workspace.yaml`
  picks it up automatically.

---

## ADR-002 · Orchestration substrate

**Status:** Accepted (recommended; user asked for the thorough version).

**Context.** The composer is a 5–7 phase linear-with-checkpoints pipeline. Two
existing substrates could host it: `UnifiedGenerationPipeline` (single-
modality, multi-provider router) or `PipelineOrchestrator` (multi-phase,
multi-chain). A custom thin state machine is also viable.

**Options considered.**

1. **Custom thin async state machine** — ~150 LoC, simple, easy to test, but
   reinvents what `operation-orchestrator` provides and forks the eventual
   operations dashboard.
2. **PipelineOrchestrator with a real checkpoint primitive** — uses the existing
   substrate; requires extending operation-orchestrator first to add
   `AWAITING_APPROVAL`, `approveCheckpoint`, etc. (block A).
3. **Hybrid**: thin state machine that emits orchestrator-shaped events for
   forward compatibility — half-measure that doesn't actually integrate.

**Decision.** Option 2 — extend `operation-orchestrator` (block A) and build the
composer on `PipelineOrchestrator`.

**Reasoning.** The user asked for the long-term option. The composer is one of N
future multi-phase Isis operations; building bespoke state machines for each is
the predictable two-years-in regret. The half-implemented checkpoint machinery
(`PhaseType.CHECKPOINT`, `injectCheckpointPhases`) is a clear invitation to
finish the primitive. The ~2 weeks to extend the orchestrator returns value
across composer + lipsync pipeline + cosmetic pipeline + venue capture
pipeline + every future multi-phase op.

**Consequences.**

- Block A is a hard gate for block B and downstream.
- The composer's state machine becomes a configuration of chains and phases, not
  a procedural function. Slightly heavier to define, much easier to observe and
  persist.
- Other multi-phase operations can adopt the same primitive immediately.
- The composer benefits from `retry-orchestrator`, `failure-orchestrator`, batch
  composition, dependency graphs — none of which we'd have built into the thin
  state machine.

---

## ADR-003 · Object source modes

**Status:** Accepted (user asked for the thorough version; all four modes ship).

**Context.** "Where do the foreground objects come from?" has at least three
distinct production use cases, each with different ergonomics.

**Options considered.**

1. **Explicit prompts only** — caller supplies an array of `{ prompt, anchor }`
   objects. Deterministic, scriptable, simple.
2. **Cropped image regions** — caller supplies bboxes; composer crops the source
   image and feeds each crop to the provider's image-to-3D path. Preserves
   visual identity ("that specific tree", not "a tree").
3. **VLM-detected salient objects** — composer auto-detects via the
   grounding-gate's VLM adapter. Best for batch unattended runs.
4. **Skip object generation** — environment-only mode. For panoramas with no
   foreground.
5. **Mixed** — caller supplies a heterogeneous list spanning the above modes.

**Decision.** All five modes ship. They are unified through a single
`ObjectSpec` discriminated union in
[block C](./block-plans/C-foreground-detection-and-provider-routing.md):

```ts
type ObjectSpec =
  | { source: 'explicit'; prompt: string; anchor: AnchorSpec }
  | {
      source: 'cropped';
      bbox: BBox;
      prompt?: string;
      anchor: AnchorSpec;
      removeBackground?: boolean;
    }
  | {
      source: 'vlm-detected';
      bbox: BBox;
      label: string;
      confidence: number;
      anchor: AnchorSpec;
    }
  | { source: 'mixed'; specs: ObjectSpec[] };
```

The "skip object generation" path is just `objects: []`.

**Reasoning.** The three primary modes cover non-overlapping use cases: explicit
for CI/golden tests, cropped for designer-driven workflows, VLM- detected for
batch. Investing in all three up front avoids retrofitting the phase contract
later.

**Consequences.**

- Block C is larger (~5w) than a single-mode version.
- The `ForegroundDetectionSubphase` becomes a first-class citizen in the
  composer, not a v1.1 retrofit.
- Mode selection happens at composer config; downstream phases stay
  mode-agnostic — they all consume `PlacedObject` after detection + provider
  call.

---

## ADR-004 · Splat ↔ mesh frame bridging strategy

**Status:** Accepted (user asked for the thorough version).

**Context.** Marble emits a Gaussian-splat scene with bounds in metres.
Foreground meshes are unit-normalised local-frame GLBs. Producing a "composed
scene" requires choosing how to bridge them.

**Options considered.**

1. **Don't bridge** — manifest lists splat and meshes as independent files.
   Consumer assembles at runtime. Forever an asterisk on "composed".
2. **Compute a transform per mesh, write to manifest** — each mesh gets a 4×4
   transform; consumer applies at load time. Lean, ~40 LoC.
3. **Bake meshes into the splat as Gaussians, merge into single `.spz`** — true
   single-file composed scene; works in any splat viewer. Requires `.spz`
   codec + mesh-to-Gaussians rasteriser + splat merging + lighting consistency.
   ~6–10 weeks.

**Decision.** Option 3 — bake meshes into the splat. Block E does the
rasterisation + merging; block F does the lighting consistency; block D does the
`.spz` codec.

**Reasoning.** The user asked for the long-term option. Option 2 produces
artefacts that every consumer (web viewer, Unreal plugin, mobile AR app, Blender
importer) has to specifically support. Option 3 produces artefacts that work in
any conformant splat viewer with no per-consumer code. Lighting mismatch
(Marble's baked radiance vs Hunyuan's PBR) is also fundamentally unfixable in
the consumer layer — it has to happen at bake time.

**Consequences.**

- Hardest block to ship; ~6–10 weeks for E + F + the dependency on D.
- The composer also keeps the source meshes + transforms + un-baked env splat in
  a `sources/` subdirectory of the manifest. Bake is destructive; sources let
  you re-bake later with different mesh placements without re-running Marble.
- The "transform-only" intermediate output (option 2) ships in Phase 1 as a
  stop-gap so block B is testable before block E lands. Phase 2 deprecates it in
  favour of the baked `.spz`.

---

## ADR-005 · Job dispatch

**Status:** Accepted.

**Context.** Heavy phases (Marble call, mesh generation, splat bake) are GPU
workloads. Calling them in-process from the BFF blocks the BFF event loop and
prevents horizontal scaling.

**Options considered.**

1. **In-process direct calls** — simplest; appropriate for a v1 demo.
2. **Dispatch via `@isis/job-envelope` to Redis-backed GPU workers** — the
   established Isis pattern for GPU work.
3. **Dispatch via Kubernetes Job objects** — full K8s native; over-engineered
   for the use case.

**Decision.** Option 2 — every heavy phase dispatches via `@isis/job-envelope`.
The composer waits on job IDs, not in-process Promises. Block G wires this.

**Consequences.**

- Composer can run on a small BFF instance; GPU workers scale independently.
- Phase-completion events come back through the job-envelope event bus, not
  directly from the provider's polling loop.
- Block H (state persistence) becomes essential — if the BFF restarts, the
  composer needs to resume against still-running GPU jobs.

---

## ADR-006 · Pipeline state persistence layer

**Status:** Accepted.

**Context.** `PipelineOrchestrator` is in-memory only. Composer runs are 5–30
minutes; deploys happen during that window. State must survive process restart
and worker restart.

**Options considered.**

1. **Postgres via `@isis/database`'s Prisma layer.**
2. Redis (already used for the GPU job queue).
3. Custom file-backed storage.
4. None — restart loses all in-flight pipelines.

**Decision.** Option 1 — Postgres via Prisma. New tables: `Pipeline`, `Chain`,
`Phase`, `PhaseArtifact`, `Checkpoint`, `CheckpointDecision`.

**Reasoning.** Postgres is already the system of record for manifests
(`manifest-storage.ts`) and gives us atomicity, indexes, and a stable backup
story. Redis is fine for ephemeral GPU job state but not for multi-day audit
trails. File-backed storage is a footgun in a multi-replica environment.

**Consequences.**

- Block H ships a Prisma migration + a `PipelineRepository` class + changes to
  `PipelineOrchestrator` to read/write state through it.
- The `resumePipeline(pipelineId, fromPhaseId, modifications)` API becomes the
  primary tool for handling checkpoint rejections.
- Backwards compat: pipelines started in-memory before block H lands are
  abandoned on restart. Acceptable since composer isn't yet in production.

---

## ADR-007 · License/budget/anomaly integration point

**Status:** Accepted.

**Context.** The composer pulls together outputs from multiple providers, each
with its own license. It also accumulates real cost across multiple provider
calls. Drift in object distributions over time signals model or input changes.

**Options considered.**

1. **Per-block ad hoc integration** — every block wires its own license check,
   budget check, anomaly sample. Inconsistent, duplicated.
2. **Composer-level cross-cutting integration** — the composer hooks into
   `model-governance-3d`, `token-budget`, `anomaly-detection` once, and the
   integration is shared across all blocks.

**Decision.** Option 2 — single integration point in the composer.
[Block I](./block-plans/I-observability-policy-and-consensus.md) owns the
wiring.

**Consequences.**

- The composer's `compose` call accepts (optionally) a `policyContext` carrying
  budget, license-policy ID, anomaly-detection scope.
- Each phase reports back cost + license-tags; composer aggregates and enforces.
- A single hard-fail rule: if any artefact's license is incompatible with the
  destination license, the whole pipeline halts at the next checkpoint.

---

## ADR-008 · Agent-consensus checkpoint policy

**Status:** Provisional — opt-in for v1, default for v2.

**Context.** Checkpoints (block A) can be approved by a human, by an auto-policy
timeout, or by automated agents (`@isis/agent-consensus`). Whether to default to
human approval, auto-approve, or agent-consensus is a product decision.

**Options considered.**

1. **Human-only.** Always wait for an explicit `approveCheckpoint` call.
2. **Auto-approve with timeout.** If no decision in N minutes, proceed.
3. **Agent-consensus.** Three LLM agents debate the intermediate artifact and
   vote.

**Decision.** v1 default = auto-approve with `auto-approve-after: 5min`
(per-checkpoint configurable). Agent-consensus available as opt-in via
`policy: 'consensus-3'` per checkpoint. Human-only configurable per checkpoint.
v2 may flip the default.

**Consequences.**

- v1 composer is usable unattended without writing operator-UI code.
- Agent-consensus integration is real but not on the critical path.
- Human-approval UI is its own product surface, deferred outside this plan.

---

## ADR-009 · Foreground detection adapter sourcing

**Status:** Accepted.

**Context.** Block C's VLM-detected mode needs an object-detection VLM. The
grounding gate already defines `ThreeDQualityGroundingVlmAdapter` with a similar
contract.

**Options considered.**

1. **Reuse `ThreeDQualityGroundingVlmAdapter`** — single adapter contract across
   QA gate and detection.
2. **New `SceneFromImageDetectionAdapter`** — distinct contract, possibly richer
   (segmentation masks, depth estimation).

**Decision.** Option 1 — reuse `ThreeDQualityGroundingVlmAdapter`. Detection
needs `label + confidence + boundingBox`, all of which the grounding adapter
already returns.

**Consequences.**

- Adapter authors implement one contract, satisfy two consumers.
- If detection ever needs richer outputs (masks, depth), we extend the shared
  contract rather than fork it.

---

## ADR-010 · Where placement transforms are stored in the manifest

**Status:** Accepted.

**Context.** `OutputManifest.metadata` is a free-form `Record<string, unknown>`
(`outputs/src/types.ts:366`). The composer needs a typed extension point.

**Decision.** Composer-specific data lives under `metadata.sceneFromImage.*`
with a typed schema declared in the composer library and re-exported for
downstream consumers. Specifically:

```ts
metadata: {
  sceneFromImage: {
    schemaVersion: '1.0.0';
    placements: Array<{ fileId: OutputFileId; transform: number[16]; anchor: AnchorSpec }>;
    sceneBoundsMeters: { min: [n,n,n]; max: [n,n,n] };
    bakeMetadata?: { method: 'surface-sampling' | 'differentiable' | 'hybrid'; gaussianCount: number };
    lightingMetadata?: { shCoefficients: number[]; estimationMethod: string };
    checkpointHistory: Array<{ phase: string; decision: 'approved' | 'rejected' | 'modified'; reviewer: string; reviewedAt: string }>;
    policyContext?: { licensePolicyId: string; budgetCents: number; spentCents: number };
  }
}
```

**Consequences.**

- Manifest validator (`manifest-validator.ts`) gets an optional rule set for the
  composer extension; non-composer manifests are unaffected.
- Downstream consumers (Three.js viewer, Unreal plugin, gallery UI) read this
  typed shape via the composer's exported types.

---

## ADR-011 · Composer public API shape

**Status:** Accepted.

**Context.** Composer entry point should be (a) callable from a single function,
(b) compatible with operation-orchestrator's Pipeline-based model, (c)
extensible to streaming / long-running scenarios.

**Decision.** Two-tier API.

1. **Convenience:**

   ```ts
   async function composeSceneFromImage(
     input: SceneFromImageRequest
   ): Promise<SceneFromImageResult>;
   ```

   Synchronous from the caller's perspective; awaits the pipeline.

2. **Advanced:**
   ```ts
   function createSceneFromImagePipeline(
     input: SceneFromImageRequest
   ): PipelineDefinition;
   ```
   Returns a pipeline definition that the caller can hand to a configured
   `PipelineOrchestrator`, listen to events on, persist via block H, etc.

The convenience API is implemented as a thin wrapper around the advanced API
plus an in-process orchestrator with auto-approving checkpoints.

**Consequences.**

- Test code uses the convenience API.
- Production code uses the advanced API to get persistence, observability,
  resume, agent-consensus checkpoints.
- Both surfaces are stable and exported from the lib's public index.
