# 07 — Risks and Open Questions

Known unknowns. Each entry: what the risk is, why it matters, how we'd mitigate,
who decides. Open questions are flagged where they need user input before a
block can start.

---

## Risks

### R1 · Niantic `.spz` spec ambiguities

**Block:** D **Severity:** medium **Probability:** medium

The public `.spz` spec at `https://github.com/nianticlabs/spz` covers the binary
layout but the exact quantisation constants for alpha (`SCALE = 8.0`), scale
(`(u8 - 64) / 16.0`), and rotation (`(u8 - 127.5) / 127.5`) are sourced from
public implementations rather than from the upstream source directly. The
reference loader's exact behavior for edge cases (degenerate quaternions,
position overflow, SH degree mismatch) is not documented in prose.

**Mitigation:**

1. Build a reference corpus of known-good `.spz` files (Niantic samples, public
   viewer demos). Round-trip every file through our codec; bit-exact diffs flag
   any spec deviation.
2. If round-trip fails, lift the quantisation math from a permissive-license
   reference implementation (e.g. `gsplat.js`) and re-test.
3. If any bit pattern remains ambiguous, document the ambiguity in the exporter
   source and flag in the PR.

**Decision needed:** none up front; the codec ships when round-trip is bit-
exact for ≥ 5 reference files.

---

### R2 · Differentiable splat fitting convergence

**Block:** E (hybrid path) **Severity:** high **Probability:** medium

The hybrid bake path runs gradient descent on Gaussian parameters against PBR
mesh renders. Convergence depends on initialization (surface sampling), loss
function (L2 + SSIM), learning rate, and step count. Real meshes (PBR characters
with translucency, hair, fine detail) may not converge in the default 200 steps;
quality may not exceed surface-sampling baseline by enough to justify the GPU
cost.

**Mitigation:**

1. Phase 2 ships the surface-sampling baseline as the default; hybrid is opt-in.
2. Quality regression tests assert hybrid > baseline on the mesh fixture corpus.
   If hybrid doesn't win on a fixture, the test fails.
3. If hybrid never beats baseline by a worthwhile margin, defer it indefinitely
   — surface-sampling alone is usable.

**Decision needed:** if phase 2 visual QA shows the bake quality is adequate at
surface-sampling, do we still ship hybrid? Default: yes (it's opt-in cost), but
worth a re-evaluation point.

---

### R3 · Lighting estimation accuracy

**Block:** F **Severity:** medium **Probability:** medium

The gaussian-weighted SH aggregation in `estimate-environment-light-field.ts` is
a coarse approximation — treats each Gaussian's outgoing radiance as if it were
incoming irradiance at the placement point. Visual QA may show shadows in wrong
places, missing colour bleed, or bake artifacts in heavily directional lighting.

**Mitigation:**

1. Phase 2 ships SH9 by default and adds env-map encoding for high- roughness or
   metallic surfaces.
2. If artifacts persist, implement a path-traced irradiance sampler via the Rust
   binding from `libs/isis/3d-inference-local`. Adds 2 weeks.
3. Composition coherence gate catches gross failures and routes to re-bake with
   different settings.

**Decision needed:** acceptable bake quality threshold. Recommended: coherence
score ≥ 0.85 against a benchmark corpus.

---

### R4 · Marble API surface drift

**Block:** B (depends on shipped Marble provider) **Severity:** low–medium
**Probability:** low (provider API is young; could change)

We don't control World Labs' API. They could rename `marble-1.1`, change the
`.spz` output format version, change cost model, deprecate `environment-objects`
mode.

**Mitigation:**

1. The Marble provider already isolates the API surface behind `MarbleProvider`;
   composer code uses only the typed `Generated3DModel` interface.
2. If the API breaks, the provider needs an update; composer is unaffected so
   long as the `Generated3DModel` shape is preserved.
3. Add a daily smoke test (real provider) that flags drift.

**Decision needed:** none structural.

---

### R5 · GPU worker pool capacity

**Block:** G **Severity:** medium **Probability:** depends on infra

If the GPU worker pool isn't sized for composer load, composer requests queue
and time out. Current Isis GPU pool capacity isn't documented in this repo.

**Mitigation:**

1. Block G's `JobDispatcher.dispatch` honours `responseTimeoutMs` (default 30
   min); composer surfaces timeout as `JobTimeoutError`.
2. Composer's checkpoint policy can accept "queue depth too high" as a reject
   reason — operator decides whether to retry later.
3. Phase 3 load testing surfaces capacity ceiling.

**Decision needed (ops-side):** GPU worker pool sizing for production composer
load. Out of scope for this folder.

---

### R6 · Postgres schema drift across libs

**Block:** H **Severity:** low **Probability:** low

Block H adds 6 new Prisma models. Other libs (`outputs`, `output-gallery`,
`isis/database`) have their own models. If two libs both add a model with the
same name (`Pipeline` is generic enough), migrations conflict.

**Mitigation:**

1. Prefix the new models if there's a collision: `OperationPipeline`,
   `OperationChain`, etc.
2. Migration name includes block letter (`add_block_h_pipeline_state`) so review
   catches conflicts.

**Decision needed:** confirm no existing `Pipeline` model in `@isis/database`.
(Spot-check at `01-current-state-survey.md` shows `prisma/schema.prisma` exists;
need to verify contents before block H.)

---

### R7 · Agent-consensus latency

**Block:** I **Severity:** low **Probability:** medium

Spawning 3+ LLM agents per checkpoint adds 30–120s per checkpoint to the
composer run. For unattended batch this is acceptable (the alternative is human
review at much higher latency); for interactive use it's a noticeable delay.

**Mitigation:**

1. Consensus is opt-in. Default checkpoint policy is `auto-approve`.
2. Agent calls run in parallel; total latency = max single-agent latency.
3. Consensus quorum can be set so the first 2/3 majority returns without waiting
   for the slowest agent.

**Decision needed:** none structural.

---

### R8 · Stub-scanner false positives in lighting math

**Block:** F **Severity:** low **Probability:** medium

Lighting code has comments like "approximate environment irradiance" and
variable names like `placeholder` for intermediate buffers. The stub-indicator
scan may fire on these even though they're not stubs.

**Mitigation:** per `oshun_stub_scanner_strictness` and
`feedback_stub_legitimate_honesty` memories, annotate legitimate hits with
`// stub:legitimate <reason>`. Honest stubs (genuinely incomplete math) must be
solved or split out — never annotated to bypass.

**Decision needed:** none structural.

---

### R9 · Composer test fixture image licensing

**Block:** B (corpus) onwards **Severity:** low **Probability:** medium

The 20-image corpus needs to ship in the repo. Real photographs may be
copyright-encumbered; using them in test fixtures is a license risk.

**Mitigation:**

1. Source corpus from explicitly-permissive sources: Unsplash (CC0), Wikimedia
   public domain, AI-generated (no copyright).
2. Document image source in `test-fixtures/corpus/SOURCES.md`.
3. If a permissive source can't be found for a failure mode, synthesise the
   image with an internal SD model.

**Decision needed:** confirm acceptable image-source policy with the user before
populating the corpus.

---

### R10 · Long-running composer state grows large

**Block:** H + I **Severity:** low **Probability:** medium

A composer run with 10 placed meshes + grounding + bake + coherence eval
generates ~50 MB of artifact data. Persisted to Postgres inline, this bloats the
DB. Persisted to MinIO/S3 as URIs, the DB stays small but the storage layer
needs lifecycle management.

**Mitigation:** block H's open-question already flags ≥ 1 MB → URI threshold.
Storage lifecycle (TTL, cold tier) is ops-side.

**Decision needed:** retention policy for composer-run artifacts.
Recommendation: 30 days hot, 1 year cold, then delete unless tagged "keep". Out
of scope for this plan; flagged for ops follow-up.

---

## Open questions for the user

Tracked here so each block knows what to ask before starting. Same questions
appear inline in the relevant block-plans.

### Q1 · Default checkpoint policy in production

Per ADR-008: v1 default is `auto-approve-after: 5min`. Production may want
`escalate` to a human approver via UI surface. Decision needed before Phase 3.

### Q2 · License policy registry source of truth

Per block I open-questions: hard-coded in `model-governance-3d` registry, or
per-tenant config layered over a global default? Recommended: per-tenant
overlay. Decision needed before block I.

### Q3 · Reference splat corpus sourcing

Per R1: are we OK redistributing Niantic-sample `.spz` files, or do we need to
synthesise our own with the encoder? Decision needed before block D ships
(corpus is the test gate).

### Q4 · Bake quality bar

Per R2 + R3: what's the minimum visual-quality threshold for "composed scene"?
Recommended: composition coherence score ≥ 0.85 on the benchmark corpus.
Decision needed before block F final sign-off.

### Q5 · GPU worker pool capacity

Per R5: ops decision; not blocking the planning but blocking production
deployment. Decision needed before phase 3 exit.

### Q6 · Image fixture license policy

Per R9: confirm permissive-source-only policy. Decision needed before the corpus
is committed.

### Q7 · Artifact retention TTL

Per R10: 30/365/delete recommendation OK? Decision needed before block H ships
the storage helper.

### Q8 · Consensus agent backend

Per block I open-questions: production agents back onto Claude Anthropic API by
default. Decision needed about credential sourcing, rate limits, fallback
behavior. Decision needed before block I phase 3 ship.

---

## Deferred decisions (re-visit after phase 1)

### D1 · Hybrid bake "always opt-in" or "auto-promote on coherence fail"?

If coherence gate fails at surface-sampling output, should the composer
auto-retry with hybrid? Adds cost but improves quality automatically.
Recommended: yes for production, no for test/dev. Re-visit once phase 2 visual
QA is in.

### D2 · Schema versioning for `metadata.sceneFromImage`

`schemaVersion: '1.0.0'` is the v1 contract. When fields are added (e.g. in
phase 3) do we bump to `1.1.0` or `2.0.0`? Recommended: `1.1.0` for additive,
`2.0.0` for breaking. Re-visit when first additive change lands.

### D3 · Composer as a service vs library

v1 ships as a library (`@isis/scene-from-image-composer`). Phase 3 may warrant a
wrapping service (HTTP endpoint, request queue, multi-tenant isolation).
Recommended: defer to a separate planning doc once production load patterns are
observed.

### D4 · Multi-image input

v1 takes a single source image. Some use cases want multiple input images
(stereo pairs, time-series). Out of scope for v1; defer.

---

## Research items

Not blocking; explore as time permits.

- **Niantic's published roadmap for `.spz`** — version 3 spec might add features
  (transparency channels, animated splats) we should accommodate.
- **`gsplat`'s differentiable rasteriser** — current Rust port, integration
  effort, performance vs `nerfstudio`.
- **VLM grounding adapters** — production-quality adapters for Claude vision,
  Gemini, local VLMs (Florence-2, Qwen-VL); benchmark on our corpus.
- **Splat compression beyond `.spz`** — newer formats (e.g. SOG, PCC) for
  size-sensitive deployments; not v1 but worth a read.
- **Mesh-to-Gaussians via neural networks** — there are recent papers on
  end-to-end mesh→Gaussian baking with learned models. Could replace block E's
  analytical pipeline at higher fidelity. Long-term research.
