Eve library

07 — Risks and Open Questions

Other

Revision 1

On this page

Known unknowns. Each entry: what the risk is, why it matters, how we'd mitigate, who decides. Open questions are flagged where they need user input before a block can start.


Risks#

R1 · Niantic .spz spec ambiguities#

Block: D Severity: medium Probability: medium

The public .spz spec at https://github.com/nianticlabs/spz covers the binary layout but the exact quantisation constants for alpha (SCALE = 8.0), scale ((u8 - 64) / 16.0), and rotation ((u8 - 127.5) / 127.5) are sourced from public implementations rather than from the upstream source directly. The reference loader's exact behavior for edge cases (degenerate quaternions, position overflow, SH degree mismatch) is not documented in prose.

Mitigation:

  1. Build a reference corpus of known-good .spz files (Niantic samples, public viewer demos). Round-trip every file through our codec; bit-exact diffs flag any spec deviation.
  2. If round-trip fails, lift the quantisation math from a permissive-license reference implementation (e.g. gsplat.js) and re-test.
  3. If any bit pattern remains ambiguous, document the ambiguity in the exporter source and flag in the PR.

Decision needed: none up front; the codec ships when round-trip is bit- exact for ≥ 5 reference files.


R2 · Differentiable splat fitting convergence#

Block: E (hybrid path) Severity: high Probability: medium

The hybrid bake path runs gradient descent on Gaussian parameters against PBR mesh renders. Convergence depends on initialization (surface sampling), loss function (L2 + SSIM), learning rate, and step count. Real meshes (PBR characters with translucency, hair, fine detail) may not converge in the default 200 steps; quality may not exceed surface-sampling baseline by enough to justify the GPU cost.

Mitigation:

  1. Phase 2 ships the surface-sampling baseline as the default; hybrid is opt-in.
  2. Quality regression tests assert hybrid > baseline on the mesh fixture corpus. If hybrid doesn't win on a fixture, the test fails.
  3. If hybrid never beats baseline by a worthwhile margin, defer it indefinitely — surface-sampling alone is usable.

Decision needed: if phase 2 visual QA shows the bake quality is adequate at surface-sampling, do we still ship hybrid? Default: yes (it's opt-in cost), but worth a re-evaluation point.


R3 · Lighting estimation accuracy#

Block: F Severity: medium Probability: medium

The gaussian-weighted SH aggregation in estimate-environment-light-field.ts is a coarse approximation — treats each Gaussian's outgoing radiance as if it were incoming irradiance at the placement point. Visual QA may show shadows in wrong places, missing colour bleed, or bake artifacts in heavily directional lighting.

Mitigation:

  1. Phase 2 ships SH9 by default and adds env-map encoding for high- roughness or metallic surfaces.
  2. If artifacts persist, implement a path-traced irradiance sampler via the Rust binding from libs/isis/3d-inference-local. Adds 2 weeks.
  3. Composition coherence gate catches gross failures and routes to re-bake with different settings.

Decision needed: acceptable bake quality threshold. Recommended: coherence score ≥ 0.85 against a benchmark corpus.


R4 · Marble API surface drift#

Block: B (depends on shipped Marble provider) Severity: low–medium Probability: low (provider API is young; could change)

We don't control World Labs' API. They could rename marble-1.1, change the .spz output format version, change cost model, deprecate environment-objects mode.

Mitigation:

  1. The Marble provider already isolates the API surface behind MarbleProvider; composer code uses only the typed Generated3DModel interface.
  2. If the API breaks, the provider needs an update; composer is unaffected so long as the Generated3DModel shape is preserved.
  3. Add a daily smoke test (real provider) that flags drift.

Decision needed: none structural.


R5 · GPU worker pool capacity#

Block: G Severity: medium Probability: depends on infra

If the GPU worker pool isn't sized for composer load, composer requests queue and time out. Current Isis GPU pool capacity isn't documented in this repo.

Mitigation:

  1. Block G's JobDispatcher.dispatch honours responseTimeoutMs (default 30 min); composer surfaces timeout as JobTimeoutError.
  2. Composer's checkpoint policy can accept "queue depth too high" as a reject reason — operator decides whether to retry later.
  3. Phase 3 load testing surfaces capacity ceiling.

Decision needed (ops-side): GPU worker pool sizing for production composer load. Out of scope for this folder.


R6 · Postgres schema drift across libs#

Block: H Severity: low Probability: low

Block H adds 6 new Prisma models. Other libs (outputs, output-gallery, isis/database) have their own models. If two libs both add a model with the same name (Pipeline is generic enough), migrations conflict.

Mitigation:

  1. Prefix the new models if there's a collision: OperationPipeline, OperationChain, etc.
  2. Migration name includes block letter (add_block_h_pipeline_state) so review catches conflicts.

Decision needed: confirm no existing Pipeline model in @isis/database. (Spot-check at 01-current-state-survey.md shows prisma/schema.prisma exists; need to verify contents before block H.)


R7 · Agent-consensus latency#

Block: I Severity: low Probability: medium

Spawning 3+ LLM agents per checkpoint adds 30–120s per checkpoint to the composer run. For unattended batch this is acceptable (the alternative is human review at much higher latency); for interactive use it's a noticeable delay.

Mitigation:

  1. Consensus is opt-in. Default checkpoint policy is auto-approve.
  2. Agent calls run in parallel; total latency = max single-agent latency.
  3. Consensus quorum can be set so the first 2/3 majority returns without waiting for the slowest agent.

Decision needed: none structural.


R8 · Stub-scanner false positives in lighting math#

Block: F Severity: low Probability: medium

Lighting code has comments like "approximate environment irradiance" and variable names like placeholder for intermediate buffers. The stub-indicator scan may fire on these even though they're not stubs.

Mitigation: per oshun_stub_scanner_strictness and feedback_stub_legitimate_honesty memories, annotate legitimate hits with // stub:legitimate <reason>. Honest stubs (genuinely incomplete math) must be solved or split out — never annotated to bypass.

Decision needed: none structural.


R9 · Composer test fixture image licensing#

Block: B (corpus) onwards Severity: low Probability: medium

The 20-image corpus needs to ship in the repo. Real photographs may be copyright-encumbered; using them in test fixtures is a license risk.

Mitigation:

  1. Source corpus from explicitly-permissive sources: Unsplash (CC0), Wikimedia public domain, AI-generated (no copyright).
  2. Document image source in test-fixtures/corpus/SOURCES.md.
  3. If a permissive source can't be found for a failure mode, synthesise the image with an internal SD model.

Decision needed: confirm acceptable image-source policy with the user before populating the corpus.


R10 · Long-running composer state grows large#

Block: H + I Severity: low Probability: medium

A composer run with 10 placed meshes + grounding + bake + coherence eval generates ~50 MB of artifact data. Persisted to Postgres inline, this bloats the DB. Persisted to MinIO/S3 as URIs, the DB stays small but the storage layer needs lifecycle management.

Mitigation: block H's open-question already flags ≥ 1 MB → URI threshold. Storage lifecycle (TTL, cold tier) is ops-side.

Decision needed: retention policy for composer-run artifacts. Recommendation: 30 days hot, 1 year cold, then delete unless tagged "keep". Out of scope for this plan; flagged for ops follow-up.


Open questions for the user#

Tracked here so each block knows what to ask before starting. Same questions appear inline in the relevant block-plans.

Q1 · Default checkpoint policy in production#

Per ADR-008: v1 default is auto-approve-after: 5min. Production may want escalate to a human approver via UI surface. Decision needed before Phase 3.

Q2 · License policy registry source of truth#

Per block I open-questions: hard-coded in model-governance-3d registry, or per-tenant config layered over a global default? Recommended: per-tenant overlay. Decision needed before block I.

Q3 · Reference splat corpus sourcing#

Per R1: are we OK redistributing Niantic-sample .spz files, or do we need to synthesise our own with the encoder? Decision needed before block D ships (corpus is the test gate).

Q4 · Bake quality bar#

Per R2 + R3: what's the minimum visual-quality threshold for "composed scene"? Recommended: composition coherence score ≥ 0.85 on the benchmark corpus. Decision needed before block F final sign-off.

Q5 · GPU worker pool capacity#

Per R5: ops decision; not blocking the planning but blocking production deployment. Decision needed before phase 3 exit.

Q6 · Image fixture license policy#

Per R9: confirm permissive-source-only policy. Decision needed before the corpus is committed.

Q7 · Artifact retention TTL#

Per R10: 30/365/delete recommendation OK? Decision needed before block H ships the storage helper.

Q8 · Consensus agent backend#

Per block I open-questions: production agents back onto Claude Anthropic API by default. Decision needed about credential sourcing, rate limits, fallback behavior. Decision needed before block I phase 3 ship.


Deferred decisions (re-visit after phase 1)#

D1 · Hybrid bake "always opt-in" or "auto-promote on coherence fail"?#

If coherence gate fails at surface-sampling output, should the composer auto-retry with hybrid? Adds cost but improves quality automatically. Recommended: yes for production, no for test/dev. Re-visit once phase 2 visual QA is in.

D2 · Schema versioning for metadata.sceneFromImage#

schemaVersion: '1.0.0' is the v1 contract. When fields are added (e.g. in phase 3) do we bump to 1.1.0 or 2.0.0? Recommended: 1.1.0 for additive, 2.0.0 for breaking. Re-visit when first additive change lands.

D3 · Composer as a service vs library#

v1 ships as a library (@isis/scene-from-image-composer). Phase 3 may warrant a wrapping service (HTTP endpoint, request queue, multi-tenant isolation). Recommended: defer to a separate planning doc once production load patterns are observed.

D4 · Multi-image input#

v1 takes a single source image. Some use cases want multiple input images (stereo pairs, time-series). Out of scope for v1; defer.


Research items#

Not blocking; explore as time permits.

  • Niantic's published roadmap for .spz — version 3 spec might add features (transparency channels, animated splats) we should accommodate.
  • gsplat's differentiable rasteriser — current Rust port, integration effort, performance vs nerfstudio.
  • VLM grounding adapters — production-quality adapters for Claude vision, Gemini, local VLMs (Florence-2, Qwen-VL); benchmark on our corpus.
  • Splat compression beyond .spz — newer formats (e.g. SOG, PCC) for size-sensitive deployments; not v1 but worth a read.
  • Mesh-to-Gaussians via neural networks — there are recent papers on end-to-end mesh→Gaussian baking with learned models. Could replace block E's analytical pipeline at higher fidelity. Long-term research.