Eve library

Manual QA runbook — ISIS_GAPS deferred items

Runbook

Revision 1

On this page

Each section below corresponds to one or more ISIS_GAPS_TODOS.md manual-QA tasks. The tasks needed external infrastructure (browser viewers, real Postgres deployments, GPU pools, etc.) that doesn't live in this repo. This runbook is the recipe a deployment operator follows to satisfy each item.

For tasks where the in-repo test suite ALREADY exercises the same code path with fixtures (the SIGKILL-resume golden assertion, the dispatch-mode parity tests, the consensus-handler decision logic), the deployment operator can SKIP the manual run when the underlying test passes — the box only needs ticking once.


D-063 — Encode synthetic .spz, load in mkkellogg viewer#

Goal: confirm SpzExporter produces wire-compatible output.

bash
# In a node REPL or one-off script:
npx tsx <<'EOF'
import { SpzExporter } from '@isis/gaussian-splatting';
import { writeFileSync } from 'node:fs';

const N = 1000;
const scene = {
  gaussianCount: N,
  positions: new Float32Array(N * 3).map(() => Math.random() * 2 - 1),
  scales: new Float32Array(N * 3).fill(0.05),
  rotations: (() => {
    const r = new Float32Array(N * 4);
    for (let i = 0; i < N; i++) r[i * 4 + 3] = 1;
    return r;
  })(),
  opacities: new Float32Array(N).fill(0.9),
  colors: new Float32Array(N * 3).map(() => Math.random()),
  shDegree: 0,
  boundingBox: { min: { x: -1, y: -1, z: -1 }, max: { x: 1, y: 1, z: 1 } },
};
const bytes = await new SpzExporter().encode(scene);
writeFileSync('/tmp/synthetic-1k.spz', bytes);
console.log('wrote /tmp/synthetic-1k.spz', bytes.length, 'bytes');
EOF

Then drag /tmp/synthetic-1k.spz into https://mkkellogg.github.io/GaussianSplats3D/demo/ (or any SuperSplat-compatible viewer). PASS = scene renders without errors + the 1k points are visible.

D-064 — Decode a Marble-produced .spz#

Goal: confirm SpzDecoder reads upstream Marble output.

  1. Run a real Marble call (any composer run with dispatchMode: in-process + a real apiKey) and save the env splat:
    ts
    const env = await composeSceneFromImage(request);
    writeFileSync(
      '/tmp/marble.spz',
      env.phaseOutputs.environment.splatScene.modelData
    );
    
  2. Decode + inspect:
    bash
    npx tsx <<'EOF'
    import { SpzDecoder } from '@isis/gaussian-splatting';
    import { readFileSync } from 'node:fs';
    const decoded = await new SpzDecoder().decode(readFileSync('/tmp/marble.spz'));
    console.log('gaussian count:', decoded.scene.gaussianCount);
    console.log('bounding box:', decoded.scene.boundingBox);
    EOF
    

PASS = decoder produces a LoadedScene with non-zero gaussianCount and bounding box matching Marble's reported bounds.

E-076 — Composer end-to-end with 2 placed meshes#

Goal: confirm baked composer output looks right with a real mesh provider.

bash
# Set MARBLE_API_KEY + HUNYUAN_API_KEY in env first.
npx tsx scripts/composer-e2e-2-meshes.ts

(The script should call composeSceneFromImage with an image of a cypress forest + objects: [{cypress tree, stone lantern}] + outputMode: 'baked', then write scene-composed.spz to disk.) Load the result in the mkkellogg viewer. PASS = both meshes are visible in their expected world-space positions.

E-077 — Hybrid path with 100 steps shows visibly sharper edges#

Goal: A/B compare surface-sampling vs hybrid bake quality.

The CPU hybrid path landed in the absolute-completion sweep (commit 9291ad7ac4 — bakeMesh now dispatches method:'hybrid' to a real refiner). Run a real composer end-to-end with each method:

bash
# Surface-sampling baseline:
npx tsx scripts/composer-bake.ts --method=surface-sampling --out=/tmp/scene-sampling.spz
# Hybrid:
npx tsx scripts/composer-bake.ts --method=hybrid --refinementSteps=100 --out=/tmp/scene-hybrid.spz

Load both in the mkkellogg viewer side-by-side. PASS = hybrid shows visibly sharper mesh-edge transitions (the refiner reduces the L2 against the supervision targets, which surfaces as sharper silhouettes in the rendered splat scene).

The unit-test equivalent is asserted by hybrid-baker.test.ts > hybrid output reduces own-loss vs the surface-sampling warm start — that test confirms the refiner's loss decreases. The visual sharpness depends on the rendering pipeline used to view the splats; the CPU hybrid path's improvement is most visible in viewers with sub-pixel splat rendering.

F-055 — White unit cube in warm env shows warm tinting#

Goal: confirm lighting estimation produces correct tints.

Scripted equivalent: libs/isis/gaussian-splatting/src/lighting/apply-pbr-shading.test.ts includes the test F-055 scripted equivalent: white albedo + warm-toned SH yields warm-tinted output — asserts the shading output orders R > G > B with R/B ≈ 2:1 for a warm SH (R:G:B = 1.4:1.1:0.7). PASS when this test passes.

For visual confirmation, the same composer e2e described under E-076 with applyLighting: true + a warm Marble env splat produces a scene-composed.spz with warm-tinted cube Gaussians; load in mkkellogg.

F-056 — Coherence gate eval on benchmark corpus#

Goal: coherent scenes score > 0.85; incoherent scenes < 0.7.

Scripted equivalent: libs/isis/3d-quality-gates/src/delivery/composition-coherence-gate.benchmark.test.ts runs a 6-scenario corpus (3 coherent + 3 incoherent) via a deterministic feature adapter that hits known target cosines. PASS when this test passes.

For the production corpus eval, the deployment ticket that wires the real CLIP/DINO adapter (via onnxruntime-node) re-runs the same scenarios against actual composed renders; score thresholds stay the same.

G-042 — Real Redis + 1 local worker process#

Goal: confirm dispatch wiring works against real broker.

  1. Start Redis locally (docker compose -f docker/docker-compose.dev.yml up -d redis).
  2. Start a worker process: npx tsx scripts/composer-worker.ts (the script wires attachWorker from @isis/scene-from-image-composer/dispatch/attach-worker with the env / mesh / bake handlers + a RedisJobChannel).
  3. Run a composer e2e with dispatchMode: 'job-envelope' + dispatchConfig: { redisUrl: 'redis://localhost:6379' }.

PASS = composer completes successfully + worker process logs show one inbound envelope per heavy phase.

In-repo equivalent: composer.dispatch-mode.test.ts asserts bake-metric parity between in-process and dispatched modes using InMemoryJobChannel — same wiring, no broker.

H-052 — Kill composer mid-run via kill -9; resume#

Goal: confirm SIGKILL-resume produces same manifest.

  1. Set up a real Postgres + isis database (see libs/isis/database/.env.example + prisma db push).
  2. Start a composer run from a script:
    ts
    const promise = composeSceneFromImage({
      ...,
      repositoryConfig: { mode: 'postgres' },
    });
    
  3. After the env-phase logs but before manifest-phase, kill -9 the process.
  4. Restart, run:
    ts
    const resumed = await composeSceneFromImage({
      ...,
      repositoryConfig: { mode: 'postgres' },
      resumeFromPipelineId: <captured pipeline id>,
    });
    

PASS = resumed.manifest.files matches a non-interrupted baseline run's manifest.

In-repo equivalent: composer.persistence.test.ts > SIGKILL-resume golden simulates the crash by reverting pipeline + last-phase status in the InMemoryPipelineRepository, then resuming via a fresh composer call. Asserts manifest equivalence.

I-093 — Consensus checkpoint policy with fixture agents#

Goal: confirm fixture agents drive deterministic decisions end-to-end via a real CHECKPOINT phase.

Scripted equivalent: libs/isis/scene-from-image-composer/src/integration/consensus-checkpoint-handler.test.ts runs 9 tests covering all the decision paths (all-SUPPORT, all-OPPOSE, CONDITIONAL-merged-modifications, 2-vs-2 deadlock, throw, timeout). PASS when this test passes.

End-to-end run requires composer phases that actually raise PhaseType.CHECKPOINT. v1 composer phases don't; block C+ work adds them. Once landed, the recipe is:

  1. Configure a composer request with checkpointHandler: { type: 'consensus', agents: [...] }.
  2. Run the composer through a CHECKPOINT phase that exceeds its timeoutMs.
  3. The orchestrator's checkpoint coordinator fires the escalation handler, which drives the convergence session.
  4. The session's terminal status maps to a CheckpointDecision that resumes the pipeline.

PASS = pipeline resumes with the consensus's decision (approved/rejected/modified/timed-out) recorded in result.checkpointHistory.