How we know each block works before signing it off, plus the cross-block integration story.
Per-block test types#
Every block plan declares unit, integration, and adversarial tests. The expectations:
| Type | Purpose | Per-block target |
|---|---|---|
| Unit | One file, one behaviour. Pure functions where possible. | ≥ 90% line coverage on new code |
| Integration | Multiple files in concert. Stub providers/adapters. | One per public API entry point |
| Adversarial | Failure paths, misuse, malformed input. | Every public function has at least one |
| End-to-end | Full composer run against fixture providers. | One per phase combination of interest |
| Visual QA | Manual screenshot in PR. | After block E and block F |
All unit + integration + adversarial tests run via npx vitest run from the
affected library directory (per oshun_nx_broken memory: bypass Nx).
Golden test corpora#
Image corpus (composer end-to-end)#
20 source images covering the failure modes we expect to hit:
| ID | Description | Why included |
|---|---|---|
| C01 | Tight crop of a single subject (e.g. close-up of a vase) | High-novelty Marble failure mode |
| C02 | Wide vista with no clear foreground | Tests "skip objects" path |
| C03 | Crowded interior (10+ objects) | Stress test for detection IoU merge |
| C04 | Abstract / non-photorealistic | Negative case — should fail soft on grounding |
| C05 | Dark / low contrast | Tests detection robustness |
| C06 | Blurry / motion blur | Tests grounding-gate confidence floor |
| C07 | Architectural blueprint / ortho | Tests Marble's response to non-perspective input |
| C08 | Outdoor scene with sky as ⅔ of frame | Background classifier unit test data |
| C09 | Single character against plain background | Target for Hunyuan high-detail routing |
| C10 | Complex scene with reflections | Tests lighting consistency (block F) |
| C11–C20 | Variations covering edge cases (small objects, occluded subjects, etc.) |
Stored under libs/isis/scene-from-image-composer/test-fixtures/corpus/. Each
image accompanied by a sidecar JSON of expected outputs (object counts,
grounding score ranges, manifest file counts) that integration tests assert
against.
Splat reference corpus (block D)#
5–10 known-good .spz files for the encoder/decoder round-trip tests. See
block D for sourcing constraints.
Stored under libs/isis/gaussian-splatting/test-fixtures/spz-reference/.
Mesh fixtures (block E)#
GLBs with known properties for bake tests:
| File | Properties | Used for |
|---|---|---|
unit-cube.glb |
12 triangles, white diffuse | sampling-baker correctness |
textured-cube.glb |
12 triangles, checker texture | colour sampling tests |
100k-character.glb |
100k tris, PBR materials | performance baseline |
degenerate.glb |
All zero-area triangles | adversarial test |
Stored under libs/isis/gaussian-splatting/test-fixtures/meshes/.
Fixture adapters#
Production code uses real providers (Marble, Hunyuan, ElevenLabs SFX, Claude/Gemini for grounding, CLIP/DINO for coherence). Tests use fixture adapters that satisfy the same TypeScript interface but return scripted data. Every adapter has both a real and a fixture implementation.
| Surface | Fixture | Where it lives |
|---|---|---|
MarbleProvider |
FixtureMarbleProvider (extends Base3DProvider, returns canned Generated3DModel) |
composer test-utils |
I3DProvider (mesh) |
FixtureMeshProvider |
composer test-utils |
ISFXProvider |
FixtureSfxProvider |
composer test-utils |
ThreeDQualityGroundingVlmAdapter |
already exists in input-grounding-gate.test.ts |
reused |
ThreeDQualityCoherenceFeatureAdapter |
FixtureCoherenceAdapter |
block F tests |
JobChannel |
InMemoryJobChannel (block G dispatcher tests) |
block G |
PipelineRepository |
InMemoryPipelineRepository (also production-real for ephemeral runs) |
block H |
ConsensusAgent |
FixtureConsensusAgent (block I) |
block I |
Each fixture adapter ships under
libs/isis/scene-from-image-composer/src/test-utils/.
Cross-block integration matrix#
A single end-to-end test exercises every block's contribution in one run. Lives
at libs/isis/scene-from-image-composer/src/composer.e2e.test.ts.
| Step | Block(s) involved | Assertion |
|---|---|---|
| 1. Submit request with fixture providers | B | composeSceneFromImage returns within timeout |
| 2. Environment phase | B | EnvironmentPhaseOutput shape matches contract |
| 3. Grounding phase | B | GroundingPhaseOutput.decision === 'pass' for benign fixture |
| 4. Detection phase (vlm-detected mode) | C | 3 detections after IoU merge from 5 raw |
| 5. Objects phase | B + C routing | 3 meshes generated; routed providers match scoring rule expectations |
| 6. Align phase | B + C | 3 placements with deterministic transforms |
| 7. Bake phase (surface-sampling) | E + F lighting | Baked Gaussians colour-shifted relative to raw albedo (lighting applied) |
| 8. Audio phase | B | 1 ambient + 3 per-object SFX |
| 9. Manifest phase | B | OutputManifest with 5 files; metadata.sceneFromImage.* populated |
| 10. Coherence eval | F | decision === 'pass' for fixture adapter returning matched features |
| 11. Gallery registration | I | One OutputRecord created |
| 12. Lineage build | I | Graph has 5 nodes + 5 edges |
| 13. Anomaly sample | I | One sample emitted to fixture adapter |
| 14. Persistence | H | All phase artifacts persisted; pipeline status COMPLETED |
| 15. Resume after kill (separate test) | H | Killed before phase 5; resumed; same final manifest |
| 16. Checkpoint approval flow (separate test) | A + B | Auto-approve at default; rejection halts pipeline |
| 17. Consensus checkpoint (separate test) | A + I | 3-agent fixture consensus, 2 approve → proceeds |
The matrix is the acceptance gate for the thorough version: until every row
asserts cleanly, the rollout (06-rollout-plan.md) doesn't move past phase 2.
Performance targets#
Per phase, on the test-fixture sizes:
| Phase | Mode | Target |
|---|---|---|
| Environment | Real Marble | < 5 min wall clock (provider's SLA) |
| Environment | Fixture | < 10 ms |
| Grounding | Real VLM | < 10 s |
| Grounding | Fixture | < 5 ms |
| Detection | Real VLM | < 15 s |
| Detection | Fixture | < 5 ms |
| Objects (per mesh) | Real Hunyuan | < 90 s |
| Objects (per mesh) | Fixture | < 10 ms |
| Align (per object) | — | < 1 ms |
| Bake (per mesh, surface-sampling, 5k Gaussians) | CPU | < 60 s |
| Bake (per mesh, hybrid 200 steps) | GPU | < 5 min |
| Audio (per track) | Real ElevenLabs | < 30 s |
| Audio (per track) | Fixture | < 10 ms |
| Manifest | — | < 100 ms |
| Coherence | Real CLIP | < 30 s |
| Coherence | Fixture | < 5 ms |
End-to-end fixture run target: < 5 s (so we can run on every push without making CI painful).
End-to-end real-provider run (single-mesh scene) target: < 10 min.
CI integration#
Each block's tests run as part of the existing Vitest CI matrix. New suites added per the existing repo conventions; no new CI scaffolding required beyond:
- Postgres test database (block H) — already provisioned via
docker/docker-compose.dev.ymlper CLAUDE.md. - Redis (block G) — same.
- Real-provider tests are not in the default CI run (cost + flake); they
live in
*.real-providers.test.tsfiles and run in a nightly job.
Verification commands per block#
| Block | Command |
|---|---|
| A | cd libs/isis/operation-orchestrator && npx vitest run src/checkpoint/ src/execution-orchestrator/ |
| B | cd libs/isis/scene-from-image-composer && npx vitest run |
| C | cd libs/isis/scene-from-image-composer && npx vitest run src/detection/ src/objects/ src/align/ src/routing/ |
| D | cd libs/isis/gaussian-splatting && npx vitest run src/exporters/ |
| E | cd libs/isis/gaussian-splatting && npx vitest run src/bake/ + cd libs/isis/scene-from-image-composer && npx vitest run src/phases/bake-phase.test.ts |
| F | cd libs/isis/gaussian-splatting && npx vitest run src/lighting/ + cd libs/isis/3d-quality-gates && npx vitest run src/delivery/composition-coherence-gate.test.ts |
| G | cd libs/isis/scene-from-image-composer && npx vitest run src/dispatch/ |
| H | cd libs/isis/operation-orchestrator && npx vitest run src/persistence/ src/resume/ |
| I | cd libs/isis/scene-from-image-composer && npx vitest run src/integration/ |
| All | cd libs/isis/scene-from-image-composer && npx vitest run (after all blocks land) |
Adversarial / mutation testing#
Per CLAUDE.md's stub-scanner conventions, every block runs the
scripts/stub-indicator-scan.sh against the new files before commit. Hits must
be either fixed or annotated // stub:legitimate <reason> per the
oshun_stub_scanner_strictness memory (and per
feedback_stub_legitimate_honesty — never abuse the annotation).
Optionally (future hardening): introduce mutation testing via stryker-js
against the composer's pure-function modules (align helper, constraint solver,
IoU merge, irradiance estimator). Out of scope for v1; flagged in
07-risks-and-open-questions.md.