Eve library

05 — Test Strategy

Other

Revision 1

On this page

How we know each block works before signing it off, plus the cross-block integration story.

Per-block test types#

Every block plan declares unit, integration, and adversarial tests. The expectations:

Type Purpose Per-block target
Unit One file, one behaviour. Pure functions where possible. ≥ 90% line coverage on new code
Integration Multiple files in concert. Stub providers/adapters. One per public API entry point
Adversarial Failure paths, misuse, malformed input. Every public function has at least one
End-to-end Full composer run against fixture providers. One per phase combination of interest
Visual QA Manual screenshot in PR. After block E and block F

All unit + integration + adversarial tests run via npx vitest run from the affected library directory (per oshun_nx_broken memory: bypass Nx).

Golden test corpora#

Image corpus (composer end-to-end)#

20 source images covering the failure modes we expect to hit:

ID Description Why included
C01 Tight crop of a single subject (e.g. close-up of a vase) High-novelty Marble failure mode
C02 Wide vista with no clear foreground Tests "skip objects" path
C03 Crowded interior (10+ objects) Stress test for detection IoU merge
C04 Abstract / non-photorealistic Negative case — should fail soft on grounding
C05 Dark / low contrast Tests detection robustness
C06 Blurry / motion blur Tests grounding-gate confidence floor
C07 Architectural blueprint / ortho Tests Marble's response to non-perspective input
C08 Outdoor scene with sky as ⅔ of frame Background classifier unit test data
C09 Single character against plain background Target for Hunyuan high-detail routing
C10 Complex scene with reflections Tests lighting consistency (block F)
C11–C20 Variations covering edge cases (small objects, occluded subjects, etc.)

Stored under libs/isis/scene-from-image-composer/test-fixtures/corpus/. Each image accompanied by a sidecar JSON of expected outputs (object counts, grounding score ranges, manifest file counts) that integration tests assert against.

Splat reference corpus (block D)#

5–10 known-good .spz files for the encoder/decoder round-trip tests. See block D for sourcing constraints.

Stored under libs/isis/gaussian-splatting/test-fixtures/spz-reference/.

Mesh fixtures (block E)#

GLBs with known properties for bake tests:

File Properties Used for
unit-cube.glb 12 triangles, white diffuse sampling-baker correctness
textured-cube.glb 12 triangles, checker texture colour sampling tests
100k-character.glb 100k tris, PBR materials performance baseline
degenerate.glb All zero-area triangles adversarial test

Stored under libs/isis/gaussian-splatting/test-fixtures/meshes/.

Fixture adapters#

Production code uses real providers (Marble, Hunyuan, ElevenLabs SFX, Claude/Gemini for grounding, CLIP/DINO for coherence). Tests use fixture adapters that satisfy the same TypeScript interface but return scripted data. Every adapter has both a real and a fixture implementation.

Surface Fixture Where it lives
MarbleProvider FixtureMarbleProvider (extends Base3DProvider, returns canned Generated3DModel) composer test-utils
I3DProvider (mesh) FixtureMeshProvider composer test-utils
ISFXProvider FixtureSfxProvider composer test-utils
ThreeDQualityGroundingVlmAdapter already exists in input-grounding-gate.test.ts reused
ThreeDQualityCoherenceFeatureAdapter FixtureCoherenceAdapter block F tests
JobChannel InMemoryJobChannel (block G dispatcher tests) block G
PipelineRepository InMemoryPipelineRepository (also production-real for ephemeral runs) block H
ConsensusAgent FixtureConsensusAgent (block I) block I

Each fixture adapter ships under libs/isis/scene-from-image-composer/src/test-utils/.

Cross-block integration matrix#

A single end-to-end test exercises every block's contribution in one run. Lives at libs/isis/scene-from-image-composer/src/composer.e2e.test.ts.

Step Block(s) involved Assertion
1. Submit request with fixture providers B composeSceneFromImage returns within timeout
2. Environment phase B EnvironmentPhaseOutput shape matches contract
3. Grounding phase B GroundingPhaseOutput.decision === 'pass' for benign fixture
4. Detection phase (vlm-detected mode) C 3 detections after IoU merge from 5 raw
5. Objects phase B + C routing 3 meshes generated; routed providers match scoring rule expectations
6. Align phase B + C 3 placements with deterministic transforms
7. Bake phase (surface-sampling) E + F lighting Baked Gaussians colour-shifted relative to raw albedo (lighting applied)
8. Audio phase B 1 ambient + 3 per-object SFX
9. Manifest phase B OutputManifest with 5 files; metadata.sceneFromImage.* populated
10. Coherence eval F decision === 'pass' for fixture adapter returning matched features
11. Gallery registration I One OutputRecord created
12. Lineage build I Graph has 5 nodes + 5 edges
13. Anomaly sample I One sample emitted to fixture adapter
14. Persistence H All phase artifacts persisted; pipeline status COMPLETED
15. Resume after kill (separate test) H Killed before phase 5; resumed; same final manifest
16. Checkpoint approval flow (separate test) A + B Auto-approve at default; rejection halts pipeline
17. Consensus checkpoint (separate test) A + I 3-agent fixture consensus, 2 approve → proceeds

The matrix is the acceptance gate for the thorough version: until every row asserts cleanly, the rollout (06-rollout-plan.md) doesn't move past phase 2.

Performance targets#

Per phase, on the test-fixture sizes:

Phase Mode Target
Environment Real Marble < 5 min wall clock (provider's SLA)
Environment Fixture < 10 ms
Grounding Real VLM < 10 s
Grounding Fixture < 5 ms
Detection Real VLM < 15 s
Detection Fixture < 5 ms
Objects (per mesh) Real Hunyuan < 90 s
Objects (per mesh) Fixture < 10 ms
Align (per object) — < 1 ms
Bake (per mesh, surface-sampling, 5k Gaussians) CPU < 60 s
Bake (per mesh, hybrid 200 steps) GPU < 5 min
Audio (per track) Real ElevenLabs < 30 s
Audio (per track) Fixture < 10 ms
Manifest — < 100 ms
Coherence Real CLIP < 30 s
Coherence Fixture < 5 ms

End-to-end fixture run target: < 5 s (so we can run on every push without making CI painful).

End-to-end real-provider run (single-mesh scene) target: < 10 min.

CI integration#

Each block's tests run as part of the existing Vitest CI matrix. New suites added per the existing repo conventions; no new CI scaffolding required beyond:

  • Postgres test database (block H) — already provisioned via docker/docker-compose.dev.yml per CLAUDE.md.
  • Redis (block G) — same.
  • Real-provider tests are not in the default CI run (cost + flake); they live in *.real-providers.test.ts files and run in a nightly job.

Verification commands per block#

Block Command
A cd libs/isis/operation-orchestrator && npx vitest run src/checkpoint/ src/execution-orchestrator/
B cd libs/isis/scene-from-image-composer && npx vitest run
C cd libs/isis/scene-from-image-composer && npx vitest run src/detection/ src/objects/ src/align/ src/routing/
D cd libs/isis/gaussian-splatting && npx vitest run src/exporters/
E cd libs/isis/gaussian-splatting && npx vitest run src/bake/ + cd libs/isis/scene-from-image-composer && npx vitest run src/phases/bake-phase.test.ts
F cd libs/isis/gaussian-splatting && npx vitest run src/lighting/ + cd libs/isis/3d-quality-gates && npx vitest run src/delivery/composition-coherence-gate.test.ts
G cd libs/isis/scene-from-image-composer && npx vitest run src/dispatch/
H cd libs/isis/operation-orchestrator && npx vitest run src/persistence/ src/resume/
I cd libs/isis/scene-from-image-composer && npx vitest run src/integration/
All cd libs/isis/scene-from-image-composer && npx vitest run (after all blocks land)

Adversarial / mutation testing#

Per CLAUDE.md's stub-scanner conventions, every block runs the scripts/stub-indicator-scan.sh against the new files before commit. Hits must be either fixed or annotated // stub:legitimate <reason> per the oshun_stub_scanner_strictness memory (and per feedback_stub_legitimate_honesty — never abuse the annotation).

Optionally (future hardening): introduce mutation testing via stryker-js against the composer's pure-function modules (align helper, constraint solver, IoU merge, irradiance estimator). Out of scope for v1; flagged in 07-risks-and-open-questions.md.