# 05 — Test Strategy

How we know each block works before signing it off, plus the cross-block
integration story.

## Per-block test types

Every block plan declares unit, integration, and adversarial tests. The
expectations:

| Type        | Purpose                                                 | Per-block target                       |
| ----------- | ------------------------------------------------------- | -------------------------------------- |
| Unit        | One file, one behaviour. Pure functions where possible. | ≥ 90% line coverage on new code        |
| Integration | Multiple files in concert. Stub providers/adapters.     | One per public API entry point         |
| Adversarial | Failure paths, misuse, malformed input.                 | Every public function has at least one |
| End-to-end  | Full composer run against fixture providers.            | One per phase combination of interest  |
| Visual QA   | Manual screenshot in PR.                                | After block E and block F              |

All unit + integration + adversarial tests run via `npx vitest run` from the
affected library directory (per `oshun_nx_broken` memory: bypass Nx).

## Golden test corpora

### Image corpus (composer end-to-end)

20 source images covering the failure modes we expect to hit:

| ID      | Description                                                             | Why included                                     |
| ------- | ----------------------------------------------------------------------- | ------------------------------------------------ |
| C01     | Tight crop of a single subject (e.g. close-up of a vase)                | High-novelty Marble failure mode                 |
| C02     | Wide vista with no clear foreground                                     | Tests "skip objects" path                        |
| C03     | Crowded interior (10+ objects)                                          | Stress test for detection IoU merge              |
| C04     | Abstract / non-photorealistic                                           | Negative case — should fail soft on grounding    |
| C05     | Dark / low contrast                                                     | Tests detection robustness                       |
| C06     | Blurry / motion blur                                                    | Tests grounding-gate confidence floor            |
| C07     | Architectural blueprint / ortho                                         | Tests Marble's response to non-perspective input |
| C08     | Outdoor scene with sky as ⅔ of frame                                    | Background classifier unit test data             |
| C09     | Single character against plain background                               | Target for Hunyuan high-detail routing           |
| C10     | Complex scene with reflections                                          | Tests lighting consistency (block F)             |
| C11–C20 | Variations covering edge cases (small objects, occluded subjects, etc.) |                                                  |

Stored under `libs/isis/scene-from-image-composer/test-fixtures/corpus/`. Each
image accompanied by a sidecar JSON of expected outputs (object counts,
grounding score ranges, manifest file counts) that integration tests assert
against.

### Splat reference corpus (block D)

5–10 known-good `.spz` files for the encoder/decoder round-trip tests. See
[block D](./block-plans/D-spz-exporter.md) for sourcing constraints.

Stored under `libs/isis/gaussian-splatting/test-fixtures/spz-reference/`.

### Mesh fixtures (block E)

GLBs with known properties for bake tests:

| File                 | Properties                    | Used for                   |
| -------------------- | ----------------------------- | -------------------------- |
| `unit-cube.glb`      | 12 triangles, white diffuse   | sampling-baker correctness |
| `textured-cube.glb`  | 12 triangles, checker texture | colour sampling tests      |
| `100k-character.glb` | 100k tris, PBR materials      | performance baseline       |
| `degenerate.glb`     | All zero-area triangles       | adversarial test           |

Stored under `libs/isis/gaussian-splatting/test-fixtures/meshes/`.

## Fixture adapters

Production code uses real providers (Marble, Hunyuan, ElevenLabs SFX,
Claude/Gemini for grounding, CLIP/DINO for coherence). Tests use fixture
adapters that satisfy the same TypeScript interface but return scripted data.
Every adapter has both a real and a fixture implementation.

| Surface                                | Fixture                                                                               | Where it lives      |
| -------------------------------------- | ------------------------------------------------------------------------------------- | ------------------- |
| `MarbleProvider`                       | `FixtureMarbleProvider` (extends `Base3DProvider`, returns canned `Generated3DModel`) | composer test-utils |
| `I3DProvider` (mesh)                   | `FixtureMeshProvider`                                                                 | composer test-utils |
| `ISFXProvider`                         | `FixtureSfxProvider`                                                                  | composer test-utils |
| `ThreeDQualityGroundingVlmAdapter`     | already exists in `input-grounding-gate.test.ts`                                      | reused              |
| `ThreeDQualityCoherenceFeatureAdapter` | `FixtureCoherenceAdapter`                                                             | block F tests       |
| `JobChannel`                           | `InMemoryJobChannel` (block G dispatcher tests)                                       | block G             |
| `PipelineRepository`                   | `InMemoryPipelineRepository` (also production-real for ephemeral runs)                | block H             |
| `ConsensusAgent`                       | `FixtureConsensusAgent` (block I)                                                     | block I             |

Each fixture adapter ships under
`libs/isis/scene-from-image-composer/src/test-utils/`.

## Cross-block integration matrix

A single end-to-end test exercises every block's contribution in one run. Lives
at `libs/isis/scene-from-image-composer/src/composer.e2e.test.ts`.

| Step                                         | Block(s) involved | Assertion                                                                |
| -------------------------------------------- | ----------------- | ------------------------------------------------------------------------ |
| 1. Submit request with fixture providers     | B                 | `composeSceneFromImage` returns within timeout                           |
| 2. Environment phase                         | B                 | `EnvironmentPhaseOutput` shape matches contract                          |
| 3. Grounding phase                           | B                 | `GroundingPhaseOutput.decision === 'pass'` for benign fixture            |
| 4. Detection phase (vlm-detected mode)       | C                 | 3 detections after IoU merge from 5 raw                                  |
| 5. Objects phase                             | B + C routing     | 3 meshes generated; routed providers match scoring rule expectations     |
| 6. Align phase                               | B + C             | 3 placements with deterministic transforms                               |
| 7. Bake phase (surface-sampling)             | E + F lighting    | Baked Gaussians colour-shifted relative to raw albedo (lighting applied) |
| 8. Audio phase                               | B                 | 1 ambient + 3 per-object SFX                                             |
| 9. Manifest phase                            | B                 | `OutputManifest` with 5 files; `metadata.sceneFromImage.*` populated     |
| 10. Coherence eval                           | F                 | `decision === 'pass'` for fixture adapter returning matched features     |
| 11. Gallery registration                     | I                 | One `OutputRecord` created                                               |
| 12. Lineage build                            | I                 | Graph has 5 nodes + 5 edges                                              |
| 13. Anomaly sample                           | I                 | One sample emitted to fixture adapter                                    |
| 14. Persistence                              | H                 | All phase artifacts persisted; pipeline status `COMPLETED`               |
| 15. Resume after kill (separate test)        | H                 | Killed before phase 5; resumed; same final manifest                      |
| 16. Checkpoint approval flow (separate test) | A + B             | Auto-approve at default; rejection halts pipeline                        |
| 17. Consensus checkpoint (separate test)     | A + I             | 3-agent fixture consensus, 2 approve → proceeds                          |

The matrix is the **acceptance gate** for the thorough version: until every row
asserts cleanly, the rollout (`06-rollout-plan.md`) doesn't move past phase 2.

## Performance targets

Per phase, on the test-fixture sizes:

| Phase                                           | Mode            | Target                              |
| ----------------------------------------------- | --------------- | ----------------------------------- |
| Environment                                     | Real Marble     | < 5 min wall clock (provider's SLA) |
| Environment                                     | Fixture         | < 10 ms                             |
| Grounding                                       | Real VLM        | < 10 s                              |
| Grounding                                       | Fixture         | < 5 ms                              |
| Detection                                       | Real VLM        | < 15 s                              |
| Detection                                       | Fixture         | < 5 ms                              |
| Objects (per mesh)                              | Real Hunyuan    | < 90 s                              |
| Objects (per mesh)                              | Fixture         | < 10 ms                             |
| Align (per object)                              | —               | < 1 ms                              |
| Bake (per mesh, surface-sampling, 5k Gaussians) | CPU             | < 60 s                              |
| Bake (per mesh, hybrid 200 steps)               | GPU             | < 5 min                             |
| Audio (per track)                               | Real ElevenLabs | < 30 s                              |
| Audio (per track)                               | Fixture         | < 10 ms                             |
| Manifest                                        | —               | < 100 ms                            |
| Coherence                                       | Real CLIP       | < 30 s                              |
| Coherence                                       | Fixture         | < 5 ms                              |

End-to-end fixture run target: < 5 s (so we can run on every push without making
CI painful).

End-to-end real-provider run (single-mesh scene) target: < 10 min.

## CI integration

Each block's tests run as part of the existing Vitest CI matrix. New suites
added per the existing repo conventions; no new CI scaffolding required beyond:

- Postgres test database (block H) — already provisioned via
  `docker/docker-compose.dev.yml` per CLAUDE.md.
- Redis (block G) — same.
- Real-provider tests are **not** in the default CI run (cost + flake); they
  live in `*.real-providers.test.ts` files and run in a nightly job.

## Verification commands per block

| Block | Command                                                                                                                                                               |
| ----- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| A     | `cd libs/isis/operation-orchestrator && npx vitest run src/checkpoint/ src/execution-orchestrator/`                                                                   |
| B     | `cd libs/isis/scene-from-image-composer && npx vitest run`                                                                                                            |
| C     | `cd libs/isis/scene-from-image-composer && npx vitest run src/detection/ src/objects/ src/align/ src/routing/`                                                        |
| D     | `cd libs/isis/gaussian-splatting && npx vitest run src/exporters/`                                                                                                    |
| E     | `cd libs/isis/gaussian-splatting && npx vitest run src/bake/` + `cd libs/isis/scene-from-image-composer && npx vitest run src/phases/bake-phase.test.ts`              |
| F     | `cd libs/isis/gaussian-splatting && npx vitest run src/lighting/` + `cd libs/isis/3d-quality-gates && npx vitest run src/delivery/composition-coherence-gate.test.ts` |
| G     | `cd libs/isis/scene-from-image-composer && npx vitest run src/dispatch/`                                                                                              |
| H     | `cd libs/isis/operation-orchestrator && npx vitest run src/persistence/ src/resume/`                                                                                  |
| I     | `cd libs/isis/scene-from-image-composer && npx vitest run src/integration/`                                                                                           |
| All   | `cd libs/isis/scene-from-image-composer && npx vitest run` (after all blocks land)                                                                                    |

## Adversarial / mutation testing

Per CLAUDE.md's stub-scanner conventions, every block runs the
`scripts/stub-indicator-scan.sh` against the new files before commit. Hits must
be either fixed or annotated `// stub:legitimate <reason>` per the
`oshun_stub_scanner_strictness` memory (and per
`feedback_stub_legitimate_honesty` — never abuse the annotation).

Optionally (future hardening): introduce mutation testing via `stryker-js`
against the composer's pure-function modules (align helper, constraint solver,
IoU merge, irradiance estimator). Out of scope for v1; flagged in
`07-risks-and-open-questions.md`.
