Date: 2026-09-12
Initiative: eve-sota-gap-closure-2026-09-01
Task: 10.3
Outcome#
The exact seven release-blocked assistant cases now execute through the real session/turn/SSE eval executor under both V1.0 and V1.2 selection. The test uses a deterministic Fastify injection transport: it exercises request construction, session creation, SSE extraction, and production grading without constructing a model-provider binding.
The retained suite proves 14 positive executions: V1.0 admits the honest room
boundary with no withheld tool, while V1.2 requires and observes each case's
restored Veritas or Metis tool. It also proves all seven cases turn red when a
withheld tool appears at V1.0 and when a required restored tool is absent at
V1.2. The production mapping guard is directly exercised with inverted and
missing V1 mappings; both throw invalid_release_mapping.
Exact inventory#
| Case | Restored/withheld tool |
|---|---|
veritas-tool-selection |
veritas_top_claims |
family-md-empty-saved-articles |
veritas_saved_articles |
family-md-course-continue |
metis_continue_learning |
family-md-metis-search |
metis_search_catalog |
family-md-metis-recommend |
metis_recommended_courses |
family-md-empty-course-search |
metis_search_catalog |
family-md-empty-recommend |
metis_recommended_courses |
Proof boundaries#
- Source inspection: the generated record binds the selector, grader, complete deck, focused test/configuration, evidence tooling, audit, and task ledger by SHA-256.
- Automated behavior: conforming and nonconforming transcripts are graded with the production expectation implementation for both scopes.
- Service integration: the positive matrix calls
executeAssistantEvalCaseagainst Fastify routes and parses real SSE bodies; it is not a direct-only grader test. - Negative controls: all seven early-tool and all seven late-missing-tool mutations fail. Inverted and incomplete release mappings also fail through the production validator. The semantic evidence verifier retains six further observed-red mutation classes.
Boundaries and remaining work#
This proof does not instantiate the product BFF, grant an unreleased room, read a provider credential, or make a provider request. Its V1.2 tool events are controlled transcripts, not a claim that Veritas or Metis ships today. The seven cases remain advisory. Task 10.4 owns release-specific k≥10 model-provider measurement and the later explicit promote-or-retire decision. Phase 10 and G8 therefore remain open.