# Eve Models, Quality, and Evaluation

Eve's model strategy is “the smallest measured model that meets the task-family
bar.” Cost or speed alone cannot select a model, and a global benchmark cannot
replace evaluation of routing, tools, grounding, refusals, and product-specific
behavior.

```mermaid
flowchart LR
  Request[Observed task-family request] --> Route[Deterministic route when reliable]
  Route --> Candidate[Smallest eligible model leg]
  Candidate --> Eval[Family deck: answer tool grounding refusal format]
  Eval --> Floor{All release floors met?}
  Floor -- yes --> Canary[Bounded canary with quality latency token and cost signals]
  Floor -- no --> Repair[Repair skill examples prompt tools or routing]
  Repair --> Eval
  Canary --> Drift{Current cohort stays above floor?}
  Drift -- yes --> Default[Eligible default for that family and leg]
  Drift -- no --> Escalate[Escalate model or roll back candidate]
  Escalate --> Diagnose[Capture failures for distillation and judge review]
  Diagnose --> Repair
```

Selection is per task family and model leg. A cheaper model is acceptable only
after it meets the same product-specific floors and continues to meet them under
canary and maintenance evidence.

## Small-model doctrine

The
[Small-Model Excellence design](../../EVE_SMALL_MODEL_EXCELLENCE_DESIGN_2026-08-16.md)
defines the architecture and its evidence standard. The key principles are:

- route deterministically before asking a model when the signal is reliable;
- provide only the skill and tools relevant to the selected family;
- validate tool arguments at a constrained boundary;
- check grounding and visible claims independently of fluent prose;
- escalate only after a verified failure and within a bounded ladder;
- choose models per leg/family, not once for the entire assistant;
- keep prompts cache-ordered and enforce explicit context budgets;
- continuously distill strong-model findings into skills, examples, and tests;
- measure quality, latency, tokens, and cost together.

## Runtime quality components

| Component          | Responsibility                                                            | Principal failure it prevents                           |
| ------------------ | ------------------------------------------------------------------------- | ------------------------------------------------------- |
| Task-family router | Select the behavioral contract for the turn.                              | Wrong prompt/toolset and misleading capability claims.  |
| Skill registry     | Bind instructions, schemas, budgets, tools, and eval cases.               | Ad hoc prompt growth and unmeasured capability drift.   |
| Toolset scoper     | Withhold irrelevant tools; force a relevant tool when policy requires it. | Over-broad authority and tool-avoidant hallucination.   |
| Argument boundary  | Validate, instruct, and perform at most the bounded repair.               | Malformed or smuggled action payloads.                  |
| Grounding checkers | Compare visible claims and evidence state.                                | Fluent unsupported answers and misleading badges.       |
| Escalation ladder  | Retry/escalate verified failures with logged reasons.                     | Permanent frontier binding and arbitrary retry loops.   |
| Model registry     | Pin model/provider/settings per leg.                                      | Undocumented provider drift and irreproducible results. |
| Context budgeter   | Order cacheable instructions and cap history/retrieval/tools.             | Latency/cost blowout and attention dilution.            |

## Evaluation deck

The deck is a product instrument, not a collection of pleasant prompts. It
contains golden, battery, family, adversarial, held-out, and ledger cases that
exercise:

- routing and task-family selection;
- exact tool availability/use and mutation restraint;
- constrained argument validity;
- grounding, citations, evidence chips, and unsupported-claim handling;
- persona/tone, crisis, refusal, and safety supersession;
- page context, selection, memory, and continuity behavior;
- builder reads/writes, confirmation-card affordance, and docs search;
- degraded providers, timeouts, stale state, and budget limits;
- member versus operator versus agent authority boundaries;
- copy/string quality where wording itself changes safe product behavior.

Cases name their fixtures and environment dependencies. An environment-blocked
case is not a pass, and a case that cannot exercise its asserted capability must
be repaired or removed from the release claim.

## Floors and release gates

Quality floors are set per family because failure costs differ. A navigation
hint, a grounded factual answer, a crisis turn, and a workbench mutation should
not share one undifferentiated average. Gates should include hard invariants
where one violation is unacceptable and statistical floors where model variance
is expected.

A release record captures:

- deck and prompt/skill hashes;
- model registry and provider settings;
- fixture/environment version;
- sample count and randomization posture;
- per-family pass/fail and hard-invariant failures;
- latency, token, and cost distribution;
- judge model/version and judge-validation status;
- waivers with owner and expiry.

## Escalation

Escalation follows an observed failure signal such as invalid arguments,
grounding hold, refusal-policy mismatch, or failed family rubric. It is capped
and logged. A retry cannot silently change the task family in a way that removes
required tools or policy.

The ladder may use a stronger or different model, a constrained repair leg, a
deterministic fallback, or human review. The correct response to a high-risk
unresolved failure can be refusal rather than escalation.

## Judge validation

Model judges are calibrated against human-labeled examples before their scores
gate releases. Validation measures false accepts and false rejects on the
behavior that matters, not just correlation with an aggregate score. Judge
prompts and versions are pinned, and disputed cases enter the ledger.

No judge should grade evidence it cannot see. Tool traces, context, expected
policy, and rendered disclosures must be present when the rubric depends on
them.

## Prompt and skill change control

Model-facing strings include system prompts, skill text, tool descriptions,
examples, repair instructions, and sometimes UI-supplied context. The prompt
inventory and hash ratchet make those bytes reviewable. Any material change
reruns the affected family floors; a formatting-only claim is not accepted when
the model-visible bytes changed.

The builder prompt inventory is recorded in
[Eve Builder Prompt Inventory](../../../docs/audits/EVE_BUILDER_PROMPT_INVENTORY_2026-08.md),
and the standing procedure is
[Eve SMX maintenance](../../../docs/agents/eve-smx-maintenance.md).

## Cost and performance

Cost reporting separates model/provider, task family, leg, escalation, cached
versus uncached tokens, tool cost, and outcome. Optimize only after quality
floors are green. The cheapest raw call is not cheaper if it triggers repeated
turns, incorrect actions, human cleanup, or routine escalation.

Latency budgets include routing, retrieval, provider time, tools, grounding
checks, streaming start, and finalization. Sentence-safe streaming improves
perceived progress but does not hide an unbounded total turn.

## Standing maintenance loop

The weekly loop reruns ratchets/decks, inspects misroutes and refusals, reviews
cost/latency, validates judge drift, triages new production failures into cases,
and tests a cheaper candidate only under the measurement protocol. Outcomes are
recorded in the [Eve SMX scorecard](../../../docs/audits/EVE_SMX_SCORECARD.md).

## Related

- [Operations, observability, and maintenance](./operations-observability-and-maintenance.md)
- [Runtime, tools, and actions](./runtime-tools-and-actions.md)
- [Analytics and testing](../../features/analytics-and-testing.md)
- [Eve Small-Model Excellence TODO record](../../../EVE_SMALL_MODEL_EXCELLENCE_TODOS_2026-08-16.md)