Eve's model strategy is “the smallest measured model that meets the task-family bar.” Cost or speed alone cannot select a model, and a global benchmark cannot replace evaluation of routing, tools, grounding, refusals, and product-specific behavior.
Selection is per task family and model leg. A cheaper model is acceptable only after it meets the same product-specific floors and continues to meet them under canary and maintenance evidence.
Small-model doctrine#
The Small-Model Excellence design defines the architecture and its evidence standard. The key principles are:
- route deterministically before asking a model when the signal is reliable;
- provide only the skill and tools relevant to the selected family;
- validate tool arguments at a constrained boundary;
- check grounding and visible claims independently of fluent prose;
- escalate only after a verified failure and within a bounded ladder;
- choose models per leg/family, not once for the entire assistant;
- keep prompts cache-ordered and enforce explicit context budgets;
- continuously distill strong-model findings into skills, examples, and tests;
- measure quality, latency, tokens, and cost together.
Runtime quality components#
| Component | Responsibility | Principal failure it prevents |
|---|---|---|
| Task-family router | Select the behavioral contract for the turn. | Wrong prompt/toolset and misleading capability claims. |
| Skill registry | Bind instructions, schemas, budgets, tools, and eval cases. | Ad hoc prompt growth and unmeasured capability drift. |
| Toolset scoper | Withhold irrelevant tools; force a relevant tool when policy requires it. | Over-broad authority and tool-avoidant hallucination. |
| Argument boundary | Validate, instruct, and perform at most the bounded repair. | Malformed or smuggled action payloads. |
| Grounding checkers | Compare visible claims and evidence state. | Fluent unsupported answers and misleading badges. |
| Escalation ladder | Retry/escalate verified failures with logged reasons. | Permanent frontier binding and arbitrary retry loops. |
| Model registry | Pin model/provider/settings per leg. | Undocumented provider drift and irreproducible results. |
| Context budgeter | Order cacheable instructions and cap history/retrieval/tools. | Latency/cost blowout and attention dilution. |
Evaluation deck#
The deck is a product instrument, not a collection of pleasant prompts. It contains golden, battery, family, adversarial, held-out, and ledger cases that exercise:
- routing and task-family selection;
- exact tool availability/use and mutation restraint;
- constrained argument validity;
- grounding, citations, evidence chips, and unsupported-claim handling;
- persona/tone, crisis, refusal, and safety supersession;
- page context, selection, memory, and continuity behavior;
- builder reads/writes, confirmation-card affordance, and docs search;
- degraded providers, timeouts, stale state, and budget limits;
- member versus operator versus agent authority boundaries;
- copy/string quality where wording itself changes safe product behavior.
Cases name their fixtures and environment dependencies. An environment-blocked case is not a pass, and a case that cannot exercise its asserted capability must be repaired or removed from the release claim.
Floors and release gates#
Quality floors are set per family because failure costs differ. A navigation hint, a grounded factual answer, a crisis turn, and a workbench mutation should not share one undifferentiated average. Gates should include hard invariants where one violation is unacceptable and statistical floors where model variance is expected.
A release record captures:
- deck and prompt/skill hashes;
- model registry and provider settings;
- fixture/environment version;
- sample count and randomization posture;
- per-family pass/fail and hard-invariant failures;
- latency, token, and cost distribution;
- judge model/version and judge-validation status;
- waivers with owner and expiry.
Escalation#
Escalation follows an observed failure signal such as invalid arguments, grounding hold, refusal-policy mismatch, or failed family rubric. It is capped and logged. A retry cannot silently change the task family in a way that removes required tools or policy.
The ladder may use a stronger or different model, a constrained repair leg, a deterministic fallback, or human review. The correct response to a high-risk unresolved failure can be refusal rather than escalation.
Judge validation#
Model judges are calibrated against human-labeled examples before their scores gate releases. Validation measures false accepts and false rejects on the behavior that matters, not just correlation with an aggregate score. Judge prompts and versions are pinned, and disputed cases enter the ledger.
No judge should grade evidence it cannot see. Tool traces, context, expected policy, and rendered disclosures must be present when the rubric depends on them.
Prompt and skill change control#
Model-facing strings include system prompts, skill text, tool descriptions, examples, repair instructions, and sometimes UI-supplied context. The prompt inventory and hash ratchet make those bytes reviewable. Any material change reruns the affected family floors; a formatting-only claim is not accepted when the model-visible bytes changed.
The builder prompt inventory is recorded in Eve Builder Prompt Inventory, and the standing procedure is Eve SMX maintenance.
Cost and performance#
Cost reporting separates model/provider, task family, leg, escalation, cached versus uncached tokens, tool cost, and outcome. Optimize only after quality floors are green. The cheapest raw call is not cheaper if it triggers repeated turns, incorrect actions, human cleanup, or routine escalation.
Latency budgets include routing, retrieval, provider time, tools, grounding checks, streaming start, and finalization. Sentence-safe streaming improves perceived progress but does not hide an unbounded total turn.
Standing maintenance loop#
The weekly loop reruns ratchets/decks, inspects misroutes and refusals, reviews cost/latency, validates judge drift, triages new production failures into cases, and tests a cheaper candidate only under the measurement protocol. Outcomes are recorded in the Eve SMX scorecard.