How the assistant brain stays excellent on the smallest passing model after the
SMX initiative closed. The loop runs on telemetry, not enthusiasm: every step
below reads a real instrument and ends in a recorded artifact. Companion
records: EVE_SMALL_MODEL_EXCELLENCE_TODOS_2026-08-16.md (the ledger),
docs/audits/EVE_SMX_SCORECARD.md (every measurement),
docs/audits/eve-smx-ratchet.json (prompt-bytes hash + family floors),
docs/audits/EVE_SMX_DECK_SOURCES.md (case provenance + failure taxonomies).
Standing pins#
- Turn leg:
deepseek/deepseek-v4-flash-0731(champion). Escalation + judge:deepseek/deepseek-v4-pro-0813(judge pin is provisional — see "Judge validation" below). Pins live inapps/oshun/bff/src/assistant/model-registry.tswith price snapshots andchosenByprovenance; env overrides (OSHUN_ASSISTANT_OPENROUTER_MODEL,OSHUN_ASSISTANT_ESCALATION_MODEL,OSHUN_ASSISTANT_JUDGE_MODEL) win over pins and are the rollback lever. - Measurement pins, always:
OPENROUTER_PROVIDER_SORT=price;OPENROUTER_PROVIDER_QUANTIZATIONS=fp8on every arm feeding a comparison. LeaveOPENROUTER_PROVIDER_ONLYunset to use the registry's measured endpoint set; setting it explicitly replaces that set and therefore requires a new same-window quality record. - Serving default (2026-08-23; endpoint boundary repaired 2026-08-30): the
turn registry's
providerPreferencesaresort: price,quantizations: ['fp8'], andonly: ['baidu/fp8', 'deepinfra/fp8', 'streamlake/fp8'].agent-provider-configapplies them by default. Environment values override only their matching field, so a deployment changing sort or quantization cannot accidentally erase the measured-provider boundary. Why: the first P8.4 drift-alarm re-check breached (cache-read 26.5% vs the 70% floor over 246 cost-reported drawer turns); on the same afternoon the unpinned price-sorted route failed 0/7 asks (native tool-call markup leaked as text, a 201 s stall) while the fp8-pinned route measured 11/11 and 10/10, with a byte-identical HEAD control confirming the route, not the bytes. The deep completion audit later caught the auction admitting newly cheapest OpenInference: the original seven builder cases fell to 4/7 and recovered to 7/7 on the measured allowlist. The raw before/after evidence and executable binding check live underdocs/audits/eve-phase-2/. - The champion's moat is BEHAVIORAL price, not sticker price: ~85–97% implicit cache-read on its routed endpoints. Any cost comparison that does not measure CACHED-run billed cost is wrong (P7.3: a challenger with a cheaper sticker billed 2.4× because its route served no cache).
The weekly loop#
Order matters: each step feeds the next.
- Cost report + drift alarm. With the BFF up (
:4010):EVE_SMX_ADMIN_TOKEN=<admin-scoped bearer> node tools/eve-smx-cost-report.mjs --check-floors(add--jsonfor the raw model). Read: cost/turn and cache-read rate (trend, not level — a cache collapse is a route change), outcome breakdown, top tool-error shapes, escalation-tier tally, checker-verdict tally.--check-floors(P8.4) compares each serving route against thetelemetryFloorsstamped ineve-smx-ratchet.jsonand exits 3 on a breach — a breach goes straight to the demotion protocol below. The scorecard-staleness gate (a ratchet re-stamp whose story never reached the scorecard) fails closed in CI viaeve-smx-prompt-hash.spec.ts.GET /v1/assistant/telemetry(admin scope) has the durable event counters and a redacted recent sample. - Escalation queue. Every tier-3 escalation carries a mandatory reason slug
(P5.3 — a reasonless escalation throws). A reason slug that recurs week over
week is the distillery's work order: it names a task the cheap model could
not finish and exactly why. Zero escalations with rising checker refusals
means the ladder is not reaching — look at
groundingRefusedturns in the trace before assuming health. - Distillery edits (the P6 discipline, binding):
- Pre-register the expectation BEFORE editing (which case(s) improve, by how much, what must not regress).
- Measure at k=10 on the target battery, not k=3 (point floors at k=3 are a lottery — scorecard P2.11).
- Before attributing any effect to authored bytes, run a BYTE-IDENTICAL control in the same window (P6.4: both "regressions" reproduced with zero authored bytes — it was provider mix). Pair arms by serving ENDPOINT; fp8+price does not pin behavior.
- A regression on any ledger-born lock reverts the edit, full stop.
- Skills: ≤700 heuristic tokens, ≤2 exemplars,
evalCaseIdsmust trace to real deck cases, changelog entry per change — the registry spec enforces all of it. - NEVER edit anything to make a held-out case pass (see rotation rules).
- Deck run (operator-invoked, never CI):
memory_pressure | tail -2first; below 35% free, tear down before running. Postgres up (docker compose -f docker/docker-compose.dev.yml up -d postgres); no Next dev server is needed.- Secrets via a throwaway
/tmp/<task>-env.shsourced from the drawer; verify by length, delete at task end. - Routine cadence: full deck at k=3 in foreground quarter-slices
(
EVE_SMX_EVAL_SLICE=i/4, concurrency 4 — the session-create limiter 429s at 8). Lock re-verification and any promotion/attribution claim: k=10. Runner:pnpm --filter @oshun/bff eval:assistant(the script selects onlyassistant-golden.eval.ts; it never launches the transcript writer). - A killed run salvages: grades are extractable from the log (validated
recipe in scorecard P2.11); finish stragglers via
EVE_SMX_EVAL_CASE_IDS. - Builder families ride the same deck (EVE*EVERYWHERE 2.1–2.4): the
eval's globalSetup clones
oshun_devinto a run-uniqueoshun_eval_PID_TIMESTAMPdatabase (force-dropped by teardown; needs Postgres up and NO live connections to oshun_dev — kill the dev BFF first) and mounts the frozen docs slice (src/assistant/evals/fixtures/docs-index/). When either is missing the docs/workbench cases degrade to their old blocked state and say so with a[builder-eval-env]log line — a run whose builder cases all skipped is NOT a clean run, read the log. The builder deck (deck-builder-cases.ts: the original 33-case Phase-2 slice plus later admitted locks) feeds the 2.4 family Wilson ratchet. A full-deck pool with at least the stamped run count and case diversity is compared automatically; a partial or smaller pool says the floors were not evaluated. The frozen docs slice must be re-cut when the docs estate shifts (docs/audits/EVE_BUILDER_EVAL_ENV_NOTES_2026-08.mdhas the recipe).node tools/eve-everywhere/verify-phase-2-route-repair.mjsadditionally locks the current seven-case route regression, allowlist, parameter-support boundary, and honest partial-run claim. - Read the held-out section of the output SEPARATELY (below).
- Ratchet. Any change to what the model READS (conduct core, tool
descriptions, skills, registry surface) re-stamps
docs/audits/eve-smx-ratchet.json(hash +hashScopenote) and adds the scorecard entry for the why. The hash gate (eve-smx-prompt-hash.spec.ts) fails closed on drift, so a forgotten re-stamp reds CI rather than aging silently. Family floor updates come only from FULL-deck runs at the pins.
Filed weekly evidence contract#
A live command scrolling past in a terminal is not a filed loop. Before a weekly
pass is closed, keep one bundle under docs/audits/eve-smx-weekly/:
- the cost report's exact stdout, the command, exit code, and a SHA-256 in the run manifest;
- the story-drift test file and pass count; if its raw log was not retained, say so explicitly rather than reconstructing one;
- escalation-tier counts, the resulting distillery work-order count, and the glm-4.7-flash re-review verdict with its measured reason;
- every breach's required follow-up. A provisional attribution is not a closed
alarm until a later manifest links back to it and records the remedy; missing
raw follow-up output remains
nullwith a limitation, never synthesized.
The canonical historical bundle is 2026-08-21-run.json →
2026-08-23-follow-up.json. Run pnpm verify:eve-smx-weekly to bind those
manifests to the raw report, ratchet floor, prompt/story gate, scorecard,
checklist, and executable serving-route pin. A new pass should extend the same
contract rather than overwriting either historical record.
Held-out set rotation (P8.2)#
The heldout-* cases are the overfitting alarm: authored after the P6 skill
freeze, never tuned against (enforced by admission rules, the skill-trace spec,
and the misroute audit's exclusion). Rules:
- A held-out decay (a previously-10/10 case dropping while the graded deck stays green) is INVESTIGATED, not fixed. The finding goes in the scorecard.
- The moment anything is tuned against a held-out case — a skill, a checker,
router vocabulary, a prompt byte aimed at its failure — the case retires into
the graded deck and a FRESH held-out case replaces it. The disjointness spec
(
deck-heldout-cases.spec.ts) is the tripwire, and weakening it is never the fix. - Keep the set ≥10 cases across ≥5 families. Baseline (2026-08-18): 12/13 at
k=10; the standing miss (
heldout-docs-internals-absent, unsearched docs-absence claims) is a recorded mechanism gap, not a defect to prompt around.
Monthly route-price note#
Per docs/agents/model-cost-openrouter.md, plus the P7 correction: price the
route by CACHED-run billed cost from real deck-run usage, never the sticker
table. Record: which endpoints served the champion (servedBy in the deck spend
summary), the cache-read rate, and billed $/arm on a fixed battery. Standing
next-review candidate: z-ai/glm-4.7-flash (P7.3 — better on six locks, cheaper
billed, 1.7× faster; eliminated on three narrow lock regressions). Reopen the
tournament when either (a) a harness mechanism ships for the navigate
claim-order / envelope-after-tools shapes, or (b) the champion's route degrades
(cache collapse or a lock going provider-sick).
Demotion triggers → registry rollback#
Trigger on any of, measured not vibed:
- a ledger-born lock failing at k=10 on the pinned champion (re-run once in a different window first — provider-sick ≠ model-sick; scorecard P7.3's ledger-129 note is the precedent);
- a family's Wilson lower bound falling below its
eve-smx-ratchet.jsonfloor on a full-deck run; - production telemetry: cost/turn or cache-read rate stepping (route change),
provider_error/refusedoutcome share rising, or checker-refusal (groundingRefused) spiking without a matching adapter incident.
Rollback protocol, in order:
- Same hour: set the leg's env override to the last-good slug (the registry
documents each leg's
envOverride) — serving changes immediately, no deploy. - Same day: edit the registry pin with a
chosenBynote naming the trigger measurement; add the scorecard entry; if prompt-facing bytes changed in the same window, re-stamp the ratchet. - Then diagnose at leisure with the tournament method (P7.3): staged locks first, k=10, same-window endpoint-paired arms.
Never roll forward and back silently: every registry change carries its
measurement in chosenBy, or it did not happen.
Judge validation protocol (unblocks 7.2 and 8.1)#
The judge leg's pin is provisional until validated against human labels. This is the operator's own act — the harness must never author the labels it is graded by. Pre-registered protocol:
- Corpus: ≥40 conversation transcripts. Capture them with the committed
tool —
EVE_SMX_EVAL_K=4 pnpm --filter @oshun/bff eval:assistant:battery— which runs the ten battery conversations k times on the serving binding and writesdocs/audits/eve-smx-judge-transcripts.jsonlplus the labeling sheetdocs/audits/eve-smx-judge-labels.SHEET.md(blank verdicts; the tool captures, it never labels). Supplement with redacted real member conversations from the telemetry sample when live traffic exists. - Label first, blind: for each transcript, record a PASS/FAIL per the
battery's quality question (does the arc actually help; is the follow-up
coherent) BEFORE running any judge on it. Labels live in
docs/audits/eve-smx-judge-labels.jsonl(one row: transcript ref, class, verdict, one-line why). - Run the judge (registry judge pin, structured output) over the same transcripts; never show it the labels.
- Agreement bar, per class: raw agreement ≥ 0.85 AND Wilson 95% lower bound
≥ 0.70 (the
eval-stats.tshelper computes it). A class below the bar stays advisory — the judge does not gate what it cannot match humans on. - Record the agreement table in the scorecard; flip 7.2/8.1 only then;
promote the 0.11
advisorybattery cases to graded ONLY for qualifying classes.
Machine limits (this Mac, binding)#
16 GB / 2 GB swap. One heavy job at a time; memory_pressure before every deck
run or typecheck; the bff's real typecheck needs
NODE_OPTIONS=--max-old-space-size=10240 npx tsc --noEmit -p tsconfig.json --noCheck false
(its tsconfig has noCheck: true — a bare tsc there is syntax-only). Dev
servers die with the session; deck runs need only Postgres + the in-process app.