# Eve SMX maintenance runbook (P8.3) How the assistant brain stays excellent on the smallest passing model after the SMX initiative closed. The loop runs on telemetry, not enthusiasm: every step below reads a real instrument and ends in a recorded artifact. Companion records: `EVE_SMALL_MODEL_EXCELLENCE_TODOS_2026-08-16.md` (the ledger), `docs/audits/EVE_SMX_SCORECARD.md` (every measurement), `docs/audits/eve-smx-ratchet.json` (prompt-bytes hash + family floors), `docs/audits/EVE_SMX_DECK_SOURCES.md` (case provenance + failure taxonomies). ## Standing pins - Turn leg: `deepseek/deepseek-v4-flash-0731` (champion). Escalation + judge: `deepseek/deepseek-v4-pro-0813` (judge pin is provisional — see "Judge validation" below). Pins live in `apps/oshun/bff/src/assistant/model-registry.ts` with price snapshots and `chosenBy` provenance; env overrides (`OSHUN_ASSISTANT_OPENROUTER_MODEL`, `OSHUN_ASSISTANT_ESCALATION_MODEL`, `OSHUN_ASSISTANT_JUDGE_MODEL`) win over pins and are the rollback lever. - Measurement pins, always: `OPENROUTER_PROVIDER_SORT=price`; `OPENROUTER_PROVIDER_QUANTIZATIONS=fp8` on every arm feeding a comparison. Leave `OPENROUTER_PROVIDER_ONLY` unset to use the registry's measured endpoint set; setting it explicitly replaces that set and therefore requires a new same-window quality record. - **Serving default (2026-08-23; endpoint boundary repaired 2026-08-30):** the turn registry's `providerPreferences` are `sort: price`, `quantizations: ['fp8']`, and `only: ['baidu/fp8', 'deepinfra/fp8', 'streamlake/fp8']`. `agent-provider-config` applies them by default. Environment values override only their matching field, so a deployment changing sort or quantization cannot accidentally erase the measured-provider boundary. Why: the first P8.4 drift-alarm re-check breached (cache-read 26.5% vs the 70% floor over 246 cost-reported drawer turns); on the same afternoon the unpinned price-sorted route failed 0/7 asks (native tool-call markup leaked as text, a 201 s stall) while the fp8-pinned route measured 11/11 and 10/10, with a byte-identical HEAD control confirming the route, not the bytes. The deep completion audit later caught the auction admitting newly cheapest OpenInference: the original seven builder cases fell to 4/7 and recovered to 7/7 on the measured allowlist. The raw before/after evidence and executable binding check live under `docs/audits/eve-phase-2/`. - The champion's moat is BEHAVIORAL price, not sticker price: ~85–97% implicit cache-read on its routed endpoints. Any cost comparison that does not measure CACHED-run billed cost is wrong (P7.3: a challenger with a cheaper sticker billed 2.4× because its route served no cache). ## The weekly loop Order matters: each step feeds the next. 1. **Cost report + drift alarm.** With the BFF up (`:4010`): `EVE_SMX_ADMIN_TOKEN= node tools/eve-smx-cost-report.mjs --check-floors` (add `--json` for the raw model). Read: cost/turn and cache-read rate (trend, not level — a cache collapse is a route change), outcome breakdown, top tool-error shapes, escalation-tier tally, checker-verdict tally. `--check-floors` (P8.4) compares each serving route against the `telemetryFloors` stamped in `eve-smx-ratchet.json` and exits 3 on a breach — a breach goes straight to the demotion protocol below. The scorecard-staleness gate (a ratchet re-stamp whose story never reached the scorecard) fails closed in CI via `eve-smx-prompt-hash.spec.ts`. `GET /v1/assistant/telemetry` (admin scope) has the durable event counters and a redacted recent sample. 2. **Escalation queue.** Every tier-3 escalation carries a mandatory reason slug (P5.3 — a reasonless escalation throws). A reason slug that recurs week over week is the distillery's work order: it names a task the cheap model could not finish and exactly why. Zero escalations with rising checker refusals means the ladder is not reaching — look at `groundingRefused` turns in the trace before assuming health. 3. **Distillery edits** (the P6 discipline, binding): - Pre-register the expectation BEFORE editing (which case(s) improve, by how much, what must not regress). - Measure at k=10 on the target battery, not k=3 (point floors at k=3 are a lottery — scorecard P2.11). - Before attributing any effect to authored bytes, run a BYTE-IDENTICAL control in the same window (P6.4: both "regressions" reproduced with zero authored bytes — it was provider mix). Pair arms by serving ENDPOINT; fp8+price does not pin behavior. - A regression on any ledger-born lock reverts the edit, full stop. - Skills: ≤700 heuristic tokens, ≤2 exemplars, `evalCaseIds` must trace to real deck cases, changelog entry per change — the registry spec enforces all of it. - NEVER edit anything to make a held-out case pass (see rotation rules). 4. **Deck run** (operator-invoked, never CI): - `memory_pressure | tail -2` first; below 35% free, tear down before running. Postgres up (`docker compose -f docker/docker-compose.dev.yml up -d postgres`); no Next dev server is needed. - Secrets via a throwaway `/tmp/-env.sh` sourced from the drawer; verify by length, delete at task end. - Routine cadence: full deck at k=3 in foreground quarter-slices (`EVE_SMX_EVAL_SLICE=i/4`, concurrency 4 — the session-create limiter 429s at 8). Lock re-verification and any promotion/attribution claim: k=10. Runner: `pnpm --filter @oshun/bff eval:assistant` (the script selects only `assistant-golden.eval.ts`; it never launches the transcript writer). - A killed run salvages: grades are extractable from the log (validated recipe in scorecard P2.11); finish stragglers via `EVE_SMX_EVAL_CASE_IDS`. - **Builder families ride the same deck** (EVE\*EVERYWHERE 2.1–2.4): the eval's globalSetup clones `oshun_dev` into a run-unique `oshun_eval_PID_TIMESTAMP` database (force-dropped by teardown; needs Postgres up and NO live connections to oshun_dev — kill the dev BFF first) and mounts the frozen docs slice (`src/assistant/evals/fixtures/docs-index/`). When either is missing the docs/workbench cases degrade to their old blocked state and say so with a `[builder-eval-env]` log line — a run whose builder cases all skipped is NOT a clean run, read the log. The builder deck (`deck-builder-cases.ts`: the original 33-case Phase-2 slice plus later admitted locks) feeds the 2.4 family Wilson ratchet. A full-deck pool with at least the stamped run count and case diversity is compared automatically; a partial or smaller pool says the floors were not evaluated. The frozen docs slice must be re-cut when the docs estate shifts (`docs/audits/EVE_BUILDER_EVAL_ENV_NOTES_2026-08.md` has the recipe). `node tools/eve-everywhere/verify-phase-2-route-repair.mjs` additionally locks the current seven-case route regression, allowlist, parameter-support boundary, and honest partial-run claim. - Read the held-out section of the output SEPARATELY (below). 5. **Ratchet.** Any change to what the model READS (conduct core, tool descriptions, skills, registry surface) re-stamps `docs/audits/eve-smx-ratchet.json` (hash + `hashScope` note) and adds the scorecard entry for the why. The hash gate (`eve-smx-prompt-hash.spec.ts`) fails closed on drift, so a forgotten re-stamp reds CI rather than aging silently. Family floor updates come only from FULL-deck runs at the pins. ### Filed weekly evidence contract A live command scrolling past in a terminal is not a filed loop. Before a weekly pass is closed, keep one bundle under `docs/audits/eve-smx-weekly/`: - the cost report's exact stdout, the command, exit code, and a SHA-256 in the run manifest; - the story-drift test file and pass count; if its raw log was not retained, say so explicitly rather than reconstructing one; - escalation-tier counts, the resulting distillery work-order count, and the glm-4.7-flash re-review verdict with its measured reason; - every breach's required follow-up. A provisional attribution is not a closed alarm until a later manifest links back to it and records the remedy; missing raw follow-up output remains `null` with a limitation, never synthesized. The canonical historical bundle is `2026-08-21-run.json` → `2026-08-23-follow-up.json`. Run `pnpm verify:eve-smx-weekly` to bind those manifests to the raw report, ratchet floor, prompt/story gate, scorecard, checklist, and executable serving-route pin. A new pass should extend the same contract rather than overwriting either historical record. ## Held-out set rotation (P8.2) The `heldout-*` cases are the overfitting alarm: authored after the P6 skill freeze, never tuned against (enforced by admission rules, the skill-trace spec, and the misroute audit's exclusion). Rules: - A held-out decay (a previously-10/10 case dropping while the graded deck stays green) is INVESTIGATED, not fixed. The finding goes in the scorecard. - The moment anything is tuned against a held-out case — a skill, a checker, router vocabulary, a prompt byte aimed at its failure — the case retires into the graded deck and a FRESH held-out case replaces it. The disjointness spec (`deck-heldout-cases.spec.ts`) is the tripwire, and weakening it is never the fix. - Keep the set ≥10 cases across ≥5 families. Baseline (2026-08-18): 12/13 at k=10; the standing miss (`heldout-docs-internals-absent`, unsearched docs-absence claims) is a recorded mechanism gap, not a defect to prompt around. ## Monthly route-price note Per `docs/agents/model-cost-openrouter.md`, plus the P7 correction: price the route by CACHED-run billed cost from real deck-run usage, never the sticker table. Record: which endpoints served the champion (`servedBy` in the deck spend summary), the cache-read rate, and billed $/arm on a fixed battery. Standing next-review candidate: `z-ai/glm-4.7-flash` (P7.3 — better on six locks, cheaper billed, 1.7× faster; eliminated on three narrow lock regressions). Reopen the tournament when either (a) a harness mechanism ships for the navigate claim-order / envelope-after-tools shapes, or (b) the champion's route degrades (cache collapse or a lock going provider-sick). ## Demotion triggers → registry rollback Trigger on any of, measured not vibed: - a ledger-born lock failing at k=10 on the pinned champion (re-run once in a different window first — provider-sick ≠ model-sick; scorecard P7.3's ledger-129 note is the precedent); - a family's Wilson lower bound falling below its `eve-smx-ratchet.json` floor on a full-deck run; - production telemetry: cost/turn or cache-read rate stepping (route change), `provider_error`/`refused` outcome share rising, or checker-refusal (`groundingRefused`) spiking without a matching adapter incident. Rollback protocol, in order: 1. Same hour: set the leg's env override to the last-good slug (the registry documents each leg's `envOverride`) — serving changes immediately, no deploy. 2. Same day: edit the registry pin with a `chosenBy` note naming the trigger measurement; add the scorecard entry; if prompt-facing bytes changed in the same window, re-stamp the ratchet. 3. Then diagnose at leisure with the tournament method (P7.3): staged locks first, k=10, same-window endpoint-paired arms. Never roll forward and back silently: every registry change carries its measurement in `chosenBy`, or it did not happen. ## Judge validation protocol (unblocks 7.2 and 8.1) The judge leg's pin is provisional until validated against human labels. This is the operator's own act — the harness must never author the labels it is graded by. Pre-registered protocol: 1. **Corpus:** ≥40 conversation transcripts. Capture them with the committed tool — `EVE_SMX_EVAL_K=4 pnpm --filter @oshun/bff eval:assistant:battery` — which runs the ten battery conversations k times on the serving binding and writes `docs/audits/eve-smx-judge-transcripts.jsonl` plus the labeling sheet `docs/audits/eve-smx-judge-labels.SHEET.md` (blank verdicts; the tool captures, it never labels). Supplement with redacted real member conversations from the telemetry sample when live traffic exists. 2. **Label first, blind:** for each transcript, record a PASS/FAIL per the battery's quality question (does the arc actually help; is the follow-up coherent) BEFORE running any judge on it. Labels live in `docs/audits/eve-smx-judge-labels.jsonl` (one row: transcript ref, class, verdict, one-line why). 3. **Run the judge** (registry judge pin, structured output) over the same transcripts; never show it the labels. 4. **Agreement bar, per class:** raw agreement ≥ 0.85 AND Wilson 95% lower bound ≥ 0.70 (the `eval-stats.ts` helper computes it). A class below the bar stays advisory — the judge does not gate what it cannot match humans on. 5. **Record** the agreement table in the scorecard; flip 7.2/8.1 only then; promote the 0.11 `advisory` battery cases to graded ONLY for qualifying classes. ## Machine limits (this Mac, binding) 16 GB / 2 GB swap. One heavy job at a time; `memory_pressure` before every deck run or typecheck; the bff's real typecheck needs `NODE_OPTIONS=--max-old-space-size=10240 npx tsc --noEmit -p tsconfig.json --noCheck false` (its tsconfig has `noCheck: true` — a bare tsc there is syntax-only). Dev servers die with the session; deck runs need only Postgres + the in-process app.