Eve library

Eve SMX maintenance runbook (P8.3)

Guide

Revision 1

On this page

How the assistant brain stays excellent on the smallest passing model after the SMX initiative closed. The loop runs on telemetry, not enthusiasm: every step below reads a real instrument and ends in a recorded artifact. Companion records: EVE_SMALL_MODEL_EXCELLENCE_TODOS_2026-08-16.md (the ledger), docs/audits/EVE_SMX_SCORECARD.md (every measurement), docs/audits/eve-smx-ratchet.json (prompt-bytes hash + family floors), docs/audits/EVE_SMX_DECK_SOURCES.md (case provenance + failure taxonomies).

Standing pins#

  • Turn leg: deepseek/deepseek-v4-flash-0731 (champion). Escalation + judge: deepseek/deepseek-v4-pro-0813 (judge pin is provisional — see "Judge validation" below). Pins live in apps/oshun/bff/src/assistant/model-registry.ts with price snapshots and chosenBy provenance; env overrides (OSHUN_ASSISTANT_OPENROUTER_MODEL, OSHUN_ASSISTANT_ESCALATION_MODEL, OSHUN_ASSISTANT_JUDGE_MODEL) win over pins and are the rollback lever.
  • Measurement pins, always: OPENROUTER_PROVIDER_SORT=price; OPENROUTER_PROVIDER_QUANTIZATIONS=fp8 on every arm feeding a comparison. Leave OPENROUTER_PROVIDER_ONLY unset to use the registry's measured endpoint set; setting it explicitly replaces that set and therefore requires a new same-window quality record.
  • Serving default (2026-08-23; endpoint boundary repaired 2026-08-30): the turn registry's providerPreferences are sort: price, quantizations: ['fp8'], and only: ['baidu/fp8', 'deepinfra/fp8', 'streamlake/fp8']. agent-provider-config applies them by default. Environment values override only their matching field, so a deployment changing sort or quantization cannot accidentally erase the measured-provider boundary. Why: the first P8.4 drift-alarm re-check breached (cache-read 26.5% vs the 70% floor over 246 cost-reported drawer turns); on the same afternoon the unpinned price-sorted route failed 0/7 asks (native tool-call markup leaked as text, a 201 s stall) while the fp8-pinned route measured 11/11 and 10/10, with a byte-identical HEAD control confirming the route, not the bytes. The deep completion audit later caught the auction admitting newly cheapest OpenInference: the original seven builder cases fell to 4/7 and recovered to 7/7 on the measured allowlist. The raw before/after evidence and executable binding check live under docs/audits/eve-phase-2/.
  • The champion's moat is BEHAVIORAL price, not sticker price: ~85–97% implicit cache-read on its routed endpoints. Any cost comparison that does not measure CACHED-run billed cost is wrong (P7.3: a challenger with a cheaper sticker billed 2.4× because its route served no cache).

The weekly loop#

Order matters: each step feeds the next.

  1. Cost report + drift alarm. With the BFF up (:4010): EVE_SMX_ADMIN_TOKEN=<admin-scoped bearer> node tools/eve-smx-cost-report.mjs --check-floors (add --json for the raw model). Read: cost/turn and cache-read rate (trend, not level — a cache collapse is a route change), outcome breakdown, top tool-error shapes, escalation-tier tally, checker-verdict tally. --check-floors (P8.4) compares each serving route against the telemetryFloors stamped in eve-smx-ratchet.json and exits 3 on a breach — a breach goes straight to the demotion protocol below. The scorecard-staleness gate (a ratchet re-stamp whose story never reached the scorecard) fails closed in CI via eve-smx-prompt-hash.spec.ts. GET /v1/assistant/telemetry (admin scope) has the durable event counters and a redacted recent sample.
  2. Escalation queue. Every tier-3 escalation carries a mandatory reason slug (P5.3 — a reasonless escalation throws). A reason slug that recurs week over week is the distillery's work order: it names a task the cheap model could not finish and exactly why. Zero escalations with rising checker refusals means the ladder is not reaching — look at groundingRefused turns in the trace before assuming health.
  3. Distillery edits (the P6 discipline, binding):
    • Pre-register the expectation BEFORE editing (which case(s) improve, by how much, what must not regress).
    • Measure at k=10 on the target battery, not k=3 (point floors at k=3 are a lottery — scorecard P2.11).
    • Before attributing any effect to authored bytes, run a BYTE-IDENTICAL control in the same window (P6.4: both "regressions" reproduced with zero authored bytes — it was provider mix). Pair arms by serving ENDPOINT; fp8+price does not pin behavior.
    • A regression on any ledger-born lock reverts the edit, full stop.
    • Skills: ≤700 heuristic tokens, ≤2 exemplars, evalCaseIds must trace to real deck cases, changelog entry per change — the registry spec enforces all of it.
    • NEVER edit anything to make a held-out case pass (see rotation rules).
  4. Deck run (operator-invoked, never CI):
    • memory_pressure | tail -2 first; below 35% free, tear down before running. Postgres up (docker compose -f docker/docker-compose.dev.yml up -d postgres); no Next dev server is needed.
    • Secrets via a throwaway /tmp/<task>-env.sh sourced from the drawer; verify by length, delete at task end.
    • Routine cadence: full deck at k=3 in foreground quarter-slices (EVE_SMX_EVAL_SLICE=i/4, concurrency 4 — the session-create limiter 429s at 8). Lock re-verification and any promotion/attribution claim: k=10. Runner: pnpm --filter @oshun/bff eval:assistant (the script selects only assistant-golden.eval.ts; it never launches the transcript writer).
    • A killed run salvages: grades are extractable from the log (validated recipe in scorecard P2.11); finish stragglers via EVE_SMX_EVAL_CASE_IDS.
    • Builder families ride the same deck (EVE*EVERYWHERE 2.1–2.4): the eval's globalSetup clones oshun_dev into a run-unique oshun_eval_PID_TIMESTAMP database (force-dropped by teardown; needs Postgres up and NO live connections to oshun_dev — kill the dev BFF first) and mounts the frozen docs slice (src/assistant/evals/fixtures/docs-index/). When either is missing the docs/workbench cases degrade to their old blocked state and say so with a [builder-eval-env] log line — a run whose builder cases all skipped is NOT a clean run, read the log. The builder deck (deck-builder-cases.ts: the original 33-case Phase-2 slice plus later admitted locks) feeds the 2.4 family Wilson ratchet. A full-deck pool with at least the stamped run count and case diversity is compared automatically; a partial or smaller pool says the floors were not evaluated. The frozen docs slice must be re-cut when the docs estate shifts (docs/audits/EVE_BUILDER_EVAL_ENV_NOTES_2026-08.md has the recipe). node tools/eve-everywhere/verify-phase-2-route-repair.mjs additionally locks the current seven-case route regression, allowlist, parameter-support boundary, and honest partial-run claim.
    • Read the held-out section of the output SEPARATELY (below).
  5. Ratchet. Any change to what the model READS (conduct core, tool descriptions, skills, registry surface) re-stamps docs/audits/eve-smx-ratchet.json (hash + hashScope note) and adds the scorecard entry for the why. The hash gate (eve-smx-prompt-hash.spec.ts) fails closed on drift, so a forgotten re-stamp reds CI rather than aging silently. Family floor updates come only from FULL-deck runs at the pins.

Filed weekly evidence contract#

A live command scrolling past in a terminal is not a filed loop. Before a weekly pass is closed, keep one bundle under docs/audits/eve-smx-weekly/:

  • the cost report's exact stdout, the command, exit code, and a SHA-256 in the run manifest;
  • the story-drift test file and pass count; if its raw log was not retained, say so explicitly rather than reconstructing one;
  • escalation-tier counts, the resulting distillery work-order count, and the glm-4.7-flash re-review verdict with its measured reason;
  • every breach's required follow-up. A provisional attribution is not a closed alarm until a later manifest links back to it and records the remedy; missing raw follow-up output remains null with a limitation, never synthesized.

The canonical historical bundle is 2026-08-21-run.json → 2026-08-23-follow-up.json. Run pnpm verify:eve-smx-weekly to bind those manifests to the raw report, ratchet floor, prompt/story gate, scorecard, checklist, and executable serving-route pin. A new pass should extend the same contract rather than overwriting either historical record.

Held-out set rotation (P8.2)#

The heldout-* cases are the overfitting alarm: authored after the P6 skill freeze, never tuned against (enforced by admission rules, the skill-trace spec, and the misroute audit's exclusion). Rules:

  • A held-out decay (a previously-10/10 case dropping while the graded deck stays green) is INVESTIGATED, not fixed. The finding goes in the scorecard.
  • The moment anything is tuned against a held-out case — a skill, a checker, router vocabulary, a prompt byte aimed at its failure — the case retires into the graded deck and a FRESH held-out case replaces it. The disjointness spec (deck-heldout-cases.spec.ts) is the tripwire, and weakening it is never the fix.
  • Keep the set ≥10 cases across ≥5 families. Baseline (2026-08-18): 12/13 at k=10; the standing miss (heldout-docs-internals-absent, unsearched docs-absence claims) is a recorded mechanism gap, not a defect to prompt around.

Monthly route-price note#

Per docs/agents/model-cost-openrouter.md, plus the P7 correction: price the route by CACHED-run billed cost from real deck-run usage, never the sticker table. Record: which endpoints served the champion (servedBy in the deck spend summary), the cache-read rate, and billed $/arm on a fixed battery. Standing next-review candidate: z-ai/glm-4.7-flash (P7.3 — better on six locks, cheaper billed, 1.7× faster; eliminated on three narrow lock regressions). Reopen the tournament when either (a) a harness mechanism ships for the navigate claim-order / envelope-after-tools shapes, or (b) the champion's route degrades (cache collapse or a lock going provider-sick).

Demotion triggers → registry rollback#

Trigger on any of, measured not vibed:

  • a ledger-born lock failing at k=10 on the pinned champion (re-run once in a different window first — provider-sick ≠ model-sick; scorecard P7.3's ledger-129 note is the precedent);
  • a family's Wilson lower bound falling below its eve-smx-ratchet.json floor on a full-deck run;
  • production telemetry: cost/turn or cache-read rate stepping (route change), provider_error/refused outcome share rising, or checker-refusal (groundingRefused) spiking without a matching adapter incident.

Rollback protocol, in order:

  1. Same hour: set the leg's env override to the last-good slug (the registry documents each leg's envOverride) — serving changes immediately, no deploy.
  2. Same day: edit the registry pin with a chosenBy note naming the trigger measurement; add the scorecard entry; if prompt-facing bytes changed in the same window, re-stamp the ratchet.
  3. Then diagnose at leisure with the tournament method (P7.3): staged locks first, k=10, same-window endpoint-paired arms.

Never roll forward and back silently: every registry change carries its measurement in chosenBy, or it did not happen.

Judge validation protocol (unblocks 7.2 and 8.1)#

The judge leg's pin is provisional until validated against human labels. This is the operator's own act — the harness must never author the labels it is graded by. Pre-registered protocol:

  1. Corpus: ≥40 conversation transcripts. Capture them with the committed tool — EVE_SMX_EVAL_K=4 pnpm --filter @oshun/bff eval:assistant:battery — which runs the ten battery conversations k times on the serving binding and writes docs/audits/eve-smx-judge-transcripts.jsonl plus the labeling sheet docs/audits/eve-smx-judge-labels.SHEET.md (blank verdicts; the tool captures, it never labels). Supplement with redacted real member conversations from the telemetry sample when live traffic exists.
  2. Label first, blind: for each transcript, record a PASS/FAIL per the battery's quality question (does the arc actually help; is the follow-up coherent) BEFORE running any judge on it. Labels live in docs/audits/eve-smx-judge-labels.jsonl (one row: transcript ref, class, verdict, one-line why).
  3. Run the judge (registry judge pin, structured output) over the same transcripts; never show it the labels.
  4. Agreement bar, per class: raw agreement ≥ 0.85 AND Wilson 95% lower bound ≥ 0.70 (the eval-stats.ts helper computes it). A class below the bar stays advisory — the judge does not gate what it cannot match humans on.
  5. Record the agreement table in the scorecard; flip 7.2/8.1 only then; promote the 0.11 advisory battery cases to graded ONLY for qualifying classes.

Machine limits (this Mac, binding)#

16 GB / 2 GB swap. One heavy job at a time; memory_pressure before every deck run or typecheck; the bff's real typecheck needs NODE_OPTIONS=--max-old-space-size=10240 npx tsc --noEmit -p tsconfig.json --noCheck false (its tsconfig has noCheck: true — a bare tsc there is syntax-only). Dev servers die with the session; deck runs need only Postgres + the in-process app.