Two consecutive runs of the admin-turn battery
(apps/oshun/web/e2e-inspect/admin-turn-battery.spec.ts), 2026-08-19, champion
binding (deepseek/deepseek-v4-flash-0731 via OpenRouter,
OPENROUTER_PROVIDER_SORT=price), admin app :3020, BFF :4010, dev Postgres.
Ten workbench-write intents, three phrasings each (natural → explicit-tool →
imperative-tool), every parked card DECLINED; both runs ended with
work_item/decision/thread row counts unchanged (hard-asserted in the
spec). Raw reports: docs/audits/eve-builder-affordance/report-348795.json (run
A, 14.0m) and report-260394.json (run B, 9.6m).
Ask-to-card, per intent#
| intent | tool | run A | run B | stable? |
|---|---|---|---|---|
| create-task | create_work_item | 2nd ask | 2nd ask | yes |
| create-issue | create_work_item | 1st ask | 1st ask | yes |
| update-status | update_work_item | 2nd ask | 2nd ask | yes |
| update-priority | update_work_item | 2nd ask | 2nd ask | yes |
| draft-decision | draft_decision | NEVER | 1st ask | no — model-side, see below |
| transition-decision | transition_decision | 2nd ask | 1st ask | no — title-verb, see below |
| feature-proposal | create_feature_proposal | 1st ask | 1st ask | yes |
| content-brief | create_content_brief | 2nd ask | 2nd ask | yes |
| dispatch-brief | dispatch_content_brief | grounded (no such brief) | grounded | yes |
| open-thread | open_thread | NEVER | NEVER | yes |
Totals — run A: 7/9 card-expected intents carded (first-ask 2), 11 capability denials. Run B: 8/9 (first-ask 4), 8 denials. Zero fabrication-risk outcomes: the ghost-brief dispatch was answered honestly both times, and nothing was ever written without a card.
Root cause (proven, three instruments agreeing)#
The denials are the ROUTER's, not the persona's. Verified by (1) running
resolveTaskFamily (pure) on the exact battery phrasings, (2) joining the BFF's
per-turn assistant task family routed telemetry to each intent's unique
session id, and (3) the behavioral outcomes:
- Natural write phrasings that carry a workbench NOUN but a verb missing from
WORKBENCH_MUTATION_VERBS(log,open,move,bump,capture) route to workbench-read, whose allowlist holds no write tools. The execution clamp then makes the model HONESTLY read-only: run B's open-thread reply — "I only have read-only workbench tools … so I can't create anything" — is a correct description of its clamped turn. Every "read-only"/"can't create" denial in both runs sits on a workbench-read-routed turn. - Explicit-tool phrasings (
create_work_item,update_work_item,create_content_briefby name) defeat BOTH regexes — underscores are word characters, so\bwork items?\band\bdraft\bnever match inside underscored names — and fall through to general, which has no skill and therefore NO CLAMP. That is why the second ask "rescues" five intents: not better prompting, an unclamped toolset. create-issuecards on the FIRST ask for the same reason: "issue" is absent fromWORKBENCH_NOUNS, so the natural phrasing skips Tier 2 entirely and lands unclamped in general.- Correctly-routed workbench-write turns behave well:
feature-proposal(noun + "propose"/"file") carded first-ask in both runs;draft-decisionroutes to workbench-write on the natural ask and carded first-ask in run B. Its run-A never-fire (two denials, zero tool calls, tool offered) is a WITHIN-FAMILY model refusal — the residual affordance flake worked-examples should fix (workbench-write ships zero exemplars). - Title-verb landmine: run B's transition ask routed workbench-write only because the DB-resolved decision title — "Self-accepting draft" — contains the verb "draft". The router reads quoted titles as routing text; a title can flip the family in either direction.
- The retry ladder can DOWNGRADE routing: draft-decision's explicit second phrasing ("call the draft_decision tool…") routes workbench-read (underscore defeats the verb match; "decision" in the quoted title supplies the noun) while its natural first phrasing routes workbench-write.
What Phase 1 should therefore change#
- Router coverage (primary): add the missing mutation verbs and the missing
nouns (
issues?, and match underscored registered tool names as an explicit routing signal to the family that carries the tool). - Keep the general-family rescue in mind when measuring: a fix that merely re-routes explicit-tool asks from general to workbench-write must not LOWER carding (general is unclamped today; workbench-write is clamped but carries every write tool, so parity is expected — measure it).
- Worked examples for workbench-write (zero exemplars today) to close the run-A-style within-family refusal.
- Decide and document the quoted-title behavior.
Exit bar for 1.5 stays as written in the TODOS: ≥8/10 first-ask cards, zero capability denials.
Post-fix re-run (EVE_EVERYWHERE 1.5 — run 705227, 2026-08-19)#
Same instrument, same binding, after the Phase-1 changes (router verb/noun/
tool-name coverage + quoted-title stripping; workbench-write allowlist grown to
the full read surface; skill v2 with served exemplars; draft_decision's
direct-drafting example). Raw report:
docs/audits/eve-builder-affordance/report-705227.json.
| intent | baseline A / B | post-fix | consulted (first ask) |
|---|---|---|---|
| create-task | 2nd / 2nd | 1st | create_work_item |
| create-issue | 1st / 1st | 1st | create_work_item |
| update-status | 2nd / 2nd | 1st | list_work_items → update_work_item |
| update-priority | 2nd / 2nd | 1st | list_work_items → update_work_item |
| draft-decision | NEVER / 1st | 1st | draft_decision |
| transition-decision | 2nd / 1st | 1st | list_decisions → transition_decision |
| feature-proposal | 1st / 1st | 1st | create_feature_proposal |
| content-brief | 2nd / 2nd | 1st | create_content_brief |
| dispatch-brief | grounded / grounded | grounded (5 real lookups, honest miss) | list_work_items ×5 |
| open-thread | NEVER / NEVER | 1st | open_thread |
Totals: 9/9 card-expected intents on the FIRST ask (baseline: 2 and 4);
capability denials 11 / 8 → 0; never-fired 2 / 1 → 0; fabrication risk 0
throughout; every decline resolved; work_item/decision/thread row counts
unchanged (hard assert). Wall clock 14.0m → 6.4m — the retry ladder stopped
being load-bearing. The 1.5 exit bar (≥8/10 first-ask, zero capability denials)
is exceeded; the regression lock is admin-affordance-floor.spec.ts (1.6).
Completion re-audit (2026-08-28 — schema 2)#
The original three reports remain valid evidence of their conversational
outcomes, but their schemaVersion: 1 database proof was only row counts. It
cannot rule out an unauthorized in-place update, and the raw files do not bind
the requested provider/model configuration. The instrument also inherited its
open-task and draft-decision prerequisites from the developer database, could
reuse an old assistant bubble if a turn rendered no new reply, and initialized
the ghost-brief outcome as grounded before observing either a tool call or a
reply. Those are measurement defects, not product passes.
The repaired schema-2 instrument now:
- fingerprints every complete row of
work_item,decision,thread,thread_message, andworkbench_event, with a calibrated negative control proving that an in-place update changes the digest while the row count stays fixed; - installs one disposable open task and draft decision, snapshots around the
decline-only turns, removes those exact rows in
finally, and proves the harness restored the outer database state; - requires one newly rendered assistant reply per ask and classifies the ghost
miss as grounded only when the exact missing brief is named, explicit absence
language is visible, and
list_work_itemsappears in the consulted-tool trail; an unrelated tool consultation cannot satisfy the contract; - records the requested provider, model, provider sort, and quantization in each raw report (the live BFF process environment was checked independently for the same values).
Two consecutive live runs on an isolated clone of the dev Postgres used
OpenRouter deepseek/deepseek-v4-flash-0731, provider sort price, and
quantization fp8:
| run | schema | mutation cards | first ask | grounded ghost | denials | full-state equal | fixture restored |
|---|---|---|---|---|---|---|---|
report-441275.json |
2 | 9/9 | 9/9 | yes | 0 | yes | yes |
report-623442.json |
2 | 9/9 | 9/9 | yes | 0 | yes | yes |
Every parked card resolved as declined; both reports contain the same ten intent
ids in the same order; neither contains an ungrounded, fabrication-risk,
never-fired, or no-prerequisite outcome. Schema 1 and schema 2 are intentionally
not raw-schema comparable. The baseline-to-post-fix behavioral comparison above
remains useful, while the schema-2 pair is the current reproducibility and
no-write proof. tools/eve-everywhere/verify-admin-affordance-reports.mts
recomputes all five reports' totals, denial classifications, historical
agreement, current outcome parity, exact ghost grounding, and full-state
equality from the checked JSON rather than trusting the summary tables.