Disciplines · Audits

Eve builder-plane affordance baseline (EVE_EVERYWHERE 0.2)

Totals — run A: 7/9 card-expected intents carded (first-ask 2), 11 capability denials.

5sections6 minread

On this page

Two consecutive runs of the admin-turn battery (apps/oshun/web/e2e-inspect/admin-turn-battery.spec.ts), 2026-08-19, champion binding (deepseek/deepseek-v4-flash-0731 via OpenRouter, OPENROUTER_PROVIDER_SORT=price), admin app :3020, BFF :4010, dev Postgres. Ten workbench-write intents, three phrasings each (natural → explicit-tool → imperative-tool), every parked card DECLINED; both runs ended with work_item/decision/thread row counts unchanged (hard-asserted in the spec). Raw reports: docs/audits/eve-builder-affordance/report-348795.json (run A, 14.0m) and report-260394.json (run B, 9.6m).

Ask-to-card, per intent#

intent tool run A run B stable?
create-task create_work_item 2nd ask 2nd ask yes
create-issue create_work_item 1st ask 1st ask yes
update-status update_work_item 2nd ask 2nd ask yes
update-priority update_work_item 2nd ask 2nd ask yes
draft-decision draft_decision NEVER 1st ask no — model-side, see below
transition-decision transition_decision 2nd ask 1st ask no — title-verb, see below
feature-proposal create_feature_proposal 1st ask 1st ask yes
content-brief create_content_brief 2nd ask 2nd ask yes
dispatch-brief dispatch_content_brief grounded (no such brief) grounded yes
open-thread open_thread NEVER NEVER yes

Totals — run A: 7/9 card-expected intents carded (first-ask 2), 11 capability denials. Run B: 8/9 (first-ask 4), 8 denials. Zero fabrication-risk outcomes: the ghost-brief dispatch was answered honestly both times, and nothing was ever written without a card.

Root cause (proven, three instruments agreeing)#

The denials are the ROUTER's, not the persona's. Verified by (1) running resolveTaskFamily (pure) on the exact battery phrasings, (2) joining the BFF's per-turn assistant task family routed telemetry to each intent's unique session id, and (3) the behavioral outcomes:

  • Natural write phrasings that carry a workbench NOUN but a verb missing from WORKBENCH_MUTATION_VERBS (log, open, move, bump, capture) route to workbench-read, whose allowlist holds no write tools. The execution clamp then makes the model HONESTLY read-only: run B's open-thread reply — "I only have read-only workbench tools … so I can't create anything" — is a correct description of its clamped turn. Every "read-only"/"can't create" denial in both runs sits on a workbench-read-routed turn.
  • Explicit-tool phrasings (create_work_item, update_work_item, create_content_brief by name) defeat BOTH regexes — underscores are word characters, so \bwork items?\b and \bdraft\b never match inside underscored names — and fall through to general, which has no skill and therefore NO CLAMP. That is why the second ask "rescues" five intents: not better prompting, an unclamped toolset.
  • create-issue cards on the FIRST ask for the same reason: "issue" is absent from WORKBENCH_NOUNS, so the natural phrasing skips Tier 2 entirely and lands unclamped in general.
  • Correctly-routed workbench-write turns behave well: feature-proposal (noun + "propose"/"file") carded first-ask in both runs; draft-decision routes to workbench-write on the natural ask and carded first-ask in run B. Its run-A never-fire (two denials, zero tool calls, tool offered) is a WITHIN-FAMILY model refusal — the residual affordance flake worked-examples should fix (workbench-write ships zero exemplars).
  • Title-verb landmine: run B's transition ask routed workbench-write only because the DB-resolved decision title — "Self-accepting draft" — contains the verb "draft". The router reads quoted titles as routing text; a title can flip the family in either direction.
  • The retry ladder can DOWNGRADE routing: draft-decision's explicit second phrasing ("call the draft_decision tool…") routes workbench-read (underscore defeats the verb match; "decision" in the quoted title supplies the noun) while its natural first phrasing routes workbench-write.

What Phase 1 should therefore change#

  1. Router coverage (primary): add the missing mutation verbs and the missing nouns (issues?, and match underscored registered tool names as an explicit routing signal to the family that carries the tool).
  2. Keep the general-family rescue in mind when measuring: a fix that merely re-routes explicit-tool asks from general to workbench-write must not LOWER carding (general is unclamped today; workbench-write is clamped but carries every write tool, so parity is expected — measure it).
  3. Worked examples for workbench-write (zero exemplars today) to close the run-A-style within-family refusal.
  4. Decide and document the quoted-title behavior.

Exit bar for 1.5 stays as written in the TODOS: ≥8/10 first-ask cards, zero capability denials.

Post-fix re-run (EVE_EVERYWHERE 1.5 — run 705227, 2026-08-19)#

Same instrument, same binding, after the Phase-1 changes (router verb/noun/ tool-name coverage + quoted-title stripping; workbench-write allowlist grown to the full read surface; skill v2 with served exemplars; draft_decision's direct-drafting example). Raw report: docs/audits/eve-builder-affordance/report-705227.json.

intent baseline A / B post-fix consulted (first ask)
create-task 2nd / 2nd 1st create_work_item
create-issue 1st / 1st 1st create_work_item
update-status 2nd / 2nd 1st list_work_items → update_work_item
update-priority 2nd / 2nd 1st list_work_items → update_work_item
draft-decision NEVER / 1st 1st draft_decision
transition-decision 2nd / 1st 1st list_decisions → transition_decision
feature-proposal 1st / 1st 1st create_feature_proposal
content-brief 2nd / 2nd 1st create_content_brief
dispatch-brief grounded / grounded grounded (5 real lookups, honest miss) list_work_items ×5
open-thread NEVER / NEVER 1st open_thread

Totals: 9/9 card-expected intents on the FIRST ask (baseline: 2 and 4); capability denials 11 / 8 → 0; never-fired 2 / 1 → 0; fabrication risk 0 throughout; every decline resolved; work_item/decision/thread row counts unchanged (hard assert). Wall clock 14.0m → 6.4m — the retry ladder stopped being load-bearing. The 1.5 exit bar (≥8/10 first-ask, zero capability denials) is exceeded; the regression lock is admin-affordance-floor.spec.ts (1.6).

Completion re-audit (2026-08-28 — schema 2)#

The original three reports remain valid evidence of their conversational outcomes, but their schemaVersion: 1 database proof was only row counts. It cannot rule out an unauthorized in-place update, and the raw files do not bind the requested provider/model configuration. The instrument also inherited its open-task and draft-decision prerequisites from the developer database, could reuse an old assistant bubble if a turn rendered no new reply, and initialized the ghost-brief outcome as grounded before observing either a tool call or a reply. Those are measurement defects, not product passes.

The repaired schema-2 instrument now:

  • fingerprints every complete row of work_item, decision, thread, thread_message, and workbench_event, with a calibrated negative control proving that an in-place update changes the digest while the row count stays fixed;
  • installs one disposable open task and draft decision, snapshots around the decline-only turns, removes those exact rows in finally, and proves the harness restored the outer database state;
  • requires one newly rendered assistant reply per ask and classifies the ghost miss as grounded only when the exact missing brief is named, explicit absence language is visible, and list_work_items appears in the consulted-tool trail; an unrelated tool consultation cannot satisfy the contract;
  • records the requested provider, model, provider sort, and quantization in each raw report (the live BFF process environment was checked independently for the same values).

Two consecutive live runs on an isolated clone of the dev Postgres used OpenRouter deepseek/deepseek-v4-flash-0731, provider sort price, and quantization fp8:

run schema mutation cards first ask grounded ghost denials full-state equal fixture restored
report-441275.json 2 9/9 9/9 yes 0 yes yes
report-623442.json 2 9/9 9/9 yes 0 yes yes

Every parked card resolved as declined; both reports contain the same ten intent ids in the same order; neither contains an ungrounded, fabrication-risk, never-fired, or no-prerequisite outcome. Schema 1 and schema 2 are intentionally not raw-schema comparable. The baseline-to-post-fix behavioral comparison above remains useful, while the schema-2 pair is the current reproducibility and no-write proof. tools/eve-everywhere/verify-admin-affordance-reports.mts recomputes all five reports' totals, denial classifications, historical agreement, current outcome parity, exact ghost grounding, and full-state equality from the checked JSON rather than trusting the summary tables.