# Builder-family eval environment notes (EVE_EVERYWHERE 0.4) Why the docs and workbench families are "environment-blocked" in the eval (`task-families.ts`'s standing comment), what each blocked case actually needs, and what happened when the environment was supplied locally on 2026-08-19 (k=1, champion `deepseek/deepseek-v4-flash-0731`, dev Postgres, `EVE_SMX_EVAL_CASE_IDS` selection; the harness itself prints the honest caveat — "[PARTIAL RUN] … floors and scorecards need the FULL deck" — so nothing below is a floor, only an unblocking probe). ## The blocking mechanism (all structural, none in the harness) The eval boots the real app (`createApp`) and inherits process env. The harness already forwards everything the cases declare — operator `authScopes`, `clientCapabilities: ['action_confirm']` — so the block is never machinery, always a missing runtime surface: 1. **Workbench tools**: the route offers them only when `OSHUN_ADMIN_DATABASE_URL ?? OSHUN_V1_DATABASE_URL` yields a live store ("no DB, no workbench tools" — `routes/assistant.ts`). No eval file sets a DB URL, and CI has no seeded Postgres. 2. **Docs corpus**: `search_docs` is offered only when a corpus loads with ≥1 chunk. The MEMBER corpus is empty BY DESIGN (EVE-VIS-126 — a docs tool with zero excerpts invites confident fabrication, so it is not offered). The FULL corpus (`generated/docs-search-corpus.full.jsonl`, 54k lines here) is a machine-local build artifact and is NOT git-tracked — absent on any fresh checkout, whereupon `full` falls back to the empty member file and the tool vanishes for operators too. ## Per-case dependencies and the observed local run | case | observed (k=1, env supplied) | missing dependency | | ----------------------------- | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `family-docs-member-scope` | FAIL 0/1 | UNSATISFIABLE AS AUTHORED: it expects a member turn to CALL `search_docs`, but the member corpus is empty by design so the tool is never offered to members. Needs a Phase-2 re-scope decision (expect the honest tool-less absence instead), not an environment. | | `family-docs-admin-grounding` | FAIL 0/1 then PASS 1/1 (two invocations) | The untracked full corpus (build step in the eval env), plus k≥10 Wilson margins — at k=1 it is a coin worth nothing. | | `family-wbr-decisions` | PASS 1/1 | DB URL + Postgres + ≥1 `decision` row with an `adr-` id (dev DB holds 7). Env alone unblocks it. | | `family-wbr-explorer-link` | FAIL 0/1 | CORRECTED during 1.4: not a model flake. "Give me a graph explorer LINK" matches the mutation verb "link" → routed workbench-write, whose allowlist did NOT carry `open_graph_explorer` — the tool was clamped away, the read-clamp defect mirrored. Fixed by the 1.4 superset allowlist; re-measure at k≥10 in Phase 2. | | `family-wbr-member-refusal` | PASS 1/1 | None — it never needed an environment (a member session has no workbench store by construction). Its "blocked" label was inherited, not earned. | | `family-wbw-held-create` | PASS 1/1 | DB URL + Postgres. The confirm bridge parks in-process; the declared `action_confirm` capability is forwarded by the harness. | | `family-wbw-no-capability` | PASS 1/1 | DB URL + Postgres. | Run artifacts: `/tmp` logs from the 2026-08-19 session; verdict lines quoted in the EVE_EVERYWHERE ledger entry for 0.4. Side observation, recorded for accuracy: the original `eval:assistant` script selected every `.eval.ts` file. With `EVE_SMX_EVAL_CASE_IDS` set, `battery-transcripts.eval.ts` failed loudly ("names X — not a battery case"); without a selection it also captured and overwrote the human-labeling corpus as an unintended side effect of a deck run. The completion re-audit split the two entry points so each operator command selects exactly one live suite. ## What Phase 2.1 therefore builds - A seeded-Postgres fixture leg for eval runs: `OSHUN_V1_DATABASE_URL` pointing at a disposable schema seeded with the minimum the cases name (one `adr-` decision, one open task) — NOT the developer's live dev DB, so eval runs cannot eat local state. - A corpus step: build (or check in a frozen slice of) the full docs corpus for eval environments, sized for the two docs cases. - The `family-docs-member-scope` re-scope decision goes to Phase 2.2 with the EVE-VIS-126 rationale attached; until then the case cannot gate. - k≥10 with Wilson bounds before any floor claims — the two flaky observations above are exactly why (the SMX k=3 lottery lesson, re-demonstrated at k=1). ## Chunk-runner hazards measured 2026-08-20 (the 2.4 foreground pool) Three operational truths the 2.4 chunk runs proved, recorded here so the next runner does not rediscover them: - **The BFF must be DOWN when a chunk starts.** `eval-env.config.ts` clones `oshun_dev` → `oshun_eval` at global setup, and Postgres refuses `CREATE DATABASE … TEMPLATE` while any connection holds the source — a running BFF is exactly such a holder. The chunk then runs with "workbench families stay blocked" and silently grades a narrower deck. Boot the BFF AFTER the clone line appears in the log, or between chunks. - **The runnable deck file is `assistant-golden.eval.ts`.** The deck-case sources (`deck-*-cases.ts`) are data, not test files; `battery-transcripts.eval.ts` is the judge-corpus capture. The `eval:assistant` and `eval:assistant:battery` package scripts now select those files separately, so a deck run cannot spend on or overwrite the labeling corpus. - **Foreground chunks that overflow the 560s window survive in background inconsistently.** Grade lines print per case as they finish, so a killed run's completed cases salvage from the log (`grep 'ADVISORY'`) — append to the pool and relaunch only the missing ids, never regrade salvaged ones. ## Completion re-audit (2026-08-28) All seven docs/workbench family cases still exist, and every one still has a dependency/verdict row above. Later phases resolved the gaps exactly along the lines this note supplied as design input: - the eval global setup now points operators at a checked-in frozen full-corpus slice while preserving the metadata-only member corpus (`pages: 0`, `chunks: 0`) by design; - workbench cases receive a disposable eval clone when the operator has not supplied a database URL; the clone is run-unique, deterministically seeds the minimum ledger-backed state, and is force-dropped by global teardown; setup failures are narrated rather than silently graded as a narrower surface; - `family-docs-member-scope` now grades honest tool-less absence; `family-wbr-explorer-link` has the superset-allowlist repair and remains advisory at its measured 9/10; `family-wbr-member-refusal` remains correctly environment-independent. `eval-env-notes.spec.ts` now locks the note to the live family-case inventory and checks both corpus fixtures (full nonempty, member semantically empty). This prevents a new docs/workbench family case from inheriting an unearned "blocked" label or acquiring an unnamed fixture dependency. `eval-runner-isolation.spec.ts` also pins the separate deck and transcript entry points. A partial deck selector can no longer fail an unrelated capture suite, and a routine deck run can no longer rewrite human-labeling artifacts.