Residual Audit — Agentic AI Studio (V1/features.md 4523–4942)

Date: 2026-06-11. Read-only static audit of the residual AFTER the 57-task 2026-06-10 ground-truth backlog landed (A5 server-side kill-switch authority, C9 envelope persistence + console bridge). Evidence = current code only; every delegation chain below was read to the bottom.

Verdict on the central question — "does a REAL executor now exist?"#

Yes, narrowly, and it is honest. POST /v1/agentic/runs/execute (apps/oshun/bff/src/agentic/runs-route.ts:148) runs a client-submitted tool plan through runGuardedToolPlan → dispatchGuardedToolCall → admitToolCall (libs/oshun/agentic-studio/src/runs/{orchestrator,dispatcher,executor}.ts), and the tools are real bounded mutations over real stores (apps/oshun/bff/src/agentic/studio-tool-catalog.ts wired in server.ts:900–934: workspace-state read/write, real enqueueGenerationJob, real outputGalleryAdminStore.applyBulkAction). Kill-switch admission genuinely prevents tool execution (dispatcher.ts:55–67 runs the tool IFF admitted). Nothing fabricates a tool result; unconfigured tools 503 fail-closed. No result-faking stub was found anywhere on the execution path — the adversarial grep over the lib + BFF agentic files returned zero actionable hits.

But the executor is an API-only island: no product UI calls it (only apps/oshun/web/e2e/agentic-pipeline-invocation.spec.ts), it is disconnected from the approval-gated customer lifecycle, budgets remain client-authority, and the executed envelope records no transcript. The findings below are that residual.

F1. Privilege escalation: any customer can run operator-admin gallery mutations through the agentic executor#

Severity: P0-SEC

Evidence:

• apps/oshun/bff/src/agentic/runs-route.ts:146,148–151 — /v1/agentic/runs/execute is gated by createAuthPreHandler() only; no scope/role/tier check.
• apps/oshun/bff/src/server.ts:917–930 — the injected retagOutputs dep calls outputGalleryAdminStore.applyBulkAction with authorization: { operatorId: input.operatorId, tier: 'operator-admin' } — the tier is fabricated server-side for whoever the caller is.
• libs/isis/output-gallery/src/bulk-actions.ts:56–65 — re-tag requires curated-creator | aaa-creator | operator-admin; a plain customer has none.
• Contrast: the real admin route apps/oshun/bff/src/routes/admin-isis-output-gallery-actions.ts:54–58,93,116 requires admin:*/admin:studio/admin:workspace:isis before passing the same tier: 'operator-admin'.

Spec promise: features.md ~4612–4615 (operator action authorization tests), ~4699 (per-call authorization checks under "Tool isolation").

What the code actually does: any authenticated user POSTs a one-step plan {toolId:'output-gallery.retag', ...} with payload outputIds/addTags/removeTags and mutates the admin output gallery as a synthetic operator-admin, bypassing the tier gate the dedicated admin route enforces. (generation.enqueue-job via the same route is no worse than the existing unauthenticated POST /v1/generation/jobs, generation/jobs-route.ts:204 — that hole belongs to the generation slice.)

Fix sketch: gate output-gallery.retag (or the whole execute route per-tool) on the caller's real scopes; derive the gallery tier from authContext.scopes instead of hardcoding operator-admin.

F2. Operator kill switches are evadable by request shaping — A5 fixed switch storage, not switch targeting#

Severity: P0-SEC

Evidence:

• apps/oshun/bff/src/agentic/runs-route.ts:197–209 — the ExecutionTarget the switches match against is built from client-supplied fields: family: step.family ?? 'default', providerId: step.providerId ?? 'default', region: step.region ?? 'default', agentId: body.rootAgentId (free text, never validated against any registry).
• runs-route.ts:176 — tenantId = authContext.tenantId ?? body.tenantId: when the principal carries no tenant claim, the body picks the tenant.
• libs/oshun/agentic-studio/src/budgets/kill-switch.ts:57–69 — matching is plain string equality (s.scopeRef === target.family, etc.).

Spec promise: features.md ~4670–4673 ("Kill switches — per agent, per agent family, per tenant, per provider, per region") and ~4938 ("platform-wide tool kill propagates ≤ 5 s to all in-flight runs").

What the code actually does: an operator arming agent_family: isis (dev seed ks-agent-isis) stops only callers who volunteer family: 'isis' in their plan step; omit the field and the run sails through. Per-agent switches match a caller-chosen rootAgentId string; tenant switches are evadable by any principal without a tenant claim. Only global scope is server-authoritative end to end.

Fix sketch: resolve family/providerId/region server-side from the tool registry + agent registry entry for rootAgentId (reject unregistered agents); require a server-resolved tenant; treat client-supplied target attributes as untrusted.

F3. Any authenticated user can overwrite any stored AgentRun (including other users') via recordExecutedRun upsert#

Severity: P0-SEC

Evidence:

• apps/oshun/bff/src/agentic/runs-route.ts:160–171,234–243 — runId is client-supplied free text; after execution the route unconditionally calls agentRunLifecycleStore.recordExecutedRun(...).
• apps/oshun/bff/src/agentic/run-lifecycle-store.ts:147–155 — recordExecutedRun is a blind upsert by runId: "a re-execution of the same runId replaces the prior envelope." No ownership/tenant check against the existing record.
• run-lifecycle-store.ts:169–175 — ownership is inputs.invokedByUserId; the new envelope's invoker is the overwriting caller (runs-route.ts buildRun:115), so the original owner's run vanishes from their GET /v1/agentic/runs history.

Spec promise: features.md ~4607–4608 ("Immutable per-run audit events").

What the code actually does: user B can POST execute with user A's runId (runIds are visible in the operator console and predictable in shape) and replace A's persisted envelope + approval history wholesale — destroying the audit record and also laundering an awaiting_approval run into a completed one.

Fix sketch: in recordExecutedRun, refuse (409) when an existing run's invokedByUserId/tenantId differs from the caller; better, have the server mint execution runIds and treat lifecycle runIds as approval-checked input (see F4).

F4. The approval-gated lifecycle and the executor are two disconnected machines — gates are not enforced at execution time, and approved runs never execute#

Severity: P0-STRUCT

Evidence:

• apps/oshun/bff/src/agentic/run-lifecycle-store.ts:195–243 — approve() transitions awaiting_approval → 'queued' ("released to the executor" / "Released to Lilith's composer"). Grep finds no consumer of queued runs anywhere — no worker, no dispatch loop; status === 'queued' is read only for display mapping (routes/admin-agentic-operations.ts:57) and pause-ability (lib run-controls).
• apps/oshun/bff/src/agentic/runs-route.ts:188–195 — the execute route builds a fresh envelope with status: 'running', never consulting the lifecycle store's approval state for that runId. Nothing requires a run to have passed its gate before tools fire.
• apps/oshun/web/src/components/lilith/AreteReviewDraftRun.tsx:55 — customer copy for queued is "Released to Lilith's composer"; states running/completed (lines 56–59) are unreachable for lifecycle runs because nothing ever executes them.

Spec promise: features.md ~4590–4596 (operator-defined approval gates gating stages), ~4900–4905 (approvalRequirement pauses in awaiting_approval; "resolution gates resumption"), ~4860–4866 (arete.weekly_review_draft produces an actual draft).

What the code actually does: approval is a real, role-checked gate on a queue that goes nowhere, while actual execution runs on a parallel road with no gate at all. The flagship customer journey honestly dead-ends at "queued" forever (the component admits this in its own docstring, AreteReviewDraftRun.tsx:23–27 — honest, not a stub, but the product loop is open).

Fix sketch: make the executor consume the lifecycle store — approve() enqueues the run's plan into the guarded executor (or execute requires lifecycle.status ∈ {queued} for gated runs and stamps results back); delete the fresh-envelope path for runIds that exist in the store.

F5. Budgets and quotas have no server authority and meters are never debited — budget enforcement is effectively vacuous at runtime#

Severity: P1

Evidence:

• apps/oshun/bff/src/agentic/runs-route.ts:87,223 — budgets: body.budgets ?? []: the client supplies every BudgetEnvelope, BudgetMeter, and requestUnits; omit the field and zero budgets apply. No server-side budget registry exists (A5's own DONE note concedes "no server budget registry exists yet"; no follow-up task was filed — this is the residual).
• libs/oshun/agentic-studio/src/runs/orchestrator.ts:64–77 — the same static input.budgets (meters included) is re-checked at every step; nothing accumulates consumption between steps, so a 100-step plan each individually under the cap passes with unbounded cumulative cost.
• consumeBudget has zero runtime callers in the agentic path (executor.ts:100 and dispatcher.ts:49 explicitly delegate it to "the caller"; runs-route.ts never calls it; repo-wide grep confirms only lilith/yemaya unrelated modules).
• The admin store's per-tenant budgetUnits30d ledger (admin-agentic-operations-store.ts:355–383) is dev-seed display data — never resolved into the execute path and never updated by executed runs.

Spec promise: features.md ~4660–4677 ("Per-tenant, per-role, per-agent, per-run, and per-tool budgets and quotas", "Tests for budget enforcement").

What the code actually does: the budget engine (checkBudget verdicts, warn/grace, hard-stop kill) is real and the executor honors a verdict — but the only budgets it ever sees are ones the caller chooses to volunteer, checked against a meter the caller also supplies, and never debited.

Fix sketch: add a server budget registry (per tenant/agent/tool) resolved on every execute like resolveOperatorKillSwitches; thread consumeBudget after each executed call so meters accumulate within and across runs; persist meters.

F6. Executed runs persist with empty transcripts — toolCalls, evidence, cost ledger, and the termination audit event are all dropped#

Severity: P1

Evidence:

• libs/oshun/agentic-studio/src/runs/dispatcher.ts:58–66 — on executed it returns admission.run (the unchanged input envelope) plus the result; nothing appends a toolCalls entry, evidence, or cost ledger line. executor.ts:167–172 likewise returns run: input.run untouched on admit.
• apps/oshun/bff/src/agentic/runs-route.ts:234–243 — the persisted "executed envelope" therefore has toolCalls: [], evidenceTrail: [], costLedger: emptyCostLedger(), plan: [] always.
• On termination, result.terminated.auditEvent (a real RunControlAuditEvent) is partially echoed in the HTTP response (runs-route.ts:248–254) and never stored — recordExecutedRun takes only the envelope.
• Downstream: routes/admin-agentic-operations.ts:44,96–98 maps real runs with costUnits = 0 (band "low"), evidenceRefCount = 0, planNodeCount = 0.

Spec promise: features.md ~4563–4566 (AgentRun captures "tool calls, intermediate artifacts, evidence trails, cost ledger"), ~4607–4608 ("which tools fired"), ~4577 (replay with "full intermediate state and tool-call transcripts").

What the code actually does: C9 persisted the envelope, but the envelope was never populated — the operator console's "real runs" carry no record of what executed. Replay/time-travel is impossible by construction.

Fix sketch: in runGuardedToolPlan, append a toolCalls entry (toolId, startedAt, outcome, result ref) + cost ledger line per executed step and thread the enriched envelope forward; persist auditEvents alongside the envelope in the lifecycle store.

F7. Operator console renders every approved run as "rejected" — nonexistent field read#

Severity: P1

Evidence:

• apps/oshun/bff/src/routes/admin-agentic-operations.ts:88–91 — latestApproval ? latestApproval.approved ? 'approved' : 'rejected' : ….
• libs/oshun/agentic-studio/src/handoff/handoff-protocol.ts:315–322 — ApprovalDecision has verdict: 'approved' | 'rejected'; there is no approved property, so the access yields undefined → falsy → 'rejected'.
• run-lifecycle-store.ts:234–241 pushes { verdict: 'approved', … } on approve.

Spec promise: features.md ~4599–4603 (dashboard approval-state filter fidelity).

What the code actually does: the only approval verdict the store can record is 'approved' (discard does not append a decision), yet the console maps every approved run's approvalState to rejected.

Fix sketch: latestApproval.verdict === 'approved' ? 'approved' : 'rejected'. Add a test asserting an approved lifecycle run surfaces approvalState: 'approved'.

F8. Gold sets and champion-challenger are state machines over display rows — no traffic routing, no evaluation, and in production nothing to operate on#

Severity: P1

Evidence:

• apps/oshun/bff/src/admin/admin-agentic-operations-store.ts:115–195 — promoteGoldSet/rolloutChampionChallenger validate transitions + rationale and mutate the stored row. Nothing reads the result: repo-wide, trafficSharePct routes zero traffic, no shadow execution exists, fixtureCount/acceptanceRatePct are static numbers never produced by an evaluation harness.
• The lib's real engines — libs/oshun/agentic-studio/src/feedback/{gold-sets, champion-challenger,drift-detection,feedback}.ts — have no importer outside the lib (module-consumption sweep, this audit).
• admin-agentic-operations-store.ts:258–273 — production seed is empty and the routes expose no create verb (only promote/rollout on existing rows), so the prod console's gold-set and champion-challenger panels are permanently empty and the operator buttons can never have a target.

Spec promise: features.md ~4706–4720 (operator decisions promoted into versioned gold sets; champion-challenger with "parallel evaluation, statistical readiness gates"; promotion/freeze/rollback "with bound evaluation evidence").

What the code actually does: dev-seeded rows can be walked through a correct state machine; no runtime behavior anywhere changes as a consequence. Stored-config-never- enforced, and in prod, stored-config-never-exists.

Fix sketch: minimum honest V1: add create/ingest paths (operator decision → gold-set row; agent-version pair → experiment) and have the run dispatch path consult the active experiment's trafficSharePct for agent-version selection; otherwise label the panels read-only-registry and defer with a recorded V1.x note.

F9. The contract tool catalog (19 tools) has zero runtime implementations; the 4 live tools bypass it; no grant checks exist at dispatch#

Severity: P1

Evidence:

• libs/contracts/src/agent/tools.ts:89–306 — V1_TOOL_IDS / V1_TOOL_CATALOG faithfully enumerate the spec catalog (web.fetch, sophia.ground, memory.read/write, handoff, approval.request, agent.terminate, …) with semver + dependency resolution. No runtime imports it outside contracts tests and a maya lib.
• apps/oshun/bff/src/agentic/studio-tool-catalog.ts:18–23 — the live registry is 4 studio.*/generation.*/output-gallery.* ids, none present in the contract catalog; runs-route.ts:180–185 validates plan toolIds only against the injected registry, never the catalog.
• Grant semantics: libs/oshun/agentic-studio/src/grants/resolver.ts + capabilities/tool-grants.ts (scoped grants, revocation cascade, queued-run recheck) have no runtime consumer; admitToolCall checks kill-switch/budget/ throttle but no grant — any authenticated caller may invoke any registered tool (see F1).

Spec promise: features.md ~4745–4800 (enumerated AgentToolCatalog "exported from libs/contracts/src/agent/tools.ts" — done — and per-tool grant records, revocation propagation, expiry; ~4884–4906 grant-scope tests).

What the code actually does: catalog and grant engines are real, tested, and unreachable; runtime authority is "tool exists in the injected map".

Fix sketch: validate plan steps against V1_TOOL_CATALOG (unknown → 400, known-but- unimplemented → 503); add a grant-resolution step to admitToolCall's input built from a server-side grant store keyed by (agentId, toolId, scope).

F10. Cross-domain pipelines: six named specs, registry, observability — no runtime registers, schedules, or executes any of them#

Severity: P1

Evidence:

• libs/oshun/agentic-studio/src/pipelines/v1-pipelines.ts (259 lines: the six spec'd pipelines incl. ARETE_WEEKLY_REVIEW_DRAFT, VERITAS_STORY_DRAFTING) + pipeline-registry.ts + observability.ts — no importer outside the lib except libs/oshun/agent-pipelines, which is a 91-line pure re-export shim (src/pipelines/ index.ts:1–28, src/grants/resolver.ts:1–22) satisfying the spec's directory name with no added behavior.
• No BFF route mentions pipelines beyond free-text pipelineId pass-through (routes/agentic-runs-lifecycle.ts:29 — any ≤200-char string accepted; nothing checks it against V1_PIPELINES); nextFireFor scheduling has no caller.

Spec promise: features.md ~4727–4742 (pipeline registry, fixture-bound rehearsal, tenant-scoped scheduling), ~4845–4878 (the six concrete pipelines with declared stages and gates).

What the code actually does: the only pipeline with any product presence is arete.weekly_review_draft as a string label on a lifecycle run that never executes (F4). Stage declarations, budget caps, gate declarations are inert data.

Fix sketch: at minimum validate submitted pipelineId against V1_PIPELINES and attach the declared approval gate + budget cap to the run; a pipeline-stage executor over the guarded tool plan is the real (larger) closure, likely V1.x — record it.

F11. No agent registry/catalog at runtime — agent identity is unvalidated free text everywhere#

Severity: P1

Evidence:

• libs/oshun/agentic-studio/src/registry/{agent-registry,agent-catalog, agent-families}.ts — no importer outside the lib.
• routes/agentic-runs-lifecycle.ts:27 + agentic/runs-route.ts:162 — rootAgentId is any non-empty string; no registry lookup, no lifecycle state, no version pin, no tool-grant inheritance, no cost class.
• No BFF route or web/admin surface exposes a browsable agent catalog (repo-wide grep: no /v1/agents-style route; the web "agents" route is a prod-guarded fixture per C3).

Spec promise: features.md ~4538–4558 (canonical registry, browsable catalog, per-tenant views, version pins).

What the code actually does: every governance feature that should key off the registry (family kill-switch targeting F2, default grants F9, catalog UX) has nothing to key off.

Fix sketch: mount a read-only registry route over the lib's registry module with a small curated seed of the agents the product actually invokes (arete.weekly-review-composer, the studio executor agent), and make execute/submit reject unregistered ids.

F12. Throttles are never engaged on the runtime path#

Severity: P2

Evidence: libs/oshun/agentic-studio/src/runs/orchestrator.ts:72 hardcodes throttle: null for every step; runs-route.ts accepts no throttle config; the token bucket (budgets/throttle.ts acquireToken) and the executor's throttled outcome are reachable only from tests.

Spec promise: features.md ~4666–4669 (throttle/back-pressure on provider saturation).

What the code actually does: the throttled branch of the admission result is dead at runtime.

Fix sketch: accept a server-side per-tool/per-tenant throttle config (same pattern as resolveOperatorKillSwitches) and persist ThrottleState between requests.

F13. Spec pillars with engines but zero runtime: agent memory, anomaly quarantine, usage meters, modes, run controls/replay, dashboard query#

Severity: P2

Evidence (module-consumption sweep, no importer outside the lib for any of):

• capabilities/agent-memory.ts (116 lines: inspect/export/redact/scoped-clear), capabilities/capability-audit.ts, capabilities/tool-isolation.ts
• budgets/anomaly-detection.ts (quarantine), budgets/usage-meter.ts (customer-facing meters/grace/warnings)
• modes/modes.ts (fast/balanced/deep/exhaustive/rehearsal — no UI affordance, no route parameter anywhere)
• runs/run-controls.ts — pause/resume/cancel/branch/fork/retry-with-changes/ operator-step exist and are tested; only kill is invoked at runtime (via the executor). No operator route or console button drives any of them.
• dashboard/{dashboard-query,operator-actions,run-detail-page,audit-events}.ts — the admin console (F14) uses none of them.

Spec promise: features.md ~4683–4700 (memory audit/export/redaction), ~4674–4676 (anomaly quarantine), ~4662–4665 (usage meters), ~4715–4726 (modes incl. rehearsal isolation), ~4571–4577 (pause/branch/replay).

What the code actually does: honest absence — no surface claims these exist. This is a scope gap, not a honesty gap; listed so the residual map is complete.

Fix sketch: not closable this turn as a batch; pick per-pillar. The cheapest real wins: wire pause/cancel operator actions onto the lifecycle store (run-controls is ready), and a mode field on submit that maps to budget envelopes once F5 lands.

F14. Operator console: none of the spec'd operator actions, filters, or per-run drill-down; plan-DAG and evidence panels can never show real-run data#

Severity: UX

Evidence:

• apps/oshun/admin/src/components/AgenticOperationsPanel.tsx:280–344 — the run inventory is a read-only table: no pause/kill/escalate/override-gate/mark-reviewed/ reassign per-run or bulk actions (spec ~4604–4606), no filters by family/tenant/ band/severity/approval (spec ~4596–4599), no per-run page, no cross-run search or export (~4609–4611).
• PlanDagSection/EvidenceTrailSection (lines 763–851) read snapshot.planSnapshots/snapshot.evidenceTrail, which only the dev seed populates (admin-agentic-operations-store.ts:481–553); recordExecutedRun writes neither (and the envelope is empty anyway, F6) — so in production these panels are permanently "No plan snapshots captured" / "No evidence entries" while real runs sit in the table above with cost 0 / evidence 0 (F6) and wrong approval badges (F7). Kill-switch arm/disarm is the one fully real operator control (and it now genuinely gates execution, modulo F2).

Spec promise: features.md ~4582–4612 (Operator Job Dashboard, Replay, and Audit).

Fix sketch: add per-run kill/pause buttons calling new lifecycle verbs (run-controls is ready, F13); client-side facet filters over the existing snapshot are an afternoon; bridge real plan/evidence rows when F6 populates the envelope.

F15. Customer/creator agent surface cohesion: one pipeline, one button, no pre-flight estimates, no progress, no output review; zero creator-side agentic actions#

Severity: UX

Evidence:

• The entire customer/creator agentic product surface is apps/oshun/web/src/components/lilith/AreteReviewDraftRun.tsx (one hardcoded pipeline on /arete/review). No pre-flight cost/latency estimate, declared source-set/persona/tone (spec ~4634–4638); no in-progress run card with interrupt/redirect (~4639–4641 — moot while F4 holds, since nothing progresses); no output review/diff/regenerate (~4642–4646); no completion notifications surfaced (the store builds a real RunNotification on discard only, and nothing delivers it); no run-history surface beyond this one component's filtered list.
• Creator-side: no authoring-workspace agentic actions exist anywhere (spec ~4628–4633); the studio "agents" pages (apps/oshun/web/src/app/studio/yemaya/*) are Yemaya bot-domain consoles, a different spec area, not Agentic-Studio invocation.

Spec promise: features.md ~4617–4651 (Customer- and Creator-Facing Agent Invocation).

Fix sketch: the honest near-term move is to keep the single surface but close its loop (F4) and add a completion notification + run-history read; declare the broader invocation matrix V1.x.

Explicit non-findings (checked, clean)#

• No result-faking executor: the mandatory adversarial grep over libs/oshun/agentic-studio/src + apps/oshun/bff/src/agentic + the three routes returned zero actionable hits; tool results come from real stores; failures throw (StudioAgentToolError) rather than fabricate.
• Fail-loud seams are correct: unconfigured tools → 503 agent_tools_not_configured; the lifecycle never auto-completes on a clock; AreteReviewDraftRun never fabricates a draft.
• Dev seeds are dev-gated (admin-agentic-operations-store.ts:250–273, recorded in C9) — prod starts empty; not re-reported. The dev-mode mixing of seed rows with real runs in one table is a known dev aid.
• A5's kill-switch storage authority is genuinely server-side (runs-route.ts:143–145, 215–217 read the durable admin store per request); F2 is about target attributes, a different hole.
• admin/agentic-operations routes are properly admin-scope-gated (routes/admin-agentic-operations.ts:256–281) and mounted (app.ts:606,652); the admin Next proxies forward real sessions. No dead routes found.
• Yemaya multi-agent orchestration (routes/admin-yemaya-multi-agent- orchestration.ts, studio workspaces) is the Yemaya messaging domain's slice, not features.md §Agentic — out of scope here.

Severity counts#