Naming, official as of 2026-08-04: the omnipresent assistant is called Eve. The name was collision-checked against the whole monorepo (domains, apps, libs, docs) — no prior "Eve" exists. Everywhere this document and
V1_PRODUCT_GRAPH_RESTRUCTURE_TODOS_2026-08-04.mdsay "the assistant", read Eve. See §12 for the naming record and for Eve's coding-agent harness (Codex CLI).Brand split, later the same day (2026-08-04): on the customer shell the assistant presents as Lilith — the V1 consumer app is itself named Lilith and she is its main persona, hosting her rooms as her supporting cast; see
V1/BRAND.md. V1.0 opens four of them — Tara, Nyx, Arete, Nisaba — with Veritas and Metis deferred to V1.2, so the assistant must not offer, hand off into, or name a deferred room. Eve remains the builder/operator-side identity: the intent-planeassistantactor in the builder workbench and the codex coding-agent harness. Same underlying assistant infrastructure, two faces — members talk to Lilith; the builder works with Eve. Customer-facing surfaces must never say "Eve".
Eve is an intelligent assistant present on every page of Oshun V1 — customer web, admin console, studio, and (later) mobile — that the user can talk to in text or voice, that sees what is on the current page, that can act on the UI (navigate, highlight, fill, run commands), and that can give live guided tours of any feature. This document records the audit of what already exists, the state of the art for in-product assistants as of mid-2026, the target architecture, a phased build plan, and feature proposals beyond the original ask.
Headline verdict: V1 already owns roughly 70% of the chassis for this product — panels, invocation points, safety, memory, durable history, personas, disclosure, even provider wrappers for Deepgram/Whisper/ElevenLabs/Cartesia — but the brain is a deterministic regex intent-router with canned replies, and almost every advanced asset is dormant and unwired. The SOTA build here is mostly a wiring, unification, and "one real agent loop" project, not a greenfield one.
1. What exists today (audit, 2026-08-03)#
1.1 Wired and real#
| Asset | Where | State |
|---|---|---|
| Chat UI (customer web) | apps/oshun/web/src/components/assistant/AssistantPanel.tsx (4,234 lines), assistant-dock/AssistantDock.tsx |
Real, rich: sessions, history, personas, disclosure chips, grounding badges, save-to-notebook, journey rail |
| Shell mount | ShellLayout.tsx renders dock (≥1360px) or slide-over panel |
Wired on ~519/717 pages — but 501 of those are /studio/*; only ~18 customer pages have it |
| Engine | libs/oshun/shell-assistant (~14.6k LOC) |
Deterministic: 96 regex/keyword intents (domain-intents.ts), canned templates (response-formatter.ts). No model call in the library |
| BFF assistant API | apps/oshun/bff/src/routes/assistant.ts (1,447 lines) |
Real pipeline: auth → abuse guard → Lilith safety (crisis frames, refusals) → engine → persona → Iris memory recall → optional LLM rewrite → durable history → single JSON reply. No streaming |
| LLM touchpoint | bff/src/assistant/llm-reply-composer.ts |
Fail-closed paraphraser of the deterministic reply (may not add facts). OpenRouter-compatible HTTP, default model deepseek/deepseek-v4-flash. No creds → honestly degrades |
| Safety | Lilith analyzeMessageSafety, crisis frames, suppression |
Real, on the message path — keep as-is around any new brain |
| Memory | Iris memory bridge, consent scopes off/session/profile, pause state |
Real, enforced server-side |
| Conversation history | bff/src/conversation/* |
Durable append-only log with retention rules, erasure fences, event-bus consumer |
| Invocation registry | shell-assistant/invocation-points.ts |
19 declared entry points with guards — only 4 wired; most launches go through a stringly CustomEvent bus (46 call sites) that bypasses the guard |
| Context handoff | ShellLayout.buildAssistantContextHandoffForIntent() |
Client collects route, artifact (kind/id/path/metadata), evidence state, user's text selection, permittedToolGrants — BFF consumes only activeDomain and notebookId; the rest is discarded |
| Voice (browser) | AssistantPanel.tsx |
webkitSpeechRecognition STT, speechSynthesis TTS, real mic-level meter. Chrome-shaped; no server audio path |
| Command palette | libs/oshun/shell-core/command-surface.ts, CommandPalette.tsx (1,227 lines) |
Typed registry of navigation/search/create/assistant commands — for humans only; the assistant cannot enumerate or invoke it |
1.2 Built but dormant (the buried treasure)#
| Asset | Where | What it is |
|---|---|---|
| Full agentic runtime | libs/oshun/assistant |
Hermes-parity spine: real ask() loop over @iris/agents-core AgentRunManager, tools (browser, web-search/fetch, code-exec, filesystem, vision, clock), skills with embedder/retriever/postgres store, subagents, cron and NL scheduling, postgres conversation memory, SSRF guard. Only consumer: the V10 Ori presence beta |
| Agent governance | @iris/agents-core |
Run manager, token budgets, kill switches, audit envelopes — already registered in the BFF (app.ts:606) |
| STT providers | libs/psyche/speech-recognition |
Deepgram streaming WS client, Whisper HTTP client, WebSpeech fallback, VAD — zero importers |
| TTS providers | libs/psyche/voice-synthesis |
ElevenLabs and Cartesia clients, multi-provider strategies (lowest-latency, cost-optimized…), SSML — zero importers |
| Captions and visemes | libs/psyche/caption-streaming, viseme-generator |
Real engines, zero importers |
| RAG pipeline | libs/iris/conversation-rag (5.6k LOC) |
Pipeline, context injector, relevance scorer, source tracker — retriever interface, no concrete index |
| Orchestration and routing | libs/iris/conversation-orchestration, model-routing |
Circuit breaker, load balancer, cost/latency routing — dormant |
| MCP client | libs/iris/mcp |
Real stdio and streamable-HTTP client with tool discovery — dormant |
| Psyche session bridge | shell-assistant/psyche-session-bridge.ts |
Real logic, zero importers |
| Realtime WS | bff/src/psyche/realtime-route.ts (GET /v1/psyche/realtime) |
Session/turn envelope over @fastify/websocket — no frontend consumer |
| Tour components | web/src/components/onboarding/FeatureTour.tsx |
Real TourStep[] component — dead code, zero importers |
| Walkthrough engine | libs/oshun/shell-core/feature-education.ts + ShellFeatureEducationHub.tsx |
Declares target selectors and placements, but the renderer is a centered modal that prints the selector as text — no anchoring, spotlight, or scroll-into-view |
| Proactive-help triggers | libs/psyche/help-offer-triggers, proactive-assistance-triggers, engagement-detector |
Dormant |
| Training feedback | shell-assistant/training-feedback.ts, AssistantEvent stream |
Emitters exist; no listener or sink registered anywhere |
1.3 The gap list (what "SOTA on every page" requires that is missing)#
- No LLM agent loop. The brain is regex + templates; the only model call is a paraphrase that may not add facts. No tool calling, planning, multi-step reasoning, or citations.
- No streaming transport. One-shot JSON replies; the UI hardcodes
streamingTransport: truebut nothing streams. The WS route has no client. - "Every page" is actually ~18 customer pages.
ShellLayoutis imported per-page; deep domain routes (/arete/*,/veritas/*,/domains/*,/tara/*,/atelier/*) have no assistant at all. Dock needs ≥1360px. - Page context is collected then thrown away (route, artifact, selection,
tool grants → only
activeDomainandnotebookIdconsumed). - No action capability. The engine can return
navigateTopayloads which the panel executes withwindow.location.href(full reload, session lost). No highlight, no form fill, no command invocation, no client tool protocol. - Voice is Chrome-only browser APIs; the production provider wrappers are unwired and there is no BFF audio endpoint.
- No coachmark/spotlight engine — the one wired walkthrough renders a
centered modal printing
Target: [data-shell-nav-activity]as body text. - Three divergent assistants: web (BFF-backed deterministic), admin
(display-only panel, no input, no endpoint), mobile (local keyword-matched
canned strings, no
/v1/assistant/*calls at all). - Entitlement gating is client-side theater — literal
['assistant.customer']constants are passed into the guard, so it can never fail; no plan/quota enforcement server-side beyond generic abuse protection. - No observability or feedback loop — the
AssistantEventstream and training pipeline have no sinks; analytics has no assistant module. - Invocation-point registry mostly bypassed by the CustomEvent bus.
2. SOTA landscape (research, mid-2026)#
What industry-leading in-product assistants look like now, and what we adopt:
- Agent ↔ UI protocols. The frontier pattern is a bi-directional event stream between the agent runtime and the frontend: streaming text, tool calls, frontend-executed tools, shared state patches, generative UI, and human-in-the-loop interrupts. AG-UI (CopilotKit ecosystem, adopted by Google ADK, AWS Bedrock AgentCore integrations) is the reference vocabulary; MCP covers agent→tool, A2A/A2UI cover agent→agent and agent-rendered widgets. We adopt the shape (typed event stream + client tool registry + HITL confirmations) in-house rather than the dependency — our BFF and panel are bespoke and already close.
- Page grounding: accessibility tree over screenshots. Browser-agent research and production systems (Playwright MCP snapshots, OpenAI CUA hybrid, rtrvr/Fazm) converge on structured a11y/DOM snapshots as the primary grounding — hundreds of bytes instead of 500KB screenshots, precise element refs, works with the LLM's text-native strengths. Vision stays a selective fallback. For an assistant embedded in our own app we can do even better than generic agents: a curated anchor registry plus typed app-state selectors, with raw a11y-tree snapshots as the escape hatch.
- Tours. Commercial tools (Pendo "Leo", Stonly, Intercom Fin, Tourial) now use AI to author walkthroughs but still execute them deterministically against anchored elements — the LLM plans, a dumb player drives the spotlight. Open-source anchoring/spotlight is a solved problem (driver.js-class libraries). This "LLM plans / deterministic player executes" split is the reliability trick we adopt.
- Voice. Async voice (record → STT → text agent → TTS) is commodity: Deepgram / AssemblyAI / Whisper-class STT at ~90–300ms-class latencies and cents/hour; ElevenLabs / Cartesia / Deepgram Aura TTS at sub-250ms TTFB. Realtime speech-to-speech (single WebSocket pipelines) exists when we want it later; the async design below upgrades cleanly.
- Embedded copilot products (ChatKit/Apps SDK, CopilotKit, assistant-ui) validate the UX grammar: persistent dock + slide-over, context chips, streaming, inline citations, action confirmation cards, generative UI islands. We keep our own panel (it is already better integrated than any drop-in) and match the grammar.
Sources: see §11.
3. Target architecture#
┌─────────────────────────── every page (web, admin, studio, mobile) ──────────────────────────┐
│ AssistantHost (root layout mount) │
│ ├── AssistantPanel / Dock / Sheet (existing UIs, now streaming) │
│ ├── PageContext collector — route, artifact, selection, viewport, app-state selectors │
│ ├── AnchorRegistry — data-assistant-anchor ids + generated manifest │
│ ├── ClientToolExecutor — navigate, highlight, open_command, fill_field, start_tour │
│ ├── TourPlayer — deterministic spotlight/coachmark engine │
│ └── VoiceCapture — MediaRecorder + VAD (async), WebSpeech fallback │
│ ▲ SSE turn stream / tool-result POSTs │ │
└────────────────────┼───────────────────────────────────────┼─────────────────────────────────┘
│ ▼
┌──────────────────────────────── BFF: Assistant Agent Service ────────────────────────────────┐
│ POST /v1/assistant/sessions/:id/turns → text/event-stream (typed events) │
│ POST /v1/assistant/sessions/:id/turns/:t/tool-results (client-tool round trip) │
│ POST /v1/assistant/sessions/:id/audio → STT → normal turn │
│ GET /v1/assistant/tts?turnId=… → streamed audio │
│ │
│ Agent loop: Anthropic TS SDK tool runner, claude-opus-5, adaptive thinking, prompt caching │
│ Wrapped by (all existing): Lilith safety pre/post · Iris memory bridge · persona policy │
│ · conversation-history recorder · session store · @iris/agents-core budgets + kill switches │
│ │
│ Server tools: domain adapters (tara/veritas/nyx/arete/nisaba/metis) · search_docs (RAG over │
│ docs-center via pgvector) · command-surface introspection · profile/entitlements · │
│ admin/operator adapters (admin surface only, RBAC-scoped, audited) │
│ Client tools (executed in browser): navigate · highlight · read_page (a11y snapshot) · │
│ open_command · fill_field (confirm-first) · start_tour(plan) │
└──────────────────────────────────────────────────────────────────────────────────────────────┘
Principle: one brain, many surfaces. A single BFF agent service and a shared
headless client package (@oshun/assistant-client: session state machine, SSE
transport, tool executor interfaces) consumed by web, admin, and mobile. Surface
differences are toolsets and policies, not separate brains.
3.1 The brain: real agent loop in the BFF#
- SDK and model. Official
@anthropic-ai/sdk(drop the OpenRouter shim for the assistant path — the iris "AnthropicProvider" currently delegates to OpenRouter with a DeepSeek default). Model:claude-opus-5($5/$25 per MTok, 1M context, adaptive thinking on by default). Utility subtasks that don't need frontier quality (turn titling, follow-up chips, STT cleanup) can runclaude-haiku-4-5— a routing decision to make explicitly, not a default downgrade.effort: "medium"for chat turns,"high"for tour planning and admin/operator tasks; sweep later against evals. - Loop.
client.beta.messages.toolRunner({ stream: true, tools, messages })— the SDK handles the request→execute→loop cycle; per-turn hooks give us approval gates and result modification. Server tools execute in-process; client tools pause the loop, emit atool_inputSSE event, and resume when the browser POSTs the result (same shape as Managed Agents'custom_tool_use/custom_tool_resultround trip). - Streaming. SSE (
text/event-stream) from Fastify — simplest thing that works with the existing auth prehandlers; the dormant WS route is the later upgrade path for realtime voice. Event vocabulary (AG-UI-shaped):turn.delta(text),turn.thinking(optional summarized),turn.tool_call,turn.tool_result,turn.client_tool_input,turn.ui(highlight / tour-step / navigation intents),turn.citation,turn.complete,turn.error. - Prompt caching. Stable prefix: system prompt (persona + policy + communication style) then deterministic tool list, with a cache breakpoint; volatile page context and the user turn go after. Opus 5's 512-token cache minimum makes even the compact prompt cacheable; multi-turn breakpoint on the latest turn. Never interpolate timestamps/UUIDs into the system prompt.
- Safety wrap (keep, don't rebuild). Lilith
analyzeMessageSafety+ crisis frames stay in front of the model; model output passes the same post-check the composer uses today; refusal/crisis paths keep their existing canned, clinically-reviewed responses. Handlestop_reason: "refusal"before reading content. - Memory and history. Iris memory bridge stays (consent scopes already
enforced); promote recall from string-append to a
recall_memorytool + system context. Durable conversation history recorder stays verbatim. - Budgets. Per-session and per-org token budgets and kill switches via
@iris/agents-core(already registered in the BFF for Ori) — this is the quota enforcement that is currently missing. - Fallback. The deterministic engine is demoted to the honest no-key / outage fallback (it already degrades gracefully), and its six domain adapters are re-exposed as typed tools for the agent. Nothing is thrown away.
3.2 Ubiquity: actually on every page#
- Mount once at the root. A new
AssistantHostinapp/layout.tsx(alongsideAccessibilityShell) so the assistant exists on all 717 pages, including deep domain routes that currently render preview banners. Pages keep contributing context via the collector;ShellLayoutpages get the docked experience, non-shell pages get the floating launcher + slide-over. - Kill the 1360px cliff — panel as slide-over at any width; dock remains a ≥1360 enhancement.
- Admin becomes a real copilot.
AdminAssistantPanelgains the shared client package (input, streaming, history) against the same BFF service with anadmintoolset (operator read adapters, triage/draft actions), gated by real server-side scope checks and audited per tool call. The declared invocation points (review-detail,inbox-triage,incident-acknowledgement) become live entry buttons with prefilled context. - Route launches through the registry. Replace the raw CustomEvent bus call sites with a thin API that resolves an invocation point and evaluates the guard first — making the 19-point registry (and its analytics) real.
- Mobile (later phase). Replace
buildMobileAssistantReply's canned matcher with the shared client against/v1/assistant/*; the sheet, launcher, and handoff plumbing already exist.
3.3 Page context protocol (the assistant can see the page)#
Formalize what buildAssistantContextHandoffForIntent already collects, and
actually consume it:
- Ambient context (every turn, cheap): route, artifact (kind/id/path/metadata), active domain, selection text, viewport, surface, persona, entitlements. Injected as a compact context block after the cached prefix. This alone unlocks "what am I looking at?", "explain this selection", "why is this empty?".
- App-state selectors (per-route, typed): routes register a
getAssistantContext()returning a small typed summary (e.g. the Veritas article id + claim states on screen; the Arete check-in streak). Declared next to the page, versioned in the anchor manifest. - On-demand
read_pagetool (agent-pulled): returns a sanitized accessibility-tree snapshot of the current view (roles, names, states, anchor ids) — the SOTA browser-agent grounding, but scoped to our own DOM with PII redaction (mask input values, respectdata-assistant-private). No screenshots in v1; the a11y tree is cheaper and more precise.
3.4 Client tool registry: the assistant can act#
Typed, guarded, frontend-executed tools (the AG-UI pattern):
| Tool | Effect | Guard |
|---|---|---|
navigate({ route }) |
Client-side router push (no reload; session survives) | Route must exist in nav registry |
highlight({ anchorId, note? }) |
Spotlight + optional callout on a registered anchor | Anchor must be present on page |
read_page() |
A11y-tree snapshot (see §3.3) | Redaction rules |
open_command({ commandId, args? }) |
Invoke a ConsumerShellCommandDescriptor — the human palette registry becomes the assistant's verb set for free |
Command suppression rules already exist |
fill_field({ anchorId, value }), toggle_setting(...) |
Form assistance | Confirm-first: renders an inline confirmation card; destructive/irreversible actions always require explicit user tap |
start_tour({ plan }) |
Hands a validated tour plan to the TourPlayer | Plan schema-validated; anchors verified |
Server-side enforcement: the session's permittedToolGrants (already in the
handoff, currently decorative) becomes the authoritative allowlist — the BFF
refuses to offer a tool the surface didn't grant, and every executed tool call
is written to the audit trail. This is the "promote actions to dedicated tools
so the harness can gate, render, and audit them" principle.
3.5 Real-time guided tours#
The reliability trick: the LLM plans; a deterministic player executes.
- Anchor registry.
data-assistant-anchor="<id>"on tour-worthy elements; a generatedanchors.jsonmanifest (id, route pattern, human label, one-line description) that ships to the agent as a tool resource. A CI guard test (same style as the existing Maestro coverage guards) fails when a manifest anchor disappears from the DOM tree — tours can't silently rot. - TourPlayer. Upgrade the wired-but-broken walkthrough renderer
(
ShellFeatureEducationHub) into a real spotlight engine: dim overlay with cutout, scroll-into-view, popper placement (thetop|right|bottom|left|centerplacements are already in the shell-core types), advance on next/back/user-action,aria-livenarration for screen-reader symmetry, cross-route steps vianavigate. Either adopt a driver.js-class library or finish ours — the shell-core step model is already the right shape. - Tour planning. "Give me a tour of Veritas evidence review" → the agent
calls
plan_tour, producing a structured plan (Zod schema viazodOutputFormat,strict: true) whose steps may only reference manifest anchors and registered routes; invalid plans are rejected and retried. Narration text is generated per-step; optional TTS speaks it. - Adaptive tours. The player reports progress/deviation events back into the turn stream; if the user wanders, the agent re-plans from the current page state — this is the "real-time tour" the ask calls for, and it is exactly one loop iteration, not a new system.
- Tour flywheel. Successful generated tours get persisted as named, human-curated tours (review queue) — LLM output hardens into deterministic assets, cutting cost and variance over time. Curated tours are also offered contextually ("New here? Take the 60-second tour").
3.6 Voice (async now, realtime-ready)#
Async is explicitly acceptable, which keeps v1 simple and cheap:
- Input: push-to-talk (and VAD auto-stop via
@psyche/speech-recognition's VAD) →MediaRecorder(webm/opus) →POST /v1/assistant/sessions/:id/audio→ server STT via the already-written Deepgram/Whisper providers → transcript enters the normal turn pipeline (safety, memory, agent). Fixes the Chrome-only limitation; Web Speech API remains the zero-cost fallback. - Output: per-turn TTS via the already-written ElevenLabs/Cartesia
providers (
lowest-latency/cost-optimizedstrategies are built in), streamed as audio; captions via@psyche/caption-streaming; the existing synthetic-voice disclosure chip covers labeling.speechSynthesisstays as fallback. - Realtime later: the dormant
GET /v1/psyche/realtimeWS + Deepgram streaming client + Cartesia low-latency TTS are the ingredients for a speech-to-speech upgrade (barge-in, sub-second turns) without re-architecting — the turn pipeline is shared; only the transport and chunking change. Viseme generator is ready for the avatar-embodied mode when that ships.
3.7 Grounded knowledge (RAG)#
- Embed the docs-center corpus (3,000+ pages),
features.mdtrees, domain registry purposes, and walkthrough docs into pgvector (already in the dev stack) behind asearch_docstool; wire it as the concrete retriever for@iris/conversation-rag(pipeline exists, retriever is an injected interface today). - Answers about product behavior cite sources; the panel's existing grounding chips render them. "I don't have a doc for that" beats hallucinated UI instructions — instruct and eval for it.
3.8 Governance, cost, observability#
- Entitlements for real: server-side plan/scope check on session create and
per turn (assistant tiers can differ by plan); wrap client entry in
ShellFeatureGate; per-org and per-user token budgets with kill switches (@iris/agents-core). - Cost model (order of magnitude, Opus 5): typical turn ≈ 8k prompt (mostly cache-read at 0.1×) + ~300 fresh input + ~600 output ≈ $0.02 per turn warm, ~$0.06 cold. 100k turns/month ≈ $2–3k. Haiku-routed utility calls are ~5× cheaper. STT is cents/audio-hour; TTS is the swing cost — cache narration audio for curated tours. (Verify provider list prices at build time.)
- Observability: register a BFF sink for the existing
AssistantEventstream (intent.classified,action.executed,turn.completed…) and the training-feedback pipeline; add anassistantTelemetrymodule to web analytics (open/close, invocation point, tool usage, tour completion, thumbs). Evals: golden-turn suite per domain (answer grounding, tool-choice correctness, tour-plan validity rate, safety regression suite reusing the Lilith cases).
4. Build plan#
Each phase ships user-visible value and is independently landable.
P0 — The brain (agent loop + streaming). SHIPPED 2026-08-03, multi-provider.
POST /v1/assistant/sessions/:sessionId/turns streams SSE
(turn.meta/delta/tool_call/tool_result/complete/error) through the full
governance pipeline (Lilith safety supersede, consented memory recall injected
pre-model, durable turn + transcript recording). The loop is provider-agnostic
over LLMProviderInterface: Anthropic native (new AnthropicNativeProvider
in @oshun/ai over @anthropic-ai/sdk, default claude-opus-5), OpenAI
native (official openai SDK), and OpenRouter (DeepSeek et al.) —
selected by OSHUN_ASSISTANT_PROVIDER or first-available API key
(ANTHROPIC_API_KEY → OPENAI_API_KEY → OPENROUTER_API_KEY), kill-switched
by OSHUN_ASSISTANT_AGENT_ENABLED=0. The 24 domain-adapter tools (plus guarded
navigate) inject the authenticated user id server-side and are filtered to the
session's authorized domains. No keys → 503 assistant_agent_not_configured and
the panel falls back to the deterministic message route; refusals and provider
errors surface as turn.error, never a fabricated reply. AssistantPanel
streams into a live bubble (NEXT_PUBLIC_OSHUN_ASSISTANT_STREAMING to force
on/off) with the completed payload authoritative. Exit met in tests: a member
question streams a grounded, safety-checked, tool-using answer end-to-end.
P1 — Everywhere + context + first actions. SHIPPED 2026-08-03. Root-layout
AssistantHost (floating launcher + overlay panel on every page without shell
chrome; stands down where ShellLayout manages its own dock). Page grounding:
the panel sends a size-capped page outline (route, title, headings, on-page
anchors, selected text — data-assistant-private subtrees never read) with
every turn; the BFF coerces and injects it into the agent prompt alongside the
shared anchor registry (@oshun/shell-assistant/assistant-anchors, six seed
anchors, CI anchor-coverage guard in web). Tools: highlight (validated against
registry ∪ page anchors, streamed as turn.ui, executed by the new
AnchorSpotlight — scroll-into-view, dim-cutout spotlight, note bubble,
aria-live announcement), read_page (true paused-turn client tool:
turn.client_tool_input → browser snapshot → client-tool-results POST →
AssistantClientToolBridge resumes the loop; honest timeout result if the
client never answers), and navigate now executes as a client-side router push
(session survives — no more full-page reload; suggested-action navigation
likewise). Admin: same brain via same-origin streaming proxy routes
(/api/assistant/..., cookie credential stays server-side), a real chat in
AdminAssistantPanel, admin-scoped sessions authorized across all domains, and
two operator tools over the canonical admin state (admin_workspace_overview,
admin_review_queue). The shared SSE wire vocabulary moved to
@oshun/shell-assistant/turn-stream-frames so BFF, web, and admin cannot drift.
Deferred from P1: invocation-point bus cleanup (the 46 CustomEvent call sites
still bypass the guard). Exit criteria met in tests: page-grounded "what am I
looking at?", reload-free navigate, admin review-queue answers over real store
data.
P2 — Tours. SHIPPED 2026-08-03. The LLM-plans/deterministic-player split is
live. @oshun/shell-assistant/tour-plans defines the plan schema, the validator
(unregistered anchors rejected with model-correctable errors), and the curated
catalog — three code-reviewed tours (shell-orientation, home-daily-flow,
assistant-orientation); a CI spec runs every curated tour through the same
validator model plans face, so a catalog edit that references a missing anchor
fails the build. The BFF offers start_tour (curated id or composed steps;
invalid plans bounce back as error tool results and the model retries — covered
by a route test) and list_curated_tours, both gated behind the client's
tour_player capability; a started tour streams as a turn.ui tour_start
intent and rides turn.complete.response.tourPlan. The web TourPlayer
executes plans deterministically: per-step route navigation (client-side push),
anchor wait-with-timeout, dim-cutout spotlight (shared geometry with
AnchorSpotlight), narration card with Back/Next/End + arrow keys + Escape,
aria-live narration, optional TTS via the panel's speech. Missing anchors are
honest skips — surfaced in place, reported in the outcome, and (once per tour)
fed back to the agent as a visible [tour-player] replan message: the adaptive
loop. The panel hides while a tour runs and posts a completion summary to the
transcript. Deferred to later: dynamically persisted member-authored tours with
an operator review queue (curated tours ship as reviewed source for now —
pull-request review is the gate). Exit met in tests: "give me a tour of the
shell" streams a validated, anchored, narrated tour executed on live UI.
P3 — Voice. SHIPPED 2026-08-03. Async voice-in/voice-out on the shared turn
pipeline. STT: POST /v1/assistant/sessions/:id/audio (raw
webm/ogg/mp4/mpeg/wav, 10 MB cap, 415 on anything else) through a new
fail-closed voice binding — Deepgram prerecorded (a real REST client added to
@psyche/speech-recognition, which previously only spoke the live WebSocket
protocol) or OpenAI Whisper, selected by OSHUN_ASSISTANT_STT_PROVIDER or key
detection. TTS: POST /v1/assistant/sessions/:id/tts (1,500-char cap) through
@psyche/voice-synthesis — ElevenLabs (documented default voice) or Cartesia
(requires OSHUN_ASSISTANT_TTS_VOICE_ID; no honest default exists). No
credentials → 503 *_not_configured. The panel records push-to-talk via
MediaRecorder (60 s hard cap, mic released on stop), shows the real mic-level
meter, uploads, and sends the transcript through the normal turns route — so
every voice turn gets the same safety/agent/ recording treatment as a typed one;
replies play server audio with the on-screen reply text as the caption and the
existing synthetic-voice disclosure chip as the label. Web Speech APIs remain
the always-available fallback (server 503s hand over with an honest system
notice; empty transcripts say so instead of guessing). Scope notes: VAD
auto-stop and word-timed captions via @psyche/caption-streaming are deferred —
the latter needs provider word timings end-to-end; transcript text serves as
captions today. Exit met in tests: record → transcribe → agent turn →
synthesized reply, in any evergreen browser, fail-closed at every seam.
P4 — Unification + intelligence. Mobile parity (shared client in the sheet); RAG over docs-center; proactive help triggers (rate-limited, opt-out); observability sinks + eval harness; entitlement/budget enforcement; selection-to-ask affordance. Exit: one brain on four surfaces with measurable quality and cost dashboards.
Feature catalog + audit mode. SHIPPED 2026-08-03. A typed graph of
everything a member can do — surface → domain → feature → flow → step, with
explicit contains/follows edges, a canonical depth-first walk order, and CI
integrity gates (duplicate ids, unregistered anchors, empty containers,
malformed routes all fail the build). The curated graph
(@oshun/shell-assistant/feature-catalog, 100+ nodes across the shell and all
six domains, versioned via FEATURE_CATALOG_VERSION) syncs into a durable audit
store (snapshot-persisted like the session checkpoint, subject erasure fenced):
each member gets one active audit run materializing every flow as pending →
visited/skipped, with skips REQUIRING a reason. The assistant's audit mode
drives it through audit_begin / audit_status / audit_mark tools — strict
catalog order, one flow at a time, per-domain coverage rollups restated every
turn, tours chained in when steps carry anchors — and
GET /v1/assistant/feature-catalog + /v1/assistant/audit-runs* expose the
same state over HTTP.
Codex CLI subscription routing. SHIPPED 2026-08-03. CodexExecProvider in
@oshun/ai wraps the real codex exec binary (read-only sandbox, model
- reasoning-effort flags, injectable runner) behind the shared
LLMProviderInterface, so OpenAI-model turns can ride a ChatGPT subscription instead of metered API billing:OSHUN_ASSISTANT_PROVIDER=codex-cli(explicit-only — never auto-detected, since it shells out) withOSHUN_ASSISTANT_CODEX_MODEL/OSHUN_ASSISTANT_CODEX_EFFORT/CODEX_CLI_COMMAND. Capability honesty: the CLI cannot do tool calling or token streaming, so the binding carriessupportsTools: false, tool-carrying requests are rejected loudly, the turns route swaps in a tool-less conduct prompt ("you cannot read member data in this mode — say so"), and the finished reply streams as one delta.
Codex subscription routing v2 — DESIGN (2026-08-03, after user critique).
The exec wrapper above is the shallow design: it treats the CLI binary as the
capability boundary, when the real boundary is the ChatGPT backend Responses API
that the CLI itself talks to. Codex CLI authenticates via OAuth (tokens in
~/.codex/auth.json, maintained by codex login) and calls that API with full
tool calling, streaming, and reasoning — that is how its own agent loop works.
Any client presenting those tokens gets the same capability; the subscription is
account-bound, not CLI-bound. Proof in the wild: pi (earendil-works/pi, formerly
badlogic/pi-mono) ships an "OpenAI Codex" provider in its standalone
@earendil-works/pi-ai library (MIT) — OAuth device-code/browser login, tokens
auto-refreshed, full tool calling + token streaming through the subscription,
documented as "officially endorsed by OpenAI: Codex for OSS". Three candidate
architectures, evaluated:
- A (recommended) — native subscription transport provider. New
CodexSubscriptionProviderin@oshun/aiimplementingLLMProviderInterfaceover the ChatGPT-backend Responses API, using@earendil-works/pi-aias the transport/auth layer (thin adapter: itstext_delta/toolcall_*/donestream events map 1:1 onto ourStreamCallbackvocabulary; itstoolCall/toolResultblocks map onto our ChatMessage tool roles; TypeBox tool params ← our JSON-schema ToolDefinitions pass through). The ENTIRE existing stack — agent-turn-runner bounded loop, all 30+ domain/UI/audit tools, paused-turn client tools, tours, safety supersede, turn metrics — works unchanged; the assistant binding flips tosupportsTools: trueand full SSE streaming. pi-ai absorbs endpoint/header churn and token refresh. Login is a one-time operator step (device-code headless supported). Caveat stated honestly: this uses the subscription outside the official CLI; pi's flow follows the OpenAI-endorsed Codex-for-OSS pattern, but it is the operator's call — the binding stays explicit-only (OSHUN_ASSISTANT_PROVIDER=codex-subscription), never auto-detected. - B — MCP inversion over the official CLI. Codex CLI is an MCP client
(
mcp_serversin~/.codex/config.toml): spawncodex exec --jsonper turn with a generated config pointing at a local MCP server that exposes that turn's assistant toolset, parse the JSONL event stream into turn frames. Fully official surface, but per-turn process spawn + MCP handshake latency, codex's coding-oriented loop replaces our runner (losing bounded rounds, per-round safety recheck, metrics granularity, paused-turn client tools), and tool auth context must round-trip through an extra process boundary. Kept as the documented fallback if the transport path is ever foreclosed. - C — text-only exec wrapper (shipped above). Honest but toolless; remains
as the zero-dependency fallback (
codex-clibinding id unchanged).
Verdict: A. The critique was correct — wrapping stdout was designing to the CLI's surface instead of to the subscription's actual capability.
Codex subscription routing v2 — SHIPPED 2026-08-03 (option A).
CodexSubscriptionProvider (@oshun/ai providers/codex-subscription.ts)
implements the shared LLMProviderInterface over pi-ai's openai-codex
provider: full tool calling, incremental token streaming, reasoning effort, and
backend-reported usage on subscription billing. CodexCliCredentialStore
(providers/codex-cli-auth.ts) bridges pi-ai's CredentialStore contract to
the Codex CLI's own login file ($CODEX_HOME/auth.json): reads map the CLI
format to an OAuth credential (expiry decoded from the access-token JWT),
refresh write-backs persist in the CLI's format under a serialized write queue
with permissions clamped to 0600 — one login, two consumers, tokens rotated by
either stay valid for both. Scope refusals are explicit: the store manages
exactly the openai-codex credential, refuses non-OAuth overwrites, and refuses
delete() (that is codex logout's job). The provider rejects loudly what the
transport cannot do (named-tool forcing, responseFormat, topP, stopSequences,
URL images) and never surfaces reasoning ("thinking") output as member-visible
text. BFF binding: OSHUN_ASSISTANT_PROVIDER=codex-subscription (explicit-only,
never auto-detected), supportsTools: true, sharing
OSHUN_ASSISTANT_CODEX_MODEL / OSHUN_ASSISTANT_CODEX_EFFORT with the exec
binding. Per the 2026-08-03 model-family directive, defaults moved off gpt-5.4
to GPT-5.6 Terra (balanced tier) across codex-subscription, codex-cli, and
the native OpenAI assistant path; Sol (flagship) and Luna (fast) are
model-override choices. pi-ai (@earendil-works/pi-ai, MIT) rides the pnpm
catalog. Verified by unit specs (context/event/error mapping, credential
round-trip) and a live tool-calling round-trip on the real subscription.
P4 — Unification + intelligence. SHIPPED 2026-08-03 (core slice). One brain
now serves four surfaces: the turns route gained a buffered JSON mode
(Accept: application/json — same pipeline, one reply carrying the terminal
payload + UI intents) for clients that cannot consume SSE, and the mobile
sheet upgraded from the deterministic route to the agent brain
(sendAssistantTurn first, deterministic sendAssistantMessage as the
in-protocol fallback, the on-device keyword model remaining the labeled offline
fallback). Measurable quality and cost: a durable turn-metrics ledger (per
provider/model turns, outcomes, tokens, tool usage, latency avg/max) records
every terminal — completed, refused, provider_error, crisis, blocked,
budget_exhausted — and serves operators at GET /v1/assistant/metrics (admin
scope). Budget enforcement: OSHUN_ASSISTANT_DAILY_TOKEN_BUDGET sets a
per-member daily output-token cap, checked BEFORE provider spend; over-budget
turns answer 429 assistant_budget_exhausted and clients fall back to the
deterministic assistant — restart-safe, honest, never silent rationing.
Selection-to-ask shipped: select text anywhere (outside
data-assistant-private subtrees), an "Ask Oshun" chip appears, and the
assistant opens with the excerpt prefilled. Deferred from P4 with reasons:
docs-center RAG (needs an embedding pipeline and a decision on shipping the
corpus alongside the BFF), proactive help triggers (psyche detectors still
dormant), invocation-point bus cleanup, and an eval harness beyond the metrics
ledger.
Docs-center RAG — SHIPPED 2026-08-04 (deferred-tail item 1). The assistant
now grounds documentation answers in the docs-center reader estate via a
search_docs tool. tools/build-assistant-docs-search.mjs joins the
generator-maintained page registry (vdocs-search-index.js, 3,130 reader pages,
freshness-gated by pnpm docs:center:check) with body text extracted from the
uniform generated HTML (<main> sections split at headings, chunked ≤900 chars)
into two JSONL corpora: member (docs-center/** only — 317 curated pages,
6,485 chunks, 5.1MB, checked in and copied into dist/generated at build) and
**full** (the whole estate incl. engineering docs — 54,268 chunks, 44MB,
gitignored, built per machine). The scope split is an information-disclosure
boundary: member sessions can never surface engineering-internal pages;
admin-scoped sessions overlay the full corpus where present. Retrieval is BM25
(k1=1.2, b=0.75) with weighted field frequencies (title ×2.5, heading ×1.75 —
simplified BM25F) in bff/src/assistant/docs-search.ts — a real, named lexical
algorithm chosen so retrieval needs no runtime credentials; an embedding
reranker can layer on later without changing the contract. Per-page result cap
(2) diversifies sources; snippets window the first match; links carry section
anchors. Fail-closed: no corpus → the tool is not offered; an explicit
OSHUN_ASSISTANT_DOCS_INDEX_DIR override is authoritative and refuses to fall
back elsewhere. The conduct prompt instructs grounding + naming the source page,
and saying so when the docs have no answer. Verified: 13 unit tests
(known-correct BM25 orderings, boost/diversification/snippets, loader
fail-closed semantics, member-safety integrity guard over the committed corpus),
route test feeding real curated excerpts back to a scripted model (15/15), full
assistant suite 63/63. A live-model grounding run was attempted and blocked by
the ChatGPT subscription's rolling usage limit (the probe DID harden the
provider: "usage limit has been reached" now classifies as retryable
rate_limit_error); the identical route+toolset live path was proven earlier
the same day.
Eval harness — SHIPPED 2026-08-04 (deferred-tail item 2). Golden-turn
behavioral evals at the real route surface (auth → safety → toolset → runner →
SSE) in bff/src/assistant/evals/: a PURE grader over a transcript observation
extracted from the SSE frames (tool selection, forbidden tools, final-text
vocabulary, crisis supersede, tour-plan validity through the shared validator,
UI intents — every failure a specific sentence), an executor that runs one case
through a Fastify instance, and a 7-case golden catalog (tara/veritas tool
selection, navigate, docs grounding, honest-absence heuristic, provider-free
crisis supersede, tour startup). The live run (assistant-golden.eval.ts,
eval:assistant script, vitest.eval.config.ts) is excluded from the normal
vitest include via the .eval.ts suffix and credential-gated — no binding →
reported SKIPPED, never passed; tokens are spent only on explicit operator
request against whatever provider the environment resolves, making
provider/model swaps regression-checkable. The harness itself is fully tested
deterministically (14 specs) including negative controls: a deliberately
wrong-tool scripted run must FAIL its case with the right failure sentences.
Stated limits: the SSE wire carries tool names/ outcomes but not arguments, so
v1 grades selection and effects, not argument values (runner-event extension is
the path to arg-level grading); honest-absence is a negation-vocabulary
heuristic. Landmine recorded: vitest mergeConfig CONCATENATES test.include —
the eval config assigns its include after the merge, else the entire normal
suite rides along.
Audit coverage board — SHIPPED 2026-08-04 (deferred-tail item 3). The
chat-driven audit mode gained its visual counterpart at /assistant/audit
(web/src/components/assistant/AuditCoverageBoard.tsx +
lib/assistant/audit-coverage.ts): overall and per-domain segmented coverage
bars (done/skipped/pending with a real progressbar ARIA contract), the
next-pending flow card (breadcrumb, steps with routes, expected outcome), a
catalog-version drift warning, and honest states throughout — loading, typed
errors shown verbatim with retry, and a no-run state that reports the real
catalog scale from the graph endpoint. Both directions of the chat handoff work:
"Start the audit" begins the durable run and opens the assistant with an audit
prompt; "Continue this flow in chat" dispatches the shared assistant-open event
with a resume prompt. Chat and board read/write ONE durable audit authority, so
they can never disagree. Verified: 11 unit specs AND a full Claude-in-Chrome
pass against the live dev stack (signup → onboarding → redirect-preserved
landing → no-run state with real counts → begin → live board → chat handoff
advancing the panel transcript; zero console errors). Browser verification
caught a real bug unit tests could not: Fastify rejects an empty POST body under
a JSON content-type — the begin call now sends {{}} and the spec locks it.
Dev-stack landmines recorded in memory (web MUST run on port 3010/3011 for BFF
CORS; BFF durable boot needs redis + postgres URLs, the signup HMAC secret, and
the autonomy snapshot key pair).
Proactive help triggers — SHIPPED 2026-08-04 (deferred-tail item 4).
Reactive→proactive, restraint-first. lib/assistant/proactive-triggers.ts is a
PURE detector engine (unit-testable with synthetic event streams, no DOM):
rage_click (≥4 clicks/1.2s inside a 24px radius), nav_thrash (A→B→A→B route
bounce within 12s), error_dwell (an error surface visible ≥8s with no
interaction). Restraint is enforced in the engine, not the caller: per-kind
5-minute cooldowns, a 2-offer-per-page-load cap, suppression while the panel is
open, and a persistent member opt-out honored before any detection. Detection is
deliberately client-side — deterministic, private (no interaction telemetry
leaves the browser) — rather than waiting on the dormant psyche server
detectors. ProactiveHelpChip.tsx (root layout) is presentation only: a small
dismissible chip that NEVER auto-opens the panel — accept opens the assistant
with a grounded struggle prompt (source proactive-help), "Not now" dismisses,
"Don't offer again" persists the opt-out; clicks inside data-assistant-private
subtrees are ignored. Verified: 14 unit specs (every threshold with matching
negative cases, the full restraint battery) + a live Claude-in-Chrome pass where
the error_dwell detector fired on a genuinely visible alert, the chip rendered,
and "Not now" removed it without opening the panel.
Invocation-point bus cleanup — SHIPPED 2026-08-04 (deferred-tail item 5,
deferred since P1). Every assistant launch in the customer web app now goes
through openAssistantInvocation(pointId, detail) in
navigation/assistant-entry.ts — 32 call sites across 20 files migrated off the
raw dispatchOshunAssistantOpen CustomEvent bus. The wrapper enforces what the
client can honestly know (the point must exist in the registry; the viewport
must meet the point's minimum — refusals warn and dispatch nothing) and
deliberately does NOT claim auth/entitlement checks the browser cannot verify
(the BFF enforces those on every request; client-side claims would be theater).
Each launch now carries a canonical AssistantLaunchIntent (point id, entry
source, active path, seed) built by the registry's own
buildAssistantLaunchIntent, replacing ad-hoc source strings. The registry
gained three real points for surfaces built this week
(customer-web.selection-ask, customer-web.proactive-help,
customer-web.audit-board). Drift is CI-blocked by
invocation-bus-guard.spec.ts: a source scan fails on any raw dispatch outside
the sanctioned dispatcher, and on any openAssistantInvocation id missing from
the registry (with a floor assertion so an empty scan cannot masquerade as
cleanliness). Verified: shell-assistant 440 specs + web assistant 89 specs
green. Landmine: web tsc --noEmit does NOT cover app components (a deliberate
type error in HeroBanner passed) — component type safety rides Next build +
eslint, so migration sweeps must be verified by parse/lint + tests, not tsc
alone.
Per-org budgets + plan entitlements — SHIPPED 2026-08-04 (deferred-tail item
6). Two additions to the P4 cost machinery, both enforced BEFORE provider
spend. (1) Tenant (org) budgets: the turn-metrics ledger gained a per-tenant
daily ledger keyed by the auth token's tenant claim
(OshunAuthContext.tenantId), snapshot schema v2 with an honest v1 migration
(no historical tenant usage invented);
OSHUN_ASSISTANT_TENANT_DAILY_TOKEN_BUDGET caps the org-wide sum — a member of
a spent org gets 429 assistant_budget_exhausted with scope: 'tenant' while
other orgs pass; the admin metrics endpoint now reports tenantsToday rollups.
(2) Plan entitlements (assistant/plan-entitlements.ts): the product's real
billing tiers (free|pro|premium via the same effectiveBillingPlan chain the
domain entitlement middleware rides) map to assistant capabilities — free: 20k
output tokens/day + browser-only voice; pro: 200k + server voice; premium:
unmetered + server voice. Member budget precedence: operator env override > plan
default; no billing record fails closed to free; admin-scoped sessions are
operators, not metered members. Server voice routes (STT upload + TTS) 403
assistant_voice_not_in_plan for free members — browser Web Speech remains
their honest fallback. Route seams (planResolver, tenantDailyTokenBudget)
injectable via createApp for tests. Verified: 12 store/plan specs (tenant
summation across members, UTC day roll, v1→v2 snapshot round-trip, precedence
table) + 23 route specs including cross-member tenant exhaustion with another
org passing, free→429/premium→200 plan budgets, and free→403/pro→200 voice
gating.
Telemetry sinks + feedback pipeline — SHIPPED 2026-08-04 (deferred-tail item
7). The engine's AssistantEvent stream, emitted since P0 with no consumer,
now has a durable one: assistant/telemetry-sink.ts subscribes via the engine's
listener seam and keeps snapshot-durable COUNTERS (per event type, totals,
first/last timestamps; persisted at most once per 25 events so chat cannot
write-amplify the store) plus an in-memory redacted sample ring (200). Redaction
is allowlist-by-shape — numbers, booleans, strings ≤80 chars — message content
never enters telemetry. Served at GET /v1/assistant/telemetry (admin scope).
The feedback pipeline records member thumbs as LABELS
(assistant/feedback-store.ts): verdict + turn reference + provider/model,
never message text (the transcript already lives in the conversation log with
its own retention/erasure fences; training export joins by sessionId/turnId).
One label per (member, turn) with upsert (members may change their mind), a
5,000-entry FIFO cap, eraseSubject so a deleted member's labels disappear too,
and snapshot durability. Routes: POST …/turns/:turnId/feedback (session-owned,
400 on invalid verdicts, provider/model attribution from the live binding) and
GET /v1/assistant/feedback (admin export: summary by verdict and
provider/model + newest-first entries). Web: the streamed turn id now rides
AssistantTurnCompletePayload, and assistant bubbles with a turn id render
Helpful / Not helpful actions — a failed POST reverts the local verdict rather
than pretending. Verified: 9 store specs, 19 route specs (upsert, validation,
cross-member 403, admin-only export/telemetry, member 403), panel + turn-stream
suites green (38).
Mobile parity depth — SHIPPED 2026-08-04 (deferred-tail item 8). Three gaps
between the mobile sheet and the web panel closed at the client layer. (1)
Screen grounding: agent turns now carry a pageContext built from the
sheet's context descriptor (route, label, summary), mirroring the web's page
grounding — no client-tool round-trip needed. (2) Structured navigation: the
JSON turn payload's {domain, path} navigateTo (and new turnId + uiIntents,
which the server's turn.complete now includes for JSON-mode clients) is typed in
the mobile client, and the sheet follows it with a real route push through the
domain-route builder — guarded by a navigable-domain allowlist; the
deterministic route's string form stays advisory text. (3) Server voice
methods: transcribeAssistantAudio (raw bytes → the same fail-closed STT
route web uses; 403 plan / 503 unconfigured reasons surface typed so callers
fall back to on-device speech honestly) and fetchAssistantSpeech (TTS →
base64 + provider content type for expo-audio playback, via a pure tested base64
encoder — RN has no Buffer/btoa). Verified: 28 jest tests green (client turn
body incl. pageContext, audio upload + 403 mapping, speech base64 round-trip,
base64 padding vectors, sheet agent-path grounding + navigation push with the
deterministic fallback proven unused, and the original S11 offline-fallback
contract preserved). Honest scope note: the sheet's mic/speaker UI atop these
methods is not yet wired — that lands with an emulator-verified pass (the voice
transport and its refusal semantics are what shipped here).
Member tour authoring + operator review — SHIPPED 2026-08-04 (deferred-tail
item 9, deferred since P2). The authoring loop the curated catalog was waiting
for: submit_tour_for_review lets a member's composed tour enter a durable
review queue (assistant/tour-store.ts), validated at submission against the
SAME anchor-registry validator the player uses — invalid plans are refused with
correctable reasons, never queued. Restraint built in: 5 pending per author,
duplicate pending titles refused, 500-entry queue cap, eraseSubject removes a
deleted member's submissions in every status. Operators review at
GET /v1/assistant/tour-submissions + POST …/:tourId/review (admin scope;
rejections REQUIRE a note the author can act on; no double review). Approved
tours join list_curated_tours labeled authoredBy: 'member' and start via
start_tour({approvedTourId}) — with REVALIDATION at start so a tour whose
anchors drifted after approval fails with reasons, not mid-playback. Nothing a
member wrote runs on another member's screen without a human decision — the same
doctrine as the curated catalog. Verified: 4 store specs (lifecycle, caps,
durable round-trip, erasure) + a full route lifecycle test (tool submit → member
403 on review → operator approve → queue reflects → start-by-id fires the
tour_start UI intent with the member's plan); assistant suite 98 specs green.
"Do it" tier — confirm-first actions — SHIPPED 2026-08-04 (deferred-tail item
10). The tell/show/do escalation is complete. The four write-class domain
tools (tara_add_favorite, veritas_save_article, veritas_follow_topic,
nyx_log_observation) — which previously executed on the model's say-so — are now
marked mutating with human confirmation sentences. When the client declares
the new action_confirm capability, a mutating call does NOT execute: it parks
in assistant/action-confirm.ts (in-memory, 5-minute TTL, one-shot,
session+member-bound — a confirmation is a seconds-scale UI handshake, not
durable state), a turn.ui action_confirm intent renders a card in the panel,
and the tool result tells the model the truth ("held for member confirmation —
never claim it happened"). The member's decision POSTs to
…/sessions/:sessionId/actions/:actionId: confirm executes the parked adapter
call exactly once and returns its real result; decline discards without side
effects; both land in engine-style telemetry
(action.executed/action.failed). Clients WITHOUT the capability keep direct
execution (mobile until it grows a card) — the gate is explicit, not silent.
Web: panel declares the capability, renders the card above the composer, and
reports outcomes as system notes — success, decline, and failure alike.
Verified: 5 bridge specs (one-shot, decline purity, TTL, session/member binding,
honest failure), 2 route lifecycle tests (card frame
- intruder 403 + confirm executes + replay 404; capability-absent direct path preserved), panel + stream + shell-assistant suites green (38 + 440).
Voice depth: VAD auto-stop + word timings — SHIPPED 2026-08-04 (deferred-tail
item 11, closing the P3 deferrals). (1) VAD auto-stop:
SilenceAutoStopDetector in web lib/assistant/voice.ts — a pure
energy-threshold detector with hangover (speech ≥300ms cumulative must be heard
before silence can ever trigger; 1.8s continuous quiet after speech stops the
mic; a shorter mid-sentence pause keeps it open). Wired into
startVoiceRecording({autoStopOnSilence}) via an AnalyserNode RMS loop that
degrades to manual stop if audio-graph setup fails; the panel opts in, the
one-minute hard cap still applies, and the manual stop button remains. (2)
Word timings: Deepgram's per-word timings (already produced by
DeepgramPrerecordedProvider) now flow through the STT binding and the audio
route as an optional words array — absent when the provider (Whisper) does not
report them, never fabricated — and captionWindowAt(words, positionMs) is the
pure caption engine: active-word window with neighbors, holds the last spoken
word between words, honest null/-1 for empty and pre-speech positions. Verified:
12 voice-lib specs (detector streams incl. never-stop-without- speech and
pause-tolerance negatives; caption windows incl. tail and pre-speech), BFF voice
suites 11 green, panel 27 green. Karaoke-style caption RENDERING atop this
engine is UI polish for a browser-verified pass; the timing engine, transport,
and auto-stop are what shipped here.
E2E journey inventory → exhaustive audit catalog — SHIPPED 2026-08-04.
Answering "are the e2e suites loaded into the audit system?" — they were not;
now they are. tools/build-assistant-journey-inventory.mjs statically mines
every e2e suite in the repo — 544 customer-web + 82 admin-web + 1 tenant-admin
- 1 telegram-miniapp Playwright specs and 84 mobile Maestro flows — into a
checked-in inventory (712 journeys, 5,857 proven checks, content-hashed):
titles, describe blocks, in-app
gotoroutes, Maestro commands; nothing invented (a check-less suite appears with an explicit marker step).bff/src/assistant/journey-inventory.tscomposes it into the catalog graph undere2e-<app>surfaces grouped by the suites' own filename taxonomy — validated by the SAME validator as the curated catalog, failing closed to the curated spine, version-pinned as<curated>+e2e.<hash>so in-flight audits stay coherent when suites change. The audit singleton and the feature-catalog endpoint now serve the composed graph: audit mode walks 735 flows (curated spine first, canonical order preserved) instead of 23. Composition immediately exposed a REAL bug: full per-domain rollups in audit tool results blew the 8,000-char tool-output cap and truncated mid-JSON — audit_begin/status/mark now return a bounded summary (top-10 pending-heaviest domains + omission note) while the HTTP endpoint keeps the complete rollup. Verified: 4 composition specs (inventory floors, validator pass, spine-leads-walk ordering, 735-flow audit run) + full assistant/route suites 129 green.
Journey correctness + PRD traceability + graph guarantees — SHIPPED
2026-08-04. Three analysis layers over the e2e-composed catalog, answering "is
this correct, tied to the feature model, and properly encoded for graph
processing?". (1) Extraction fidelity: the miner is now AST-based (real
TypeScript syntax tree — modifier chains, parameterized template-literal titles
marked ⟨param⟩) with a regex cross-check pass that FAILS THE BUILD on
disagreement; the upgrade immediately recovered 23 checks the regex missed
(5,880 total). (2) Traceability: a new verifies edge kind ties e2e
journeys into the curated (PRD-level) feature model through two real signals —
filename-group → curated domain, and route-overlap → specific curated flow
(prefix-normalized). Live report: 354 verifies edges; 7/7 curated domains
verified; 9/23 curated flows route-verified with the 14 gaps LISTED BY ID;
219/712 journeys mapped and 493 honestly reported unmapped (no wishful grouping
— coverage claims come from edges only). Served at
GET /v1/assistant/catalog-traceability (admin scope). (3) Graph processing
guarantees: checkCatalogGraphIntegrity proves contiguous canonical walk
order, full connectivity to surface roots with cycle detection, edge-endpoint
existence including appended cross-edges, and verifies-edge directionality (e2e
→ curated only) — 0 errors on the live graph, and the checker itself is
negative-control tested (dangling edge, backwards verify, broken order must all
go red). Verified: 7 composition/ traceability specs, assistant suites 132,
shell-assistant 440.
Total curation pass — SHIPPED 2026-08-04. "Make sure everything is mapped
and curated" is now an enforced invariant, not an aspiration. The curated
catalog grew from the member spine to the FULL product estate
(FEATURE_CATALOG_VERSION 2026-08-04.1): six new customer-web platform domains
(studio, account & membership, shell platform services, quality/access/
resilience, assistant & intelligence, creative & cross-product content) and four
new surfaces (admin console with operations/governance/content-ops, tenant
admin, Telegram mini-app, customer mobile) — every domain with a real
representative flow and steps. e2e-curation-map.ts records ~120 explicit
group→target curation decisions plus file-level flow aliases, and
CURATED_FLOW_WAIVERS holds the five honest admissions (command palette,
assistant voice, assistant tour, nyx reminders, metis enroll — each says "write
the e2e suite", and a stale-waiver check fails when one becomes verified).
Coverage propagates up (a verified flow verifies its ancestors); curation-map
targets are integrity-checked against the graph. Final state: 1,117 verifies
edges; 712/712 journeys mapped (0 unmapped); 19/19 curated domains verified;
30/35 flows verified + 5 reasoned waivers; 0 stale waivers; 0 integrity errors
— and the TOTALITY spec enforces all of it in CI, so any future e2e group lands
only with a curated home. Suites: BFF assistant 133, shell-assistant 440
(catalog structural spec generalized to multi-surface: spine leads, contains =
nodes − surfaces).
Later / optional: realtime speech-to-speech over the WS route; avatar-embodied mode (tavus/viseme stack); MCP tool federation for operator integrations; A2UI-style generative UI islands in the panel.
5. Features beyond the ask (recommended)#
- Three-tier answers: "Tell me / Show me / Do it." Every capable answer offers escalation: explain in text → run a live tour → execute the action (confirm-first). This is the single biggest UX differentiator and falls out of §3.4 + §3.5 almost for free.
- Selection-to-ask. The handoff already carries the user's text selection — add a floating "Ask Oshun" affordance on selection anywhere (explain this claim, summarize this passage, is this consistent with my plan?).
- Proactive rescue, politely. Wire the dormant psyche
help-offer-triggers/engagement-detector: repeated failed searches, rage-clicks, error banners, or long stalls on a form trigger a quiet assistant chip ("Stuck on X? I can walk you through it") — heavily rate-limited, dismissible, off by default for accessibility users who opt out. - "What's new" narration. On login after a release, the assistant offers a 30-second personalized changelog tour of features relevant to that user's usage — generated from docs-center release notes + their activity.
- Deep-linkable prompts. The declared
customer-web.deep-linkinvocation point becomes/assistant?prompt=…&source=…— support docs, emails, and error states can link straight into a prefilled assistant conversation ("this error explained"). - Admin triage copilot. Draft responses in the inbox, summarize a review item with evidence links, one-tap incident acknowledgment with a generated timeline — the invocation points for exactly these three exist already.
- Background watchers. "Tell me when my Veritas story gets new evidence" —
the dormant
libs/oshun/assistantscheduling stack (cron, NL schedules, redis leases) runs agent tasks that deliver into notifications and the assistant thread. - Voice memos → structured actions. Async voice is better than realtime for "capture this thought": record → transcript → agent turns it into a journal entry, reminder, or library save with a confirm card.
- Cross-surface continuity. Start on web, continue on mobile: sessions are already durable server-side; the continuity-journey rail and iris presence/sync libs make "hand this conversation to my phone" a small feature, not a project.
- Why-did-you-say-that provenance. Expand the grounding chips into a per-answer provenance panel: which tools ran, which docs were cited, which memory was recalled (memory consent UI already exists) — trust UX that doubles as a debugging surface.
- Screen-reader-symmetric tours. Because tours run on the a11y-labeled
anchor registry with
aria-livenarration, the same tour works for sighted and screen-reader users — a genuine accessibility differentiator over every commercial tour product. - Tour analytics as product telemetry. Step-level drop-off from the TourPlayer is a direct map of where the product confuses people — feed it back into the docs-center and design backlog.
6. Key decisions and rationale#
| Decision | Choice | Why |
|---|---|---|
| Brain location | BFF service (not client-side, not separate service) | Auth, safety, memory, history, budgets all already live there; one hop from domain adapters |
| Model / providers | Anthropic, OpenAI, and OpenRouter all first-class behind one provider interface (P0: shipped) | Anthropic via the native @anthropic-ai/sdk (default claude-opus-5, $5/$25 per MTok, 0.1× cache reads → ~$0.02/turn warm), OpenAI via the official openai SDK, OpenRouter for aggregated models (DeepSeek et al.); OSHUN_ASSISTANT_PROVIDER or first-available API key selects; fail-closed to the deterministic engine with no keys |
| Protocol | In-house typed SSE events, AG-UI-shaped | Panel and BFF are bespoke and 80% built; adopt the pattern, skip the dependency; can adopt AG-UI proper later if we want third-party agent frontends |
| Page grounding | Anchor registry + typed selectors + a11y-tree tool | More precise and radically cheaper than screenshots; we own the DOM so curation beats generic scraping |
| Tours | LLM plans (schema-validated) + deterministic player | Reliability: generation-time intelligence, execution-time determinism; matches the pattern every serious tour product converged on |
| Voice v1 | Async STT/TTS via dormant psyche provider libs | Explicitly allowed by requirements; ~90% of the value at ~10% of the complexity; upgrade path to realtime preserved |
| Reuse vs rebuild | Reuse safety/memory/history/panel/invocation/persona wholesale | They are real, tested, and governance-reviewed; the gap is the brain and the wiring |
Main risks: action safety (mitigate: grant allowlists, confirm-first cards, audit trail, no destructive tools in v1); tour rot (mitigate: manifest CI guard); cost creep (mitigate: caching discipline, budgets/kill switches, Haiku routing, curated-tour flywheel); prompt-injection via page content (mitigate: a11y snapshots are data-not-instructions framing, redaction, tool grants scoped per surface); the three-surface drift recurring (mitigate: the shared headless client package is the only sanctioned transport).
7. Sources#
Research consulted 2026-08-03:
- AG-UI protocol — docs.ag-ui.com, github.com/ag-ui-protocol/ag-ui, CopilotKit on AG-UI, AWS Bedrock AgentCore + AG-UI, Google A2UI, AG-UI vs MCP-UI vs A2UI
- Copilot frameworks — CopilotKit, assistant-ui alternatives 2026, OpenAI ChatKit, ChatKit.js, Apps SDK examples
- Page grounding — Playwright MCP snapshots, Building browser agents (arXiv 2511.19477), DOM vs screenshots, rtrvr DOM intelligence
- Tours — open-source tour libraries 2026, AI onboarding platforms 2026, product walkthrough tools
- Voice — STT APIs 2026 benchmarks, Deepgram STT comparison, AssemblyAI TTS roundup, voice-agent pipelines
- Anthropic API — current model catalog, pricing, tool runner, prompt caching, and streaming semantics from the bundled Claude API reference (verified 2026-08).
8. Product-graph restructure. SHIPPED 2026-08-04 (phases 0–7).#
The five parallel node vocabularies (curated catalog, e2e inventory, docs
corpora, anchors/tours/invocation points, TODOS/architecture prose) are now ONE
typed, content-addressed property graph — @oshun/product-graph — with a
kind-discriminated node model (18 kinds spanning both planes), 12 edge kinds
with validator-enforced endpoint rules, and REQUIRED provenance on every record.
What moved where:
- Kernel (
libs/oshun/product-graph/src/): types + kind-aware validator (containment forest, pre-order DFS blocks, knowledge/intent plane split, route path uniqueness + orphan rules, supersedes acyclicity), index-backed traversal (propagateUpgeneralizes the coverage rollup), canonical-JSON content addressing, andcomposeVersion— the Merkle version scheme:pg1.<composed12>+<section>.<hash12>+…over ELEVEN section hashes. Any change in any source changes the composed version; string-stitching is gone. - Compilers, one pure function per source: curated catalog (exact
flattenFeatureCatalogparity, ids preserved for audit-store rows), e2e inventory (journeys/checks as REAL node kinds), curation tables, routes (first-classroutenodes +visitsedges + observed-prefix containment), anchors, tours, invocation points (new call-site minertools/build-assistant-invocation-inventory.mjs), docs reader estate (3,130docPagenodes from the new checked-indocs-graph-inventory.json), TODOS corpus (~95ktasknodes, checkbox state verbatim as DATA, honestunlinkedcount), and architecture/design prose (SHIPPED sections + dated decision paragraphs mined verbatim for the future ADR backfill — nothing fabricated). - Route overlap is a JOIN now (journey —visits→ route ←visits— step —parent→
flow), surface-scoped. The old string-prefix heuristic survives only as the
reference oracle in
route-join-parity.spec.ts, which accounts all 32 consciously-dropped cross-surface accidents edge by edge; two journeys whose only mapping was accidental got explicitadmin-web:auth/admin-web:rbacaliases. Totality after the swap: 712/712 journeys mapped, 30/35 flows verified, unverified set == the five reasoned waivers EXACTLY. - Curation moved into the lib (
src/curation/e2e-curation-map.ts+graph-curation-map.ts— tour/invocation/docs/arch-doc/TODOS tables, all totality-enforced with reasoned waiver classes; stale entries fail). - Artifact (
src/generated/): manifest + per-section JSONL, built bytools/build-product-graph.mjs(121k nodes / 123k edges). The 67 MB todos bulk is a gitignored derived cache whose hashes stay pinned in the manifest (verify: truereads refuse stale/tampered sections). CI gates: staleness (fresh recompile == disk) and TOTALITY on the composed graph. - Consumers are views (phase 5):
journey-inventory.tsis a pure adapter reconstructing the audit taxonomy FROM graph nodes —journey-inventory-graph-parity.spec.tsproves the served views are byte-identical to the pre-rewrite golden (node-id sequence hash) — andanalyzeCatalogTraceabilityruns on kernelpropagateUp. Docs-search hits carrydocumentsCuratedannotations joined throughdocumentsedges (curated vocabulary only — the member/full boundary is spec-pinned). Coverage board verified live in Chrome against the graph-backed BFF. Version-string decision (5.1): audit runs keep pinning${FEATURE_CATALOG_VERSION}+e2e.<hash>— the string's meaning (curated content + e2e inventory content) is unchanged, so no run migration was needed; the Merklepg1.…version is the ARTIFACT's identity, carried in the manifest and both projections. - Projections (phases 6–7): Postgres
product_graph_node/_edgeinoshun_dev(Prisma migration20260804140000), transactional full-replace loader behindOSHUN_PRODUCT_GRAPH_PG_SYNC=1, SQL pack asserting exact kernel answers incl. the TOTALITY restatement; Neo4j viatools/load-product-graph-neo4j.mjs(label per kind, relationship per edge kind, constraint-before-load) with a gated Cypher pack matching the analyzer exactly. Both are disposable projections of the artifact. - Landmines found: the compose
graphprofile had NEVER booted Neo4j 5 (read-only conf mount broke the entrypoint chown; env-driven APOC settings trip v5 strict validation;kalika_knowledgeis an illegal v5 database name — all three repaired, database renamedkalika-knowledge). PostgresORDER BYcollation differs from JS sort on punctuation — SQL/kernel diffs must compare sets. Marking a checkbox in ANY tracked TODOS file changes the todos section hash: regenerate the artifact in the same commit (the staleness gate enforces it).
9. Workbench MCP dogfood transcript (12.3b). RUN 2026-08-04.#
Live evidence for the coding-agent handoff surface: a REAL MCP stdio connection
(SDK client → tools/workbench-mcp/server.mjs → live BFF on :4010 with a scoped
workbench token → real Postgres rows). The server is registered project-scope in
.mcp.json (token via env), so any fresh Claude Code session picks up
mcp__oshun-workbench__* on connect; this session predates the registration, so
the sweep below drove the SAME server over an explicit SDK client connection —
every tool, with the refusals proven over the wire, plus the CLI fallback sweep.
MCP client sweep (verbatim)#
=== tools/list (real MCP handshake) ===
[
"workbench_queue",
"workbench_lease",
"workbench_brief",
"workbench_report",
"workbench_shipped",
"workbench_verify"
]
=== workbench_queue ===
{
"items": [
{
"id": "wi-5085ae62-8dae-463b-a987-f5ddc5198f90",
"kind": "issue",
"title": "Dogfood: tara courses screen lacks a retry state on failed loads",
"priority": "high",
"graphRefs": [
"tara",
"tara.courses.progress"
],
"orphanedRefs": [],
"createdAt": "2026-08-04T15:19:19.848Z"
},
{
"id": "wi-eeac8712-6e40-40e1-b97b-8e4f4faf22e2",
"kind": "issue",
"title": "Tara enroll flow loses the schedule panel on narrow viewports",
"priority": null,
"graphRefs": [
"tara",
"tara.courses.progress"
],
"orphanedRefs": [],
"createdAt": "2025-08-04T09:33:21.000Z"
},
{
"id": "wi-18c55602-0f70-4214-bb1b-c97997c41643",
"kind": "task",
"title": "Dogfood: short-TTL lease expiry check",
"priority": null,
"graphRefs": [
"tara"
],
"orphanedRefs": [],
"createdAt": "2026-08-04T15:18:19.444Z"
},
{
"id": "wi-41a83b56-62db-4f5d-b434-84eb9bdce139",
"kind": "task",
"title": "Dogfood: short-TTL lease expiry check",
"priority": null,
"graphRefs": [
"tara"
],
"orphanedRefs": [],
"createdAt": "2026-08-04T15:19:19.878Z"
}
]
}
=== workbench_lease wi-5085ae62-8dae-463b-a987-f5ddc5198f90 (ttl 300) ===
{
"id": "wi-5085ae62-8dae-463b-a987-f5ddc5198f90",
"status": "leased",
"lease": {
"agentId": "claude-code-dogfood",
"expiresAt": "2026-08-04T15:24:20.623Z"
}
}
=== workbench_brief wi-5085ae62-8dae-463b-a987-f5ddc5198f90 (neighborhood summary) ===
{
"itemTitle": "Dogfood: tara courses screen lacks a retry state on failed loads",
"historyEvents": 4,
"threads": [
{
"id": "th-96226c79-ade7-4242-b5b0-d4d647b6fedb",
"messages": 1
},
{
"id": "th-e49f6b73-180e-46e3-bb87-aa884d2b1e48",
"messages": 1
}
],
"neighborhoods": [
{
"anchor": "tara",
"entries": 23,
"sampleEntries": [
"contains:ancestor customer-web",
"contains:child tara.sessions",
"contains:child tara.courses",
"contains:child tara.favorites"
],
"truncated": {
"documents:in": 4,
"verifies:in": 13
}
},
{
"anchor": "tara.courses.progress",
"entries": 7,
"sampleEntries": [
"contains:ancestor tara.courses",
"contains:ancestor tara",
"contains:ancestor customer-web",
"contains:child tara.courses.progress.review"
],
"truncated": {}
}
],
"truncated": {}
}
=== workbench_lease wi-5085ae62-8dae-463b-a987-f5ddc5198f90 by rival-agent (must refuse) ===
{
"isError": true,
"message": "workbench API POST /v1/workbench/items/wi-5085ae62-8dae-463b-a987-f5ddc5198f90/lease → 409: {\"message\":\"work item \\\"wi-5085ae62-8dae-463b-a987-f5ddc5198f90\\\" is already leased by \\\"claude-code-dogfood\\\" until 2026-08-04T15:24:20.623Z\",\"code\":\"illegal_transition\"}"
}
=== workbench_report wi-5085ae62-8dae-463b-a987-f5ddc5198f90 progress ===
{
"status": "in-progress"
}
=== workbench_lease wi-41a83b56-62db-4f5d-b434-84eb9bdce139 (ttl 1s) ===
{
"id": "wi-41a83b56-62db-4f5d-b434-84eb9bdce139",
"status": "leased",
"lease": {
"agentId": "claude-code-dogfood",
"expiresAt": "2026-08-04T15:19:21.982Z"
}
}
=== queue after 1s-TTL expiry (itemB must be BACK) ===
{
"itemBInQueue": true,
"queueIds": [
"wi-eeac8712-6e40-40e1-b97b-8e4f4faf22e2",
"wi-18c55602-0f70-4214-bb1b-c97997c41643",
"wi-41a83b56-62db-4f5d-b434-84eb9bdce139"
]
}
=== workbench_verify ===
{
"graphVersion": "pg1.8c3f44a11a84+anchors.f5d5922e70d8+arch-docs.ab135da7e1ea+curated.c3fc500e4d92+curation.3bd274307036+docs.35c525030c48+e2e.94982c015422+invocation.56ab5df656f4+routes.abdd052c7eb2+signals.0325c5978804+todos.42ec1e7da930+tours.a9b52fab0ca5",
"outcomes": [
{
"id": "wi-06f10615-5b5b-4a35-941d-af029afdcc39",
"outcome": "verified",
"detail": "all 2 expected nodes present"
},
{
"id": "wi-2e207a47-5831-4eb2-b221-fb9982c0ab42",
"outcome": "verified",
"detail": "all 2 expected nodes present"
},
{
"id": "wi-a8924894-81a5-4d36-9088-811f3b3413de",
"outcome": "ship-verify-gap",
"detail": "missing from the artifact: tara.ghost-feature-xyz"
},
{
"id": "wi-f4327532-0163-468e-bb99-97ff996446d6",
"outcome": "ship-verify-gap",
"detail": "missing from the artifact: tara.ghost-feature-xyz"
}
]
}
=== workbench_lease with BAD token (must refuse 401) ===
{
"isError": true,
"message": "workbench API POST /v1/workbench/items/wi-5085ae62-8dae-463b-a987-f5ddc5198f90/lease → 401: {\"message\":\"invalid workbench token\"}"
}
DOGFOOD COMPLETE: every tool exercised over a real MCP stdio connection.
CLI fallback sweep (verbatim, trimmed to responses)#
--- pnpm workbench next
{
"items": [
{
"id": "wi-eeac8712-6e40-40e1-b97b-8e4f4faf22e2",
"kind": "issue",
"title": "Tara enroll flow loses the schedule panel on narrow viewports",
"priority": null,
"graphRefs": [
--- pnpm workbench lease wi-41a83b56-62db-4f5d-b434-84eb9bdce139 600
{
"id": "wi-41a83b56-62db-4f5d-b434-84eb9bdce139",
"status": "leased",
"lease": {
"agentId": "cli-dogfood",
"expiresAt": "2026-08-04T15:29:47.190Z"
}
}
--- pnpm workbench brief wi-41a83b56-62db-4f5d-b434-84eb9bdce139 (keys)
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 2 column 1 (char 1)
--- pnpm workbench report progress
{
"status": "in-progress"
}
--- pnpm workbench report completion
{
"status": "in-review"
}
--- pnpm workbench verify
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 2 column 1 (char 1)
{
"item": "Dogfood: short-TTL lease expiry check",
"status": "in-review",
"neighborhoods": [
"tara"
],
"historyEvents": 10
}
{
"graphVersion": "pg1.8c3f44a11a84+anchors",
"outcomes": [
{
"id": "wi-a8924894-81",
"outcome": "ship-verify-gap"
},
{
"id": "wi-f4327532-01",
"outcome": "ship-verify-gap"
}
]
}
10. Flagship promotion run (12.6). RUN 2026-08-04. VERIFIED.#
The end-to-end proof that intent becomes knowledge only through code:
- Created in conversation —
create_work_itemthrough the Phase-10 toolset + ActionConfirmBridge (the identical machinery the in-app assistant runs; the on-screen click is provider-credential-blocked on this dev Mac and annotated at 10.5). Actor: assistant, conversationflagship-12-6, turnturn-flagship-2, model claude-fable-5. - Leased over MCP by
claude-code-flagship(workbench_lease). - Implemented for real — the
workbench-orientationcurated tour + itsTOUR_CURATIONentry (commit29b2636143, pushed to origin/main; shell-assistant suite 440/440). - Reported over MCP — progress → in-progress, completion → in-review.
- Shipped on observation —
workbench_shippedwith the merge sha actually observed on origin/main (29b2636143). - Artifact rebuilt →
tour:workbench-orientationexists → artifact-diff verifier grantedverified("all 1 expected nodes present"). One honest bump along the way: the verifier initially read the SERVING runtime's partial graph (no tours composed) and surfaced a ship-verify-gap event rather than lying; it was fixed to hash-verify and read the FULL artifact, then verified. Both events remain in the append-only ledger.
Event-ledger status walk (verbatim)#
seq | event_type | actor.type | to/phase
168 | work-item.created | assistant |
169 | work-item.updated | human |
170 | work-item.transitioned | human | triaged
171 | work-item.transitioned | human | ready
172 | work-item.transitioned | coding-agent | leased
173 | work-item.report | coding-agent | progress
174 | work-item.transitioned | coding-agent | in-progress
175 | work-item.report | coding-agent | completion
176 | work-item.transitioned | coding-agent | in-review
177 | work-item.transitioned | coding-agent | shipped
181 | work-item.ship-verify-gap | artifact-verifier |
185 | work-item.transitioned | artifact-verifier | verified
MCP walk transcript (verbatim)#
=== workbench_lease {"id":"wi-e6f0628b-2aee-4b27-8fe5-fc6c1734cceb","ttlSeconds":3600} ===
{
"id": "wi-e6f0628b-2aee-4b27-8fe5-fc6c1734cceb",
"status": "leased",
"lease": {
"agentId": "claude-code-flagship",
"expiresAt": "2026-08-04T16:31:58.602Z"
}
}
=== workbench_report {"id":"wi-e6f0628b-2aee-4b27-8fe5-fc6c1734cceb","phase":"progress","branch":"oshun-mobile-parity-2026-07-04","notes":"Implemented the workbench-orientation curated tour + TOUR_CURATION entry."} ===
{
"status": "in-progress"
}
=== workbench_report {"id":"wi-e6f0628b-2aee-4b27-8fe5-fc6c1734cceb","phase":"completion","branch":"oshun-mobile-parity-2026-07-04","commits":["29b263614323b46a82b116fe3f72eabcab1e2916"],"notes":"Tour + curation landed; artifact rebuilt with tour:workbench-orientation."} ===
{
"status": "in-review"
}
=== workbench_shipped {"id":"wi-e6f0628b-2aee-4b27-8fe5-fc6c1734cceb","observedCommitSha":"29b263614323b46a82b116fe3f72eabcab1e2916"} ===
{
"id": "wi-e6f0628b-2aee-4b27-8fe5-fc6c1734cceb",
"status": "shipped"
}
=== workbench_verify {} ===
{
"graphVersion": "pg1.5ae9e4b71d8a+anchors.f5d5922e70d8+arch-docs.0953a3213920+curated.c3fc500e4d92+curation.3bd274307036+docs.35c525030c48+e2e.94982c015422+invocation.56ab5df656f4+routes.4ea3826ae610+signals.19a442cb9f96+todos.80e48aa8a093+tours.0a15ae4a8687",
"outcomes": [
{
"id": "wi-4ecaf82b-45c3-4920-90d9-3fe1fcc4537c",
"outcome": "not-machine-checkable",
"detail": "no machine-checkable expectation — the item stays honestly at shipped (state the gap; never fabricate a verify)"
},
{
"id": "wi-c83bd9c8-9b67-49c0-b851-3b458b7973b9",
"outcome": "not-machine-checkable",
"detail": "no machine-checkable expectation — the item stays honestly at shipped (state the gap; never fabricate a verify)"
},
{
"id": "wi-d9592614-bfc9-4bc5-808f-044f3099593f",
"outcome": "ship-verify-gap",
"detail": "missing from the artifact: tara.ghost-node-xyz"
},
{
"id": "wi-e6f0628b-2aee-4b27-8fe5-fc6c1734cceb",
"outcome": "ship-verify-gap",
"detail": "missing from the artifact: tour:workbench-orientation"
}
]
}
file:///Users/elikemagudogo/Desktop/workspace/oshun/tools/workbench-mcp/flagship.tmp.mjs:45
throw new Error(`FLAGSHIP FAILURE: expected verified, got ${JSON.stringify(mine)}`);
^
Error: FLAGSHIP FAILURE: expected verified, got {"id":"wi-e6f0628b-2aee-4b27-8fe5-fc6c1734cceb","outcome":"ship-verify-gap","detail":"missing from the artifact: tour:workbench-orientation"}
at file:///Users/elikemagudogo/Desktop/workspace/oshun/tools/workbench-mcp/flagship.tmp.mjs:45:9
at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
Node.js v24.13.1
{
"graphVersion": "pg1.5ae9e4b71d8a+anc",
"outcomes": [
{
"id": "wi-e6f0628b-2aee-4b27-8fe5-fc6c1734cceb",
"outcome": "verified",
"detail": "all 1 expected nodes present"
}
]
}
11. Builder workbench. SHIPPED 2026-08-04 (phases 9–15).#
The two-plane workbench on top of §8's unified graph is live end to end:
- Intent plane (phase 9) —
workbench_eventappend-only ledger +work_item/decision/thread/thread_messageprojections (migration20260804200000), every write actor-attributed (human / assistant / coding-agent / artifact-verifier, CHECK-enforced). Reducers are shared by live writes andreplayLedger; replay(ledger) == rows is proven by comparison against a red corruption control. Lifecycles run on workbench-kit stage machines (WORK_ITEM_GRAPH,DECISION_GRAPH);verifiedis reachable ONLY with the branded artifact-verifier capability, whose sole production constructor caller isartifact-verifier.ts. - Assistant write tools + confirm bridge (phase 10) — create/update work items, threads, comments, graph-ref links, decision drafts and transitions, feature proposals, content briefs; every mutation passes the operator confirm bridge and records conversation provenance. Refs are validated against the live graph projection; live LLM legs remain credential-blocked on this Mac (annotated at 10.5).
- Coding-agent loop (phases 12, MCP) — DB-backed queue (idempotent TTL
leases, report ladder,
markShippedwith the observed commit sha), self-contained task briefs with graph neighborhoods,workbench-mcpstdio server +pnpm workbenchCLI + fail-closed token-authed BFF routes. Closure is machine-checked by the artifact-diff verifier (catalog-change / e2e-suite-exists / nodes-exist / doc-page-exists); unmet expectations flagship-verify-gapEVENTS; no expectation ⇒ honestlynot-machine-checkable. §10 documents the flagship run, including the honest first failure and the post-rebuildverified. - Graph explorer (phase 13) —
/assistant/graphwith four lenses (explore tree + inspector, work kanban, decisions timeline, manifest changes diff), operator-scoped, live-verified in Chrome. - Content lane (phase 14) — brief → grounded thread draft → dispatch to an
existing pipeline → publish-path closure when the docs page exists in a later
artifact (
doc-page-exists), proven at real Postgres with a negative ship-verify-gap control. - Cross-plane queries (phase 15) —
work_item_graph_refsview (migration20260804230000), loader--intent-onlysync of:WorkItem/:Decision/:Thread(plane: 'intent') into the same Neo4j graph, and a three-engine pack (cross-plane.integration.spec.ts): open work touching/tara, accepted decisions on unverified curated flows, and the repo-backlog vs live-backlog reconciliation — TypeScript kernel, SQL, and Cypher return EXACTLY the same sets (run 2026-08-04: 2/2 green, Neo4j half live, not skipped). The intent-only loader mode refuses when the knowledge plane is missing or at a stale artifact version.
Honest seams that remain: live LLM-driven browser legs (no provider credentials on this box), and the graph profile is on-demand dev infra — the Cypher pack skips honestly when Neo4j is down.
12. Eve — the name, and her coding agent. OFFICIAL 2026-08-04.#
Naming. The omnipresent assistant described throughout this document and
built across the product-graph restructure is officially named Eve.
Collision check ran 2026-08-04: no domain in domains.json, no app or lib
directory, and no doc in the repo used the name (word-boundary search across
*.ts/*.tsx/*.json/*.md). On the intent plane Eve is the assistant
actor type — every work item, decision, thread, and content brief she drafts is
attributed to her with conversation provenance, confirmed by a human through the
action-confirm bridge before it lands.
Brand split (2026-08-04, same day): Eve is the builder's assistant — the
name belongs to the intent plane, the workbench, and this coding-agent harness.
The customer-facing persona of the shell assistant is Lilith: the V1
consumer app was renamed Lilith and the assistant is its main persona (system
prompt identity, panel title, persona labels, and disclosure copy all say Lilith
— implemented in libs/oshun/shell-assistant/src/system-prompt.ts and the
customer web surfaces; naming record in V1/BRAND.md). Members never see the
name Eve; the builder never confuses Lilith with the intent-plane actor.
Eve's coding agent is OpenAI Codex CLI, not Claude Code. Decision
2026-08-04: the autonomous leg of the builder loop — the agent that leases a
work item from the queue, implements it, and reports back — runs on
codex exec, whose terms are friendlier to unattended automation and less
likely to trip terms-of-service issues than driving Claude Code headlessly.
Claude Code remains the interactive builder tool a human drives (and the
.mcp.json registration stays for those sessions); the §9 MCP dogfood
transcript was historically a Claude Code session and is unchanged.
The sanctioned harness is tools/eve-codex-agent.mjs:
- Same intent-plane discipline as every other actor. The harness wires the
oshun-workbenchMCP server into codex through a per-run0600profile underCODEX_HOME(token never on argv), agent ideve-codex. Lease → brief → report all flow through the ledger; the agent contract forbids callingworkbench_shipped(needs the sha OBSERVED on origin/main) and cannot grantverified— that stays machine-only via the artifact-diff verifier. - Sandbox:
workspace-write, repo-rooted, network off (codex default);danger-full-accessis refused by the harness. The agent commits locally and reports the sha; pushing and shipping are the outer loop's job. - Modes:
--list(queue),--smoke(spawns the MCP server over stdio EXACTLY as codex would — same command/args/env — and runs initialize → tools/list →workbench_queueagainst the live BFF),--dry-run(print argv + agent prompt), default/--item <id>(run codex on a ready item).
Verification, run live 2026-08-04: --smoke PASSED against the booted dev
BFF (6 tools listed, real queue returned); --list and --dry-run verified
(profile file created 0600 and removed on exit). The codex config override
syntax (-c mcp_servers.…) was accepted by codex-cli 0.146.0 in a live
invocation. The one leg not yet exercised end to end is codex actually driving
the MCP tools with a model behind it: the account's codex usage limit was
exhausted mid-verification (resets 2026-08-08); the model-driven leg is
credential/quota-blocked until then, stated honestly rather than claimed.
Polish gate 2026-08 (added 2026-08-15)#
The Eve deep-test-and-polish initiative (2026-08-04 → 2026-08-15,
EVE_DEEP_TEST_AND_POLISH_TODOS_2026-08-04.md) closed with the Phase-15 gate:
the happy-path spine — signup → onboarding → two rooms → a held-for-confirm
write → the curated shell-orientation tour → a voice turn → audit begin + three
honour-marks + the board → the operator loop on the admin app (log issue →
approve → read tools, verified in Postgres) → the graph explorer painting the
new item — walked end to end on an account created by the run, with ZERO
findings, on 2026-08-15 (spine run 15 + 15.1b + 15.1c). The same night, every
route the spine touched was certified console-clean on a PRODUCTION build, both
themes (15.3, 11/11 cells) — a certification that first required making the
production build exist (EVE-VIS-213 and two fresh client-graph breaks, all
closed).
The method that got it there (the initiative's own words): compare what the
product SAYS against what the server HOLDS; findings are collected, not thrown;
every closed defect carries a calibrated regression lock — a lock that cannot
fail proves nothing. 279 ledger rows — S1 68 · S2 133 · S3 49, plus
found-in-passing and withdrawn tables
(docs/audits/EVE_POLISH_DEFECT_LEDGER_2026-08.md), every one closed with a
named lock, checker-enforced (tools/eve-polish/ledger-lock-audit.mjs).
Mechanisms this gate left in the product — the assistant now carries a family of server-side completion checkers, all on one seam, each born from a measured live failure and locked by wire cells:
- audit-claim check (EVE-VIS-130): stated progress numbers compared to the run; overstatements corrected in the transcript AND on the stream.
- announce-before-act hold (EVE-VIS-282): a "Marked … as visited" claim is
HELD off the wire until the turn's own
audit_marksucceeds, then flushed after the tool_result frame — record-then-announcement as a wire property. 079's sentence-atomic streaming (the held tail) is what makes the intercept clean. - lookup-claim check (EVE-VIS-080/095): a reply claiming a lookup that never ran is superseded.
- deferred-room check (EVE-VIS-092): a deferred room asked for by name and answered from another room earns one appended true sentence.
- interior-location check (EVE-VIS-083): screen geography described without
a
read_pagein the same turn is owned as a guess, appended. - incremental safety (EVE-VIS-242): the policy catalog runs over the accumulated stream before each delta write; a flagged reply goes quiet from the first flagged sentence.
Around the assistant: engine-parity as tests (Nyx tool/intent/router comparison table, agent-vs-read-role matrices across six rooms), the admin session cookie as a second credential for the builder endpoints on member web (EVE-VIS-216), mobile turn budgets and pending states sized to a no-stream transport (224/225), crash/diagnostics ingest that actually lands (226), and a refusal that leaves a server-side trace (281).
Residuals, stated honestly: EVE-VIS-280 — the harness model (deepseek-v4-flash-0731, bound per the cost rule) re-authors curated tours it was asked to start by id, at a measured per-run rate; the tool contract now states the curated preference and the row holds the telemetry for re-judging against the production model. Model-conditioned residuals are labelled throughout; the production binding decides what survives.