VAC: The Communication Pipeline

In V6 — Egbe, the agentic-companion universe — you do not pick options from a wheel; you talk to an autonomous being (an Ori), and it talks back. Vac is the substrate that makes that exchange trustworthy in both directions. On the inbound side it owns the whole path from a player's spoken or typed words to a structured agent objective the world can act on: microphone capture, transport through the realtime gateway, speech recognition, and the parse that turns natural language into a constrained intent grammar — verb, target, constraints, priority, deadline, and forbidden lines. On the outbound side it owns the negotiation and conversation routing that carries an Ori's reply — an acceptance, a question, a counter-offer, a deferral, or a refusal — back to the player as a spoken, lip-synced dialogue turn, plus the persistent squad-comms channel a household of agents talks over. The load-bearing design commitment is that a misheard word never becomes a misunderstood life: every parsed intent is a draft that the steward must confirm or correct before it can bind, the acknowledgement is held to a hard ≤ 400 ms budget, and every Vac capability has a complete non-voice equivalent so voice is never required. The recognizer, the speech synthesizer, and the language model that powers free conversation are all injected, governed seams — not hard-coded dependencies — and this page is honest about exactly where each one sits. It belongs to the Communication, Story, and Lifecycle group and is the deep companion to the "The Vac Communication Pipeline" section of the hub, ../V6_ARCHITECTURE.md.

What ships, honestly#

Implemented and tested (real today). The Vac intent pipeline is dual-authored in two places that must agree, exactly mirroring V6's split between Mind-layer logic and the engine runtime. The TypeScript canon is @oshun/vac-intent (libs/v6/vac-intent/src/index.ts, ~1,750 lines, 11 unit tests): a five-stage voice→intent pipeline, a strict constrained-function-calling grammar with a full domain validator, a natural-language objective parser, the steward-confirmation gate, and the agent-side negotiation router — all pure functions. The C++ mirror is V6Voice (V6/ue/Source/V6Voice): FV6VacVoiceIntentPipelineRuntime re-implements the same five stages on the engine side, and FV6SquadCommsRuntime owns the squad channel. Both are covered by real UE automation — V6.Vac.VoiceIntentPipeline.MicGatewayAsrParserBudget, V6.Vac.ConversationRouting.PsycheTtsLipSyncBudget, and the V6VacSquadCommsTests realm-parity scenario (all under V6/ue/Source/V6Tests/Private). The parsed objective is validated against the shared ObjectiveSchema from @oshun/contracts, so the same v6.ori.objective.1 shape round-trips between TS, the contract package, and the engine struct.

The injected seams (honest, not faked). Three boundaries in this pipeline are deliberately not implemented here, and the code says so out loud. (1) ASR is an injected adapter. VacAsrAdapter (vac-intent/src/index.ts:266) is a one-method interface (transcribe(packet)); when none is wired, transcribeVacGatewayPacket (:736) returns a deterministic echo of the captured utterance at a fixed 0.94 confidence — a test/wiring stand-in, not a speech recognizer. The real recognizer is passed in via options.asrAdapter, and the same is true engine-side, where TranscribeGatewayPacket takes the recognizer's confidence as a parameter. (2) TTS and the conversation LLM are seams owned by Psyche. The spoken reply (provider v6-local-tts, Opus 48 kHz mono) and the model that authors free dialogue are the @oshun/psyche-agent gateway boundary documented in the cognition stack; Vac routes to them, it does not contain them. (3) The transport is the Egbe Realtime Gateway (apps/v6/egbe-realtime-gateway/, Rust WebTransport/WebRTC/WS + voice SFU) — this library owns the gateway packet shape and route (egbe-vac:voice -intent), not the deployed SFU.

Honest scope limits. The stage-latency numbers Vac percentiles against the 400 ms budget are a deterministic budget model (the per-stage millisecond table in VAC_DEFAULT_STAGE_LATENCIES_MS), not a live network measurement — the same honest pattern V3's voice gateway uses. And the V6Voice module object itself is a thin IModuleInterface (V6Voice.cpp has empty startup/shutdown); the logic lives in the runtime classes, not the module shell. Nothing in this pipeline is .uasset content — it is all logic and contract.

The communication pipeline — voice to a confirmable objective#

A single voice command for one Ori runs an ordered, five-stage pipeline. Each stage is a pure function taking the previous stage's typed frame, so the whole path is testable end-to-end with the real recognizer wired only at the boundary. runVacVoiceIntentPipeline (vac-intent/src/index.ts:830) threads them; the engine's FV6VacVoiceIntentPipelineRuntime::RunVoiceIntentPipeline (V6VoiceIntentTypes.cpp:738) does the same, validating each frame's Validate(OutReason) and failing loud on the first bad stage.

1 — Capture. captureVacMicrophoneFrame (:697) normalizes a raw mic input into a VacMicrophoneCaptureFrame: it sanitizes the session and device ids, collapses whitespace in the utterance, and clamps the audio metadata to supported bounds (1–30 s duration, 8–96 kHz, 1–2 channels), defaulting to push-to-talk, 48 kHz, mono. The engine struct FV6VacMicrophoneCaptureFrame::Validate (V6VoiceIntentTypes.cpp:512) is the fail-loud twin: it rejects a frame missing session or device identity, an empty utterance, or a capture that is not bEndOfUtterance.

2 — Gateway routing. routeVacCaptureThroughGateway (:713) wraps the frame in a VacGatewayPacket bound to the fixed route egbe-vac:voice-intent, minting a deterministic streamId and traceId (FNV-1a over session + utterance) that become the spine of the trace. The engine twin pins the same route string and rejects any packet not on it — egbe-vac:voice-intent is a hard invariant on both sides, so a stray transport can't smuggle audio into the parser.

3 — ASR (the seam). transcribeVacGatewayPacket (:736) is where the recognizer plugs in. With a real VacAsrAdapter it returns that engine's VacAsrTranscript (text, confidence, language tag, isFinal); with none, it returns the deterministic echo described above. Either way the next stage trusts nothing: it re-checks isFinal and the confidence floor.

4 — Parse. parseVacTranscriptToObjective (:755) is the heart of the pipeline (its own section below). It gates on minimumAsrConfidence (default 0.7) and a resolvable command verb; failing either returns null, which surfaces as status: 'needs-clarification' rather than a guessed objective. The test 'holds low-confidence ASR and unsupported commands for clarification' (index.spec.ts:196) pins this: a 0.41-confidence transcript and an off-topic "maybe the weather is nice" both refuse to produce an objective.

5 — Budget + status. runVacVoiceIntentPipeline assembles a VacLatencyBudget from VAC_DEFAULT_STAGE_LATENCIES_MS (:24 — capture 24, gateway 46, asr 152, parser 68, ack 34 = 324 ms, inside the VAC_PARSED_INTENT_ACK_BUDGET_MS of 400) and resolves a status of 'parsed', 'needs-clarification', or 'budget-exceeded'. The crucial honesty: a parsed objective whose stages blow the budget is reported as 'budget-exceeded', not silently promoted to success — the test 'reports a budget miss without promoting the parsed intent to a successful acknowledgement' (:243) forces ASR to 360 ms and asserts the demotion. Engine-side, IsSuccessfulWithinBudget (V6VoiceIntentTypes.cpp:613) requires Parsed and a parsed objective and within-budget and an empty failure reason, all four.

The steward-confirmation gate — a draft, never a command#

A parsed objective is born a draft. parseVacTranscriptToObjective always sets status: 'draft', confirmedBySteward: false, acceptedByAgent: false, and attaches a VacIntentConfirmationPreview whose state is 'awaiting-steward-confirmation' and whose canBecomeStandingObjective is the literal false. canVacObjectiveBecomeStandingObjective (:680) returns true only once confirmVacParsedObjective (:684) has flipped both confirmedBySteward and status: 'confirmed'. The test 'keeps parsed standing-intent output behind steward confirmation' (:268) proves even an explicit "always protect the grove path" parse stays a draft until the steward acts. The engine enforces the same wall: IsSchemaReadyForDraft (V6VoiceIntentTypes.cpp:565) rejects any objective where bConfirmedBySteward or bAcceptedByAgent is already set — a parsed intent that arrives pre-confirmed is treated as malformed. This is "a misheard word never becomes a misunderstood life" encoded as a state machine, and it is rendered for the player in the V6UI intent-grammar builder (see the UE5 client modules and embodiment).

Intent parsing and grounding#

The constrained intent grammar#

Vac never lets a model emit free-form JSON. buildVacIntentGrammarFunctionSchema (:598) publishes a single strict function schema named vac_parse_objective _intent (VAC_INTENT_GRAMMAR_FUNCTION_NAME) with strict: true and additionalProperties: false at every level — the contract a constrained function-calling model is held to. Its required fields are exactly the six the hub names: verb, target, constraints, priority, deadline, forbiddenLines. The grammar is opinionated where it matters: target.targetKind and constraint.kind are enums (the nine target kinds and eight constraint kinds declared as const … satisfies arrays at :403/:415), priority is an integer bounded 0–100, deadline is an ISO-8601 string or null, and forbiddenLines has minItems: 1 — an objective with no boundary is structurally unrepresentable. The release gate also caps a parse at maxTokensPerParse: 2000 (V6/release/vac-communication-readiness.v6release.json).

That schema is enforced twice. validateVacObjectiveIntent (:623) is the runtime validator: it checks the verb and target refs against a V6 reference-id regex, label/description lengths, the priority bound, ISO timestamps, and runs validateConstraints/validateForbiddenLines for per-entry kind membership, uniqueness, and the 1–24 count window. The test 'publishes a strict constrained function-calling schema' (:100) drives a bad call — priority: 101, forbiddenLines: [] — and asserts both specific error strings come back, so the validator fails on the right reasons, not just truthiness.

From utterance to a structured objective#

The parser is a real cascade of domain-specific extractors, not a regex that relabels anything. extractCommandClause strips the addressing frame ("ask / tell / have / direct / get Abeni to …"); resolveVerb matches one of eight authored verbs (Repair, Map, Investigate, Protect, Build, Support, Befriend, Report) by token set; extractTargetLabel slices the clause before the first modifier ("without", "by", "urgently", …) so boundaries don't leak into the target; resolveTargetKind classifies the target into agent/ground/prop/ relationship/activity/custom by lexical cues. buildConstraints lifts a deadline, a careful-pace request, or a report-blockers safety constraint; resolvePriority maps urgency words to 90/78/55/25; resolveAutonomyMode reads "exactly as I say" → direct-tether, "always" → standing-intent, "use your judgment" → free, default brief; and extractDeadline resolves "by tomorrow / this week / today" or a literal ISO date against the creation time.

The most safety-relevant extractor is buildForbiddenLines (:1479). It scans for without / do not / don't / never / avoid, normalizes the captured phrase, and maps it to a value reference — "lie/deceive/mislead" → value:honesty, "private/secret/consent" → value:privacy, "harm/unsafe/danger" → value:safety — and when the player marks no boundary, it still injects a default forbidden-line:respect-agent-autonomy so the agent's own refusals stay binding. The end-to-end test (index.spec.ts:42) runs "Ask Abeni to repair the garden gate by tomorrow, without lying to visitors and keep it gentle" and asserts the verb Repair, target kind prop / label "Garden gate", a 2026-05-02 deadline, constraints [time, pace], and a forbidden line "Do not lie to visitors" tagged value:honesty. The C++ parser (ParseTranscriptToObjective, V6VoiceIntentTypes.cpp:680) reproduces the same verb tokens, forbidden-line markers, priority tiers, and ref formats, and its UE test parses "map the grove path and report blockers without crossing private gardens by tomorrow" to the identical structured shape — the proof the two authorings agree.

Negotiation — the agent's reply is not "yes"#

Because an Ori is an agent and not a tool, a confirmed objective is offered, not imposed. routeVacNegotiation (:860) is the outbound counterpart of the parser: given the objective and the agent's ranked values, needs, and relationship, it returns one of five decisions — accept, clarify, counter-offer, defer, refuse — each as a VacNegotiationRouteResult carrying a player-facing reply, the agent's reason, and an evidence list. The decisions are computed, not random: detectObjectiveValueConflict (:1028) refuses only when the objective text trips one of four value rules (honesty, privacy, safety, autonomy) and the agent actually holds that value at priority ≥ 50; shouldCounterOffer proposes a safer rest-first version when a late-night task meets low energy and an existing trust bond; shouldDefer pushes back when commitments stack or energy is depleted and priority is below 85. Crucially, coercion is a first-class, logged event: when a steward pushes past a prior refusal, the router returns refuse and emits a VacNegotiationOriEventDraft of type ObjectiveRefused with audit tags ['vac-negotiation', 'coercion', 'objective-refusal'], written to the Ori and the audit log. The test 'refuses coercion and drafts an ObjectiveRefused Ori event' (:393) pins the refusal, the value:honesty / forbidden-line:negotiation :honesty evidence, and the event draft — pressure cannot fabricate a yes. This is where Vac hands off to the agent-behavior negotiation layer documented in the cognition stack and agent behavior.

Squad comms — one channel, two realms#

A household acts as a squad, so Vac runs a persistent group channel. FV6SquadCommsRuntime (V6/ue/Source/V6Voice) routes five message kinds — status-report, callout, permission-request, inter-agent-banter, directed-order — over the gateway SFU, resolving recipients per kind (a permission-request goes to the steward; banter to the other agents). The behavior is honestly domain-shaped: a callout and a permission-request are marked bTimeSensitive, a permission-request requires a steward response, and inter-agent banter is bDuckable and throttled by a 45 s window (BanterThrottleWindowMs = 45000) so chatter never buries a real callout. The load-bearing guarantee is realm parity: CompareRealmParity runs the same script in Orun and during an Aye incarnation and asserts the routed message shapes match exactly — BuildParityScenario is what the V6VacSquadCommsTests automation drives to zero mismatches, so comms behave identically whether the squad is at home or incarnated in a tactical world.

The voice seams — ASR, TTS, and the conversation LLM#

The discipline at every model/audio boundary is fail loud or report absence, never fake. The ASR seam (VacAsrAdapter) and its engine equivalent take the recognizer's transcript and confidence as inputs; the parser then independently enforces the 0.7 confidence floor and a resolvable verb, so a low-quality recognition degrades to needs-clarification rather than a confident wrong objective. The default in-process transcriber is explicitly a deterministic echo for wiring and tests — it claims a fixed 0.94 and never pretends to have heard anything the packet did not carry.

The conversation seam is the other direction. Free conversational turns (as opposed to objective commands) route to @oshun/psyche-agent's routePsycheConversationTurn, which assembles a real PsycheAgentTtsPlan (provider v6-local-tts, codec opus, 48 kHz mono), a viseme PsycheLipSyncTrack bound to the agent pawn, and a spoken-reply latency budget capped at PSYCHE_SPOKEN_REPLY_BUDGET_MS = 1_000 (its default stage table sums to 646 ms). The engine twin is V6Audio's FV6ConversationAudioRuntime, exercised by V6.Vac.ConversationRouting.PsycheTtsLipSyncBudget: it routes a grounded turn, produces a v6-local-tts/opus plan and a >8-cue lip-sync track including the open-vowel A viseme, demotes a slow budget to BudgetExceeded, and — tellingly — rejects a turn whose Ori context is missing its MemoryRefs with an InvalidContext failure, so the agent cannot speak from a hollow context. The actual utterance text and the model that authors it come from the governed PsycheCognitionGateway seam — which fails loud on a gateway error and never substitutes a template — documented in full on the cognition stack page. Vac owns the routing, the budgets, and the lip-sync contract; the LLM and the TTS engine are injected behind it.

Where this connects#

• The mind that replies — the cognition stack and agent behavior — owns the LLM gateway, the Sophia grounding and Isis policy gates a reply passes, and the value-based negotiation Vac's routeVacNegotiation is the player-facing surface of.
• The story that remembers — the Clio story engine — reads the Ori event log Vac writes to, including the ObjectiveRefused coercion events, turning the consequences of what was said into a Chronicle a human wants to read.
• The client that hears and speaks — the UE5 client modules and embodiment — is where V6Voice mic capture, the V6UI intent-grammar confirmation builder, the squad-comms HUD, and TTS playback on the agent pawn actually live.
• The hub overview and the readiness gate (verify:v6 vac-communication-readiness, backed by V6/release/vac-communication-readiness.v6release.json) live in ../V6_ARCHITECTURE.md.