# Psyche — Real-Time Runtime Substrate Psyche is the V1 **real-time embodiment substrate**: it owns the contract for a live text / voice / avatar session — the session envelope, server-mediated turn-taking, interruption and barge-in, transcript synchronization, lip-sync and expression coherence, latency budgeting, provider failover, and continuity under reconnect or a safety crisis. It serves every surface where a member talks _with_ an assistant in real time rather than reading a response (the live assistant shell, the embodied teacher personas, conferencing, and the [Living Scenes](./living-scenes.md) frame stream). It sits among the platform-substrate deep-dives hubbed at [../ARCHITECTURE.md](../ARCHITECTURE.md), alongside [Sophia](./substrate-sophia.md), [Iris](./substrate-iris.md), [Lilith](./substrate-lilith.md), [Isis](./substrate-isis.md), and [Aje](./substrate-aje.md). A deliberate split runs through this page, and it is the single most important thing to understand about Psyche's maturity. The **contract-and-logic layer** — the pure data shapes and pure functions that define what a correct session looks like — is **real, rich, and exhaustively tested** in `libs/oshun/embodiment-psyche`. The **live wire/transport binding and the actual provider integration** (real ASR/TTS/avatar vendors, the WebSocket data plane, the operator review lane) are **spec-only / partially built**: the adapter library carries no IO, no rendering, and no transport by design. The end-to-end "tutor live session to graded record" walkthrough is rated _partial_ in the completeness audit, with the live-voice envelope, library write, and operator review lane noted as unbuilt surfaces. Where this page describes a function or constant, it exists; where it describes a deployed live session, it is honestly labeled planned or gated. > **Canonical home (§13).** `Psyche` is a cross-product substrate, so its > canonical reference home is the domain space > [`docs/domains/psyche`](../../docs/domains/psyche/deep-dive/architecture.md) > and its code-linked entity catalog at > [`systems/psyche`](../../docs-center/systems/lib-psyche.html). This page is > V1's view — how the V1 platform composes `Psyche`; the substrate itself is > documented in full at its canonical home, which this page references rather > than duplicates. ## Where Psyche lives in the codebase `@oshun/embodiment-psyche` (`libs/oshun/embodiment-psyche`, v0.1.0) is the canonical contract library. It is `type: module`, and its `main`/`types` point at `./src/index.ts` — **source, not a built `dist`** — so consumers import the contract directly. `src/index.ts` re-exports **23 modules**, which is the real scope of the substrate; the older architecture note that named only `adapter.ts`, `avatar-sync.ts`, `backpressure.ts`, and `crisis-frame.ts` materially undersells it. | Module | Responsibility | | --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | | `canonical-adapter.ts` | Canonical factory `createCanonicalPsycheEmbodimentAdapter({ apiAdapter })` — wraps an injected low-level adapter | | `adapter.ts` | Contract descriptor, availability/health, session-plan builder — **no direct IO** (no `fetch`/`ws`/`http`/`prisma`) | | `embodiment-model.ts` | Persona → embodiment profile, capabilities, live-session-state assembly | | `session-envelope.ts` | The validated session record, lifecycle graph, modality negotiation, persona-switch planning, fingerprinting | | `session-events.ts` | The event taxonomy, gap-free stream, and the server turn-state reducer | | `voice-orchestration.ts` | Voice pipeline orchestration contracts | | `reconnect-behavior.ts` | Reconnect/resume/handoff continuity | | `transcript-sync.ts` | Partial/final transcript reconciliation | | `multimodal-state.ts` | Cross-modality coherence state | | `latency-dashboard.ts` | Latency targets, sample builders, budget assessment | | `backpressure.ts` | Admission control, shed planning, provider backpressure policy | | `quality-thresholds.ts` | Quality gates that can trigger fallback | | `fallback-routing.ts` | Avatar → voice → text → unavailable cascade | | `provider-failover.ts` | Provider-failure routing | | `session-audit.ts` / `session-diagnostics.ts` | Audit trail and diagnostics | | `avatar-sync.ts` | Viseme table, pose interpolation, alignment/drift | | `emotion-modulation.ts` | Expression/emotion modulation contracts | | `screen-context.ts` | Screen-share / screen-context attachment | | `crisis-frame.ts` | Safety-boundary crisis-frame entry | | `events/` | Living-Scenes scene events, frame stream, cue-plan replay | The **service backing** — the live runtime that _consumes_ these contracts — exists with substantive content but is the less-mature half: `services/psyche/*` (real subdirectories include `orchestrator`, `avatar-engine`, `behavior-engine`, `voice-engine`, `conferencing`/`video-conferencing`, `tavus-integration`, `perception-engine`, `tool-framework`, `persona-service`, …), `infrastructure/psyche/{docker,kubernetes,terraform}`, contracts under `libs/contracts/psyche/src`, and `apps/psyche/admin` (only the `admin` app is present). Separately, `libs/psyche` is a much larger substrate tree (~139 subdirectories, including `tavus-*`, `avatar-*`, `memory-*`, `viseme-generator`, `voice-synthesis`, `speech-recognition`) — this is **not** the same thing as the `embodiment-psyche` adapter contract; do not conflate them. ## The canonical adapter — pure logic over an injected port `createCanonicalPsycheEmbodimentAdapter({ apiAdapter })` is the canonical entry point. It takes a `PsycheEmbodimentApiAdapter` — the low-level port that _does_ the IO — and returns a `PsycheEmbodimentAdapter` that layers the canonical contract logic on top: `getContractDescriptor`, `getMetadata`, `getAvailability` (derived from the injected `getHealth`), `getSessionCapabilities`, `resolveEmbodimentProfile`, `getLiveSessionState`, `planSession`, `startEmbodiedSession`, and the pause/resume/terminate lifecycle calls. The split is the whole point. `adapter.ts` contains **zero** `fetch`, `WebSocket`, `http`, or `prisma` calls — verified by grep. `startEmbodiedSession` illustrates the discipline: it resolves the embodiment profile, builds a session plan via `buildPsycheSessionPlan`, and **refuses to proceed if the plan is not approved** (`if (!plan.approved) throw …`). Only then does it hand the granted modalities and session kind to the injected `apiAdapter.createSession`. Policy and shape are decided in pure code; the network call is delegated. This is why the library is real and rich while a deployed live session is still partial: the _rules_ are implemented; a _running_ session requires the transport and provider bindings that live downstream in `services/psyche/*`. ## The session envelope The session envelope (`session-envelope.ts`, `PsycheSessionEnvelope` at line 393) is a single validated record that both endpoints of a live session agree on. It is far richer than a connection descriptor — it carries identity, residency, capability grants, entitlement, policy binding, transport, latency budget, continuity, governance, and lifecycle in one fingerprinted object. Key fields: - `sessionId`, `version`, `mode` (`PsycheSessionMode`, an alias of `PsycheSessionKind`). - `assistantIdentityId` + `assistantIdentityFingerprint` — the Iris assistant identity driving the session, with a fingerprint so both ends can confirm they are bound to the _same_ identity. - `locale`, `region` (residency/routing), `timezone`, `deviceClass` (`web`/`mobile`/`wearable`/`desktop`/`unknown`), `platformShell`, `consumer`. - `capabilities` — the wire/runtime capabilities the server may emit (`PSYCHE_SESSION_CAPABILITIES`, 21 entries, including `lip-sync`, `emotional-modulation`, `screen-context`, and `translation`). - `entitlementClass` — `PSYCHE_SESSION_ENTITLEMENTS` = `free`/`premium`/`enterprise`/`operator-admin`/`operator-studio`. - `lilithPolicyVersion` — the [Lilith](./substrate-lilith.md) policy pack bound at session open. - `embodiment`, `transport`, `latencyBudget`, `governance`, and `continuity`. - `continuity.memoryScope` — `PSYCHE_SESSION_MEMORY_SCOPES` = `off`/`session`/`profile` — plus `memoryConsentGranted`, `groundingMode` (`none`/`recommended`/`required`), `activeDomain`, `conversationHistoryId`, and `carryOverContextId`, which is how an [Iris](./substrate-iris.md) conversation carries into a live session. - `lifecycleState`, timestamps, `reconnectAttempts`, `lastError`, and a deterministic `fingerprint`. The envelope is not a passive bag of fields — the module ships the operations that keep it correct: - **Lifecycle as a transition graph.** `PSYCHE_SESSION_LIFECYCLE_TRANSITIONS` encodes the legal moves between states (`pending → ready → connecting → connected`, with `degraded` reachable from `connecting`/`connected` and recoverable back to `connected`, and the terminal set `closed`/`failed`/`timed-out`). `isPsycheSessionTransitionAllowed` and `isPsycheSessionTerminalState` enforce it. A session cannot skip from `pending` straight to `connected`. - **Fingerprinting.** `computePsycheSessionEnvelopeFingerprint` derives a deterministic hash over the identity-relevant fields, so two endpoints holding matching fingerprints know they are looking at the same envelope without reshipping the whole record. - **Modality negotiation.** `negotiatePsycheSessionModalities` reconciles requested modalities against grants and re-fingerprints the result. - **Persona-switch planning.** `planPsycheSessionPersonaSwitch` plans a mid-session persona change while preserving continuity. ## The event taxonomy and the gap-free stream Psyche ships a **dual event taxonomy**, and the architecture note that only listed the 15 canonical V1 wire types under-describes it. **Canonical V1 wire events** — `PSYCHE_V1_SESSION_EVENT_TYPES` (`session-events.ts:115`), 15 entries that match the published list exactly: `text-token-stream`, `asr-partial`, `asr-final`, `tts-chunk`, `avatar-frame`, `viseme-stream`, `expression-update`, `turn-complete`, `interruption`, `reconnect`, `transcript-sync`, `error`, `kill-switch`, `fallback-engaged`, `policy-intervention`. **Server turn events** — `PSYCHE_SERVER_TURN_EVENT_KINDS` = `turn-start`, `turn-end`. These are emitted by the server-side turn reducer (below) and are _not_ in the 15-entry V1 list, which is a documentation gap rather than a contradiction. **Legacy dotted protocol** — `PSYCHE_LEGACY_SESSION_EVENT_KINDS` is the older namespaced family that the union still carries: - Turn-taking: `turn.claim`, `turn.grant`, `turn.yield`, `turn.interrupt`, `turn.barge-in`, `turn.timeout` - Streaming: `stream.audio.frame`, `stream.transcript.partial`, `stream.transcript.final`, `stream.avatar.frame`, `stream.tool.call`, `stream.tool.result` - Reconnect: `reconnect.attempt`, `reconnect.success`, `reconnect.failed`, `reconnect.handoff` - Lifecycle: `session.state-changed`, `session.error`, `session.heartbeat` The discriminated union `PsycheSessionEvent` (and the combined `PSYCHE_SESSION_EVENT_KINDS`) covers all three families. Every event flows through a **gap-free, monotonic stream**. `buildPsycheSessionEventStream` / `appendPsycheSessionEvent` enforce that `sequence === nextSequence` (no gaps, no reordering) and that `emittedAt` is non-decreasing (monotonic time). Each event is fully traceable, carrying an `eventId`, `sessionId`, `traceId`, timing (ingress/egress plus provider attribution and `retryAttempts`), the `sequence`, and an `actor`. This is what makes the stream auditable and replayable: a kill-switch or `policy-intervention` event has a stable position and a provable provenance. ## Server-mediated turn-taking — a real state machine Turn-taking is not narrative hand-waving; it is an implemented reducer. `createPsycheServerTurnState` builds the initial state and `applyPsycheServerTurnCommand` advances it. The phases are `PsycheServerTurnPhase = idle | active | interrupted | complete`, and the commands are `turn-start`, `turn-end`, `interruption`, and `system-initiated-barge-in`: | Command | Effect | | --------------------------- | ------------------------------------------------------------------------------------------ | | `turn-start` | Opens a turn (`turnId`, `speakerId`, `turnCeilingMs`); emits a `turn-start` event | | `turn-end` | Closes a turn (`reason`, whether the final transcript is available, final TTS chunk index) | | `interruption` | A caller barges in (`interruptedBy`, `policyApproved`); fades partial TTS | | `system-initiated-barge-in` | The runtime itself interrupts (e.g. for a policy or safety reason) | The server is the authority on whose turn it is — barge-in and interruption are mediated through this reducer rather than negotiated peer-to-peer, which is what keeps a multi-party live session deterministic. The interruption commands fade out any in-flight TTS over `PSYCHE_DEFAULT_PARTIAL_TTS_FADE_OUT_MS = 180` (`session-events.ts:768`) by default — the configurable barge-in fade-out that lets the synthetic voice tail off cleanly instead of cutting dead mid-word. ## Latency budgets and the "thinking indicator" deadline Live embodiment lives or dies on latency, so the targets are codified, not aspirational prose (`latency-dashboard.ts:68-84`): | Metric | p50 | p95 | p99 | | -------------------------------------------------------------- | ------ | ------ | ------- | | Voice first-audio (`PSYCHE_VOICE_FIRST_AUDIO_LATENCY_TARGETS`) | 500 ms | 900 ms | 1500 ms | | First token (`PSYCHE_FIRST_TOKEN_LATENCY_TARGETS`) | 350 ms | 700 ms | — | | Avatar frame (`PSYCHE_AVATAR_FRAME_LATENCY_TARGETS`) | 80 ms | — | — | | ASR final text (`PSYCHE_ASR_FINAL_TEXT_LATENCY_TARGETS`) | 150 ms | — | — | Sample builders (e.g. `buildPsycheVoiceFirstAudioLatencySample`, `buildPsycheAvatarFrameLatencySample`) compute each value from raw timestamps and validate ordering (first-audio cannot precede speech-end), and `assessPsycheLatencyAgainstBudget` compares observed percentiles to the envelope's budget, returning `PsycheLatencyBudgetStatus` (`PSYCHE_LATENCY_BUDGET_STATUSES` = `ok` / `warning` / `exceeded`) for the round trip, the pipeline, and overall. A `warning`/`exceeded` status is the input that feeds quality-driven fallback. When a provider lags, the runtime must not leave the member staring at silence. The provider-backpressure policy `PSYCHE_PROVIDER_BACKPRESSURE_DEFAULT_POLICY` sets `thinkingIndicatorDeadlineMs: 200` (`backpressure.ts:322`), clamped to ≤ 200 ms — so a "thinking" indicator must surface within 200 ms of a stall, which matches the published "within 200 ms" requirement. ## Backpressure: admission control and load shedding Beyond the thinking-indicator deadline, `backpressure.ts` implements real load-management mechanics: - **Admission control.** `decidePsycheSessionAdmission` returns a typed decision (`PSYCHE_SESSION_ADMISSION_DECISIONS`) weighing `PSYCHE_SESSION_ADMISSION_PRIORITIES` — i.e. whether a new live session may start under current load. - **Shed planning.** `planPsycheBackpressureShed` picks running sessions to relieve, choosing among `PSYCHE_BACKPRESSURE_SHED_ACTIONS` = `downgrade-to-text`, `graceful-disconnect`, `queue-handoff`. - **Provider backpressure response.** `planPsycheProviderBackpressureResponse` produces a throttle / thinking-indicator / fallback plan from the default policy (the ASR-throttle ratio, model-streaming-throttle ratio, and `fallbackToTextAfterMs` all live in that policy). - **Status rollup.** `computePsycheBackpressureStatus` reduces the picture to `PSYCHE_BACKPRESSURE_STATUSES` = `healthy` / `elevated` / `critical`. ## The fallback cascade When quality, latency, a provider failure, a kill-switch, a policy, or an accessibility need degrades a modality, Psyche steps **down** a fixed cascade rather than dropping the session. `PSYCHE_FALLBACK_MODES` (`fallback-routing.ts:55`) = `avatar → voice → text → unavailable`, and the triggers are `PSYCHE_FALLBACK_TRIGGERS` = `provider-failure`, `quality`, `latency`, `kill-switch`, `policy`, `accessibility`. `resolvePsycheFallbackMode` ranks the requested mode against current capability and resolves the highest mode that can actually be served. `buildPsycheFallbackCascadePlan` and `planPsycheFallbackRenegotiation` produce the step-down plan and the user-facing decline message, and a `fallback-engaged` event is emitted onto the stream so the degradation is visible and audited. The **kill switch** has its own granularity: `PSYCHE_KILL_SWITCH_SCOPES` = `session` / `tenant` / `region` / `global`, with `PSYCHE_KILL_SWITCH_FALLBACK_MODES` = `voice` / `text` / `closed` — so an operator can collapse all avatar synthesis in a single region to text-only without taking text away. ## Avatar lip-sync and expression coherence `avatar-sync.ts` makes "viseme alignment" concrete. `PSYCHE_AVATAR_VISEMES` is an 11-viseme set (`rest`, `ah`, `ee`, `oh`, `oo`, `mb`, `fv`, `ss`, `th`, `eh`, `ay`), and `PSYCHE_AVATAR_VISEME_TABLE` maps **39 ARPABET phonemes** onto those visemes (`resolvePsycheAvatarViseme` upper-cases the phoneme id and looks it up). The pipeline is: - `buildPsycheAvatarPhonemeTimeline` orders phonemes into a timed sequence. - `interpolatePsycheAvatarPose` linearly interpolates between viseme poses so the mouth transitions smoothly rather than snapping. - `assessPsycheAvatarAlignment` measures the gap between the avatar's actual pose and the audio it should track, returning `PSYCHE_AVATAR_ALIGNMENT_STATUSES` = `aligned` / `drifting` / `desynced` — drift detection so the runtime knows when lip-sync has come unstuck. Expression coherence is handled alongside in `emotion-modulation.ts`, and `screen-context.ts` carries screen-share context for sessions that share a screen. ## Crisis-frame continuity — a first-class safety boundary `crisis-frame.ts` is a fully implemented module, not narrative. A crisis-frame entry is a **safety boundary, not a normal persona transition** (its own fileoverview says so): when [Lilith](./substrate-lilith.md) policy takes over for a member in distress, the runtime must break persona, stop synthetic output, and suspend memory writes _at the same timestamp_. `enterPsycheCrisisFrame` produces a new envelope and the audit/event trail in one deterministic step: - Sets `continuity.memoryScope → 'off'`, `embodiment.syntheticVoice → false` and `syntheticAvatar → false`, clears the voice/avatar pack ids, strips the synthesis capabilities, and moves `lifecycleState → 'degraded'`. - Emits a `policy-intervention` event onto the stream by actor `{ kind: 'system', id: 'lilith' }`. - Writes a `persona.break` audit event and a `lilith.intervention` audit event, both stamping the triple action `PSYCHE_CRISIS_FRAME_AUDIT_ACTIONS = ['break-persona', 'halt-synthesis', 'suspend-memory-writes']`. - Produces a synthesis-halt plan (the `tts`/`avatar` pipeline services and `voice`/`avatar`/`video` modalities are removed) and re-validates the new envelope before returning. The user-visible disclosure is fixed and honest: _"I need to pause the persona voice and switch to direct safety support for this moment."_ The point is that a crisis cannot half-happen — persona break, synthesis halt, and the memory-write suspension are computed together, so [Iris](./substrate-iris.md) never records anything said inside a crisis frame. ## Living Scenes integration The `events/` subtree wires Psyche into [Living Scenes](./living-scenes.md): `events/scene-events.ts` carries scene events, `events/cue-plan-replay.ts` handles cue-plan replay, and `events/frame-stream.ts` is a **real backpressure channel** for the render frame stream. `frame-stream.ts` ships a `DEFAULT_FRAME_STREAM_CONFIG`, an `evaluateBackpressure` step, and `stepFrameStream`/`consumeFrame`, with a high-water policy that **drops non-keyframes first** when the channel saturates — which is how render envelopes are kept inside Living-Scenes latency budgets without tearing the scene. ## Maturity, honestly To restate the split so nothing is oversold: - **Real and rich (contract + logic):** the 23-module adapter library — session envelope and its lifecycle graph, the full event taxonomy and gap-free stream, the server turn-state reducer, latency budgets and assessment, admission / shed / provider backpressure, the fallback cascade and kill switch, the viseme table and alignment drift detection, and crisis-frame continuity. Every claim on this page maps to an exported symbol or constant, and the modules ship with comprehensive `*.test.ts` suites. - **Spec-only / partially built (runtime):** the live WebSocket transport binding and the real ASR/TTS/avatar provider integrations (deliberately absent from the pure-logic library), and the end-to-end "tutor live session to graded record" flow — whose live-voice envelope, library write, and operator review lane are noted as unbuilt surfaces. The service trees under `services/psyche/*`, `infrastructure/psyche/*`, and `apps/psyche/admin` exist with substantive content but are the less-mature half. See the backlog (§11) for the live-session and operator-lane work that closes this gap. ## Related - [Lilith — Contemplative Policy Substrate](./substrate-lilith.md) - [Iris — Assistant Memory Substrate](./substrate-iris.md) - [Living Scenes](./living-scenes.md) - [Sophia — Grounding Substrate](./substrate-sophia.md) - [Persona, Avatar, and Voice Packs](./persona-avatar-voice-packs.md) - [Observability, Design System, Testing, and Performance](./observability-and-quality.md) - [Trust, Safety, and Privacy](./trust-safety-and-privacy.md) - [Subsystem Glossary](./glossary.md) - [`V1/features.md` § Psyche live-session event taxonomy and latency targets](../features.md) - [Hub: V1 Architecture](../ARCHITECTURE.md)