Psyche Domain — Feature Reference

Psyche (formerly codenamed Serwaa) is the Hyper-Realistic AI Virtual Assistant Platform of the Oshun ecosystem. It creates photorealistic digital humans that join video conferences, hold real-time conversations, perceive the emotional state of participants, and perform computer tasks on their behalf — indistinguishably from a real human participant. Psyche combines three foundational technologies: a neural avatar rendering engine built on 3D Gaussian Splatting (a technique that reconstructs a photorealistic 3D scene from training video), a pipelined neural voice synthesis system achieving approximately 75ms time-to-first-audio-byte latency, and Claude-powered conversational AI with 200K token context. The result is a digital human that can listen, understand, speak, express appropriate emotion, and act — all in real time, inside Zoom, Microsoft Teams, Google Meet, and Webex.

Library prefix: @psyche/* | 135 libraries under libs/psyche/, plus 16 Python services under services/psyche/

1. Digital Human Personas#

Personas are the configurable identity of Psyche. Each persona is a complete virtual assistant with a distinct identity, personality, voice, photorealistic appearance, domain expertise, memory, and behavioral profile. Multiple personas can coexist on a single Psyche deployment, each serving different use cases — a "Dr. Sarah Chen" for medical consultations, a "Marcus Webb" for sales, a "Priya Nair" for technical support.

1.1 Identity Creation#

The following fields define the persona's professional identity. This information shapes how the assistant introduces itself to meeting participants and frames its conversational responses.

1.2 Personality Engineering — OCEAN Model#

Personality is configured using the scientifically validated Big Five (OCEAN) model. Each trait is set on a continuous 0.0–1.0 scale, giving fine-grained control over the persona's behavioral tendencies. The OCEAN settings translate directly into LLM system prompt parameters, ensuring personality is consistent across all interactions.

1.3 Communication Style and Tone#

Beyond the OCEAN personality model, operators can directly configure the surface-level communication style that the persona projects in every utterance.

1.4 Personality Presets#

For operators who want a quick starting point, one-click personality configurations are available for the most common professional use cases. Each preset maps to a specific combination of OCEAN trait values and communication settings.

1.5 Expertise Configuration#

Expertise configuration shapes what the persona knows, how deeply it engages with specific topics, and what it does when a question falls outside its domain.

• Domain tagging: Assign expertise areas from a controlled vocabulary (e.g., "Product Support", "API Integration", "Insurance Claims", "Investment Strategy").
• Proficiency levels: Set per-domain depth as Beginner, Intermediate, Advanced, or Expert — this shapes how deeply the persona engages with domain-specific questions.
• Topic boundaries: Explicitly define what the persona should refuse to address and where to redirect the user (e.g., "Do not provide medical diagnoses; refer to a licensed physician").
• Escalation triggers: Configure conditions that automatically escalate to a human — repeated resolution failures, detected customer frustration, specific keyword triggers, or out-of-scope requests exceeding a threshold.

1.6 Tool Assignment#

Each persona can be granted access to a specific set of tools it is permitted to invoke during conversations. Tool permissions are scoped per persona — a customer service persona cannot access the file system or run code, while an enterprise assistant persona may have broader access.

1.7 Persona Lifecycle#

A persona moves through a well-defined lifecycle from initial creation through production use. The Training state is significant because avatar and voice model generation can take 1–4 hours — no sessions can be started until training completes and the persona transitions to Active.

2. Hyper-Realistic Avatar System#

The avatar system creates photorealistic digital humans from a short video recording using 3DGS (3D Gaussian Splatting — a neural rendering technique that represents the scene as millions of small 3D Gaussian "splats" rather than traditional mesh polygons, producing photorealistic results from short training videos without requiring a controlled studio environment).

2.1 Avatar Training — @psyche/avatar-core#

Training takes a short video of the target person and produces a GPU-resident neural model that renders the face at 30+ FPS. The pipeline runs entirely offline before the persona becomes active.

2.2 Avatar Quality Settings — @psyche/avatar-quality#

Quality can be adjusted to trade off visual fidelity against GPU resource usage, enabling the same avatar to serve both mobile-optimized and studio-grade scenarios.

2.3 Avatar Styles#

Different use cases call for different visual registers. The style setting controls whether the avatar uses the photorealistic 3DGS render or a stylized visual representation.

2.4 Facial Expression System#

The expression system drives the avatar's face to convey appropriate emotion throughout the conversation, maintaining the believability of the digital human. Multiple complementary systems work together, from coarse emotion states down to involuntary micro-expressions.

• Automatic emotion mapping (@psyche/avatar-expressions): Expressions change dynamically based on speech content, detected conversational emotion, and configured personality traits.
• FACS control (@psyche/facs-expressions): FACS (Facial Action Coding System — the scientific system for classifying all human facial movements by the specific muscle groups, called action units, they involve) support for precise, anatomically grounded expression control.
• Micro-expressions (@psyche/micro-expressions): Brief involuntary facial expressions lasting 1/25 to 1/5 second that occur before the voluntary expression — adding micro-expressions dramatically increases perceived naturalness and trustworthiness.
• Lip synchronization (@psyche/avatar-lipsync): Real-time lip sync driven by TTS audio output with less than 33ms audio-to-visual synchronization latency. Viseme sequences (the visual equivalents of phonemes) are extracted from audio and mapped to FLAME blend shapes.
• V2 commentator face animation: V2 uses @psyche/avatar-lipsync and @psyche/avatar-expressions with @iris/voice phoneme timing to animate AI commentator faces. The commentator face animation integration is presentation-only, off rollback, and owned by @v2/psyche-commentary-face-animation.
• Gaze control (@psyche/gaze-control): Natural eye gaze that maintains appropriate eye contact during active speech, looks away during apparent thought, and tracks movement in the video frame.
• Head movement (@psyche/head-movement): Realistic head nods, tilts, and turns that accompany and accentuate speech — the non-verbal communication essential for perceived attentiveness.
• Posture system (@psyche/posture-system): Upper-body posture management that shifts between attentive (slightly forward-leaning), relaxed, and engaged stances based on conversation dynamics.
• Gesture system (@psyche/gesture-system): Procedurally generated hand and arm gestures that complement speech prosody — illustrators (gestures that accompany and emphasize speech content) and adaptors (self-touching behaviors signaling cognitive processing).
• Uncanny valley mitigation (@psyche/uncanny-valley): Automatic adjustments to prevent the subtle artifact combinations that trigger the "uncanny valley" effect — the discomfort caused by a human-like entity that is almost-but-not-quite convincing.

2.5 Avatar Performance Features#

These optimizations ensure the avatar renders instantly and maintains smooth, artifact-free output under sustained load.

• Model preloading (@psyche/avatar-cache): Avatar models and neural network weights are preloaded into GPU memory at startup so the first frame renders instantly without a cold-start delay.
• Temporal anti-aliasing (@psyche/avatar-taa): TAA removes temporal flickering and edge aliasing artifacts by accumulating information across frames, producing visually stable, clean renders.
• TensorRT acceleration: GPU inference acceleration using NVIDIA TensorRT to optimize avatar rendering neural networks for maximum throughput.
• Tavus cloud fallback (@psyche/tavus-hybrid): When local GPU resources are insufficient, the system transparently routes to Tavus Phoenix cloud rendering, maintaining session continuity without interruption.

3. Natural Voice and Speech#

3.1 Voice Synthesis — @psyche/voice-synthesis#

The voice synthesis engine converts the LLM's text output into natural-sounding speech, synchronized with the avatar's lip movements. Multiple providers are supported with automatic failover so the session continues uninterrupted even if the primary provider experiences an outage.

• Custom voice cloning (@psyche/voice-consistency): Create a unique synthesized voice from 1–3 minutes of audio samples of the subject. The cloned voice is specific to the persona and cannot be shared or exported.
• Provider failover: If the primary TTS provider returns an error or exceeds latency thresholds, the system automatically switches to the next provider in the chain without interrupting the session.

3.2 Voice Parameters#

These parameters allow fine-grained control over how the synthesized voice sounds during a session. They can be configured per persona or adjusted per session.

3.3 Speech Recognition — @psyche/speech-recognition#

Real-time transcription of participant speech drives the AI's conversational understanding. Interim results arrive while the participant is still speaking, enabling responsive turn-taking before the full utterance is complete.

3.4 Voice Streaming — @psyche/voice-streaming#

• WebSocket audio streaming: Audio is streamed in real-time rather than buffered, achieving approximately 75ms from the first token being generated to the first audio byte being output.
• Streaming buffer management: Intelligent buffering ensures smooth playback despite variable network conditions, with automatic quality adaptation.

3.5 Viseme Generation — @psyche/viseme-generator#

Visemes are the visual equivalents of phonemes — the 21 canonical mouth shapes corresponding to distinct speech sounds. The viseme generator extracts the phoneme sequence from incoming audio in real time, maps each phoneme to the corresponding FLAME blend shape identifier, and outputs a time-aligned viseme sequence that drives avatar lip animation, achieving sub-33ms audio-to-visual synchronization.

4. Real-Time Conversation#

4.1 AI-Powered Dialogue — @psyche/dialogue-manager#

The dialogue manager orchestrates the full turn-taking conversation between participants and the AI persona.

• Primary LLM: Claude 3.5 Sonnet (configurable per persona) provides reasoning, response generation, and tool-calling capabilities.
• Real-time token streaming: Responses are streamed token by token to the synthesis pipeline, enabling pipelined TTS synthesis that begins producing audio before the full response is complete.
• Multi-LLM failover chain: Claude → OpenAI GPT-4 → Google Gemini → Local LLM (Ollama/vLLM). If the primary model is unavailable, the session continues uninterrupted on the next model in the chain.
• 200K token context window: With intelligent truncation that prioritizes recent exchanges, knowledge base retrievals, and high-importance historical context.
• Tool calling: The LLM can invoke any registered tool mid-conversation and incorporate the tool's response into the next utterance without breaking conversational flow.
• Persona system prompt injection: Each persona's personality configuration, expertise profile, topic boundaries, and custom instructions are injected as the LLM system prompt.

4.2 Turn-Taking Management — @psyche/tavus-turntaking#

Turn-taking governs the flow of speaking rights between the AI persona and meeting participants, creating a natural conversational rhythm that avoids both awkward silences and interruptions.

• Endpoint detection: Automatically detects when a participant has finished speaking using a combination of audio silence detection and semantic completeness assessment.
• Acknowledgment phrases: Natural filler utterances ("I see", "That's a good question", "Let me think about that") are generated while the LLM processes the full response, preventing awkward silence.
• Clarification behavior: When transcription confidence falls below a configurable threshold or a request is ambiguous, the persona asks a targeted clarifying question rather than proceeding with a low-confidence interpretation.
• Smooth topic transitions: Natural bridging phrases for transitions between topics, maintaining conversational coherence across subject shifts.
• Proactive follow-up offers: After resolving a query, the persona offers related help or anticipates the next likely question.

4.3 Response Configuration#

These parameters control the format and behavior of persona responses, independent of the LLM model or voice synthesis settings.

4.4 Behavior Coordination — @psyche/behavior-coordinator#

The behavior coordinator synchronizes the avatar's physical expressions with the spoken content, simultaneously coordinating speech generation, facial expression transitions, eye gaze direction, head movement, hand gestures, and posture. The behavioral anomaly detector (@psyche/behavior-anomaly-detector) identifies and corrects unnatural behavioral combinations — for example, an expression conflicting with the vocal sentiment, or a gesture out of phase with the speech it should accompany.

5. Knowledge Base and RAG#

5.1 Document Ingestion#

Documents are processed into a searchable vector database that the persona consults during conversations to provide accurate, source-grounded answers. Rather than relying on the LLM's general training knowledge, the persona retrieves specific content from uploaded documents, reducing hallucination and enabling accurate domain-specific responses.

5.2 Vector Search and Retrieval — @psyche/tavus-knowledge#

At conversation time, the top-K most semantically relevant document chunks are retrieved from Qdrant and injected into the LLM's context window immediately before generating a response. This is RAG (Retrieval-Augmented Generation) — grounding responses in specific knowledge rather than general training data.

• Vector embeddings: Generated using Voyage AI (primary), Cohere, or OpenAI (fallback) to create high-dimensional semantic representations of each document chunk.
• Vector database: Qdrant (a high-performance approximate nearest-neighbor vector database) for similarity search across potentially millions of document chunks.
• RAG integration: During conversations, the top-K most semantically relevant document chunks are retrieved and injected into the LLM's context window immediately before generating a response.
• Configurable retrieval: Adjustable top-K count (default: 5 chunks) and minimum relevance score threshold (default: 0.7 cosine similarity) per knowledge collection.

5.3 Knowledge Sources#

Knowledge can enter the system in several ways, supporting both one-time uploads and continuously synchronized content from external systems.

5.4 Knowledge-Persona Linking#

Knowledge collections are scoped to specific personas rather than being globally accessible, ensuring that each persona only retrieves information relevant to its role. This is important for multi-tenant deployments where different personas serve different customers.

• Per-persona knowledge scoping: Each persona has an assigned set of knowledge collections — a customer service persona sees only the product documentation, not the internal engineering wiki.
• Collection priority ranking: When multiple collections are assigned, priority determines which is searched first and which fills context space when it is limited.
• Persistent fact memory: Store specific facts the persona always knows without retrieval — company founding date, current product pricing, support hours.
• Opinion and stance memory: Define the persona's position on configurable topics with a conviction strength (0.0–1.0) controlling how confidently the persona volunteers that opinion.

6. Video Conferencing Integration#

6.1 Supported Platforms — @psyche/conferencing-core#

The conferencing layer abstracts over four major platforms plus a direct WebRTC stack. Each platform integration handles authentication, meeting join, and audio/video injection in the platform-specific way, while exposing a uniform interface to the rest of Psyche.

6.2 Meeting Capabilities#

Once connected to a meeting, the persona participates as a full meeting member with its own video and audio streams, indistinguishable from a real participant's webcam feed.

• Automatic meeting join: The virtual assistant autonomously joins scheduled meetings at the configured start time without human intervention.
• Participant management (@psyche/participant-manager): Real-time tracking of who is in the meeting, detection of participant joins and departures, and per-participant state tracking.
• Avatar video streaming: The rendered avatar video stream is injected into the conferencing platform's video feed, appearing as a regular participant's webcam feed.
• Audio injection: Synthesized speech audio is injected into the meeting audio mix, with echo cancellation to prevent feedback loops.
• State synchronization (@psyche/state-sync): All components (avatar renderer, voice engine, LLM, conferencing platform) maintain synchronized state so disruptions in one subsystem do not desynchronize others.
• Chat interaction: The persona can send and receive text chat messages within the meeting platform's chat interface (Microsoft Teams and Webex).
• Meeting summarization (@psyche/meeting-summarizer): After sessions, automatically generates structured summaries with key discussion points, decisions made, and action items.
• Recording integration (@psyche/recall-integration): Integration with Recall.ai for meeting recording access and post-session transcript retrieval.

6.3 Platform-Specific Integrations#

• Zoom integration (@psyche/zoom-integration): Full Zoom Meeting SDK integration validated and certified for production use.
• Teams integration (@psyche/teams-integration): Microsoft Teams integration with Graph API authentication, Teams bot framework, and meeting lifecycle events.
• Google Meet integration (@psyche/meet-integration): WebRTC-based Meet integration with OAuth authentication.
• Webex integration (@psyche/webex-integration): Webex REST API integration for enterprise Cisco environments.

6.4 Tavus Cloud Avatar Integration (Optional)#

Tavus provides cloud-hosted neural avatar rendering as an alternative or fallback to local 3DGS rendering when local GPU resources are insufficient. The hybrid mode switches between local and cloud rendering transparently, without the participant noticing.

• Replica creation (@psyche/tavus-replica-manager): Train cloud-hosted Tavus Phoenix avatar replicas from video recordings.
• Real-time conversations (@psyche/tavus-conversation): Conduct full interactive sessions using Tavus's Phoenix rendering pipeline.
• Hybrid rendering (@psyche/tavus-hybrid): Intelligently switch between local 3DGS and Tavus cloud rendering based on local GPU availability.
• Tavus Perception (@psyche/tavus-perception): Leverage Tavus's Raven model for participant video understanding — face detection, emotion recognition, engagement measurement.
• Tavus pipeline (@psyche/tavus-pipeline): Full end-to-end Tavus pipeline orchestration.
• Tavus Pipecat (@psyche/tavus-pipecat): Low-latency real-time audio/video pipeline built on the Pipecat framework for ultra-responsive sessions.

7. Computer Use and Task Execution#

The computer use system enables Psyche personas to browse the web, fill forms, and perform multi-step software tasks on behalf of meeting participants — live, during video calls. The persona can execute tasks while narrating its actions to the human participants, making the automation transparent and verifiable.

Under the Phase 180 Remote Creative Control Plane (TODOS/phase-180.md, owned primarily by Bellona), Psyche's screenshot/click/type/window primitives are planned to be extracted into a reusable libs/psyche/desktop-fallback library shared between Psyche Computer Use and the Bellona remote host's desktop fallback adapter, so remote creative control uses the same audited automation primitives as live persona task execution.

7.1 Browser Automation — @psyche/browser-automation#

Browser automation runs inside a sandboxed Chromium instance that has no access to host system resources, preventing runaway automation from affecting anything outside the intended task.

• Sandboxed Chromium browser: A headless Chromium instance runs in a fully isolated sandbox. The AI can only interact with the browser; it has no access to the host system's filesystem, clipboard, or other applications outside the sandbox.
• Observe-Reason-Act loop (@psyche/computer-use-core): The system operates iteratively: (1) take a screenshot of the current browser state, (2) ask the LLM to analyze the screen and decide the next action, (3) execute the action, (4) verify the outcome, (5) repeat until the task is complete or the maximum iteration limit is reached.
• Action types: Click (by element reference or screen coordinates), type text into fields, navigate to URLs, scroll pages, select from dropdowns and menus, upload files, download files.
• Action verification: After each action, the system takes a verification screenshot and asks the LLM to confirm the expected outcome occurred before proceeding.
• Automatic recovery: If an action produces an unexpected result (error message, unexpected navigation), the system detects this and attempts a recovery strategy before failing.

7.2 Screen Analysis — @psyche/screen-analysis#

The screen analysis library provides computer vision capabilities for interpreting browser and application content. It has over 1,000 tests covering all analysis capabilities.

• Layout analysis: Detect and understand the structure of web pages — navigation, main content, sidebars, forms, tables, and interactive controls.
• OCR: Extract text from rendered pages, including text in images, canvas elements, and CSS-rendered content.
• Element detection: Identify clickable elements, form fields, buttons, links, and other interactive components with their bounding boxes.
• Multi-step workflow support: Understand complex multi-page workflows (checkout flows, form wizards, search and filter operations) and execute them across multiple page transitions.

7.3 Task Safety — @psyche/action-safety#

Every action is validated before execution, and high-risk actions require explicit human confirmation. This ensures that the automation cannot accidentally or maliciously cause irreversible changes without participant awareness.

• Pre-execution rule validation: Every action is validated against a configurable safety rule set before execution. Rules can block specific URL patterns, form field types, or action categories.
• Confirmation prompts: High-risk actions (form submissions with financial implications, data deletions, purchases) surface a confirmation request to the human participant before execution.
• Maximum iteration limit: Tasks are capped at a configurable maximum of 50 actions per invocation to prevent infinite loops and runaway automation.
• Sandbox isolation: The browser runs in a container with no network access outside explicitly permitted domains and no access to host system resources.

7.4 Tool Registry and Execution#

• Tool registration (@psyche/tool-registry): Register custom tools with typed input/output schemas, descriptions, authentication configuration, and execution handlers.
• Tool execution (@psyche/tool-execution): Safe, traced execution of registered tools with timeout enforcement, error handling, and result logging.
• Built-in tools: Platform-provided tools including web search, calendar lookup, CRM query, and knowledge base search available to all personas.

8. Emotion and Perception Intelligence#

Psyche personas perceive and respond to the emotional state of meeting participants, creating genuinely empathetic interactions rather than producing the same response regardless of context. The perception pipeline runs entirely on the inbound participant video and audio — it does not analyze the avatar's own output.

8.1 Inbound Participant Perception#

Each participant's video and audio streams are analyzed independently, enabling Psyche to track different emotional states for each person in a multi-party meeting.

• Face detection (@psyche/face-detection): Real-time detection of participant faces in the video stream, including bounding box tracking across frames for multiple simultaneous participants.
• Facial emotion analysis (@psyche/face-analysis, @psyche/emotion-recognition): Classify detected facial expressions into primary emotional categories (happiness, sadness, surprise, fear, disgust, anger, neutral) with continuous probability scores rather than binary classifications.
• Voice sentiment detection: Analyze vocal characteristics — tone, speaking rate, pitch variation — to infer emotional state independent of the spoken words.
• Engagement measurement (@psyche/engagement-detector): Quantify participant engagement level from attention direction, facial animation, and speech participation frequency. Low engagement triggers more engaging response strategies.
• Attention tracking (@psyche/attention-tracker): Detect whether participants are looking at the camera (engaged) or looking away (distracted), adjusting conversation pacing accordingly.
• Head pose estimation (@psyche/head-pose-estimation): Determine the 3D orientation of participant heads to assess attentiveness.
• Gaze estimation (@psyche/gaze-estimation): Track where participants are looking — screen, notes, other participants — as a signal of focus and engagement.

8.2 Outbound Emotion Expression — @psyche/emotion-engine#

The emotion engine maintains a continuous emotional state model that transitions smoothly between states rather than snapping instantly, mimicking natural emotional inertia. The table below maps each emotion state to the conversation conditions that trigger it.

8.3 Participation Analytics — @psyche/participation-tracker#

Participation analytics surface the meeting dynamics over time, enabling the persona to balance engagement across participants who speak very different amounts.

• Per-participant contribution tracking: Measure each person's share of speaking time and message count, identifying dominant speakers and quiet participants.
• Meeting engagement score: Aggregate engagement across all participants into a single per-minute engagement metric for the overall meeting.

9. Memory and Continuity#

Psyche personas maintain memory across sessions, enabling genuine continuity — a returning participant is recognized, remembered, and engaged with the context of their history. Memory operates at three timescales: within a session, between sessions, and at the level of persistent configured knowledge.

9.1 In-Session Memory — @psyche/memory-in-context#

• Full conversation history: The complete exchange within the current session is maintained in the active context window for coherent single-session dialogue.
• Working memory (@psyche/memory-working): Key facts established during the session — participant name, problem description, stated preferences, previous decisions — are tracked in structured working memory that persists for the session duration.

9.2 Long-Term Memory — @psyche/memory-core#

After a session ends, interaction data is persisted and vectorized so that future sessions can retrieve semantically relevant history. The memory lifecycle moves from active storage through consolidation to eventual archival.

• Memory persistence (@psyche/memory-persistence): Session conversations and extracted facts are stored to PostgreSQL at session end, building a durable record of all persona-participant interactions.
• Memory embeddings (@psyche/memory-embeddings): Conversation memories are vectorized using embedding models and stored in Qdrant, enabling semantic similarity retrieval — the persona recalls past interactions by meaning, not just exact text match.
• Memory retrieval (@psyche/memory-retrieval): At the start of each session and during each conversation turn, semantically relevant past memories are retrieved and injected into the context window.
• Memory consolidation (@psyche/memory-consolidation): During low-activity periods, detailed interaction records are compressed into concise summaries using the LLM, reducing storage requirements while preserving essential information.
• Memory archival (@psyche/memory-archival): Memories beyond the active retention window are archived to cold storage and can be retrieved for auditing or compliance needs.

9.3 Cross-Session Continuity#

• Returning user recognition: When a participant who has interacted before joins a new session (identified by email, account ID, or phone number), the persona retrieves their history and can acknowledge the relationship naturally.
• Configurable retention period: Memory retention duration is configurable per persona and per organization (default: 90 days).
• Privacy controls: Participants can request deletion of all their memory data. Memories are automatically anonymized after the configured retention period.

9.4 Persistent Knowledge#

• Fact memory: Key-value facts the persona always knows regardless of conversation context — company name, current promotions, support hours, policies.
• Opinion memory: The persona's defined stance on configurable topics with conviction strength settings (0.0–1.0) controlling frequency and confidence of the opinion being voiced.

10. Session Management#

10.1 Session Lifecycle#

A session progresses through a well-defined set of states from creation through teardown. The Connecting state covers the time between a client requesting entry to a meeting and the persona becoming fully active in the call.

10.2 Session Configuration#

• Persona assignment: Select which persona configuration participates in this session.
• Platform selection: Choose the video conferencing platform (Zoom, Teams, Google Meet, Webex, or custom WebRTC).
• Meeting linkage: Associate the session with a specific meeting ID, invite URL, or scheduled calendar event.
• Computer use permissions: Session-level toggle for enabling or disabling browser automation capabilities.
• Knowledge collection override: Specify which knowledge collections to use for this specific session, overriding persona defaults.

10.3 Session Metrics#

The following metrics are captured for every session and are available through the analytics API and admin dashboard.

11. Admin Dashboard#

11.1 Overview Metrics#

The dashboard shows at-a-glance operational status: total sessions (current period), active sessions (real-time), total messages, average response time, satisfaction score, and error rate. A period selector (day, week, month, quarter, year) enables period-over-period comparison, with time-series charts for all key metrics.

11.2 Persona Management#

• Create persona: Multi-step guided form: Identity → Personality → Expertise → Voice → Avatar → Review.
• Edit persona: Update any attribute of an existing persona at any time. Changes to personality and expertise take effect immediately; voice and avatar changes require re-training.
• Clone persona: Duplicate an existing persona as the starting point for a new variant.
• Status control: Activate, deactivate, or archive personas individually or in bulk.

11.3 Knowledge Base Management#

• Document upload: Drag-and-drop upload with processing status indicators showing Uploading → Processing → Embedding → Ready.
• Collection organization: Create and manage named collections with descriptions and priority rankings.
• Chunk preview: View how any document was split into retrieval chunks to assess chunking quality.
• Persona-collection assignment: Map knowledge collections to personas with priority settings.
• Ingestion analytics: Per-document retrieval frequency to identify high-value knowledge assets.

11.4 Session Monitoring#

• Live session view: Real-time list of active sessions with monitoring capability.
• Session history: Paginated history of all past sessions with filtering by persona, platform, date range, and outcome.
• Session replay: Review the transcript and key metrics of any past session.
• Engagement heatmap: Visual timeline of participant engagement levels throughout each session.

12. Analytics and Reporting#

Psyche captures detailed operational data at every level — per-session, per-persona, and per-knowledge-collection — to support quality improvement, capacity planning, and compliance reporting.

• Session analytics: Duration, message count, tool execution count, average response time, and satisfaction score per session, aggregated across any date range.
• Persona performance comparison: Side-by-side comparison of personas across engagement rate, satisfaction score, task resolution rate, and escalation frequency.
• Knowledge utilization: Per-document and per-collection retrieval frequency, identifying the highest-value knowledge assets and coverage gaps.
• Engagement trends: Participant engagement scores over time, segmented by persona, platform, and organization.
• Satisfaction tracking (@psyche/satisfaction, @psyche/nps): NPS-style (Net Promoter Score) satisfaction scores per persona and session type, with trend analysis and anomaly alerting.
• Quality trend analysis (@psyche/quality-trend-analysis): Automated analysis of response quality trends using linguistic quality metrics and user feedback signals.
• Human evaluation framework (@psyche/human-evaluation, @psyche/crowdsourced-evaluation): Structured quality assessment by human reviewers using standardized rubrics, and crowdsourced evaluation for large-scale quality assessment of generated responses.
• Export: Download any analytics dataset as CSV or PDF for external reporting or compliance documentation.

13. Multi-Language and Localization#

Psyche supports meetings conducted in multiple languages and can bridge language gaps between participants and the persona in real time.

• Language detection (@psyche/language-detection): Automatic detection of the language each participant is speaking, with no manual configuration required.
• Speech translation (@psyche/speech-translation): Real-time translation of participant speech so the persona can respond in a different language from the input — enabling cross-language meetings.
• Translation memory (@psyche/translation-memory): Stores previously translated phrases for consistency across sessions and reduced translation latency for recurring content.
• Translation quality (@psyche/translation-quality): Automated quality scoring for translated utterances to detect and flag low-confidence translations.
• Translation streaming (@psyche/translation-streaming): Real-time streaming translation with sub-300ms latency from speech input to translated text output.
• Dialect handling (@psyche/dialect-handling): Handles regional accent and dialect variations in speech recognition, improving accuracy for non-standard accents.
• Participant language preferences (@psyche/participant-language-preference): Per-participant language preference storage so returning participants are always addressed in their preferred language.
• Cultural adaptation (@psyche/cultural-adaptation): Adjusts the persona's communication style, idioms, and conversational etiquette for different cultural contexts — not just language translation, but cultural appropriateness.

14. Accessibility#

• Screen reader support: ARIA-compliant admin dashboard with full keyboard navigation and screen reader compatibility (VoiceOver, NVDA).
• Keyboard navigation: All admin dashboard actions are accessible via keyboard with logical tab order and keyboard shortcut support.
• Closed captions: Real-time captioning of avatar speech displayed in the meeting interface, with automatic caption file generation for recordings.
• Caption streaming (@psyche/caption-streaming): Low-latency streaming caption delivery synchronized with avatar speech output.
• V2 match captions: V2 composes @psyche/caption-streaming through @v2/psyche-caption-streaming for real-time HUD captions, spectator mode overlays, replay transcripts, companion captions, and broadcast observer captions. The integration is presentation-only and off rollback.
• Accessibility compliance (@psyche/accessibility): WCAG 2.1 AA compliance auditing for the admin dashboard with automated compliance checking.

15. Integrations and Webhooks#

15.1 Knowledge and Research Integration#

Psyche uses Sophia's academic research infrastructure to enrich knowledge- intensive persona configurations with research-grade information. The boundary is clean: Sophia owns the research data and search infrastructure; Psyche owns the retrieval configuration that decides when and how to invoke it.

• Sophia integration (@psyche/sophia-integration, @psyche/sophia-search-integration): Deep integration with Sophia's academic research infrastructure for knowledge enrichment and fact verification in knowledge-intensive persona configurations.

15.2 Outbound Webhooks#

Webhook subscriptions enable external systems to react to Psyche events without polling. All payloads are HMAC-signed so receiving servers can verify that the request genuinely originated from Psyche.

• Session events: HTTP callbacks when sessions start, end, or encounter errors.
• Message events: Per-message webhooks for real-time integration with external CRM or logging systems.
• Tool execution events: Webhooks when specific tools are invoked, enabling external system updates triggered by persona actions.
• HMAC-SHA256 authentication: All webhook payloads are signed with a shared secret, enabling receiving servers to verify authenticity without a separate authentication step.
• Configurable events: Enable or disable specific event types per webhook endpoint to control delivery volume.

15.3 REST and WebSocket APIs#

• REST API: Full programmatic management of personas, sessions, knowledge bases, and analytics — complete coverage of all admin dashboard capabilities via API.
• WebSocket API: Real-time streaming of session events, live transcript delivery, and session state changes for building real-time integrations.

16. Security and Compliance#

• End-to-end encryption: Session audio, video streams, and conversation data are encrypted in transit (TLS 1.3) and at rest (AES-256).
• API key authentication: HMAC-signed API keys with configurable scopes (read-only, read-write, admin) and optional expiry timestamps.
• Role-based access control: Four permission levels — Admin (full access), Editor (persona and knowledge management), Viewer (read-only analytics), API-only (programmatic access without UI).
• Audit logging: Immutable timestamped logs of all persona changes, session access, knowledge base modifications, and administrative actions.
• Data residency: Enterprise organizations choose which geographic region (US, EU, Asia-Pacific) stores all data, supporting GDPR data sovereignty requirements.
• Memory deletion: Per-user and per-persona memory purge capabilities with cryptographic deletion verification for compliance.
• GDPR compliance: Full GDPR compliance including data subject access requests, right to erasure, processing records, and DPA (Data Processing Agreement) template.
• Kubernetes security (@psyche/k8s-security): Kubernetes security policies enforced via OPA/Gatekeeper including pod security standards, network policies, and RBAC.
• Computer use sandbox isolation: Browser automation runs in a container with no network access outside explicitly permitted domains and no access to host system resources.

17. Platform Management#

• Multi-organization support: Complete data isolation between organizations. One Psyche deployment serves multiple customer organizations with no data co-mingling.
• User roles and permissions: Scoped roles per organization with fine-grained permission sets.
• API key management: Create, name, scope, rotate, and revoke API keys from the dashboard.
• Usage quotas: Configurable limits on session counts, message volumes, and storage capacity per organization.
• White-label support: Custom organization branding (logo, colors, domain) for the admin dashboard and avatar video frame overlay.

18. Infrastructure and Reliability#

18.1 Observability#

Psyche instruments every microservice with a consistent observability stack so that performance regressions and failures are detected before users notice them.

• Structured logging (@psyche/logging): All service logs are structured JSON with correlation IDs linking logs across the avatar renderer, voice engine, LLM service, and conferencing integration for a given session.
• Distributed tracing (@psyche/tracing): OpenTelemetry-based distributed tracing for tracking request paths across microservices.
• SLO alerting (@psyche/slo-alerting): Service Level Objective definitions with automated alerts when error budgets are being consumed too quickly — before users experience a visible degradation.
• Latency analysis (@psyche/latency-analysis): Per-component latency tracking identifying bottlenecks in the end-to-end response pipeline.
• Metrics (@psyche/log-metrics): High-cardinality operational metrics with timeseries optimization (@psyche/timeseries-optimization) for efficient storage and querying.
• Predictive alerting (@psyche/predictive-alerting): ML-based anomaly detection that identifies unusual metric patterns before they become user-visible failures.
• Predictive performance modeling (@psyche/predictive-performance-modeling): Models that predict resource requirements and session quality under different load conditions.
• ML log analysis (@psyche/ml-log-analysis): Automated analysis of log streams to detect error patterns, correlate incidents, and surface root causes faster than manual inspection.

18.2 Reliability#

• Disaster recovery (@psyche/disaster-recovery): Documented recovery procedures and automated failover for all critical services.
• Database sharding (@psyche/database-sharding): Horizontal database partitioning for high-volume deployments.
• K8s federation (@psyche/k8s-federation): Multi-cluster Kubernetes federation for geographic distribution and failure isolation.
• AIOps anomaly detection (@psyche/aiops-anomaly): Automated anomaly detection across system metrics using machine learning.

Library Summary#

The table below maps every major capability area to its implementing libraries. This serves as a quick lookup when you need to find the right library for a given concern.

Neith Embodiment and Spatial Runtime Dependencies (Phases 162, 166, 167)#

Psyche owns embodied assistant behavior, conversation, perception, voice, and human-like presence. Neith owns the digital-human, spatial-computing, and runtime audio primitives that Psyche can use for higher-fidelity embodiment. The boundary exists because Neith's sovereign embodiment primitives (character rigs, spatial OS integration, audio graphs) are shared infrastructure that multiple domains use — Psyche consumes these rather than reimplementing them.

• Digital humans (Phase 162): Psyche integrates with @neith/dh-* character creator, skin/eye material systems, FACS face rig, body rig, muscle/fat/skin simulation, clothing, performance capture animator, voice and speech bridge, retargeting, runtime SDK, consent, likeness, and provenance to upgrade avatar rendering, expressions, gesture, lip sync, and ethical identity handling.
• Spatial computing (Phase 166): Psyche uses @neith/spatial-* sessions, world anchors, volumes, immersive/shared spaces, gaze, pinch, hand, and voice input, personas, avatars, presence, foveated rendering, passthrough physics, spatial audio, developer SDK, collocated sessions, and spatial OS shell integration for embodied assistants in AR/VR/spatial environments.
• Engine audio (Phase 167): Psyche can route voice, dialogue, microphone, spatialization, environmental acoustics, ducking, audio accessibility, captions, voice chat, profiler, and tooling through Neith's audio graph, source playback, 3D spatialization, dialogue, music, synth, and accessibility systems.

Neural World Model Integration (Phase 176)#

Psyche contributes computer-use trajectories and perception data into the Nous world-model flywheel and can consume JEPA visual/audio encoders for embodied assistant perception. Psyche owns embodied behavior, screen understanding, voice, avatar presence, and action safety; Nous owns reusable world-model training, latent dynamics, and serving. This boundary keeps the data pipeline and model training infrastructure centralized in Nous, while Psyche remains focused on the real-time assistant behavior that uses those models.

Training-Data Flywheel (Phases 85–86)#

libs/psyche/training-data implements Psyche's side of the ML-sovereignty data flywheel with dedicated avatar and perception signal pipelines: avatar rendering/interaction quality signals and perception (emotion/engagement) signals are captured as passive training data. Signals are consent-gated, anonymized where required, and emitted in the shared flywheel envelope that Nous dataset management (Phase 87) ingests. Nous owns the training infrastructure; Psyche owns what constitutes a high-quality persona signal.