# Psyche Domain — Architecture > **Psyche** (Serwaa) - Hyper-Realistic AI Virtual Assistant Platform Psyche is the domain responsible for photorealistic digital-human virtual assistants that join real video conference meetings, hold real-time conversations, and perform computer tasks on behalf of participants. The core engineering challenge is achieving **human indistinguishability** in real time: rendering a convincing 3D human face, synthesizing natural voice, tracking participant emotion, and generating intelligent responses — all within a sub-500ms end-to-end latency budget, inside a live Zoom, Microsoft Teams, Google Meet, or Webex call. The architecture decomposes this into six loosely coupled real-time engines (Avatar, Voice, Behavior, Perception, Computer-Use, Conferencing), each running as an independent Python microservice. An Orchestrator service coordinates session lifecycle and routes data across engines; a Persona Service manages configurable identity profiles; and a Knowledge Base service supplies domain-specific facts via Retrieval-Augmented Generation. All engines communicate via gRPC for binary streams and Redis pub/sub for state broadcasts, ensuring that no single slow component blocks the others. A key design choice is **graceful degradation**: if the local GPU is overloaded, avatar rendering falls back to Tavus cloud rendering; if the primary TTS provider is unavailable, the voice pipeline switches to the next in a failover chain; the conversation never drops. This document describes the full system topology, data flows, library organization, ML pipelines, and integration architecture. --- ## 1. Design Principles 1. **Human Indistinguishability** — All rendering, voice, and behavioral systems are tuned to avoid the uncanny valley. Quality wins over speed. 2. **Real-Time First** — Sub-500ms end-to-end latency is a hard requirement. Every component budget is tracked. 3. **Modular Pipelines** — Avatar, voice, behavior, and perception are independent services that communicate via gRPC and Redis pub/sub, not monolithic code. 4. **Privacy by Default** — Session audio and video are not retained beyond the session. Memory is opt-in. Memory deletion is always honored. 5. **Graceful Degradation** — If the avatar engine is overloaded, fall back to Tavus cloud rendering. If ElevenLabs is down, fall back to XTTS. The conversation never drops. 6. **Provider Agnosticism** — Voice providers, STT providers, LLMs, and conferencing platforms are all pluggable. Psyche does not lock in to any single vendor. --- ## 2. Technology Stack The table below summarizes every technology layer. Most backend services are Python FastAPI applications; TypeScript libraries provide client-facing SDKs, observability tooling, and cross-domain bridges. GPU-accelerated components (avatar rendering, voice synthesis, perception) run on CUDA 12.1+ with TensorRT for optimized inference. | Component | Technology | | ------------------ | ------------------------------------------------------------------------ | | Backend Services | Python 3.11+, FastAPI, uvicorn, asyncio | | TypeScript Libs | TypeScript 5+, Node.js 22+, Vitest | | Admin Dashboard | TypeScript, Next.js 14, Tailwind CSS | | Avatar Rendering | 3D Gaussian Splatting, FLAME model, CUDA 12.1+, TensorRT | | Voice Synthesis | XTTS v2, ElevenLabs, Cartesia, OpenAI TTS, Deepgram Aura | | Speech Recognition | Deepgram Nova-2 (primary), OpenAI Whisper (local/fallback) | | LLM | Anthropic Claude 3.5 Sonnet (primary), GPT-4, Gemini, Ollama | | Embeddings | Voyage AI (primary), Cohere, OpenAI (fallbacks) | | Database | PostgreSQL 16 with pgvector, SQLAlchemy ORM | | Vector Store | Qdrant | | Cache | Redis 7 | | Object Storage | MinIO (S3-compatible) | | Conferencing | Zoom SDK, Microsoft Teams (Graph API + Bot Framework), WebRTC, Webex SDK | | Cloud Avatars | Tavus Phoenix (optional) | | Build System | Nx + pnpm (TypeScript), Poetry (Python) | | Code Quality | ESLint + Prettier (TS), Ruff + mypy (Python) | | Infrastructure | Docker, Kubernetes, NVIDIA GPU nodes | --- ## 3. High-Level System Diagram The diagram below shows how external clients connect through the API Gateway to the three layers of the system: management services (Orchestrator, Persona, Knowledge Base), the real-time processing engines, and the underlying data stores. GPU memory allocations are shown in brackets for the three GPU-bound engines. ``` ┌─────────────────────────────────────────────────────────────────────────────┐ │ EXTERNAL CLIENTS │ │ │ │ Web App | Mobile App | REST API | Zoom / Teams / Meet / Webex │ └──────────────────────────────┬───────────────────────────────────────────────┘ │ ▼ ┌──────────────────────┐ │ API Gateway :8000 │ │ Auth | Rate Limit │ │ Routing | Sessions │ └──────────┬───────────┘ │ ┌────────────────────┼────────────────────┐ ▼ ▼ ▼ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────────┐ │ ORCHESTRATOR │ │ PERSONA SERVICE │ │ KNOWLEDGE BASE │ │ :8007 │ │ :8009 │ │ :8008 │ │ │ │ │ │ │ │ Session State │ │ Identity Mgmt │ │ Document Ingestion │ │ Pipeline Coord │ │ Personality │ │ Vector Embeddings │ │ Resource Alloc │ │ Voice Config │ │ Semantic Search │ └────────┬─────────┘ └──────────────────┘ └──────────────────────┘ │ │ Session Coordination ▼ ┌─────────────────────────────────────────────────────────────────────────────┐ │ REAL-TIME PROCESSING LAYER │ │ │ │ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ │ │ AVATAR ENGINE │ │ VOICE ENGINE │ │ BEHAVIOR ENGINE │ │ │ │ Port 8001 │ │ Port 8002 │ │ Port 8003 │ │ │ │ [GPU: 8 GB] │ │ [GPU: 6 GB] │ │ [CPU only] │ │ │ │ │ │ │ │ │ │ │ │ 3DGS Render │ │ XTTS / EL TTS │ │ FACS Exprs. │ │ │ │ FLAME Model │ │ Whisper STT │ │ Micro-Exprs. │ │ │ │ Lip Sync │ │ Voice Cloning │ │ Gaze Control │ │ │ │ TAA │ │ Viseme Gen │ │ Head Movement │ │ │ └────────┬────────┘ └────────┬────────┘ └────────┬────────┘ │ │ │ │ │ │ │ ┌────────┴───────┐ ┌─────────┴──────┐ ┌─────────┴──────────┐ │ │ │ PERCEPTION ENG │ │ CONFERENCING │ │ COMPUTER USE │ │ │ │ Port 8004 │ │ Port 8006 │ │ Port 8005 │ │ │ │ [GPU: 4 GB] │ │ │ │ │ │ │ │ │ │ Zoom SDK │ │ Headless Chrome │ │ │ │ Face Detect │ │ Teams SDK │ │ Screen Analysis │ │ │ │ Emotion Recog │ │ Meet WebRTC │ │ Sandboxed Exec │ │ │ │ Gaze Track │ │ Webex SDK │ │ Tool Framework │ │ │ └────────────────┘ └────────────────┘ └────────────────────┘ │ └─────────────────────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────────────┐ │ DATA LAYER │ │ │ │ ┌────────────┐ ┌───────────┐ ┌────────────┐ ┌────────────────────────┐ │ │ │ PostgreSQL │ │ Redis │ │ Qdrant │ │ MinIO (S3) │ │ │ │ (pgvector) │ │ │ │ │ │ │ │ │ │ Personas │ │ Sessions │ │ Knowledge │ │ serwaa-models │ │ │ │ Sessions │ │ Cache │ │ Memories │ │ serwaa-avatars │ │ │ │ Knowledge │ │ Pub/Sub │ │ Voices │ │ serwaa-voices │ │ │ │ Audit │ │ Queues │ │ Convs. │ │ serwaa-knowledge │ │ │ └────────────┘ └───────────┘ └────────────┘ └────────────────────────┘ │ └─────────────────────────────────────────────────────────────────────────────┘ ``` --- ## 4. Library Organization Psyche ships 135 library directories under `libs/psyche/` (predominantly TypeScript) plus 16 Python services under `services/psyche/`. Most `libs/psyche/` libraries are TypeScript packages (`package.json` + `tsup.config.ts`); a small number — including `auth`, `cache`, and `database` — are Python packages (`pyproject.toml`). The libraries are grouped below by functional area. Each group corresponds to a dedicated service or subsystem in the architecture diagram above. ``` libs/psyche/ │ ├── common/ # Shared types, utilities, constants ├── storage/ # S3-compatible file storage ├── logging/ # Structured logging ├── tracing/ # Distributed tracing ├── messaging/ # Inter-service messaging │ ├── Avatar System │ ├── avatar-core/ # Core rendering pipeline │ ├── avatar-cache/ # Model caching and preloading │ ├── avatar-expressions/ # Facial expression system │ ├── avatar-lipsync/ # Lip synchronization │ ├── avatar-quality/ # Quality metrics │ ├── avatar-taa/ # Temporal anti-aliasing │ ├── avatar-training/ # Training pipeline │ └── avatar-isis-integration/ # Cross-domain avatar generation │ ├── Voice and Speech │ ├── voice-synthesis/ # TTS synthesis │ ├── voice-streaming/ # Real-time audio streaming │ ├── voice-consistency/ # Quality monitoring │ ├── speech-recognition/# STT │ ├── viseme-generator/ # Visual phoneme generation │ └── noise-handling/ # Audio preprocessing │ ├── Behavior and Expression │ ├── emotion-engine/ # Emotional response generation │ ├── emotion-recognition/ # User emotion detection │ ├── micro-expressions/ # Subtle facial expressions │ ├── facs-expressions/ # FACS control system │ ├── gaze-control/ # Eye gaze direction │ ├── gaze-estimation/ # Participant gaze tracking │ ├── head-movement/ # Head motion generation │ ├── head-pose-estimation/ # Participant head pose detection │ ├── posture-system/ # Body posture management │ ├── gesture-system/ # Hand/body gestures │ ├── behavior-coordinator/ # Multi-channel orchestration │ ├── behavior-anomaly-detector/ # Anomaly detection │ └── uncanny-valley/ # Uncanny valley mitigation │ ├── Perception │ ├── face-detection/ # Video stream face detection │ ├── face-analysis/ # Facial feature analysis │ ├── engagement-detector/ # Engagement measurement │ ├── attention-tracker/ # Participant attention │ └── participation-tracker/ # Meeting contribution analysis │ ├── Memory System │ ├── memory-core/ # Core memory manager │ ├── memory-working/ # Short-term working memory │ ├── memory-in-context/ # In-context memory injection │ ├── memory-embeddings/ # Memory vector embedding │ ├── memory-retrieval/ # Retrieval and recall │ ├── memory-archival/ # Long-term archival │ ├── memory-consolidation/ # Memory consolidation │ ├── memory-persistence/ # Database persistence │ └── memory-tools/ # Memory tool implementations │ ├── Knowledge and Intelligence │ ├── knowledge-context/ # Knowledge context management │ ├── knowledge-ingestion/ # Document ingestion pipeline │ ├── knowledge-retrieval/ # RAG retrieval │ └── dialogue-manager/ # Conversation flow management │ ├── Computer Use │ ├── computer-use-core/ # Core orchestration │ ├── browser-automation/ # Headless browser control │ ├── screen-analysis/ # Screen content understanding │ ├── sandbox/ # Secure execution sandbox │ ├── tool-registry/ # Tool registration │ ├── tool-execution/ # Tool execution engine │ └── action-safety/ # Action safety validation │ ├── Video Conferencing │ ├── conferencing-core/ # Core conferencing abstraction │ ├── zoom-integration/ # Zoom SDK integration │ ├── teams-integration/ # Microsoft Teams integration │ ├── meet-integration/ # Google Meet integration │ ├── webex-integration/ # Webex integration │ ├── participant-manager/ # Participant management │ └── state-sync/ # State synchronization │ ├── Tavus Integration (optional cloud rendering) │ └── tavus-bridge, tavus-client, tavus-conversation, │ tavus-expression, tavus-hybrid, tavus-knowledge, │ tavus-llm, tavus-memories, tavus-objectives, │ tavus-perception, tavus-persona-manager, tavus-phoenix, │ tavus-pipecat, tavus-pipeline, tavus-replica-manager, │ tavus-tools, tavus-tts, tavus-turntaking │ ├── Observability and Quality │ ├── latency-analysis/ # End-to-end latency measurement │ ├── slo-alerting/ # SLO monitoring │ ├── nps/ # Net Promoter Score │ ├── satisfaction/ # User satisfaction │ ├── human-evaluation/ # Human evaluation framework │ └── k8s-security/ # Kubernetes security policies │ └── Cross-Domain ├── sophia-integration/ # Sophia knowledge search bridge ├── sophia-search-integration/ # Sophia semantic search bridge └── recall-integration/ # Recall.ai bridge ``` > **Note on Python libraries inside `libs/psyche/`**: `auth` > (authentication/authorization), `database` (schema, Prisma, migrations), and > `cache` (Redis caching) are Python packages (`pyproject.toml`, Python sources > only) that live **inside** `libs/psyche/` alongside the TypeScript libraries — > the `libs/psyche/` tree is therefore not exclusively TypeScript. They are > distinct from the deployable Python **services** under `services/psyche/` > listed in Section 5. --- ## 5. Service Architecture ### Python Backend Microservices All backend services are Python FastAPI applications under `services/psyche/`. There are 16 service directories; the `project.json` name is in parentheses. Three of these (Behavior Engine, Security, Error Handling) are currently service shells — the directory and `pyproject.toml` exist but `src/` has not yet been populated with logic. | Service (project name) | Responsibilities | | ------------------------------------------------ | ------------------------------------------------------------ | | API Gateway (`psyche-api-gateway`) | REST + WebSocket gateway, rate limiting, request routing | | Orchestrator (`psyche-orchestrator`) | Session state machine, pipeline routing, resource allocation | | Persona Service (`psyche-persona-service`) | Persona identity/personality/expertise/memory/constraints | | Knowledge Base (`psyche-knowledge-base`) | Document ingestion, chunking, embedding, RAG, retrieval | | Avatar Engine (`psyche-avatar-engine`) | 3DGS / NeRF rendering, FLAME model, lip sync, TAA | | Voice Engine (`psyche-voice-engine`) | TTS, STT, diarization, visemes, turn-taking, captions | | Behavior Engine (`psyche-behavior-engine`) | Service shell — `src/` is currently empty | | Perception Engine (`psyche-perception-engine`) | Face/emotion/intent/screen perception | | Computer Use (`psyche-computer-use`) | Browser/desktop automation agent, vision, sandbox | | Conferencing (`psyche-conferencing`) | Zoom/Teams/Meet/Webex/WebRTC adapters, audio/video routing | | Video Conferencing (`psyche-video-conferencing`) | Conferencing service shell | | Learning System (`psyche-learning-system`) | Continuous adaptation, persona improvement from feedback | | Tool Framework (`psyche-tool-framework`) | Tool registry, router, MCP server | | Tavus Integration (`psyche-tavus-integration`) | Tavus cloud-rendering integration | | Security (`psyche-security`) | Security service shell | | Error Handling (`psyche-error-handling`) | Error-handling service shell | > The deployable services above are distinct from the Python **libraries** > (`auth`, `cache`, `database`) that live under `libs/psyche/`. There is no > dedicated standalone "Conferencing Gateway" service; conferencing logic is in > the `psyche-conferencing` package (whose Python module is > `video_conferencing`). --- ## 6. Session Lifecycle Flow When a client requests a session, the API Gateway delegates to the Orchestrator, which allocates resources and coordinates the startup of the Avatar, Voice, and Conferencing engines in parallel before signaling readiness. The sequence below shows the request path from session creation through the active state. ``` [Client] [API Gateway] [Orchestrator] │ │ │ │ POST /sessions │ │ │ ─────────────────────>│ │ │ │ Create record │ │ │ ────────────────────>│ │ │ Allocate resources │ │ │<─────────────────────│ │ session_id, PENDING │ │ │<──────────────────────│ │ │ │ │ │ POST /sessions/{id}/join │ │ ─────────────────────>│ │ │ │ │ │ │ [Conferencing :8006] │ │ │ Connect to meeting │ │ │ │ │ │ [Avatar :8001] │ │ │ Load 3DGS model │ │ │ │ │ │ [Voice :8002] │ │ │ Warm up TTS │ │ │ │ │ status=ACTIVE │ │ │<──────────────────────│ │ │ │ │ ▼ ▼ ▼ ════════ ACTIVE SESSION PROCESSING ════════ (Real-Time Processing Loop) ``` --- ## 7. Real-Time Processing Loop Once a session is active, every frame of the meeting goes through a pipelined chain of operations. Incoming audio and video are first analyzed by the Perception Engine (detecting emotion and gaze), then speech-to-text transcription drives the LLM and TTS, and finally the Behavior Engine drives the Avatar Engine to render the response video. The total budget for this cycle is under 500ms. ``` [Video Conference] [AI Avatar Output] │ ▲ │ User Audio + Video (30 FPS) │ ▼ │ ┌──────────────┐ ┌──────────────┐ ┌─────────────────┐ │ │ CONFERENCING │───>│ PERCEPTION │───>│ ORCHESTRATOR │ │ │ Gateway │ │ Engine │ │ │ │ │ │ │ │ │ Route to LLM │ │ │ Audio/Video │ │ Face Detect │ │ Manage context │ │ │ 30 FPS │ │ Emotion │ │ Tool execution │ │ └──────────────┘ │ Head Pose │ └────────┬────────┘ │ │ Gaze │ │ │ └──────────────┘ │ │ ▼ │ ┌────────────────────┐ │ │ VOICE ENGINE │ │ │ STT (Deepgram) │ │ │ Claude LLM │ │ │ TTS (ElevenLabs) │ │ │ Viseme Generation │ │ └────────┬───────────┘ │ │ │ ┌──────────────────┼───────────────┤ │ │ │ ▼ ▼ │ ┌────────────┐ ┌────────────────┐ │ │ BEHAVIOR │ │ AVATAR ENGINE │──────┘ │ Engine │───>│ │ │ FACS │ │ 3DGS Render │ │ Gestures │ │ FLAME Model │ │ Gaze │ │ 1080p @ 30 FPS│ └────────────┘ └────────────────┘ TIMING BUDGET: Perception: < 20 ms Voice STT: < 100 ms LLM Response: < 200 ms Voice TTS: < 100 ms Behavior Compute: < 10 ms Avatar Render: < 33 ms TOTAL: < 500 ms ``` --- ## 8. Data Architecture Psyche uses three distinct persistence mechanisms, each chosen for a specific access pattern. PostgreSQL holds structured relational data (personas, sessions, audit logs). Qdrant provides approximate nearest-neighbor search over high-dimensional vectors for knowledge and memory retrieval. MinIO holds large binary objects — trained avatar models, cloned voice files, and uploaded documents. Redis sits in front of all three as a multi-TTL caching layer, keeping the hot paths well under 5ms. ``` APPLICATION LAYER ┌─────────────┬─────────────┬───────────────┬───────────────────────┐ │ API Gateway │ Orchestrator│ Avatar/Voice │ Other Microservices │ └──────┬──────┴──────┬──────┴───────┬───────┴────────────┬──────────┘ │ │ │ │ ▼ ▼ ▼ ▼ CACHING LAYER ┌────────────────────────────────────────────────────────────────────┐ │ L1: Application-Level (< 1 ms) │ L2: Redis Cluster (< 5 ms) │ │ Hot config data │ Session state (TTL: 1 hr) │ │ Recent embeddings │ Persona config (TTL: 30min) │ │ Pinned model weights │ Avatar state (TTL: 1 min) │ │ │ Voice state (TTL: 30 sec) │ └────────────────────────────────────────────────────────────────────┘ │ │ │ │ ▼ ▼ ▼ ▼ PERSISTENCE LAYER ┌─────────────┐ ┌───────────────┐ ┌────────────────────────────┐ │ PostgreSQL │ │ Qdrant │ │ MinIO (S3) │ │ (pgvector) │ │ Vector search │ │ Object storage │ │ │ │ for knowledge │ │ (avatar models, voices, │ │ Target │ │ and memory │ │ documents) │ │ store for │ │ embeddings │ │ │ │ domain │ │ │ │ │ │ data │ │ │ │ │ └─────────────┘ └───────────────┘ └────────────────────────────┘ ``` > **Implementation status.** This is the intended persistence design. As built, > the API Gateway keeps personas, sessions, knowledge documents, tools, and > webhooks in per-process in-memory dictionaries (every router file is labelled > "In-memory storage for demo purposes"), and its lifespan handler leaves > database and Redis initialization commented out. The only database DDL that > exists is the idempotent bootstrap in `api-gateway/.../db/migrate.py`, which > enables extensions (including pgvector) and creates the `persona`, > `knowledge`, `sessions`, `analytics`, and `audit` schemas plus a single > `audit.change_log` table — no domain tables. SQLAlchemy/asyncpg are declared > dependencies but no connections are opened at runtime. --- ## 9. ML Pipelines Psyche has three offline training pipelines — avatar generation, voice cloning, and knowledge ingestion — each producing artifacts that are stored in MinIO and consumed by the real-time rendering and retrieval paths at session time. ### Avatar Training Pipeline An avatar is created from a short video of the target person. The pipeline extracts face geometry, fits the FLAME parametric face model to capture expression controls, and then runs a 3D Gaussian Splatting optimization over thousands of iterations. The resulting model is TensorRT-optimized for real-time inference. ``` Input: 30–60s training video (1080p+) │ v Face Extraction (MTCNN detector, landmark tracking) │ v FLAME Model Fitting (50 expression blend shapes) │ v 3DGS Training (~30,000 iterations, ~1–4 hours) │ v Expression Blend Shape Learning (lip sync, emotion blends) │ v TensorRT Optimization (optimized for real-time rendering) │ v Model Storage (MinIO: serwaa-models) ``` ### Voice Cloning Pipeline A cloned voice is derived from audio samples of the target person. The pipeline extracts a speaker embedding that captures the vocal identity, then fine-tunes the synthesis model to reproduce that speaker's characteristics. ``` Input: 1–3 min audio samples (WAV/MP3, 44.1 kHz+, SNR > 20 dB) │ v Audio Preprocessing (noise removal, normalization) │ v Speaker Embedding Extraction (SpeakerNet or XTTS encoder) │ v Voice Model Training / Fine-tuning │ v Voice Model Storage (MinIO: serwaa-voices) ``` ### Knowledge Ingestion Pipeline Documents uploaded to the knowledge base are parsed, split into overlapping semantic chunks, embedded into high-dimensional vectors by Voyage AI, and indexed into Qdrant. At inference time, a semantic nearest-neighbor search retrieves the most relevant chunks for RAG injection. ``` Input: PDF / DOCX / TXT / HTML / URL │ v Document Parser (format-specific) │ v Text Extraction + Metadata │ v Semantic Chunking (512 tokens, 64 overlap) │ v Embedding Generation (Voyage AI) │ v Qdrant Indexing (collection: psyche_knowledge) │ v PostgreSQL Metadata Record ``` --- ## 10. Communication Patterns Services communicate using three mechanisms, each chosen for the characteristics of the data being exchanged. Synchronous REST calls handle configuration and management operations. Redis pub/sub broadcasts lightweight state-change events to all interested subscribers without coupling publishers to receivers. gRPC streaming handles high-bandwidth continuous data such as audio and video frames. ### Service-to-Service REST (Python → Python) Standard HTTP requests using `httpx.AsyncClient` for configuration queries, persona lookups, and other request-response operations. ```python async def get_persona_config(persona_id: str) -> PersonaConfig: async with httpx.AsyncClient() as client: response = await client.get( f"{PERSONA_SERVICE_URL}/api/v1/personas/{persona_id}" ) return PersonaConfig(**response.json()) ``` ### Redis Pub/Sub (real-time event broadcasting) State changes are published on named channels and consumed by any number of subscribers. For example, the Avatar Engine publishes its current state and the Behavior Engine subscribes to drive expression synchronization. ```python # Publisher (Avatar Engine) await redis.publish(f"avatar:{session_id}:state", state.model_dump_json()) # Subscriber (Behavior Engine) pubsub = redis.pubsub() await pubsub.subscribe(f"avatar:{session_id}:state") async for message in pubsub.listen(): yield AvatarState.model_validate_json(message["data"]) ``` ### gRPC Streaming (audio/video streams) Binary audio and video streams use bidirectional gRPC streaming for minimum latency. The service definitions below show both the video frame and audio chunk streams exposed by the Avatar Engine. ```protobuf service AvatarService { rpc StreamFrames(stream RenderRequest) returns (stream VideoFrame); rpc StreamAudio(stream AudioChunk) returns (stream AudioChunk); } ``` --- ## 11. Security Architecture ### Authentication - **JWT tokens** — Short-lived (1 hr) signed tokens issued by API Gateway - **API keys** — HMAC-signed keys with organization scope and expiry - **Webhook HMAC** — All outbound webhooks include HMAC-SHA256 signature header ### Data Protection - Session audio and video are processed in-memory only and not persisted - Conversation transcripts are encrypted at rest (AES-256-GCM) - Memory records are encrypted; deletion is permanent - MinIO objects use server-side encryption ### Computer Use Sandbox The computer-use browser runs in a tightly constrained environment to prevent accidental or malicious access to host resources. Key restrictions include: - Isolated headless Chromium with no access to host filesystem - Network access restricted to explicit allowlist per tool - Maximum 50 actions per task (configurable) - High-risk actions require human confirmation before execution ### Kubernetes Security - Network policies enforced via `@psyche/k8s-security` library - GPU nodes isolated from public internet - Pod security contexts restrict privilege escalation --- ## 12. Observability Architecture Psyche instruments all services with a consistent three-layer observability stack: distributed traces for request-level visibility, Prometheus metrics for aggregate health, and structured logs correlated by trace IDs. The `@psyche/slo-alerting` library enforces per-component latency budgets and fires alerts before the full 500ms budget is exhausted. | Layer | Tool | Data Captured | | ---------- | ---------------------- | ------------------------------------------------------ | | Tracing | OpenTelemetry → Jaeger | Distributed traces across all service boundaries | | Metrics | Prometheus + Grafana | Latency per component, session counts, GPU utilization | | Logs | structlog → Loki | Structured JSON with trace correlation | | SLO alerts | `@psyche/slo-alerting` | Latency budget violations, error rate thresholds | ### Key SLO Metrics The following Prometheus metric names and their targets define the system's SLOs. Any breach fires an alert via `@psyche/slo-alerting`. ``` psyche_session_e2e_latency_ms (target: < 500 ms p95) psyche_avatar_frame_latency_ms (target: < 33 ms p99) psyche_tts_first_byte_latency_ms (target: < 100 ms p95) psyche_stt_transcription_latency_ms (target: < 100 ms p95) psyche_llm_response_latency_ms (target: < 200 ms p95) psyche_session_error_rate (target: < 1%) psyche_gpu_utilization_percent (alerting threshold: > 85%) ``` --- ## 13. Deployment Architecture ### Kubernetes Topology The deployment separates CPU-bound services (Deployments, horizontally scalable) from GPU-bound engines (DaemonSets on dedicated GPU node pools). Stateful data services run as StatefulSets with persistent volumes, and the nightly memory consolidation job runs as a Kubernetes CronJob. ``` Ingress └── psyche-api-gateway (Deployment, 2–10 replicas) ├── psyche-orchestrator (Deployment, 2–8 replicas) ├── psyche-persona-service (Deployment, 2–4 replicas) ├── psyche-knowledge (Deployment, 2–4 replicas) ├── psyche-conferencing (Deployment, 2–8 replicas) └── psyche-computer-use (Deployment, 2–4 replicas) GPU Node Pool: ├── psyche-avatar-engine (DaemonSet on GPU nodes, CUDA) ├── psyche-voice-engine (DaemonSet on GPU nodes, CUDA) └── psyche-perception-engine (DaemonSet on GPU nodes, CUDA) StatefulSets: ├── postgres-psyche ├── redis-psyche └── qdrant-psyche Jobs: └── psyche-memory-consolidator (CronJob, nightly) ``` ### Admin Dashboard - Next.js app deployed to CDN - API calls proxied through API Gateway - Available at `http://localhost:3000` in development --- ## 14. Domain Dependencies ### External Services Psyche Depends On These third-party services are essential to Psyche's real-time capabilities. Each is pluggable — if a provider is unavailable, Psyche falls back to the next option in the chain rather than failing the session. | Service | Purpose | | --------------------------- | ----------------------------------------- | | Anthropic Claude | LLM for conversation, computer use vision | | ElevenLabs | Primary TTS voice synthesis | | Deepgram | Primary real-time STT | | Voyage AI | Knowledge base embeddings | | Qdrant | Vector similarity search | | Zoom / Teams / Meet / Webex | Video conferencing platforms | | Tavus | Optional cloud avatar rendering (Phoenix) | ### Oshun Domain Dependencies Psyche draws on two other Oshun domains to augment its knowledge capabilities. The boundary is clear: Psyche owns the real-time assistant behavior and rendering; Sophia owns the research and academic knowledge infrastructure that Psyche taps for knowledge-intensive persona configurations. Iris provides the conversational AI capabilities that power general assistant use cases within Psyche sessions. | Domain | Library | Purpose | | ---------- | ---------------------------- | ---------------------------------------- | | **Iris** | `@iris/integrations-psyche` | Conversation AI capabilities | | **Sophia** | `@psyche/sophia-integration` | Knowledge enrichment via research search | ### Domains That Depend on Psyche Psyche is a standalone product domain. Other domains do not currently depend on it directly, though Iris's integration library (`@iris/integrations-psyche`) references Psyche APIs for virtual assistant use cases.