# Psyche Domain: Machine Learning Model Dependencies

This document provides a comprehensive inventory of all machine learning models
used by the Serwaa platform (source codebase for the Psyche domain migration).

## Table of Contents

1. [Overview](#overview)
2. [Avatar/3D Models](#avatar3d-models)
3. [Perception/Face Analysis Models](#perceptionface-analysis-models)
4. [Voice/Audio Models](#voiceaudio-models)
5. [Embedding Models](#embedding-models)
6. [NLP/Intent Detection Models](#nlpintent-detection-models)
7. [Core ML Dependencies](#core-ml-dependencies)
8. [Model Loading Patterns](#model-loading-patterns)
9. [GPU & Hardware Requirements](#gpu--hardware-requirements)
10. [Model Download Sources](#model-download-sources)
11. [Environment Variables](#environment-variables)
12. [Migration Considerations](#migration-considerations)

---

## Overview

The Serwaa platform uses 50+ machine learning models across 6 major categories:

| Category      | Models                        | Framework            | Deployment    |
| ------------- | ----------------------------- | -------------------- | ------------- |
| Avatar/3D     | 3DGS, FLAME, NeRF, Lip Sync   | PyTorch              | Local GPU     |
| Face Analysis | MediaPipe, InsightFace, MTCNN | TensorFlow, PyTorch  | Local GPU/CPU |
| Voice/Audio   | Whisper, ECAPA-TDNN, F5-TTS   | PyTorch, SpeechBrain | Local GPU     |
| Embeddings    | Voyage AI, OpenAI, BGE        | API, HuggingFace     | API/Local     |
| NLP           | BART-MNLI, Sentiment          | HuggingFace          | Local CPU     |
| Emotion       | DeepFace, HSEmotion, FER      | TensorFlow, PyTorch  | Local GPU/CPU |

### Model Architecture Overview

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                          ML Model Pipeline                                   │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  ┌──────────────────┐    ┌──────────────────┐    ┌──────────────────┐       │
│  │  Input Pipeline  │───▶│  Feature Models  │───▶│  Output Models   │       │
│  └──────────────────┘    └──────────────────┘    └──────────────────┘       │
│         ↓                         ↓                       ↓                  │
│  ┌──────────────────┐    ┌──────────────────┐    ┌──────────────────┐       │
│  │ Audio Processing │    │ Voice Embeddings │    │ TTS Synthesis    │       │
│  │ Face Detection   │    │ Face Embeddings  │    │ Avatar Rendering │       │
│  │ Text Processing  │    │ Text Embeddings  │    │ Lip Sync         │       │
│  └──────────────────┘    └──────────────────┘    └──────────────────┘       │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘
```

---

## Avatar/3D Models

### 3D Gaussian Splatting (3DGS)

**Real-time avatar rendering using 3D Gaussian primitives.**

#### Source Files

```
serwaa/services/avatar-engine/src/avatar_engine/models/gaussian.py
```

#### Model Specification

| Property      | Value                    |
| ------------- | ------------------------ |
| **Type**      | Custom PyTorch nn.Module |
| **Framework** | PyTorch 2.x              |
| **Device**    | CUDA (GPU required)      |
| **Precision** | FP16/FP32                |

#### Parameters

| Component     | Description                   | Size   |
| ------------- | ----------------------------- | ------ |
| **Positions** | 3D Gaussian centers           | N × 3  |
| **Rotations** | Quaternion orientations       | N × 4  |
| **Scales**    | Anisotropic scales            | N × 3  |
| **Opacities** | Alpha values                  | N × 1  |
| **SH DC**     | 0th-order spherical harmonics | N × 3  |
| **SH Rest**   | Higher-order SH (degree 1-3)  | N × 45 |

Where N = number of Gaussians (typically 50,000 - 100,000)

#### Spherical Harmonics Configuration

| SH Degree | Coefficients per Gaussian | Use Case                  |
| --------- | ------------------------- | ------------------------- |
| 0         | 3 (RGB only)              | Basic color               |
| 1         | 12                        | Directional lighting      |
| 2         | 27                        | Specular highlights       |
| 3         | 48                        | Full appearance (default) |

#### Memory Usage

| Resolution | Gaussians | VRAM   | Target FPS |
| ---------- | --------- | ------ | ---------- |
| 512×512    | 50K       | 1.5 GB | 200+       |
| 768×768    | 75K       | 2.5 GB | 120+       |
| 1024×1024  | 100K      | 4.0 GB | 60+        |
| 1920×1080  | 100K      | 6.0 GB | 30+        |

#### File Format

```python
# Checkpoint structure (.pt.gz)
{
    "metadata": {
        "num_gaussians": int,
        "sh_degree": int,
        "version": str,
        "creation_time": str
    },
    "core_state": {
        "positions": torch.Tensor,      # (N, 3) float16
        "rotations": torch.Tensor,      # (N, 4) float16
        "scales": torch.Tensor,         # (N, 3) float16
        "opacities": torch.Tensor,      # (N, 1) float16
        "sh_dc": torch.Tensor           # (N, 3) float16
    },
    "sh_rest": torch.Tensor             # (N, 45) float16, separate file
}
```

#### Loading Pattern

```python
# Progressive loading for faster startup
gaussian_model = GaussianModel(sh_degree=3)

# Load core state first (essential for rendering)
await gaussian_model.load_core(f"{checkpoint_dir}/core.pt.gz")

# Load SH rest asynchronously (optional, improves quality)
await gaussian_model.load_sh_rest(f"{checkpoint_dir}/sh_rest.pt.gz")
```

---

### FLAME Parametric Head Model

**Industry-standard 3D morphable face model.**

#### Source Files

```
serwaa/services/avatar-engine/src/avatar_engine/models/flame.py
```

#### Model Specification

| Property      | Value                        |
| ------------- | ---------------------------- |
| **Source**    | Max Planck Institute         |
| **Website**   | https://flame.is.tue.mpg.de/ |
| **Framework** | PyTorch nn.Module            |
| **Device**    | CUDA/CPU                     |
| **License**   | Academic/Commercial          |

#### Model Components

| Component                  | Dimensions      | Description                 |
| -------------------------- | --------------- | --------------------------- |
| **Template Vertices**      | 5,023 × 3       | Mean face mesh              |
| **Face Indices**           | 9,976 × 3       | Triangle indices            |
| **Shape Blendshapes**      | 5,023 × 3 × 100 | Identity variation          |
| **Expression Blendshapes** | 5,023 × 3 × 50  | Facial expressions          |
| **Pose Blendshapes**       | 5,023 × 3 × 36  | Pose-dependent deformations |
| **Joint Regressor**        | 5 × 5,023       | 5-joint kinematic chain     |
| **Skinning Weights**       | 5,023 × 5       | LBS weights                 |
| **Landmark Embedding**     | 68 × 3          | Facial landmark positions   |

#### Input Parameters

| Parameter      | Shape  | Range      | Description             |
| -------------- | ------ | ---------- | ----------------------- |
| **shape**      | (100,) | [-3, 3]    | Identity coefficients   |
| **expression** | (50,)  | [-3, 3]    | Expression coefficients |
| **pose**       | (6,)   | Axis-angle | Head rotation + jaw     |
| **neck_pose**  | (3,)   | Axis-angle | Neck rotation           |
| **eye_pose**   | (6,)   | Axis-angle | Eye gaze                |

#### Required Files

```
/models/flame/
├── generic_model.pkl      # ~200MB - Main FLAME model
├── landmark_embedding.npy # ~10KB - 68-point landmarks
└── FLAME_masks.pkl        # ~5MB - Region masks (optional)
```

#### Loading Pattern

```python
import pickle
import numpy as np

# Load FLAME model
with open("models/flame/generic_model.pkl", "rb") as f:
    flame_model = pickle.load(f, encoding="latin1")

# Load landmarks
landmark_embedding = np.load("models/flame/landmark_embedding.npy")
```

#### Expression Blendshapes (50 coefficients)

| Index | Expression        |
| ----- | ----------------- |
| 0-9   | Eyebrow movements |
| 10-19 | Eye movements     |
| 20-29 | Mouth shapes      |
| 30-39 | Lip movements     |
| 40-49 | Jaw and cheek     |

---

### NeRF Models (Research/Backup)

**Neural Radiance Field implementations for background rendering.**

#### Source Files

```
serwaa/services/avatar-engine/src/avatar_engine/nerf/
├── instant_ngp.py          # Instant-NGP implementation
├── dynamic_instant_ngp.py  # Time-conditioned variant
└── hash_grid.py            # Multi-resolution hash encoding
```

#### Model Specification

| Property      | Value                          |
| ------------- | ------------------------------ |
| **Type**      | Instant-NGP (pure PyTorch)     |
| **Framework** | PyTorch                        |
| **Purpose**   | Background, 3DGS bootstrapping |
| **Device**    | CUDA                           |

#### Hash Grid Configuration

| Parameter             | Default | Description                 |
| --------------------- | ------- | --------------------------- |
| **num_levels**        | 16      | Number of resolution levels |
| **level_dim**         | 2       | Features per level          |
| **log2_hashmap_size** | 19      | ~500K entries per level     |
| **base_resolution**   | 16      | Coarsest grid resolution    |
| **finest_resolution** | 512     | Finest grid resolution      |

---

### Lip Sync Models

**Audio-driven lip synchronization for avatars.**

#### Source Files

```
serwaa/services/avatar-engine/src/avatar_engine/lipsync/
├── models.py       # Wav2Lip-style models
├── audio_encoder.py
└── sync_net.py     # Synchronization discriminator
```

#### Model Architecture

| Component              | Input                   | Output                 | Parameters |
| ---------------------- | ----------------------- | ---------------------- | ---------- |
| **Audio Encoder**      | Mel-spectrogram (80×16) | Audio embedding (512)  | ~5M        |
| **Lip Encoder**        | Face patch (96×96)      | Visual embedding (512) | ~8M        |
| **Lip Decoder**        | Fused embedding         | Lip vertices/mask      | ~10M       |
| **Sync Discriminator** | Audio + Visual          | Sync score             | ~3M        |

#### Audio Processing

| Parameter           | Value       |
| ------------------- | ----------- |
| **Sample Rate**     | 16,000 Hz   |
| **Mel Bins**        | 80          |
| **Window Size**     | 25ms        |
| **Hop Length**      | 10ms        |
| **Frames per Step** | 16 (~160ms) |

---

## Perception/Face Analysis Models

### Face Detection Models

#### MediaPipe Face Detection

**Built-in Google MediaPipe face detector.**

| Property      | Value                             |
| ------------- | --------------------------------- |
| **Framework** | MediaPipe                         |
| **Package**   | `mediapipe ^0.10.0`               |
| **Models**    | Short-range (2m), Full-range (5m) |
| **Output**    | Bounding box + 6 keypoints        |
| **Device**    | CPU/GPU                           |

```python
import mediapipe as mp

face_detection = mp.solutions.face_detection.FaceDetection(
    model_selection=0,  # 0=short-range, 1=full-range
    min_detection_confidence=0.5
)
```

#### MTCNN (Multi-task Cascaded CNN)

**Three-stage face detection cascade.**

| Property      | Value                        |
| ------------- | ---------------------------- |
| **Framework** | facenet_pytorch              |
| **Package**   | `facenet-pytorch ^2.5.0`     |
| **Stages**    | P-Net → R-Net → O-Net        |
| **Output**    | Bounding boxes + 5 keypoints |
| **Device**    | CUDA/CPU                     |

| Stage     | Purpose  | Input Size           |
| --------- | -------- | -------------------- |
| **P-Net** | Proposal | Variable (12×12 min) |
| **R-Net** | Refine   | 24×24                |
| **O-Net** | Output   | 48×48                |

```python
from facenet_pytorch import MTCNN

mtcnn = MTCNN(
    image_size=160,
    margin=0,
    min_face_size=20,
    thresholds=[0.6, 0.7, 0.7],  # P-Net, R-Net, O-Net
    factor=0.709,
    post_process=True,
    device='cuda'
)
```

#### RetinaFace (InsightFace)

**High-accuracy face detection with landmarks.**

| Property      | Value                              |
| ------------- | ---------------------------------- |
| **Framework** | InsightFace                        |
| **Package**   | `insightface ^0.7.3`               |
| **Model**     | retinaface_resnet50                |
| **Output**    | Bounding box + 5 keypoints + score |
| **Device**    | CUDA/CPU (via ONNX Runtime)        |

```python
from insightface.app import FaceAnalysis

app = FaceAnalysis(
    name='buffalo_l',
    providers=['CUDAExecutionProvider', 'CPUExecutionProvider']
)
app.prepare(ctx_id=0, det_size=(640, 640))
```

---

### Facial Landmark Detection

#### face_alignment Library

**2D and 3D facial landmark detection.**

| Property      | Value                   |
| ------------- | ----------------------- |
| **Framework** | PyTorch                 |
| **Package**   | `face-alignment ^1.4.1` |
| **Output**    | 68 facial landmarks     |
| **Variants**  | 2D, 3D                  |
| **Device**    | CUDA/CPU                |

```python
import face_alignment

fa = face_alignment.FaceAlignment(
    face_alignment.LandmarksType.TWO_D,  # or THREE_D
    device='cuda',
    flip_input=False
)
landmarks = fa.get_landmarks(image)  # Returns (68, 2) or (68, 3)
```

#### MediaPipe Face Mesh

**Dense 468-point face mesh.**

| Property      | Value                       |
| ------------- | --------------------------- |
| **Framework** | MediaPipe                   |
| **Output**    | 468 landmarks (→ 68 mapped) |
| **Features**  | 2D + 3D normalized coords   |
| **Device**    | CPU/GPU                     |

| Landmark Region  | Count |
| ---------------- | ----- |
| **Face contour** | 36    |
| **Eyebrows**     | 20    |
| **Eyes**         | 32    |
| **Nose**         | 18    |
| **Lips**         | 40    |
| **Irises**       | 10    |
| **Other**        | 312   |

```python
import mediapipe as mp

face_mesh = mp.solutions.face_mesh.FaceMesh(
    static_image_mode=True,
    max_num_faces=1,
    refine_landmarks=True,  # Enable iris landmarks
    min_detection_confidence=0.5,
    min_tracking_confidence=0.5
)
```

#### dlib Shape Predictor

**Classic 68-point landmark detector.**

| Property       | Value                                   |
| -------------- | --------------------------------------- |
| **Framework**  | dlib                                    |
| **Model File** | `shape_predictor_68_face_landmarks.dat` |
| **Size**       | ~100MB                                  |
| **Output**     | 68 landmarks                            |
| **Device**     | CPU                                     |

```bash
# Download model
wget http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2
bunzip2 shape_predictor_68_face_landmarks.dat.bz2
```

---

### Emotion Detection Models

#### DeepFace

**Multi-backend facial analysis.**

| Property      | Value                       |
| ------------- | --------------------------- |
| **Framework** | TensorFlow                  |
| **Package**   | `deepface ^0.0.79`          |
| **Output**    | 7 emotions + face embedding |
| **Device**    | CPU/GPU                     |

| Emotion  | Description         |
| -------- | ------------------- |
| angry    | Anger expression    |
| disgust  | Disgust expression  |
| fear     | Fear expression     |
| happy    | Happiness/joy       |
| sad      | Sadness expression  |
| surprise | Surprise expression |
| neutral  | Neutral expression  |

```python
from deepface import DeepFace

result = DeepFace.analyze(
    image,
    actions=['emotion', 'age', 'gender', 'race'],
    detector_backend='retinaface',
    enforce_detection=True
)
# result['emotion'] = {'angry': 0.1, 'happy': 0.8, ...}
```

#### HSEmotion

**High-accuracy 8-class emotion classifier.**

| Property      | Value                        |
| ------------- | ---------------------------- |
| **Framework** | PyTorch (EfficientNet-based) |
| **Package**   | `hsemotion ^0.3.0`           |
| **Model**     | enet_b0_8_best_vgaf          |
| **Output**    | 8 emotions                   |
| **Device**    | CUDA/CPU                     |

| Emotion   | Index |
| --------- | ----- |
| Anger     | 0     |
| Contempt  | 1     |
| Disgust   | 2     |
| Fear      | 3     |
| Happiness | 4     |
| Neutral   | 5     |
| Sadness   | 6     |
| Surprise  | 7     |

```python
from hsemotion import HSEmotionRecognizer

recognizer = HSEmotionRecognizer(model_name='enet_b0_8_best_vgaf')
emotion, scores = recognizer.predict_emotions(face_image)
```

#### FER (Facial Expression Recognition)

**Keras-based emotion classifier.**

| Property      | Value            |
| ------------- | ---------------- |
| **Framework** | TensorFlow/Keras |
| **Package**   | `fer ^22.5.0`    |
| **Output**    | 7 emotions       |
| **Device**    | CPU/GPU          |

```python
from fer import FER

detector = FER(mtcnn=True)
emotions = detector.detect_emotions(image)
# [{'box': [x, y, w, h], 'emotions': {...}}]
```

#### Custom PyTorch CNN

**MobileNetV2-based emotion classifier.**

| Property       | Value                    |
| -------------- | ------------------------ |
| **Backbone**   | MobileNetV2 (pretrained) |
| **Input**      | 224×224 RGB              |
| **Output**     | 8 emotion classes        |
| **Parameters** | ~3.5M                    |

```python
import torch
import torchvision.models as models

class EmotionClassifier(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.backbone = models.mobilenet_v2(pretrained=True)
        self.backbone.classifier = torch.nn.Linear(1280, 8)

    def forward(self, x):
        return self.backbone(x)
```

---

## Voice/Audio Models

### Whisper (Speech-to-Text)

**OpenAI's speech recognition model.**

#### Source Files

```
serwaa/services/voice-engine/src/voice_engine/stt/local_whisper.py
```

#### Model Variants

| Model      | Parameters | VRAM  | Speed | WER   |
| ---------- | ---------- | ----- | ----- | ----- |
| `tiny`     | 39M        | ~1GB  | 32x   | ~10%  |
| `base`     | 74M        | ~1GB  | 16x   | ~7%   |
| `small`    | 244M       | ~2GB  | 8x    | ~5%   |
| `medium`   | 769M       | ~5GB  | 4x    | ~4%   |
| `large`    | 1.55B      | ~10GB | 1x    | ~3%   |
| `large-v2` | 1.55B      | ~10GB | 1x    | ~2.5% |

#### Configuration

| Parameter       | Value                    |
| --------------- | ------------------------ |
| **Sample Rate** | 16,000 Hz                |
| **Input**       | 30-second chunks         |
| **Framework**   | HuggingFace Transformers |
| **Model ID**    | `openai/whisper-{size}`  |

```python
from transformers import WhisperProcessor, WhisperForConditionalGeneration

processor = WhisperProcessor.from_pretrained("openai/whisper-medium")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-medium")
model.to("cuda")

# Transcribe
input_features = processor(audio, return_tensors="pt").input_features.to("cuda")
predicted_ids = model.generate(input_features)
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)
```

#### Features

- Multi-language transcription (100+ languages)
- Language detection
- Translation to English
- Word-level timestamps
- Vocabulary priming (prompts)

---

### Speaker Embeddings

#### SpeechBrain ECAPA-TDNN

**State-of-the-art speaker embedding model.**

| Property          | Value                               |
| ----------------- | ----------------------------------- |
| **Framework**     | SpeechBrain                         |
| **Model ID**      | `speechbrain/spkrec-ecapa-voxceleb` |
| **Embedding Dim** | 192                                 |
| **Input**         | 16kHz mono audio                    |
| **Device**        | CUDA/CPU                            |

```python
from speechbrain.pretrained import EncoderClassifier

classifier = EncoderClassifier.from_hparams(
    source="speechbrain/spkrec-ecapa-voxceleb",
    savedir="models/speechbrain_ecapa/"
)
embedding = classifier.encode_batch(waveform)  # (batch, 192)
```

#### Resemblyzer

**Lightweight speaker encoder.**

| Property          | Value                |
| ----------------- | -------------------- |
| **Framework**     | PyTorch              |
| **Package**       | `resemblyzer ^0.1.3` |
| **Embedding Dim** | 256                  |
| **Device**        | CPU optimized        |

```python
from resemblyzer import VoiceEncoder, preprocess_wav

encoder = VoiceEncoder()
wav = preprocess_wav(audio_path)
embedding = encoder.embed_utterance(wav)  # (256,)
```

#### Simple Feature-Based (Fallback)

**Handcrafted audio features.**

| Property          | Value                     |
| ----------------- | ------------------------- |
| **Framework**     | librosa + NumPy           |
| **Features**      | MFCCs + spectral features |
| **Embedding Dim** | 128                       |
| **Device**        | CPU                       |

| Feature            | Dimension           |
| ------------------ | ------------------- |
| MFCCs              | 13 × 2 (mean + std) |
| Spectral centroid  | 1 × 2               |
| Spectral bandwidth | 1 × 2               |
| Spectral rolloff   | 1 × 2               |
| Zero-crossing rate | 1 × 2               |
| Delta MFCCs        | 13 × 2              |
| Delta-delta MFCCs  | 13 × 2              |
| **Total**          | 128                 |

---

### F5-TTS

**Diffusion-based text-to-speech synthesis.**

#### Source Files

```
serwaa/services/voice-engine/src/voice_engine/tts/f5_tts.py
```

#### Model Specification

| Property      | Value                  |
| ------------- | ---------------------- |
| **Type**      | Diffusion-based TTS    |
| **Framework** | PyTorch                |
| **Input**     | Text + reference audio |
| **Output**    | High-quality speech    |
| **Device**    | CUDA                   |

#### Features

- Zero-shot voice cloning
- High fidelity synthesis
- Prosody transfer from reference
- Multi-speaker support

---

## Embedding Models

### Cloud-Based Embeddings

#### Voyage AI

**Primary embedding provider.**

| Model             | Dimensions | Max Tokens | Use Case       |
| ----------------- | ---------- | ---------- | -------------- |
| `voyage-3.5-lite` | 1024       | 16,000     | Default (fast) |
| `voyage-3.5`      | 1024       | 32,000     | Standard       |
| `voyage-3`        | 1024       | 32,000     | High quality   |

```python
import voyageai

client = voyageai.AsyncClient(api_key=os.environ["VOYAGE_API_KEY"])
result = await client.embed(
    texts=["Hello world"],
    model="voyage-3.5-lite",
    input_type="document"  # or "query"
)
embeddings = result.embeddings  # List[List[float]]
```

#### OpenAI Embeddings

**Alternative embedding provider.**

| Model                    | Dimensions | Max Tokens | Cost        |
| ------------------------ | ---------- | ---------- | ----------- |
| `text-embedding-ada-002` | 1536       | 8,191      | $0.0001/1K  |
| `text-embedding-3-small` | 1536       | 8,191      | $0.00002/1K |
| `text-embedding-3-large` | 3072       | 8,191      | $0.00013/1K |

```python
import openai

response = openai.Embedding.create(
    input="Hello world",
    model="text-embedding-3-small"
)
embedding = response.data[0].embedding  # List[float]
```

#### Cohere Embeddings

**Fallback provider.**

| Model                     | Dimensions | Max Tokens |
| ------------------------- | ---------- | ---------- |
| `embed-english-v3.0`      | 1024       | 512        |
| `embed-multilingual-v3.0` | 1024       | 512        |

---

### Local Embedding Models

#### BGE (BAAI General Embedding)

**High-quality open-source embeddings.**

| Model               | Dimensions | Max Tokens | Size   |
| ------------------- | ---------- | ---------- | ------ |
| `bge-m3`            | 1024       | 8,192      | ~2GB   |
| `bge-large-en-v1.5` | 1024       | 512        | ~1.3GB |
| `bge-base-en-v1.5`  | 768        | 512        | ~440MB |
| `bge-small-en-v1.5` | 384        | 512        | ~130MB |

```python
from sentence_transformers import SentenceTransformer

model = SentenceTransformer('BAAI/bge-large-en-v1.5')
embeddings = model.encode(["Hello world"])  # (1, 1024)
```

#### Sentence Transformers

**General-purpose sentence embeddings.**

| Model                                   | Dimensions | Max Tokens |
| --------------------------------------- | ---------- | ---------- |
| `all-MiniLM-L6-v2`                      | 384        | 256        |
| `all-mpnet-base-v2`                     | 768        | 384        |
| `paraphrase-multilingual-MiniLM-L12-v2` | 384        | 128        |

```python
from sentence_transformers import SentenceTransformer

model = SentenceTransformer('sentence-transformers/all-mpnet-base-v2')
embeddings = model.encode(texts, convert_to_tensor=True)
```

---

## NLP/Intent Detection Models

### Intent Classification

**Zero-shot classification for intent detection.**

#### Source Files

```
serwaa/services/voice-engine/src/voice_engine/transcript/intent.py
```

#### BART-MNLI Model

| Property       | Value                      |
| -------------- | -------------------------- |
| **Model ID**   | `facebook/bart-large-mnli` |
| **Framework**  | HuggingFace Transformers   |
| **Parameters** | 407M                       |
| **Task**       | Zero-shot classification   |
| **Device**     | CUDA/CPU                   |

```python
from transformers import pipeline

classifier = pipeline(
    "zero-shot-classification",
    model="facebook/bart-large-mnli",
    device=0  # CUDA device
)

result = classifier(
    "What's the weather like?",
    candidate_labels=["question", "statement", "command", "request"]
)
# {'labels': ['question', ...], 'scores': [0.95, ...]}
```

#### Intent Categories

| Intent      | Example                   |
| ----------- | ------------------------- |
| question    | "What is your name?"      |
| statement   | "The sky is blue."        |
| command     | "Turn on the lights."     |
| request     | "Could you help me?"      |
| greeting    | "Hello, how are you?"     |
| farewell    | "Goodbye, see you later." |
| affirmation | "Yes, that's correct."    |
| negation    | "No, I don't think so."   |

---

### Sentiment Analysis

**Sentiment classification for text.**

#### Source Files

```
serwaa/services/voice-engine/src/voice_engine/transcript/sentiment.py
```

#### Model Options

| Model                                              | Framework   | Output            |
| -------------------------------------------------- | ----------- | ----------------- |
| `distilbert-base-uncased-finetuned-sst-2-english`  | HuggingFace | Positive/Negative |
| `cardiffnlp/twitter-roberta-base-sentiment`        | HuggingFace | Pos/Neg/Neutral   |
| `nlptown/bert-base-multilingual-uncased-sentiment` | HuggingFace | 1-5 stars         |

```python
from transformers import pipeline

sentiment = pipeline("sentiment-analysis")
result = sentiment("I love this product!")
# [{'label': 'POSITIVE', 'score': 0.99}]
```

---

## Core ML Dependencies

### Deep Learning Frameworks

```toml
# pyproject.toml dependencies
torch = "^2.2.0"           # Core deep learning
torchaudio = "^2.2.0"      # Audio processing
torchvision = "^0.17.0"    # Image processing
```

### ML Utilities

```toml
transformers = "^4.37.0"   # HuggingFace models
accelerate = "^0.26.0"     # Training acceleration
safetensors = "^0.4.0"     # Safe model serialization
huggingface-hub = "^0.20.0" # Model downloads
numpy = "^1.26.0"          # Numerical computing
scipy = "^1.12.0"          # Scientific computing
scikit-learn = "^1.4.0"    # ML utilities
```

### Computer Vision

```toml
opencv-python = "^4.9.0"   # Image processing
mediapipe = "^0.10.0"      # Face/pose detection
pillow = "^10.2.0"         # Image I/O
```

### Avatar Engine Specific

```toml
pytorch3d = "^0.7.5"       # 3D deep learning
trimesh = "^4.0.0"         # 3D mesh processing
open3d = "^0.18.0"         # Point cloud processing
pyrender = "^0.1.45"       # 3D rendering
chumpy = "^0.70"           # Auto-diff for FLAME
smplx = "^0.1.28"          # Body models
face-alignment = "^1.4.1"  # Facial landmarks
insightface = "^0.7.3"     # Face analysis
scikit-image = "^0.22.0"   # Image processing
imageio = "^2.33.0"        # Image I/O
imageio-ffmpeg = "^0.4.9"  # Video I/O
h5py = "^3.10.0"           # HDF5 files
cupy-cuda12x = "^13.0.0"   # GPU arrays (optional)
```

### Voice Engine Specific

```toml
librosa = "^0.10.0"        # Audio analysis
soundfile = "^0.12.0"      # Audio I/O
speechbrain = "^0.5.0"     # Speech processing
resemblyzer = "^0.1.3"     # Speaker verification
```

---

## Model Loading Patterns

### HuggingFace Models

```python
from transformers import AutoModel, AutoTokenizer

# Download and cache
model = AutoModel.from_pretrained("model_name")
tokenizer = AutoTokenizer.from_pretrained("model_name")

# Load from local path
model = AutoModel.from_pretrained("/path/to/model")
```

### PyTorch Checkpoints

```python
import torch

# Load state dict
state_dict = torch.load("model.pt", map_location="cuda")
model.load_state_dict(state_dict)

# Load with compression
import gzip
with gzip.open("model.pt.gz", "rb") as f:
    state_dict = torch.load(f, map_location="cuda")
```

### Pickle Models

```python
import pickle

# Load FLAME model
with open("generic_model.pkl", "rb") as f:
    model_data = pickle.load(f, encoding="latin1")

# Access components
vertices = model_data["v_template"]
faces = model_data["f"]
```

### ONNX Models

```python
import onnxruntime as ort

# Create session with GPU
providers = ['CUDAExecutionProvider', 'CPUExecutionProvider']
session = ort.InferenceSession("model.onnx", providers=providers)

# Run inference
outputs = session.run(None, {"input": input_array})
```

### Lazy Loading Pattern

```python
class LazyModel:
    def __init__(self, model_path: str):
        self._model_path = model_path
        self._model = None

    @property
    def model(self):
        if self._model is None:
            self._model = self._load_model()
        return self._model

    def _load_model(self):
        return torch.load(self._model_path)
```

---

## GPU & Hardware Requirements

### Minimum Requirements

| Component    | Minimum                | Recommended        |
| ------------ | ---------------------- | ------------------ |
| **GPU**      | NVIDIA RTX 3080 (10GB) | NVIDIA A100 (40GB) |
| **CUDA**     | 11.8                   | 12.1+              |
| **cuDNN**    | 8.6                    | 8.9+               |
| **TensorRT** | 8.5                    | 8.6+               |
| **RAM**      | 32GB                   | 64GB+              |
| **Storage**  | 100GB SSD              | 500GB NVMe         |

### GPU Memory Breakdown

| Model Category               | VRAM Usage |
| ---------------------------- | ---------- |
| **3D Gaussian Splatting**    | 2-6 GB     |
| **FLAME Model**              | 0.5 GB     |
| **Whisper Medium**           | 5 GB       |
| **Whisper Large**            | 10 GB      |
| **Face Detection (MTCNN)**   | 0.5 GB     |
| **Emotion Detection**        | 1 GB       |
| **Embedding Models (local)** | 1-2 GB     |
| **Total (typical)**          | 10-15 GB   |

### Multi-GPU Configuration

```bash
# Environment variables
CUDA_VISIBLE_DEVICES=0,1
AVATAR_GPU_ID=0
VOICE_ENGINE_GPU_ID=1
PERCEPTION_GPU_ID=0
```

### Performance Targets

| Task                 | Latency Target | Throughput |
| -------------------- | -------------- | ---------- |
| Avatar rendering     | < 33ms         | 30 FPS     |
| Face detection       | < 20ms         | 50 FPS     |
| Emotion detection    | < 50ms         | 20 FPS     |
| Speech recognition   | Real-time      | 1x+        |
| TTS synthesis        | < 100ms        | -          |
| Embedding generation | < 50ms         | -          |

---

## Model Download Sources

### HuggingFace Hub

| Model Type            | Source                     |
| --------------------- | -------------------------- |
| Whisper               | `openai/whisper-*`         |
| Transformers          | `facebook/bart-large-mnli` |
| BGE Embeddings        | `BAAI/bge-*`               |
| Sentence Transformers | `sentence-transformers/*`  |

```bash
# Pre-download models
huggingface-cli download openai/whisper-medium
huggingface-cli download BAAI/bge-large-en-v1.5
```

### Official Repositories

| Model          | Source URL                   |
| -------------- | ---------------------------- |
| FLAME          | https://flame.is.tue.mpg.de/ |
| dlib landmarks | http://dlib.net/files/       |
| MediaPipe      | Built-in (via package)       |
| InsightFace    | Built-in (via package)       |

### Cloud APIs (Runtime)

| Provider  | Models                       |
| --------- | ---------------------------- |
| OpenAI    | text-embedding-\*, whisper-1 |
| Voyage AI | voyage-3.5-\*                |
| Cohere    | embed-\*                     |

### Custom Trained Models

| Model                 | Location                     |
| --------------------- | ---------------------------- |
| 3D Gaussian Splatting | `models/avatar/checkpoints/` |
| Lip Sync              | `models/lipsync/`            |
| Voice Profiles        | `models/voice/profiles/`     |

---

## Environment Variables

### GPU Configuration

```bash
# Device selection
AVATAR_DEVICE=cuda
AVATAR_GPU_ID=0
PERCEPTION_DEVICE=cuda
VOICE_ENGINE_DEVICE=cuda

# Memory optimization
AVATAR_ENABLE_MIXED_PRECISION=true
PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:512
```

### Model Paths

```bash
# Avatar models
AVATAR_MODEL_PATH=/models/avatar
AVATAR_CHECKPOINT_PATH=/models/avatar/checkpoints
FLAME_MODEL_PATH=/models/flame/generic_model.pkl
FLAME_LANDMARK_PATH=/models/flame/landmark_embedding.npy

# Voice models
WHISPER_MODEL_PATH=/models/whisper
SPEAKER_EMBEDDING_MODEL_PATH=/models/speechbrain_ecapa
VOICE_PROFILE_PATH=/models/voice/profiles

# Perception models
FACE_DETECTION_MODEL=mediapipe
EMOTION_MODEL_PATH=/models/emotion
DLIB_LANDMARK_PATH=/models/dlib/shape_predictor_68_face_landmarks.dat
```

### Model Selection

```bash
# Voice engine
VOICE_ENGINE_STT_MODEL=openai/whisper-medium
VOICE_ENGINE_EMBEDDING_MODEL=speechbrain

# Perception engine
PERCEPTION_FACE_DETECTOR=mediapipe  # mediapipe, mtcnn, retinaface
PERCEPTION_EMOTION_MODEL=deepface   # deepface, hsemotion, fer

# Knowledge base
KNOWLEDGE_BASE_EMBEDDING_MODEL=voyage-3.5-lite
KNOWLEDGE_BASE_EMBEDDING_PROVIDER=voyage  # voyage, openai, cohere, local
```

### API Keys (for cloud models)

```bash
# Embedding providers
VOYAGE_API_KEY=pa-...
OPENAI_API_KEY=sk-...
COHERE_API_KEY=...

# LLM providers (for model-based features)
ANTHROPIC_API_KEY=sk-ant-...
```

---

## Migration Considerations

### High Priority Models

These models are critical for core functionality:

| Model                 | Priority | Complexity | Notes                         |
| --------------------- | -------- | ---------- | ----------------------------- |
| 3D Gaussian Splatting | P0       | High       | Custom PyTorch, GPU-intensive |
| FLAME                 | P0       | Medium     | Pickle format, needs license  |
| Whisper               | P0       | Low        | HuggingFace, well-supported   |
| Speaker Embeddings    | P0       | Medium     | Multiple options              |
| Face Detection        | P1       | Low        | Multiple backends             |

### TypeScript Considerations

For Psyche TypeScript migration:

| Python Library | TypeScript Option                     |
| -------------- | ------------------------------------- |
| PyTorch        | Run Python subprocess, or use ONNX.js |
| Transformers   | HuggingFace.js (limited)              |
| MediaPipe      | @mediapipe/face_detection             |
| OpenCV         | opencv4nodejs                         |
| SpeechBrain    | Keep Python, call via API             |

### Recommended Architecture

```
┌──────────────────────────────────────────────────────────────────────┐
│                      Psyche ML Service Layer                         │
├──────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐      │
│  │  TypeScript API │  │  TypeScript API │  │  TypeScript API │      │
│  │   (HTTP/gRPC)   │  │   (HTTP/gRPC)   │  │   (HTTP/gRPC)   │      │
│  └────────┬────────┘  └────────┬────────┘  └────────┬────────┘      │
│           │                    │                    │                │
│  ┌────────▼────────┐  ┌────────▼────────┐  ┌────────▼────────┐      │
│  │  Python Worker  │  │  Python Worker  │  │  Python Worker  │      │
│  │  (Avatar/3DGS)  │  │  (Voice/Audio)  │  │  (Perception)   │      │
│  └─────────────────┘  └─────────────────┘  └─────────────────┘      │
│                                                                      │
└──────────────────────────────────────────────────────────────────────┘
```

### Migration Phases

1. **Phase 1**: Keep Python ML workers, add TypeScript API layer
2. **Phase 2**: Migrate simple models to ONNX.js where possible
3. **Phase 3**: Optimize GPU workers with TensorRT
4. **Phase 4**: Consider cloud ML services for scale

### Model Versioning

```
models/
├── avatar/
│   ├── v1.0/
│   │   ├── core.pt.gz
│   │   └── sh_rest.pt.gz
│   └── v1.1/
│       ├── core.pt.gz
│       └── sh_rest.pt.gz
├── flame/
│   └── v2020/
│       ├── generic_model.pkl
│       └── landmark_embedding.npy
└── voice/
    ├── whisper-medium/
    └── speechbrain-ecapa/
```

---

## Appendix: Model File Inventory

### Required Model Files

| File                                         | Size   | Source         | Required |
| -------------------------------------------- | ------ | -------------- | -------- |
| `flame/generic_model.pkl`                    | ~200MB | FLAME website  | Yes      |
| `flame/landmark_embedding.npy`               | ~10KB  | FLAME website  | Yes      |
| `dlib/shape_predictor_68_face_landmarks.dat` | ~100MB | dlib.net       | Optional |
| `avatar/checkpoints/core.pt.gz`              | ~200MB | Custom trained | Yes      |
| `avatar/checkpoints/sh_rest.pt.gz`           | ~400MB | Custom trained | Optional |
| `voice/speechbrain_ecapa/*`                  | ~100MB | HuggingFace    | Yes      |
| `whisper-medium/*`                           | ~1.5GB | HuggingFace    | Yes      |

### Auto-Downloaded Models

| Model                    | Package               | Auto-Download |
| ------------------------ | --------------------- | ------------- |
| MediaPipe Face Detection | mediapipe             | Yes           |
| MediaPipe Face Mesh      | mediapipe             | Yes           |
| MTCNN                    | facenet-pytorch       | Yes           |
| RetinaFace               | insightface           | Yes           |
| DeepFace backends        | deepface              | Yes           |
| HSEmotion                | hsemotion             | Yes           |
| FER                      | fer                   | Yes           |
| Sentence Transformers    | sentence-transformers | Yes           |

### Docker Build Context

```dockerfile
# Voice Engine Dockerfile
FROM nvidia/cuda:12.1.1-cudnn8-devel-ubuntu22.04

# System dependencies
RUN apt-get update && apt-get install -y \
    libsndfile1 libportaudio2 ffmpeg python3.11 python3-pip

# Install PyTorch with CUDA
RUN pip install torch==2.2.0 torchaudio==2.2.0 --index-url https://download.pytorch.org/whl/cu121

# Model directories
RUN mkdir -p /app/models/{tts,stt,embeddings}
```

```dockerfile
# Perception Engine Dockerfile
FROM nvidia/cuda:12.1.1-cudnn8-devel-ubuntu22.04

# System dependencies
RUN apt-get update && apt-get install -y \
    libgl1-mesa-glx libtesseract-dev cmake ninja-build python3.11 python3-pip

# Install PyTorch with CUDA
RUN pip install torch==2.2.0 torchvision==0.17.0 --index-url https://download.pytorch.org/whl/cu121

# Model directory
RUN mkdir -p /app/models
```
