Eve library

Psyche Domain: Machine Learning Model Dependencies

Guide

Revision 1

On this page

This document provides a comprehensive inventory of all machine learning models used by the Serwaa platform (source codebase for the Psyche domain migration).

Table of Contents#

  1. Overview
  2. Avatar/3D Models
  3. Perception/Face Analysis Models
  4. Voice/Audio Models
  5. Embedding Models
  6. NLP/Intent Detection Models
  7. Core ML Dependencies
  8. Model Loading Patterns
  9. GPU & Hardware Requirements
  10. Model Download Sources
  11. Environment Variables
  12. Migration Considerations

Overview#

The Serwaa platform uses 50+ machine learning models across 6 major categories:

Category Models Framework Deployment
Avatar/3D 3DGS, FLAME, NeRF, Lip Sync PyTorch Local GPU
Face Analysis MediaPipe, InsightFace, MTCNN TensorFlow, PyTorch Local GPU/CPU
Voice/Audio Whisper, ECAPA-TDNN, F5-TTS PyTorch, SpeechBrain Local GPU
Embeddings Voyage AI, OpenAI, BGE API, HuggingFace API/Local
NLP BART-MNLI, Sentiment HuggingFace Local CPU
Emotion DeepFace, HSEmotion, FER TensorFlow, PyTorch Local GPU/CPU

Model Architecture Overview#

text
┌─────────────────────────────────────────────────────────────────────────────┐
│                          ML Model Pipeline                                   │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  ┌──────────────────┐    ┌──────────────────┐    ┌──────────────────┐       │
│  │  Input Pipeline  │───▶│  Feature Models  │───▶│  Output Models   │       │
│  └──────────────────┘    └──────────────────┘    └──────────────────┘       │
│         ↓                         ↓                       ↓                  │
│  ┌──────────────────┐    ┌──────────────────┐    ┌──────────────────┐       │
│  │ Audio Processing │    │ Voice Embeddings │    │ TTS Synthesis    │       │
│  │ Face Detection   │    │ Face Embeddings  │    │ Avatar Rendering │       │
│  │ Text Processing  │    │ Text Embeddings  │    │ Lip Sync         │       │
│  └──────────────────┘    └──────────────────┘    └──────────────────┘       │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

Avatar/3D Models#

3D Gaussian Splatting (3DGS)#

Real-time avatar rendering using 3D Gaussian primitives.

Source Files#

text
serwaa/services/avatar-engine/src/avatar_engine/models/gaussian.py

Model Specification#

Property Value
Type Custom PyTorch nn.Module
Framework PyTorch 2.x
Device CUDA (GPU required)
Precision FP16/FP32

Parameters#

Component Description Size
Positions 3D Gaussian centers N × 3
Rotations Quaternion orientations N × 4
Scales Anisotropic scales N × 3
Opacities Alpha values N × 1
SH DC 0th-order spherical harmonics N × 3
SH Rest Higher-order SH (degree 1-3) N × 45

Where N = number of Gaussians (typically 50,000 - 100,000)

Spherical Harmonics Configuration#

SH Degree Coefficients per Gaussian Use Case
0 3 (RGB only) Basic color
1 12 Directional lighting
2 27 Specular highlights
3 48 Full appearance (default)

Memory Usage#

Resolution Gaussians VRAM Target FPS
512×512 50K 1.5 GB 200+
768×768 75K 2.5 GB 120+
1024×1024 100K 4.0 GB 60+
1920×1080 100K 6.0 GB 30+

File Format#

python
# Checkpoint structure (.pt.gz)
{
    "metadata": {
        "num_gaussians": int,
        "sh_degree": int,
        "version": str,
        "creation_time": str
    },
    "core_state": {
        "positions": torch.Tensor,      # (N, 3) float16
        "rotations": torch.Tensor,      # (N, 4) float16
        "scales": torch.Tensor,         # (N, 3) float16
        "opacities": torch.Tensor,      # (N, 1) float16
        "sh_dc": torch.Tensor           # (N, 3) float16
    },
    "sh_rest": torch.Tensor             # (N, 45) float16, separate file
}

Loading Pattern#

python
# Progressive loading for faster startup
gaussian_model = GaussianModel(sh_degree=3)

# Load core state first (essential for rendering)
await gaussian_model.load_core(f"{checkpoint_dir}/core.pt.gz")

# Load SH rest asynchronously (optional, improves quality)
await gaussian_model.load_sh_rest(f"{checkpoint_dir}/sh_rest.pt.gz")

FLAME Parametric Head Model#

Industry-standard 3D morphable face model.

Source Files#

text
serwaa/services/avatar-engine/src/avatar_engine/models/flame.py

Model Specification#

Property Value
Source Max Planck Institute
Website https://flame.is.tue.mpg.de/
Framework PyTorch nn.Module
Device CUDA/CPU
License Academic/Commercial

Model Components#

Component Dimensions Description
Template Vertices 5,023 × 3 Mean face mesh
Face Indices 9,976 × 3 Triangle indices
Shape Blendshapes 5,023 × 3 × 100 Identity variation
Expression Blendshapes 5,023 × 3 × 50 Facial expressions
Pose Blendshapes 5,023 × 3 × 36 Pose-dependent deformations
Joint Regressor 5 × 5,023 5-joint kinematic chain
Skinning Weights 5,023 × 5 LBS weights
Landmark Embedding 68 × 3 Facial landmark positions

Input Parameters#

Parameter Shape Range Description
shape (100,) [-3, 3] Identity coefficients
expression (50,) [-3, 3] Expression coefficients
pose (6,) Axis-angle Head rotation + jaw
neck_pose (3,) Axis-angle Neck rotation
eye_pose (6,) Axis-angle Eye gaze

Required Files#

text
/models/flame/
├── generic_model.pkl      # ~200MB - Main FLAME model
├── landmark_embedding.npy # ~10KB - 68-point landmarks
└── FLAME_masks.pkl        # ~5MB - Region masks (optional)

Loading Pattern#

python
import pickle
import numpy as np

# Load FLAME model
with open("models/flame/generic_model.pkl", "rb") as f:
    flame_model = pickle.load(f, encoding="latin1")

# Load landmarks
landmark_embedding = np.load("models/flame/landmark_embedding.npy")

Expression Blendshapes (50 coefficients)#

Index Expression
0-9 Eyebrow movements
10-19 Eye movements
20-29 Mouth shapes
30-39 Lip movements
40-49 Jaw and cheek

NeRF Models (Research/Backup)#

Neural Radiance Field implementations for background rendering.

Source Files#

text
serwaa/services/avatar-engine/src/avatar_engine/nerf/
├── instant_ngp.py          # Instant-NGP implementation
├── dynamic_instant_ngp.py  # Time-conditioned variant
└── hash_grid.py            # Multi-resolution hash encoding

Model Specification#

Property Value
Type Instant-NGP (pure PyTorch)
Framework PyTorch
Purpose Background, 3DGS bootstrapping
Device CUDA

Hash Grid Configuration#

Parameter Default Description
num_levels 16 Number of resolution levels
level_dim 2 Features per level
log2_hashmap_size 19 ~500K entries per level
base_resolution 16 Coarsest grid resolution
finest_resolution 512 Finest grid resolution

Lip Sync Models#

Audio-driven lip synchronization for avatars.

Source Files#

text
serwaa/services/avatar-engine/src/avatar_engine/lipsync/
├── models.py       # Wav2Lip-style models
├── audio_encoder.py
└── sync_net.py     # Synchronization discriminator

Model Architecture#

Component Input Output Parameters
Audio Encoder Mel-spectrogram (80×16) Audio embedding (512) ~5M
Lip Encoder Face patch (96×96) Visual embedding (512) ~8M
Lip Decoder Fused embedding Lip vertices/mask ~10M
Sync Discriminator Audio + Visual Sync score ~3M

Audio Processing#

Parameter Value
Sample Rate 16,000 Hz
Mel Bins 80
Window Size 25ms
Hop Length 10ms
Frames per Step 16 (~160ms)

Perception/Face Analysis Models#

Face Detection Models#

MediaPipe Face Detection#

Built-in Google MediaPipe face detector.

Property Value
Framework MediaPipe
Package mediapipe ^0.10.0
Models Short-range (2m), Full-range (5m)
Output Bounding box + 6 keypoints
Device CPU/GPU
python
import mediapipe as mp

face_detection = mp.solutions.face_detection.FaceDetection(
    model_selection=0,  # 0=short-range, 1=full-range
    min_detection_confidence=0.5
)

MTCNN (Multi-task Cascaded CNN)#

Three-stage face detection cascade.

Property Value
Framework facenet_pytorch
Package facenet-pytorch ^2.5.0
Stages P-Net → R-Net → O-Net
Output Bounding boxes + 5 keypoints
Device CUDA/CPU
Stage Purpose Input Size
P-Net Proposal Variable (12×12 min)
R-Net Refine 24×24
O-Net Output 48×48
python
from facenet_pytorch import MTCNN

mtcnn = MTCNN(
    image_size=160,
    margin=0,
    min_face_size=20,
    thresholds=[0.6, 0.7, 0.7],  # P-Net, R-Net, O-Net
    factor=0.709,
    post_process=True,
    device='cuda'
)

RetinaFace (InsightFace)#

High-accuracy face detection with landmarks.

Property Value
Framework InsightFace
Package insightface ^0.7.3
Model retinaface_resnet50
Output Bounding box + 5 keypoints + score
Device CUDA/CPU (via ONNX Runtime)
python
from insightface.app import FaceAnalysis

app = FaceAnalysis(
    name='buffalo_l',
    providers=['CUDAExecutionProvider', 'CPUExecutionProvider']
)
app.prepare(ctx_id=0, det_size=(640, 640))

Facial Landmark Detection#

face_alignment Library#

2D and 3D facial landmark detection.

Property Value
Framework PyTorch
Package face-alignment ^1.4.1
Output 68 facial landmarks
Variants 2D, 3D
Device CUDA/CPU
python
import face_alignment

fa = face_alignment.FaceAlignment(
    face_alignment.LandmarksType.TWO_D,  # or THREE_D
    device='cuda',
    flip_input=False
)
landmarks = fa.get_landmarks(image)  # Returns (68, 2) or (68, 3)

MediaPipe Face Mesh#

Dense 468-point face mesh.

Property Value
Framework MediaPipe
Output 468 landmarks (→ 68 mapped)
Features 2D + 3D normalized coords
Device CPU/GPU
Landmark Region Count
Face contour 36
Eyebrows 20
Eyes 32
Nose 18
Lips 40
Irises 10
Other 312
python
import mediapipe as mp

face_mesh = mp.solutions.face_mesh.FaceMesh(
    static_image_mode=True,
    max_num_faces=1,
    refine_landmarks=True,  # Enable iris landmarks
    min_detection_confidence=0.5,
    min_tracking_confidence=0.5
)

dlib Shape Predictor#

Classic 68-point landmark detector.

Property Value
Framework dlib
Model File shape_predictor_68_face_landmarks.dat
Size ~100MB
Output 68 landmarks
Device CPU
bash
# Download model
wget http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2
bunzip2 shape_predictor_68_face_landmarks.dat.bz2

Emotion Detection Models#

DeepFace#

Multi-backend facial analysis.

Property Value
Framework TensorFlow
Package deepface ^0.0.79
Output 7 emotions + face embedding
Device CPU/GPU
Emotion Description
angry Anger expression
disgust Disgust expression
fear Fear expression
happy Happiness/joy
sad Sadness expression
surprise Surprise expression
neutral Neutral expression
python
from deepface import DeepFace

result = DeepFace.analyze(
    image,
    actions=['emotion', 'age', 'gender', 'race'],
    detector_backend='retinaface',
    enforce_detection=True
)
# result['emotion'] = {'angry': 0.1, 'happy': 0.8, ...}

HSEmotion#

High-accuracy 8-class emotion classifier.

Property Value
Framework PyTorch (EfficientNet-based)
Package hsemotion ^0.3.0
Model enet_b0_8_best_vgaf
Output 8 emotions
Device CUDA/CPU
Emotion Index
Anger 0
Contempt 1
Disgust 2
Fear 3
Happiness 4
Neutral 5
Sadness 6
Surprise 7
python
from hsemotion import HSEmotionRecognizer

recognizer = HSEmotionRecognizer(model_name='enet_b0_8_best_vgaf')
emotion, scores = recognizer.predict_emotions(face_image)

FER (Facial Expression Recognition)#

Keras-based emotion classifier.

Property Value
Framework TensorFlow/Keras
Package fer ^22.5.0
Output 7 emotions
Device CPU/GPU
python
from fer import FER

detector = FER(mtcnn=True)
emotions = detector.detect_emotions(image)
# [{'box': [x, y, w, h], 'emotions': {...}}]

Custom PyTorch CNN#

MobileNetV2-based emotion classifier.

Property Value
Backbone MobileNetV2 (pretrained)
Input 224×224 RGB
Output 8 emotion classes
Parameters ~3.5M
python
import torch
import torchvision.models as models

class EmotionClassifier(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.backbone = models.mobilenet_v2(pretrained=True)
        self.backbone.classifier = torch.nn.Linear(1280, 8)

    def forward(self, x):
        return self.backbone(x)

Voice/Audio Models#

Whisper (Speech-to-Text)#

OpenAI's speech recognition model.

Source Files#

text
serwaa/services/voice-engine/src/voice_engine/stt/local_whisper.py

Model Variants#

Model Parameters VRAM Speed WER
tiny 39M ~1GB 32x ~10%
base 74M ~1GB 16x ~7%
small 244M ~2GB 8x ~5%
medium 769M ~5GB 4x ~4%
large 1.55B ~10GB 1x ~3%
large-v2 1.55B ~10GB 1x ~2.5%

Configuration#

Parameter Value
Sample Rate 16,000 Hz
Input 30-second chunks
Framework HuggingFace Transformers
Model ID openai/whisper-{size}
python
from transformers import WhisperProcessor, WhisperForConditionalGeneration

processor = WhisperProcessor.from_pretrained("openai/whisper-medium")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-medium")
model.to("cuda")

# Transcribe
input_features = processor(audio, return_tensors="pt").input_features.to("cuda")
predicted_ids = model.generate(input_features)
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)

Features#

  • Multi-language transcription (100+ languages)
  • Language detection
  • Translation to English
  • Word-level timestamps
  • Vocabulary priming (prompts)

Speaker Embeddings#

SpeechBrain ECAPA-TDNN#

State-of-the-art speaker embedding model.

Property Value
Framework SpeechBrain
Model ID speechbrain/spkrec-ecapa-voxceleb
Embedding Dim 192
Input 16kHz mono audio
Device CUDA/CPU
python
from speechbrain.pretrained import EncoderClassifier

classifier = EncoderClassifier.from_hparams(
    source="speechbrain/spkrec-ecapa-voxceleb",
    savedir="models/speechbrain_ecapa/"
)
embedding = classifier.encode_batch(waveform)  # (batch, 192)

Resemblyzer#

Lightweight speaker encoder.

Property Value
Framework PyTorch
Package resemblyzer ^0.1.3
Embedding Dim 256
Device CPU optimized
python
from resemblyzer import VoiceEncoder, preprocess_wav

encoder = VoiceEncoder()
wav = preprocess_wav(audio_path)
embedding = encoder.embed_utterance(wav)  # (256,)

Simple Feature-Based (Fallback)#

Handcrafted audio features.

Property Value
Framework librosa + NumPy
Features MFCCs + spectral features
Embedding Dim 128
Device CPU
Feature Dimension
MFCCs 13 × 2 (mean + std)
Spectral centroid 1 × 2
Spectral bandwidth 1 × 2
Spectral rolloff 1 × 2
Zero-crossing rate 1 × 2
Delta MFCCs 13 × 2
Delta-delta MFCCs 13 × 2
Total 128

F5-TTS#

Diffusion-based text-to-speech synthesis.

Source Files#

text
serwaa/services/voice-engine/src/voice_engine/tts/f5_tts.py

Model Specification#

Property Value
Type Diffusion-based TTS
Framework PyTorch
Input Text + reference audio
Output High-quality speech
Device CUDA

Features#

  • Zero-shot voice cloning
  • High fidelity synthesis
  • Prosody transfer from reference
  • Multi-speaker support

Embedding Models#

Cloud-Based Embeddings#

Voyage AI#

Primary embedding provider.

Model Dimensions Max Tokens Use Case
voyage-3.5-lite 1024 16,000 Default (fast)
voyage-3.5 1024 32,000 Standard
voyage-3 1024 32,000 High quality
python
import voyageai

client = voyageai.AsyncClient(api_key=os.environ["VOYAGE_API_KEY"])
result = await client.embed(
    texts=["Hello world"],
    model="voyage-3.5-lite",
    input_type="document"  # or "query"
)
embeddings = result.embeddings  # List[List[float]]

OpenAI Embeddings#

Alternative embedding provider.

Model Dimensions Max Tokens Cost
text-embedding-ada-002 1536 8,191 $0.0001/1K
text-embedding-3-small 1536 8,191 $0.00002/1K
text-embedding-3-large 3072 8,191 $0.00013/1K
python
import openai

response = openai.Embedding.create(
    input="Hello world",
    model="text-embedding-3-small"
)
embedding = response.data[0].embedding  # List[float]

Cohere Embeddings#

Fallback provider.

Model Dimensions Max Tokens
embed-english-v3.0 1024 512
embed-multilingual-v3.0 1024 512

Local Embedding Models#

BGE (BAAI General Embedding)#

High-quality open-source embeddings.

Model Dimensions Max Tokens Size
bge-m3 1024 8,192 ~2GB
bge-large-en-v1.5 1024 512 ~1.3GB
bge-base-en-v1.5 768 512 ~440MB
bge-small-en-v1.5 384 512 ~130MB
python
from sentence_transformers import SentenceTransformer

model = SentenceTransformer('BAAI/bge-large-en-v1.5')
embeddings = model.encode(["Hello world"])  # (1, 1024)

Sentence Transformers#

General-purpose sentence embeddings.

Model Dimensions Max Tokens
all-MiniLM-L6-v2 384 256
all-mpnet-base-v2 768 384
paraphrase-multilingual-MiniLM-L12-v2 384 128
python
from sentence_transformers import SentenceTransformer

model = SentenceTransformer('sentence-transformers/all-mpnet-base-v2')
embeddings = model.encode(texts, convert_to_tensor=True)

NLP/Intent Detection Models#

Intent Classification#

Zero-shot classification for intent detection.

Source Files#

text
serwaa/services/voice-engine/src/voice_engine/transcript/intent.py

BART-MNLI Model#

Property Value
Model ID facebook/bart-large-mnli
Framework HuggingFace Transformers
Parameters 407M
Task Zero-shot classification
Device CUDA/CPU
python
from transformers import pipeline

classifier = pipeline(
    "zero-shot-classification",
    model="facebook/bart-large-mnli",
    device=0  # CUDA device
)

result = classifier(
    "What's the weather like?",
    candidate_labels=["question", "statement", "command", "request"]
)
# {'labels': ['question', ...], 'scores': [0.95, ...]}

Intent Categories#

Intent Example
question "What is your name?"
statement "The sky is blue."
command "Turn on the lights."
request "Could you help me?"
greeting "Hello, how are you?"
farewell "Goodbye, see you later."
affirmation "Yes, that's correct."
negation "No, I don't think so."

Sentiment Analysis#

Sentiment classification for text.

Source Files#

text
serwaa/services/voice-engine/src/voice_engine/transcript/sentiment.py

Model Options#

Model Framework Output
distilbert-base-uncased-finetuned-sst-2-english HuggingFace Positive/Negative
cardiffnlp/twitter-roberta-base-sentiment HuggingFace Pos/Neg/Neutral
nlptown/bert-base-multilingual-uncased-sentiment HuggingFace 1-5 stars
python
from transformers import pipeline

sentiment = pipeline("sentiment-analysis")
result = sentiment("I love this product!")
# [{'label': 'POSITIVE', 'score': 0.99}]

Core ML Dependencies#

Deep Learning Frameworks#

toml
# pyproject.toml dependencies
torch = "^2.2.0"           # Core deep learning
torchaudio = "^2.2.0"      # Audio processing
torchvision = "^0.17.0"    # Image processing

ML Utilities#

toml
transformers = "^4.37.0"   # HuggingFace models
accelerate = "^0.26.0"     # Training acceleration
safetensors = "^0.4.0"     # Safe model serialization
huggingface-hub = "^0.20.0" # Model downloads
numpy = "^1.26.0"          # Numerical computing
scipy = "^1.12.0"          # Scientific computing
scikit-learn = "^1.4.0"    # ML utilities

Computer Vision#

toml
opencv-python = "^4.9.0"   # Image processing
mediapipe = "^0.10.0"      # Face/pose detection
pillow = "^10.2.0"         # Image I/O

Avatar Engine Specific#

toml
pytorch3d = "^0.7.5"       # 3D deep learning
trimesh = "^4.0.0"         # 3D mesh processing
open3d = "^0.18.0"         # Point cloud processing
pyrender = "^0.1.45"       # 3D rendering
chumpy = "^0.70"           # Auto-diff for FLAME
smplx = "^0.1.28"          # Body models
face-alignment = "^1.4.1"  # Facial landmarks
insightface = "^0.7.3"     # Face analysis
scikit-image = "^0.22.0"   # Image processing
imageio = "^2.33.0"        # Image I/O
imageio-ffmpeg = "^0.4.9"  # Video I/O
h5py = "^3.10.0"           # HDF5 files
cupy-cuda12x = "^13.0.0"   # GPU arrays (optional)

Voice Engine Specific#

toml
librosa = "^0.10.0"        # Audio analysis
soundfile = "^0.12.0"      # Audio I/O
speechbrain = "^0.5.0"     # Speech processing
resemblyzer = "^0.1.3"     # Speaker verification

Model Loading Patterns#

HuggingFace Models#

python
from transformers import AutoModel, AutoTokenizer

# Download and cache
model = AutoModel.from_pretrained("model_name")
tokenizer = AutoTokenizer.from_pretrained("model_name")

# Load from local path
model = AutoModel.from_pretrained("/path/to/model")

PyTorch Checkpoints#

python
import torch

# Load state dict
state_dict = torch.load("model.pt", map_location="cuda")
model.load_state_dict(state_dict)

# Load with compression
import gzip
with gzip.open("model.pt.gz", "rb") as f:
    state_dict = torch.load(f, map_location="cuda")

Pickle Models#

python
import pickle

# Load FLAME model
with open("generic_model.pkl", "rb") as f:
    model_data = pickle.load(f, encoding="latin1")

# Access components
vertices = model_data["v_template"]
faces = model_data["f"]

ONNX Models#

python
import onnxruntime as ort

# Create session with GPU
providers = ['CUDAExecutionProvider', 'CPUExecutionProvider']
session = ort.InferenceSession("model.onnx", providers=providers)

# Run inference
outputs = session.run(None, {"input": input_array})

Lazy Loading Pattern#

python
class LazyModel:
    def __init__(self, model_path: str):
        self._model_path = model_path
        self._model = None

    @property
    def model(self):
        if self._model is None:
            self._model = self._load_model()
        return self._model

    def _load_model(self):
        return torch.load(self._model_path)

GPU & Hardware Requirements#

Minimum Requirements#

Component Minimum Recommended
GPU NVIDIA RTX 3080 (10GB) NVIDIA A100 (40GB)
CUDA 11.8 12.1+
cuDNN 8.6 8.9+
TensorRT 8.5 8.6+
RAM 32GB 64GB+
Storage 100GB SSD 500GB NVMe

GPU Memory Breakdown#

Model Category VRAM Usage
3D Gaussian Splatting 2-6 GB
FLAME Model 0.5 GB
Whisper Medium 5 GB
Whisper Large 10 GB
Face Detection (MTCNN) 0.5 GB
Emotion Detection 1 GB
Embedding Models (local) 1-2 GB
Total (typical) 10-15 GB

Multi-GPU Configuration#

bash
# Environment variables
CUDA_VISIBLE_DEVICES=0,1
AVATAR_GPU_ID=0
VOICE_ENGINE_GPU_ID=1
PERCEPTION_GPU_ID=0

Performance Targets#

Task Latency Target Throughput
Avatar rendering < 33ms 30 FPS
Face detection < 20ms 50 FPS
Emotion detection < 50ms 20 FPS
Speech recognition Real-time 1x+
TTS synthesis < 100ms -
Embedding generation < 50ms -

Model Download Sources#

HuggingFace Hub#

Model Type Source
Whisper openai/whisper-*
Transformers facebook/bart-large-mnli
BGE Embeddings BAAI/bge-*
Sentence Transformers sentence-transformers/*
bash
# Pre-download models
huggingface-cli download openai/whisper-medium
huggingface-cli download BAAI/bge-large-en-v1.5

Official Repositories#

Model Source URL
FLAME https://flame.is.tue.mpg.de/
dlib landmarks http://dlib.net/files/
MediaPipe Built-in (via package)
InsightFace Built-in (via package)

Cloud APIs (Runtime)#

Provider Models
OpenAI text-embedding-*, whisper-1
Voyage AI voyage-3.5-*
Cohere embed-*

Custom Trained Models#

Model Location
3D Gaussian Splatting models/avatar/checkpoints/
Lip Sync models/lipsync/
Voice Profiles models/voice/profiles/

Environment Variables#

GPU Configuration#

bash
# Device selection
AVATAR_DEVICE=cuda
AVATAR_GPU_ID=0
PERCEPTION_DEVICE=cuda
VOICE_ENGINE_DEVICE=cuda

# Memory optimization
AVATAR_ENABLE_MIXED_PRECISION=true
PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:512

Model Paths#

bash
# Avatar models
AVATAR_MODEL_PATH=/models/avatar
AVATAR_CHECKPOINT_PATH=/models/avatar/checkpoints
FLAME_MODEL_PATH=/models/flame/generic_model.pkl
FLAME_LANDMARK_PATH=/models/flame/landmark_embedding.npy

# Voice models
WHISPER_MODEL_PATH=/models/whisper
SPEAKER_EMBEDDING_MODEL_PATH=/models/speechbrain_ecapa
VOICE_PROFILE_PATH=/models/voice/profiles

# Perception models
FACE_DETECTION_MODEL=mediapipe
EMOTION_MODEL_PATH=/models/emotion
DLIB_LANDMARK_PATH=/models/dlib/shape_predictor_68_face_landmarks.dat

Model Selection#

bash
# Voice engine
VOICE_ENGINE_STT_MODEL=openai/whisper-medium
VOICE_ENGINE_EMBEDDING_MODEL=speechbrain

# Perception engine
PERCEPTION_FACE_DETECTOR=mediapipe  # mediapipe, mtcnn, retinaface
PERCEPTION_EMOTION_MODEL=deepface   # deepface, hsemotion, fer

# Knowledge base
KNOWLEDGE_BASE_EMBEDDING_MODEL=voyage-3.5-lite
KNOWLEDGE_BASE_EMBEDDING_PROVIDER=voyage  # voyage, openai, cohere, local

API Keys (for cloud models)#

bash
# Embedding providers
VOYAGE_API_KEY=pa-...
OPENAI_API_KEY=sk-...
COHERE_API_KEY=...

# LLM providers (for model-based features)
ANTHROPIC_API_KEY=sk-ant-...

Migration Considerations#

High Priority Models#

These models are critical for core functionality:

Model Priority Complexity Notes
3D Gaussian Splatting P0 High Custom PyTorch, GPU-intensive
FLAME P0 Medium Pickle format, needs license
Whisper P0 Low HuggingFace, well-supported
Speaker Embeddings P0 Medium Multiple options
Face Detection P1 Low Multiple backends

TypeScript Considerations#

For Psyche TypeScript migration:

Python Library TypeScript Option
PyTorch Run Python subprocess, or use ONNX.js
Transformers HuggingFace.js (limited)
MediaPipe @mediapipe/face_detection
OpenCV opencv4nodejs
SpeechBrain Keep Python, call via API
text
┌──────────────────────────────────────────────────────────────────────┐
│                      Psyche ML Service Layer                         │
├──────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐      │
│  │  TypeScript API │  │  TypeScript API │  │  TypeScript API │      │
│  │   (HTTP/gRPC)   │  │   (HTTP/gRPC)   │  │   (HTTP/gRPC)   │      │
│  └────────┬────────┘  └────────┬────────┘  └────────┬────────┘      │
│           │                    │                    │                │
│  ┌────────▼────────┐  ┌────────▼────────┐  ┌────────▼────────┐      │
│  │  Python Worker  │  │  Python Worker  │  │  Python Worker  │      │
│  │  (Avatar/3DGS)  │  │  (Voice/Audio)  │  │  (Perception)   │      │
│  └─────────────────┘  └─────────────────┘  └─────────────────┘      │
│                                                                      │
└──────────────────────────────────────────────────────────────────────┘

Migration Phases#

  1. Phase 1: Keep Python ML workers, add TypeScript API layer
  2. Phase 2: Migrate simple models to ONNX.js where possible
  3. Phase 3: Optimize GPU workers with TensorRT
  4. Phase 4: Consider cloud ML services for scale

Model Versioning#

text
models/
├── avatar/
│   ├── v1.0/
│   │   ├── core.pt.gz
│   │   └── sh_rest.pt.gz
│   └── v1.1/
│       ├── core.pt.gz
│       └── sh_rest.pt.gz
├── flame/
│   └── v2020/
│       ├── generic_model.pkl
│       └── landmark_embedding.npy
└── voice/
    ├── whisper-medium/
    └── speechbrain-ecapa/

Appendix: Model File Inventory#

Required Model Files#

File Size Source Required
flame/generic_model.pkl ~200MB FLAME website Yes
flame/landmark_embedding.npy ~10KB FLAME website Yes
dlib/shape_predictor_68_face_landmarks.dat ~100MB dlib.net Optional
avatar/checkpoints/core.pt.gz ~200MB Custom trained Yes
avatar/checkpoints/sh_rest.pt.gz ~400MB Custom trained Optional
voice/speechbrain_ecapa/* ~100MB HuggingFace Yes
whisper-medium/* ~1.5GB HuggingFace Yes

Auto-Downloaded Models#

Model Package Auto-Download
MediaPipe Face Detection mediapipe Yes
MediaPipe Face Mesh mediapipe Yes
MTCNN facenet-pytorch Yes
RetinaFace insightface Yes
DeepFace backends deepface Yes
HSEmotion hsemotion Yes
FER fer Yes
Sentence Transformers sentence-transformers Yes

Docker Build Context#

dockerfile
# Voice Engine Dockerfile
FROM nvidia/cuda:12.1.1-cudnn8-devel-ubuntu22.04

# System dependencies
RUN apt-get update && apt-get install -y \
    libsndfile1 libportaudio2 ffmpeg python3.11 python3-pip

# Install PyTorch with CUDA
RUN pip install torch==2.2.0 torchaudio==2.2.0 --index-url https://download.pytorch.org/whl/cu121

# Model directories
RUN mkdir -p /app/models/{tts,stt,embeddings}
dockerfile
# Perception Engine Dockerfile
FROM nvidia/cuda:12.1.1-cudnn8-devel-ubuntu22.04

# System dependencies
RUN apt-get update && apt-get install -y \
    libgl1-mesa-glx libtesseract-dev cmake ninja-build python3.11 python3-pip

# Install PyTorch with CUDA
RUN pip install torch==2.2.0 torchvision==0.17.0 --index-url https://download.pytorch.org/whl/cu121

# Model directory
RUN mkdir -p /app/models