This document provides a comprehensive inventory of all machine learning models
used by the Serwaa platform (source codebase for the Psyche domain migration).
Table of Contents#
Overview
Avatar/3D Models
Perception/Face Analysis Models
Voice/Audio Models
Embedding Models
NLP/Intent Detection Models
Core ML Dependencies
Model Loading Patterns
GPU & Hardware Requirements
Model Download Sources
Environment Variables
Migration Considerations
Overview#
The Serwaa platform uses 50+ machine learning models across 6 major categories:
Category
Models
Framework
Deployment
Avatar/3D
3DGS, FLAME, NeRF, Lip Sync
PyTorch
Local GPU
Face Analysis
MediaPipe, InsightFace, MTCNN
TensorFlow, PyTorch
Local GPU/CPU
Voice/Audio
Whisper, ECAPA-TDNN, F5-TTS
PyTorch, SpeechBrain
Local GPU
Embeddings
Voyage AI, OpenAI, BGE
API, HuggingFace
API/Local
NLP
BART-MNLI, Sentiment
HuggingFace
Local CPU
Emotion
DeepFace, HSEmotion, FER
TensorFlow, PyTorch
Local GPU/CPU
Model Architecture Overview#
text Copy
┌─────────────────────────────────────────────────────────────────────────────┐
│ ML Model Pipeline │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │
│ │ Input Pipeline │───▶│ Feature Models │───▶│ Output Models │ │
│ └──────────────────┘ └──────────────────┘ └──────────────────┘ │
│ ↓ ↓ ↓ │
│ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │
│ │ Audio Processing │ │ Voice Embeddings │ │ TTS Synthesis │ │
│ │ Face Detection │ │ Face Embeddings │ │ Avatar Rendering │ │
│ │ Text Processing │ │ Text Embeddings │ │ Lip Sync │ │
│ └──────────────────┘ └──────────────────┘ └──────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Avatar/3D Models#
3D Gaussian Splatting (3DGS)#
Real-time avatar rendering using 3D Gaussian primitives.
Source Files#
text Copy
serwaa/services/avatar-engine/src/avatar_engine/models/gaussian.py
Model Specification#
Property
Value
Type
Custom PyTorch nn.Module
Framework
PyTorch 2.x
Device
CUDA (GPU required)
Precision
FP16/FP32
Parameters#
Component
Description
Size
Positions
3D Gaussian centers
N × 3
Rotations
Quaternion orientations
N × 4
Scales
Anisotropic scales
N × 3
Opacities
Alpha values
N × 1
SH DC
0th-order spherical harmonics
N × 3
SH Rest
Higher-order SH (degree 1-3)
N × 45
Where N = number of Gaussians (typically 50,000 - 100,000)
Spherical Harmonics Configuration#
SH Degree
Coefficients per Gaussian
Use Case
0
3 (RGB only)
Basic color
1
12
Directional lighting
2
27
Specular highlights
3
48
Full appearance (default)
Memory Usage#
Resolution
Gaussians
VRAM
Target FPS
512×512
50K
1.5 GB
200+
768×768
75K
2.5 GB
120+
1024×1024
100K
4.0 GB
60+
1920×1080
100K
6.0 GB
30+
python Copy
# Checkpoint structure (.pt.gz)
{
"metadata": {
"num_gaussians": int,
"sh_degree": int,
"version": str,
"creation_time": str
},
"core_state": {
"positions": torch.Tensor, # (N, 3) float16
"rotations": torch.Tensor, # (N, 4) float16
"scales": torch.Tensor, # (N, 3) float16
"opacities": torch.Tensor, # (N, 1) float16
"sh_dc": torch.Tensor # (N, 3) float16
},
"sh_rest": torch.Tensor # (N, 45) float16, separate file
}
Loading Pattern#
python Copy
# Progressive loading for faster startup
gaussian_model = GaussianModel(sh_degree=3)
# Load core state first (essential for rendering)
await gaussian_model.load_core(f"{checkpoint_dir}/core.pt.gz")
# Load SH rest asynchronously (optional, improves quality)
await gaussian_model.load_sh_rest(f"{checkpoint_dir}/sh_rest.pt.gz")
FLAME Parametric Head Model#
Industry-standard 3D morphable face model.
Source Files#
text Copy
serwaa/services/avatar-engine/src/avatar_engine/models/flame.py
Model Specification#
Property
Value
Source
Max Planck Institute
Website
https://flame.is.tue.mpg.de/
Framework
PyTorch nn.Module
Device
CUDA/CPU
License
Academic/Commercial
Model Components#
Component
Dimensions
Description
Template Vertices
5,023 × 3
Mean face mesh
Face Indices
9,976 × 3
Triangle indices
Shape Blendshapes
5,023 × 3 × 100
Identity variation
Expression Blendshapes
5,023 × 3 × 50
Facial expressions
Pose Blendshapes
5,023 × 3 × 36
Pose-dependent deformations
Joint Regressor
5 × 5,023
5-joint kinematic chain
Skinning Weights
5,023 × 5
LBS weights
Landmark Embedding
68 × 3
Facial landmark positions
Parameter
Shape
Range
Description
shape
(100,)
[-3, 3]
Identity coefficients
expression
(50,)
[-3, 3]
Expression coefficients
pose
(6,)
Axis-angle
Head rotation + jaw
neck_pose
(3,)
Axis-angle
Neck rotation
eye_pose
(6,)
Axis-angle
Eye gaze
Required Files#
text Copy
/models/flame/
├── generic_model.pkl # ~200MB - Main FLAME model
├── landmark_embedding.npy # ~10KB - 68-point landmarks
└── FLAME_masks.pkl # ~5MB - Region masks (optional)
Loading Pattern#
python Copy
import pickle
import numpy as np
# Load FLAME model
with open("models/flame/generic_model.pkl", "rb") as f:
flame_model = pickle.load(f, encoding="latin1")
# Load landmarks
landmark_embedding = np.load("models/flame/landmark_embedding.npy")
Expression Blendshapes (50 coefficients)#
Index
Expression
0-9
Eyebrow movements
10-19
Eye movements
20-29
Mouth shapes
30-39
Lip movements
40-49
Jaw and cheek
NeRF Models (Research/Backup)#
Neural Radiance Field implementations for background rendering.
Source Files#
text Copy
serwaa/services/avatar-engine/src/avatar_engine/nerf/
├── instant_ngp.py # Instant-NGP implementation
├── dynamic_instant_ngp.py # Time-conditioned variant
└── hash_grid.py # Multi-resolution hash encoding
Model Specification#
Property
Value
Type
Instant-NGP (pure PyTorch)
Framework
PyTorch
Purpose
Background, 3DGS bootstrapping
Device
CUDA
Hash Grid Configuration#
Parameter
Default
Description
num_levels
16
Number of resolution levels
level_dim
2
Features per level
log2_hashmap_size
19
~500K entries per level
base_resolution
16
Coarsest grid resolution
finest_resolution
512
Finest grid resolution
Lip Sync Models#
Audio-driven lip synchronization for avatars.
Source Files#
text Copy
serwaa/services/avatar-engine/src/avatar_engine/lipsync/
├── models.py # Wav2Lip-style models
├── audio_encoder.py
└── sync_net.py # Synchronization discriminator
Model Architecture#
Component
Input
Output
Parameters
Audio Encoder
Mel-spectrogram (80×16)
Audio embedding (512)
~5M
Lip Encoder
Face patch (96×96)
Visual embedding (512)
~8M
Lip Decoder
Fused embedding
Lip vertices/mask
~10M
Sync Discriminator
Audio + Visual
Sync score
~3M
Audio Processing#
Parameter
Value
Sample Rate
16,000 Hz
Mel Bins
80
Window Size
25ms
Hop Length
10ms
Frames per Step
16 (~160ms)
Perception/Face Analysis Models#
Face Detection Models#
Built-in Google MediaPipe face detector.
Property
Value
Framework
MediaPipe
Package
mediapipe ^0.10.0
Models
Short-range (2m), Full-range (5m)
Output
Bounding box + 6 keypoints
Device
CPU/GPU
python Copy
import mediapipe as mp
face_detection = mp.solutions.face_detection.FaceDetection(
model_selection=0, # 0=short-range, 1=full-range
min_detection_confidence=0.5
)
MTCNN (Multi-task Cascaded CNN)#
Three-stage face detection cascade.
Property
Value
Framework
facenet_pytorch
Package
facenet-pytorch ^2.5.0
Stages
P-Net → R-Net → O-Net
Output
Bounding boxes + 5 keypoints
Device
CUDA/CPU
Stage
Purpose
Input Size
P-Net
Proposal
Variable (12×12 min)
R-Net
Refine
24×24
O-Net
Output
48×48
python Copy
from facenet_pytorch import MTCNN
mtcnn = MTCNN(
image_size=160,
margin=0,
min_face_size=20,
thresholds=[0.6, 0.7, 0.7], # P-Net, R-Net, O-Net
factor=0.709,
post_process=True,
device='cuda'
)
RetinaFace (InsightFace)#
High-accuracy face detection with landmarks.
Property
Value
Framework
InsightFace
Package
insightface ^0.7.3
Model
retinaface_resnet50
Output
Bounding box + 5 keypoints + score
Device
CUDA/CPU (via ONNX Runtime)
python Copy
from insightface.app import FaceAnalysis
app = FaceAnalysis(
name='buffalo_l',
providers=['CUDAExecutionProvider', 'CPUExecutionProvider']
)
app.prepare(ctx_id=0, det_size=(640, 640))
Facial Landmark Detection#
face_alignment Library#
2D and 3D facial landmark detection.
Property
Value
Framework
PyTorch
Package
face-alignment ^1.4.1
Output
68 facial landmarks
Variants
2D, 3D
Device
CUDA/CPU
python Copy
import face_alignment
fa = face_alignment.FaceAlignment(
face_alignment.LandmarksType.TWO_D, # or THREE_D
device='cuda',
flip_input=False
)
landmarks = fa.get_landmarks(image) # Returns (68, 2) or (68, 3)
Dense 468-point face mesh.
Property
Value
Framework
MediaPipe
Output
468 landmarks (→ 68 mapped)
Features
2D + 3D normalized coords
Device
CPU/GPU
Landmark Region
Count
Face contour
36
Eyebrows
20
Eyes
32
Nose
18
Lips
40
Irises
10
Other
312
python Copy
import mediapipe as mp
face_mesh = mp.solutions.face_mesh.FaceMesh(
static_image_mode=True,
max_num_faces=1,
refine_landmarks=True, # Enable iris landmarks
min_detection_confidence=0.5,
min_tracking_confidence=0.5
)
dlib Shape Predictor#
Classic 68-point landmark detector.
Property
Value
Framework
dlib
Model File
shape_predictor_68_face_landmarks.dat
Size
~100MB
Output
68 landmarks
Device
CPU
bash Copy
# Download model
wget http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2
bunzip2 shape_predictor_68_face_landmarks.dat.bz2
Emotion Detection Models#
DeepFace#
Multi-backend facial analysis.
Property
Value
Framework
TensorFlow
Package
deepface ^0.0.79
Output
7 emotions + face embedding
Device
CPU/GPU
Emotion
Description
angry
Anger expression
disgust
Disgust expression
fear
Fear expression
happy
Happiness/joy
sad
Sadness expression
surprise
Surprise expression
neutral
Neutral expression
python Copy
from deepface import DeepFace
result = DeepFace.analyze(
image,
actions=['emotion', 'age', 'gender', 'race'],
detector_backend='retinaface',
enforce_detection=True
)
# result['emotion'] = {'angry': 0.1, 'happy': 0.8, ...}
HSEmotion#
High-accuracy 8-class emotion classifier.
Property
Value
Framework
PyTorch (EfficientNet-based)
Package
hsemotion ^0.3.0
Model
enet_b0_8_best_vgaf
Output
8 emotions
Device
CUDA/CPU
Emotion
Index
Anger
0
Contempt
1
Disgust
2
Fear
3
Happiness
4
Neutral
5
Sadness
6
Surprise
7
python Copy
from hsemotion import HSEmotionRecognizer
recognizer = HSEmotionRecognizer(model_name='enet_b0_8_best_vgaf')
emotion, scores = recognizer.predict_emotions(face_image)
FER (Facial Expression Recognition)#
Keras-based emotion classifier.
Property
Value
Framework
TensorFlow/Keras
Package
fer ^22.5.0
Output
7 emotions
Device
CPU/GPU
python Copy
from fer import FER
detector = FER(mtcnn=True)
emotions = detector.detect_emotions(image)
# [{'box': [x, y, w, h], 'emotions': {...}}]
Custom PyTorch CNN#
MobileNetV2-based emotion classifier.
Property
Value
Backbone
MobileNetV2 (pretrained)
Input
224×224 RGB
Output
8 emotion classes
Parameters
~3.5M
python Copy
import torch
import torchvision.models as models
class EmotionClassifier(torch.nn.Module):
def __init__(self):
super().__init__()
self.backbone = models.mobilenet_v2(pretrained=True)
self.backbone.classifier = torch.nn.Linear(1280, 8)
def forward(self, x):
return self.backbone(x)
Voice/Audio Models#
Whisper (Speech-to-Text)#
OpenAI's speech recognition model.
Source Files#
text Copy
serwaa/services/voice-engine/src/voice_engine/stt/local_whisper.py
Model Variants#
Model
Parameters
VRAM
Speed
WER
tiny
39M
~1GB
32x
~10%
base
74M
~1GB
16x
~7%
small
244M
~2GB
8x
~5%
medium
769M
~5GB
4x
~4%
large
1.55B
~10GB
1x
~3%
large-v2
1.55B
~10GB
1x
~2.5%
Configuration#
Parameter
Value
Sample Rate
16,000 Hz
Input
30-second chunks
Framework
HuggingFace Transformers
Model ID
openai/whisper-{size}
python Copy
from transformers import WhisperProcessor, WhisperForConditionalGeneration
processor = WhisperProcessor.from_pretrained("openai/whisper-medium")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-medium")
model.to("cuda")
# Transcribe
input_features = processor(audio, return_tensors="pt").input_features.to("cuda")
predicted_ids = model.generate(input_features)
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)
Features#
Multi-language transcription (100+ languages)
Language detection
Translation to English
Word-level timestamps
Vocabulary priming (prompts)
Speaker Embeddings#
SpeechBrain ECAPA-TDNN#
State-of-the-art speaker embedding model.
Property
Value
Framework
SpeechBrain
Model ID
speechbrain/spkrec-ecapa-voxceleb
Embedding Dim
192
Input
16kHz mono audio
Device
CUDA/CPU
python Copy
from speechbrain.pretrained import EncoderClassifier
classifier = EncoderClassifier.from_hparams(
source="speechbrain/spkrec-ecapa-voxceleb",
savedir="models/speechbrain_ecapa/"
)
embedding = classifier.encode_batch(waveform) # (batch, 192)
Resemblyzer#
Lightweight speaker encoder.
Property
Value
Framework
PyTorch
Package
resemblyzer ^0.1.3
Embedding Dim
256
Device
CPU optimized
python Copy
from resemblyzer import VoiceEncoder, preprocess_wav
encoder = VoiceEncoder()
wav = preprocess_wav(audio_path)
embedding = encoder.embed_utterance(wav) # (256,)
Simple Feature-Based (Fallback)#
Handcrafted audio features.
Property
Value
Framework
librosa + NumPy
Features
MFCCs + spectral features
Embedding Dim
128
Device
CPU
Feature
Dimension
MFCCs
13 × 2 (mean + std)
Spectral centroid
1 × 2
Spectral bandwidth
1 × 2
Spectral rolloff
1 × 2
Zero-crossing rate
1 × 2
Delta MFCCs
13 × 2
Delta-delta MFCCs
13 × 2
Total
128
F5-TTS#
Diffusion-based text-to-speech synthesis.
Source Files#
text Copy
serwaa/services/voice-engine/src/voice_engine/tts/f5_tts.py
Model Specification#
Property
Value
Type
Diffusion-based TTS
Framework
PyTorch
Input
Text + reference audio
Output
High-quality speech
Device
CUDA
Features#
Zero-shot voice cloning
High fidelity synthesis
Prosody transfer from reference
Multi-speaker support
Embedding Models#
Cloud-Based Embeddings#
Voyage AI#
Primary embedding provider.
Model
Dimensions
Max Tokens
Use Case
voyage-3.5-lite
1024
16,000
Default (fast)
voyage-3.5
1024
32,000
Standard
voyage-3
1024
32,000
High quality
python Copy
import voyageai
client = voyageai.AsyncClient(api_key=os.environ["VOYAGE_API_KEY"])
result = await client.embed(
texts=["Hello world"],
model="voyage-3.5-lite",
input_type="document" # or "query"
)
embeddings = result.embeddings # List[List[float]]
OpenAI Embeddings#
Alternative embedding provider.
Model
Dimensions
Max Tokens
Cost
text-embedding-ada-002
1536
8,191
$0.0001/1K
text-embedding-3-small
1536
8,191
$0.00002/1K
text-embedding-3-large
3072
8,191
$0.00013/1K
python Copy
import openai
response = openai.Embedding.create(
input="Hello world",
model="text-embedding-3-small"
)
embedding = response.data[0].embedding # List[float]
Cohere Embeddings#
Fallback provider.
Model
Dimensions
Max Tokens
embed-english-v3.0
1024
512
embed-multilingual-v3.0
1024
512
Local Embedding Models#
BGE (BAAI General Embedding)#
High-quality open-source embeddings.
Model
Dimensions
Max Tokens
Size
bge-m3
1024
8,192
~2GB
bge-large-en-v1.5
1024
512
~1.3GB
bge-base-en-v1.5
768
512
~440MB
bge-small-en-v1.5
384
512
~130MB
python Copy
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('BAAI/bge-large-en-v1.5')
embeddings = model.encode(["Hello world"]) # (1, 1024)
General-purpose sentence embeddings.
Model
Dimensions
Max Tokens
all-MiniLM-L6-v2
384
256
all-mpnet-base-v2
768
384
paraphrase-multilingual-MiniLM-L12-v2
384
128
python Copy
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('sentence-transformers/all-mpnet-base-v2')
embeddings = model.encode(texts, convert_to_tensor=True)
NLP/Intent Detection Models#
Intent Classification#
Zero-shot classification for intent detection.
Source Files#
text Copy
serwaa/services/voice-engine/src/voice_engine/transcript/intent.py
BART-MNLI Model#
Property
Value
Model ID
facebook/bart-large-mnli
Framework
HuggingFace Transformers
Parameters
407M
Task
Zero-shot classification
Device
CUDA/CPU
python Copy
from transformers import pipeline
classifier = pipeline(
"zero-shot-classification",
model="facebook/bart-large-mnli",
device=0 # CUDA device
)
result = classifier(
"What's the weather like?",
candidate_labels=["question", "statement", "command", "request"]
)
# {'labels': ['question', ...], 'scores': [0.95, ...]}
Intent Categories#
Intent
Example
question
"What is your name?"
statement
"The sky is blue."
command
"Turn on the lights."
request
"Could you help me?"
greeting
"Hello, how are you?"
farewell
"Goodbye, see you later."
affirmation
"Yes, that's correct."
negation
"No, I don't think so."
Sentiment Analysis#
Sentiment classification for text.
Source Files#
text Copy
serwaa/services/voice-engine/src/voice_engine/transcript/sentiment.py
Model Options#
Model
Framework
Output
distilbert-base-uncased-finetuned-sst-2-english
HuggingFace
Positive/Negative
cardiffnlp/twitter-roberta-base-sentiment
HuggingFace
Pos/Neg/Neutral
nlptown/bert-base-multilingual-uncased-sentiment
HuggingFace
1-5 stars
python Copy
from transformers import pipeline
sentiment = pipeline("sentiment-analysis")
result = sentiment("I love this product!")
# [{'label': 'POSITIVE', 'score': 0.99}]
Core ML Dependencies#
Deep Learning Frameworks#
toml Copy
# pyproject.toml dependencies
torch = "^2.2.0" # Core deep learning
torchaudio = "^2.2.0" # Audio processing
torchvision = "^0.17.0" # Image processing
ML Utilities#
toml Copy
transformers = "^4.37.0" # HuggingFace models
accelerate = "^0.26.0" # Training acceleration
safetensors = "^0.4.0" # Safe model serialization
huggingface-hub = "^0.20.0" # Model downloads
numpy = "^1.26.0" # Numerical computing
scipy = "^1.12.0" # Scientific computing
scikit-learn = "^1.4.0" # ML utilities
Computer Vision#
toml Copy
opencv-python = "^4.9.0" # Image processing
mediapipe = "^0.10.0" # Face/pose detection
pillow = "^10.2.0" # Image I/O
Avatar Engine Specific#
toml Copy
pytorch3d = "^0.7.5" # 3D deep learning
trimesh = "^4.0.0" # 3D mesh processing
open3d = "^0.18.0" # Point cloud processing
pyrender = "^0.1.45" # 3D rendering
chumpy = "^0.70" # Auto-diff for FLAME
smplx = "^0.1.28" # Body models
face-alignment = "^1.4.1" # Facial landmarks
insightface = "^0.7.3" # Face analysis
scikit-image = "^0.22.0" # Image processing
imageio = "^2.33.0" # Image I/O
imageio-ffmpeg = "^0.4.9" # Video I/O
h5py = "^3.10.0" # HDF5 files
cupy-cuda12x = "^13.0.0" # GPU arrays (optional)
Voice Engine Specific#
toml Copy
librosa = "^0.10.0" # Audio analysis
soundfile = "^0.12.0" # Audio I/O
speechbrain = "^0.5.0" # Speech processing
resemblyzer = "^0.1.3" # Speaker verification
Model Loading Patterns#
HuggingFace Models#
python Copy
from transformers import AutoModel, AutoTokenizer
# Download and cache
model = AutoModel.from_pretrained("model_name")
tokenizer = AutoTokenizer.from_pretrained("model_name")
# Load from local path
model = AutoModel.from_pretrained("/path/to/model")
PyTorch Checkpoints#
python Copy
import torch
# Load state dict
state_dict = torch.load("model.pt", map_location="cuda")
model.load_state_dict(state_dict)
# Load with compression
import gzip
with gzip.open("model.pt.gz", "rb") as f:
state_dict = torch.load(f, map_location="cuda")
Pickle Models#
python Copy
import pickle
# Load FLAME model
with open("generic_model.pkl", "rb") as f:
model_data = pickle.load(f, encoding="latin1")
# Access components
vertices = model_data["v_template"]
faces = model_data["f"]
ONNX Models#
python Copy
import onnxruntime as ort
# Create session with GPU
providers = ['CUDAExecutionProvider', 'CPUExecutionProvider']
session = ort.InferenceSession("model.onnx", providers=providers)
# Run inference
outputs = session.run(None, {"input": input_array})
Lazy Loading Pattern#
python Copy
class LazyModel:
def __init__(self, model_path: str):
self._model_path = model_path
self._model = None
@property
def model(self):
if self._model is None:
self._model = self._load_model()
return self._model
def _load_model(self):
return torch.load(self._model_path)
GPU & Hardware Requirements#
Minimum Requirements#
Component
Minimum
Recommended
GPU
NVIDIA RTX 3080 (10GB)
NVIDIA A100 (40GB)
CUDA
11.8
12.1+
cuDNN
8.6
8.9+
TensorRT
8.5
8.6+
RAM
32GB
64GB+
Storage
100GB SSD
500GB NVMe
GPU Memory Breakdown#
Model Category
VRAM Usage
3D Gaussian Splatting
2-6 GB
FLAME Model
0.5 GB
Whisper Medium
5 GB
Whisper Large
10 GB
Face Detection (MTCNN)
0.5 GB
Emotion Detection
1 GB
Embedding Models (local)
1-2 GB
Total (typical)
10-15 GB
Multi-GPU Configuration#
bash Copy
# Environment variables
CUDA_VISIBLE_DEVICES=0,1
AVATAR_GPU_ID=0
VOICE_ENGINE_GPU_ID=1
PERCEPTION_GPU_ID=0
Task
Latency Target
Throughput
Avatar rendering
< 33ms
30 FPS
Face detection
< 20ms
50 FPS
Emotion detection
< 50ms
20 FPS
Speech recognition
Real-time
1x+
TTS synthesis
< 100ms
-
Embedding generation
< 50ms
-
Model Download Sources#
HuggingFace Hub#
Model Type
Source
Whisper
openai/whisper-*
Transformers
facebook/bart-large-mnli
BGE Embeddings
BAAI/bge-*
Sentence Transformers
sentence-transformers/*
bash Copy
# Pre-download models
huggingface-cli download openai/whisper-medium
huggingface-cli download BAAI/bge-large-en-v1.5
Official Repositories#
Model
Source URL
FLAME
https://flame.is.tue.mpg.de/
dlib landmarks
http://dlib.net/files/
MediaPipe
Built-in (via package)
InsightFace
Built-in (via package)
Cloud APIs (Runtime)#
Provider
Models
OpenAI
text-embedding-*, whisper-1
Voyage AI
voyage-3.5-*
Cohere
embed-*
Custom Trained Models#
Model
Location
3D Gaussian Splatting
models/avatar/checkpoints/
Lip Sync
models/lipsync/
Voice Profiles
models/voice/profiles/
Environment Variables#
GPU Configuration#
bash Copy
# Device selection
AVATAR_DEVICE=cuda
AVATAR_GPU_ID=0
PERCEPTION_DEVICE=cuda
VOICE_ENGINE_DEVICE=cuda
# Memory optimization
AVATAR_ENABLE_MIXED_PRECISION=true
PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:512
Model Paths#
bash Copy
# Avatar models
AVATAR_MODEL_PATH=/models/avatar
AVATAR_CHECKPOINT_PATH=/models/avatar/checkpoints
FLAME_MODEL_PATH=/models/flame/generic_model.pkl
FLAME_LANDMARK_PATH=/models/flame/landmark_embedding.npy
# Voice models
WHISPER_MODEL_PATH=/models/whisper
SPEAKER_EMBEDDING_MODEL_PATH=/models/speechbrain_ecapa
VOICE_PROFILE_PATH=/models/voice/profiles
# Perception models
FACE_DETECTION_MODEL=mediapipe
EMOTION_MODEL_PATH=/models/emotion
DLIB_LANDMARK_PATH=/models/dlib/shape_predictor_68_face_landmarks.dat
Model Selection#
bash Copy
# Voice engine
VOICE_ENGINE_STT_MODEL=openai/whisper-medium
VOICE_ENGINE_EMBEDDING_MODEL=speechbrain
# Perception engine
PERCEPTION_FACE_DETECTOR=mediapipe # mediapipe, mtcnn, retinaface
PERCEPTION_EMOTION_MODEL=deepface # deepface, hsemotion, fer
# Knowledge base
KNOWLEDGE_BASE_EMBEDDING_MODEL=voyage-3.5-lite
KNOWLEDGE_BASE_EMBEDDING_PROVIDER=voyage # voyage, openai, cohere, local
API Keys (for cloud models)#
bash Copy
# Embedding providers
VOYAGE_API_KEY=pa-...
OPENAI_API_KEY=sk-...
COHERE_API_KEY=...
# LLM providers (for model-based features)
ANTHROPIC_API_KEY=sk-ant-...
Migration Considerations#
High Priority Models#
These models are critical for core functionality:
Model
Priority
Complexity
Notes
3D Gaussian Splatting
P0
High
Custom PyTorch, GPU-intensive
FLAME
P0
Medium
Pickle format, needs license
Whisper
P0
Low
HuggingFace, well-supported
Speaker Embeddings
P0
Medium
Multiple options
Face Detection
P1
Low
Multiple backends
TypeScript Considerations#
For Psyche TypeScript migration:
Python Library
TypeScript Option
PyTorch
Run Python subprocess, or use ONNX.js
Transformers
HuggingFace.js (limited)
MediaPipe
@mediapipe/face_detection
OpenCV
opencv4nodejs
SpeechBrain
Keep Python, call via API
Recommended Architecture#
text Copy
┌──────────────────────────────────────────────────────────────────────┐
│ Psyche ML Service Layer │
├──────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │
│ │ TypeScript API │ │ TypeScript API │ │ TypeScript API │ │
│ │ (HTTP/gRPC) │ │ (HTTP/gRPC) │ │ (HTTP/gRPC) │ │
│ └────────┬────────┘ └────────┬────────┘ └────────┬────────┘ │
│ │ │ │ │
│ ┌────────▼────────┐ ┌────────▼────────┐ ┌────────▼────────┐ │
│ │ Python Worker │ │ Python Worker │ │ Python Worker │ │
│ │ (Avatar/3DGS) │ │ (Voice/Audio) │ │ (Perception) │ │
│ └─────────────────┘ └─────────────────┘ └─────────────────┘ │
│ │
└──────────────────────────────────────────────────────────────────────┘
Migration Phases#
Phase 1 : Keep Python ML workers, add TypeScript API layer
Phase 2 : Migrate simple models to ONNX.js where possible
Phase 3 : Optimize GPU workers with TensorRT
Phase 4 : Consider cloud ML services for scale
Model Versioning#
text Copy
models/
├── avatar/
│ ├── v1.0/
│ │ ├── core.pt.gz
│ │ └── sh_rest.pt.gz
│ └── v1.1/
│ ├── core.pt.gz
│ └── sh_rest.pt.gz
├── flame/
│ └── v2020/
│ ├── generic_model.pkl
│ └── landmark_embedding.npy
└── voice/
├── whisper-medium/
└── speechbrain-ecapa/
Appendix: Model File Inventory#
Required Model Files#
File
Size
Source
Required
flame/generic_model.pkl
~200MB
FLAME website
Yes
flame/landmark_embedding.npy
~10KB
FLAME website
Yes
dlib/shape_predictor_68_face_landmarks.dat
~100MB
dlib.net
Optional
avatar/checkpoints/core.pt.gz
~200MB
Custom trained
Yes
avatar/checkpoints/sh_rest.pt.gz
~400MB
Custom trained
Optional
voice/speechbrain_ecapa/*
~100MB
HuggingFace
Yes
whisper-medium/*
~1.5GB
HuggingFace
Yes
Auto-Downloaded Models#
Model
Package
Auto-Download
MediaPipe Face Detection
mediapipe
Yes
MediaPipe Face Mesh
mediapipe
Yes
MTCNN
facenet-pytorch
Yes
RetinaFace
insightface
Yes
DeepFace backends
deepface
Yes
HSEmotion
hsemotion
Yes
FER
fer
Yes
Sentence Transformers
sentence-transformers
Yes
Docker Build Context#
dockerfile Copy
# Voice Engine Dockerfile
FROM nvidia/cuda:12.1.1-cudnn8-devel-ubuntu22.04
# System dependencies
RUN apt-get update && apt-get install -y \
libsndfile1 libportaudio2 ffmpeg python3.11 python3-pip
# Install PyTorch with CUDA
RUN pip install torch==2.2.0 torchaudio==2.2.0 --index-url https://download.pytorch.org/whl/cu121
# Model directories
RUN mkdir -p /app/models/{tts,stt,embeddings}
dockerfile Copy
# Perception Engine Dockerfile
FROM nvidia/cuda:12.1.1-cudnn8-devel-ubuntu22.04
# System dependencies
RUN apt-get update && apt-get install -y \
libgl1-mesa-glx libtesseract-dev cmake ninja-build python3.11 python3-pip
# Install PyTorch with CUDA
RUN pip install torch==2.2.0 torchvision==0.17.0 --index-url https://download.pytorch.org/whl/cu121
# Model directory
RUN mkdir -p /app/models