Eve library

Psyche Domain: ML Model Documentation

Guide

Revision 1

On this page

This document provides comprehensive documentation for all machine learning models used in the Psyche (Serwaa) AI Virtual Assistant Platform, including architecture details, training procedures, and performance specifications.

Table of Contents#

  1. Model Overview
  2. Avatar Models
  3. Voice Models
  4. Perception Models
  5. Training Procedures
  6. Performance Specifications
  7. Model Optimization
  8. Deployment Guidelines

Model Overview#

The Psyche platform employs 50+ ML models organized into four primary categories:

text
┌─────────────────────────────────────────────────────────────────────────────┐
│                         ML MODEL HIERARCHY                                   │
└─────────────────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────────────────┐
│                             AVATAR MODELS                                    │
│  ┌──────────────────┐  ┌──────────────────┐  ┌──────────────────┐          │
│  │  3D Gaussian     │  │  FLAME           │  │  Lip Sync        │          │
│  │  Splatting       │  │  Morphable       │  │                  │          │
│  │                  │  │                  │  │                  │          │
│  │  • Rendering     │  │  • Face mesh     │  │  • Audio→Viseme  │          │
│  │  • View synth    │  │  • Expressions   │  │  • Synchronizer  │          │
│  │  • Appearance    │  │  • Identity      │  │  • Encoder/Dec   │          │
│  └──────────────────┘  └──────────────────┘  └──────────────────┘          │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                             VOICE MODELS                                     │
│  ┌──────────────────┐  ┌──────────────────┐  ┌──────────────────┐          │
│  │  Whisper         │  │  XTTS v2         │  │  Voice Cloning   │          │
│  │  (STT)           │  │  (TTS)           │  │                  │          │
│  │                  │  │                  │  │                  │          │
│  │  • Transcription │  │  • Synthesis     │  │  • Embedding     │          │
│  │  • 99 languages  │  │  • 19 languages  │  │  • Speaker ID    │          │
│  │  • 4.2% WER      │  │  • Voice clone   │  │  • Adaptation    │          │
│  └──────────────────┘  └──────────────────┘  └──────────────────┘          │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                           PERCEPTION MODELS                                  │
│  ┌──────────────────┐  ┌──────────────────┐  ┌──────────────────┐          │
│  │  RetinaFace      │  │  MediaPipe       │  │  FER+            │          │
│  │  (Detection)     │  │  (Landmarks)     │  │  (Emotion)       │          │
│  │                  │  │                  │  │                  │          │
│  │  • Face boxes    │  │  • 468 points    │  │  • 8 emotions    │          │
│  │  • 0.91 mAP      │  │  • Real-time     │  │  • Confidence    │          │
│  │  • Multi-face    │  │  • 3D coords     │  │  • Valence       │          │
│  └──────────────────┘  └──────────────────┘  └──────────────────┘          │
└─────────────────────────────────────────────────────────────────────────────┘

Model Summary Table#

Model Category Size VRAM Latency Framework
3DGS Avatar Avatar 1.5 GB 8 GB 20 ms PyTorch
FLAME 2023 Avatar 500 MB 4 GB 15 ms PyTorch
Lip Sync Avatar 100 MB 2 GB 10 ms PyTorch
Whisper Large v3 Voice 3 GB 6 GB 50 ms PyTorch
XTTS v2 Voice 2 GB 4 GB 80 ms PyTorch
RetinaFace Perception 100 MB 2 GB 8 ms ONNX
MediaPipe Mesh Perception 4 MB 0.5 GB 2 ms TensorFlow
FER+ Perception 50 MB 2 GB 10 ms PyTorch

Avatar Models#

3D Gaussian Splatting#

3D Gaussian Splatting (3DGS) is the core rendering technology for real-time avatar generation. It represents scenes as collections of 3D Gaussian primitives that can be efficiently rasterized.

Architecture#

text
┌─────────────────────────────────────────────────────────────────────────────┐
│                    3DGS ARCHITECTURE                                         │
└─────────────────────────────────────────────────────────────────────────────┘

                         Input: Camera Pose (4×4)
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                        GAUSSIAN REPRESENTATION                               │
│                                                                              │
│  For each Gaussian g_i (i = 1 to N, typically N = 50,000 - 100,000):        │
│                                                                              │
│  ┌──────────────────┐  ┌──────────────────┐  ┌──────────────────┐          │
│  │  Position μ_i    │  │  Covariance Σ_i  │  │  Appearance      │          │
│  │  (3D center)     │  │  (3D ellipsoid)  │  │                  │          │
│  │                  │  │                  │  │                  │          │
│  │  x, y, z         │  │  Rotation (quat) │  │  Opacity α       │          │
│  │  ∈ ℝ³            │  │  Scale (3D)      │  │  SH coefficients │          │
│  └──────────────────┘  └──────────────────┘  └──────────────────┘          │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                    DIFFERENTIABLE RASTERIZATION                              │
│                                                                              │
│  1. PROJECT Gaussians to 2D screen space                                    │
│     μ'_i = K × [R|t] × μ_i        (perspective projection)                  │
│     Σ'_i = J × Σ_i × J^T          (covariance transformation)               │
│                                                                              │
│  2. TILE-BASED SORTING (16×16 pixel tiles)                                  │
│     • Assign Gaussians to overlapping tiles                                 │
│     • Sort by depth within each tile                                        │
│                                                                              │
│  3. ALPHA COMPOSITING (front-to-back)                                       │
│     C(p) = Σ c_i × α_i × Π(1 - α_j)   for j < i                            │
│                                                                              │
│  4. SPHERICAL HARMONICS for view-dependent color                            │
│     c_i(d) = SH(d) · coefficients_i                                         │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
                         Output: RGB Image (H × W × 3)

Gaussian Parameters#

Parameter Shape Type Description
positions (N, 3) float32 3D center coordinates (x,y,z)
rotations (N, 4) float32 Quaternion (w,x,y,z)
scales (N, 3) float32 Anisotropic scaling (sx,sy,sz)
opacities (N, 1) float32 Alpha value [0, 1]
sh_dc (N, 3) float32 0th-order SH (base color)
sh_rest (N, 45) float32 1st-3rd order SH coefficients

Spherical Harmonics Degrees#

Degree Coefficients Visual Quality Use Case
0 3 Flat shading Preview/Debug
1 12 Basic directional Low quality
2 27 Soft specular Medium quality
3 48 Full appearance Production

Model Files#

text
s3://serwaa-models/avatar/3dgs/v1.2.0/
├── core.pt.gz                    # 800 MB - Positions, rotations, scales, opacities
├── sh_coefficients.pt.gz         # 600 MB - Full SH coefficients
├── metadata.json                 # Model metadata and config
└── camera_calibration.json       # Reference camera parameters

Memory Requirements#

Resolution Gaussians VRAM (FP16) VRAM (FP32) Target FPS
512×512 50,000 1.5 GB 3.0 GB 200+
720p 75,000 2.5 GB 5.0 GB 100+
1080p 100,000 4.0 GB 8.0 GB 45+
4K 150,000 8.0 GB 16.0 GB 15+

Code Example#

python
import torch
from avatar_engine.models.gaussian import GaussianModel

# Initialize model
model = GaussianModel(
    sh_degree=3,
    device="cuda:0"
)

# Load checkpoint
await model.load_checkpoint("s3://serwaa-models/avatar/3dgs/v1.2.0/")

# Render frame
camera_pose = torch.eye(4, device="cuda:0")  # Identity pose
camera_intrinsics = {
    "fx": 500, "fy": 500,
    "cx": 512, "cy": 512,
    "width": 1024, "height": 1024
}

rgb_image = model.render(
    camera_pose=camera_pose,
    intrinsics=camera_intrinsics
)  # Returns (1024, 1024, 3) tensor

FLAME Morphable Model#

FLAME (Faces Learned with an Articulated Model and Expressions) is a statistical 3D head model that enables identity-independent expression animation.

Architecture#

text
┌─────────────────────────────────────────────────────────────────────────────┐
│                       FLAME MODEL ARCHITECTURE                               │
└─────────────────────────────────────────────────────────────────────────────┘

      ┌────────────────┐   ┌────────────────┐   ┌────────────────┐
      │  Shape β       │   │ Expression ψ   │   │   Pose θ       │
      │  (100 dims)    │   │  (50 dims)     │   │  (15 dims)     │
      └───────┬────────┘   └───────┬────────┘   └───────┬────────┘
              │                    │                    │
              ▼                    ▼                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                         BLENDSHAPE DEFORMATION                               │
│                                                                              │
│  V = T + B_S × β + B_E × ψ + B_P × Σ(R_n(θ_n) - I)                          │
│                                                                              │
│  Where:                                                                      │
│    T     = Template vertices (5,023 × 3)                                    │
│    B_S   = Shape blendshapes (5,023 × 3 × 100)                              │
│    B_E   = Expression blendshapes (5,023 × 3 × 50)                          │
│    B_P   = Pose blendshapes (5,023 × 3 × 36)                                │
│    R_n   = Rotation matrix for joint n                                       │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                      LINEAR BLEND SKINNING (LBS)                             │
│                                                                              │
│  V' = Σ w_k × G_k × V                                                       │
│                                                                              │
│  5 Joints:                                                                   │
│    0 = Root (head orientation)                                              │
│    1 = Neck                                                                  │
│    2 = Jaw                                                                   │
│    3 = Left eye                                                              │
│    4 = Right eye                                                             │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
                         Output: Mesh (5,023 vertices)

Parameter Specifications#

Parameter Shape Range Description
shape (100,) [-3, 3] Identity PCA coefficients
expression (50,) [-3, 3] Expression blendshapes
pose (6,) axis-angle Global head + jaw rotation
neck_pose (3,) axis-angle Neck articulation
eye_pose (6,) axis-angle Left and right eye gaze

Expression Blendshapes Mapping#

Index Range Description Examples
0-9 Eyebrow movements Raise, furrow, inner raise
10-19 Eye movements Blink, squint, widen
20-29 Mouth shapes Open, smile, frown
30-39 Lip movements Pucker, stretch, roll
40-49 Jaw and cheek Jaw open, cheek puff/suck

Model Files#

text
s3://serwaa-models/flame/v2023.1.0/
├── flame2023.pkl                 # 200 MB - Main FLAME model
├── FLAME_masks.pkl               # 5 MB - Region segmentation masks
├── flame_static_embedding.pkl    # 1 MB - Static pose embedding
├── landmark_embedding.npy        # 10 KB - 68-point landmark positions
└── texture_mean.npy              # 50 MB - Mean texture (optional)

FLAME to FACS Mapping#

The platform maps FLAME expression coefficients to FACS Action Units:

FLAME Expression FACS AU Description
exp[0-2] AU1+2 Inner/outer brow
exp[3-5] AU4 Brow lowerer
exp[10-12] AU5+7 Upper lid / Squint
exp[20-22] AU12+15 Lip corner pull/dep
exp[25-27] AU17+23 Chin raise / Pucker
exp[40-42] AU26+27 Jaw drop / Stretch

Lip Sync Model#

The lip sync model generates accurate mouth movements from audio input, synchronized with speech phonemes.

Architecture#

text
┌─────────────────────────────────────────────────────────────────────────────┐
│                      LIP SYNC PIPELINE                                       │
└─────────────────────────────────────────────────────────────────────────────┘

     Audio Input (16kHz)
            │
            ▼
┌──────────────────────────────────────┐
│        MEL SPECTROGRAM               │
│  window: 25ms, hop: 10ms, bins: 80   │
└─────────────────┬────────────────────┘
                  │
                  ▼
┌──────────────────────────────────────┐       ┌──────────────────────────────┐
│        AUDIO ENCODER                 │       │    REFERENCE FACE            │
│  2D Conv + ResBlocks + LSTM          │       │    (Identity encoder)        │
│  Output: (T, 512) embeddings         │       │    Output: (512,) embedding  │
└─────────────────┬────────────────────┘       └─────────────────┬────────────┘
                  │                                              │
                  └──────────────────┬───────────────────────────┘
                                     │
                                     ▼
                     ┌───────────────────────────────────────┐
                     │          FUSION LAYER                 │
                     │  Concatenate + MLP                    │
                     │  Output: (T, 512)                     │
                     └───────────────────┬───────────────────┘
                                         │
                                         ▼
                     ┌───────────────────────────────────────┐
                     │          LIP DECODER                  │
                     │  Deconv layers → lip region           │
                     │  Output: FLAME expression deltas      │
                     │          or direct vertex offsets     │
                     └───────────────────┬───────────────────┘
                                         │
                                         ▼
                            Lip Motion Parameters

Viseme Output#

The model outputs 15 viseme classes with timing:

Viseme Phonemes Mouth Shape
sil silence Closed, neutral
PP p, b, m Lips pressed
FF f, v Lower lip under teeth
TH th (θ, ð) Tongue between teeth
DD t, d, n, l Tongue tip up
kk k, g, ng Back tongue raised
CH ch, j, sh Lips rounded, slight
SS s, z Teeth together
nn n Mouth slightly open
RR r Lips slightly rounded
aa a, ah, ar Mouth wide open
E e, eh Mouth medium, teeth
I i, ih Lips spread
O o, oh, aw Lips rounded, open
U u, oo Lips pursed

Audio Processing Parameters#

Parameter Value Description
Sample Rate 16,000 Hz Standard for speech
Mel Bins 80 Frequency resolution
Window Size 25 ms 400 samples
Hop Length 10 ms 160 samples
Window Type Hann Smooth windowing
Pre-emphasis 0.97 High-frequency boost
Normalization Per-batch Zero mean, unit variance

Voice Models#

Whisper STT#

OpenAI Whisper Large v3 provides state-of-the-art speech-to-text transcription.

Model Specifications#

Property Value
Model Whisper Large v3
Parameters 1.5 billion
Size 3 GB (FP16)
Languages 99
WER (English) 4.2%
Latency 50 ms / utterance
Framework PyTorch

Architecture#

text
┌─────────────────────────────────────────────────────────────────────────────┐
│                    WHISPER ARCHITECTURE                                      │
└─────────────────────────────────────────────────────────────────────────────┘

Audio (16kHz, 30s max)
        │
        ▼
┌───────────────────────────────────────┐
│      LOG-MEL SPECTROGRAM              │
│  80 mel bins, 25ms window, 10ms hop   │
│  Output: (3000, 80)                   │
└───────────────────┬───────────────────┘
                    │
                    ▼
┌───────────────────────────────────────┐
│      AUDIO ENCODER                    │
│  2 Conv1D layers + 32 Transformer     │
│  blocks with 20 attention heads       │
│  Hidden dim: 1280                     │
│  Output: (1500, 1280)                 │
└───────────────────┬───────────────────┘
                    │
                    ▼
┌───────────────────────────────────────┐
│      TEXT DECODER (Autoregressive)    │
│  32 Transformer decoder blocks        │
│  Cross-attention to encoder output    │
│  Hidden dim: 1280                     │
│  Vocab size: 51,864                   │
└───────────────────┬───────────────────┘
                    │
                    ▼
           Transcribed Text

Quantization Options#

Variant Size Speedup Quality Impact
FP32 6 GB 1.0x Baseline
FP16 3 GB 1.5x None
INT8 1.5 GB 2.5x Minimal (<1%)
INT4 0.8 GB 3.5x Small (~2%)

Usage#

python
from transformers import WhisperProcessor, WhisperForConditionalGeneration
import torch

processor = WhisperProcessor.from_pretrained("openai/whisper-large-v3")
model = WhisperForConditionalGeneration.from_pretrained(
    "openai/whisper-large-v3",
    torch_dtype=torch.float16,
    device_map="cuda"
)

def transcribe(audio_array, sample_rate=16000):
    inputs = processor(
        audio_array,
        sampling_rate=sample_rate,
        return_tensors="pt"
    ).input_features.to("cuda", torch.float16)

    generated_ids = model.generate(inputs, max_new_tokens=256)
    transcription = processor.batch_decode(
        generated_ids,
        skip_special_tokens=True
    )[0]
    return transcription

XTTS v2 TTS#

XTTS v2 provides high-quality text-to-speech with voice cloning capabilities.

Model Specifications#

Property Value
Model Coqui XTTS v2
Size 2 GB
Languages 19
Voice Cloning Yes (6s reference)
RTF 0.05 (20x real-time)
Latency 80 ms
Sample Rate 24,000 Hz

Supported Languages#

Code Language Code Language
en English ko Korean
es Spanish tr Turkish
fr French ru Russian
de German nl Dutch
it Italian cs Czech
pt Portuguese ar Arabic
pl Polish zh Chinese
ja Japanese hu Hungarian
hi Hindi

Voice Cloning#

python
from TTS.tts.configs.xtts_config import XttsConfig
from TTS.tts.models.xtts import Xtts

config = XttsConfig()
config.load_json("config.json")

model = Xtts.init_from_config(config)
model.load_checkpoint(config, checkpoint_dir="models/xtts-v2/")
model.cuda()

# Clone voice from reference audio
gpt_cond_latent, speaker_embedding = model.get_conditioning_latents(
    audio_path="reference_voice.wav"
)

# Synthesize with cloned voice
outputs = model.synthesize(
    text="Hello, this is my cloned voice!",
    gpt_cond_latent=gpt_cond_latent,
    speaker_embedding=speaker_embedding,
    language="en"
)

audio = outputs["wav"]  # (N,) numpy array, 24kHz

Perception Models#

RetinaFace Detection#

High-accuracy face detection with 5-point landmark localization.

Model Specifications#

Property Value
Model RetinaFace ResNet-50
Size 100 MB
mAP (WIDER FACE) 91.4%
Latency 8 ms @ 640×640
Output Boxes + 5 landmarks

Architecture#

text
┌─────────────────────────────────────────────────────────────────────────────┐
│                    RETINAFACE ARCHITECTURE                                   │
└─────────────────────────────────────────────────────────────────────────────┘

           Input Image (H × W × 3)
                    │
                    ▼
┌───────────────────────────────────────┐
│      BACKBONE (ResNet-50)             │
│  C2: 1/4, C3: 1/8, C4: 1/16, C5: 1/32│
└───────────────────┬───────────────────┘
                    │
                    ▼
┌───────────────────────────────────────┐
│      FEATURE PYRAMID NETWORK          │
│  P2, P3, P4, P5 with lateral conn.   │
└───────────────────┬───────────────────┘
                    │
        ┌───────────┼───────────┐
        │           │           │
        ▼           ▼           ▼
   ┌─────────┐ ┌─────────┐ ┌─────────┐
   │  Class  │ │  BBox   │ │Landmark │
   │  Head   │ │  Head   │ │  Head   │
   │ (face/bg)│ │ (4 coords)│ │(10 coords)│
   └─────────┘ └─────────┘ └─────────┘

Output Format#

python
{
    "boxes": [[x1, y1, x2, y2], ...],      # Face bounding boxes
    "scores": [0.98, 0.95, ...],            # Confidence scores
    "landmarks": [                           # 5-point landmarks
        [[left_eye_x, left_eye_y],
         [right_eye_x, right_eye_y],
         [nose_x, nose_y],
         [left_mouth_x, left_mouth_y],
         [right_mouth_x, right_mouth_y]],
        ...
    ]
}

FER+ Emotion Recognition#

Facial Emotion Recognition with 8 emotion classes.

Model Specifications#

Property Value
Model FER+ (ResNet-based)
Size 50 MB
Classes 8 emotions
Accuracy 87.4% (FER2013+)
Latency 10 ms
Input Size 48×48 grayscale

Emotion Classes#

Index Emotion Valence Arousal
0 Neutral 0.0 0.0
1 Happiness +1.0 +0.5
2 Surprise +0.5 +1.0
3 Sadness -0.8 -0.3
4 Anger -0.8 +0.8
5 Disgust -0.7 +0.3
6 Fear -0.5 +0.9
7 Contempt -0.3 +0.2

Training Procedures#

3DGS Avatar Training#

Data Requirements#

Dataset Component Size Description
Multi-view images 100-500 Images from different angles
Camera calibration JSON Intrinsics + extrinsics
Point cloud init 10K-50K pts Initial Gaussian positions
Reference FLAME PKL Identity-specific FLAME

Training Pipeline#

text
┌─────────────────────────────────────────────────────────────────────────────┐
│                    3DGS TRAINING PIPELINE                                    │
└─────────────────────────────────────────────────────────────────────────────┘

Phase 1: Initialization (5 minutes)
├── Load point cloud from COLMAP/structure-from-motion
├── Initialize Gaussians at point positions
├── Set initial scales, rotations, opacities
└── Initialize SH coefficients from colors

Phase 2: Coarse Training (30 minutes, 5K iterations)
├── Optimize positions, rotations, scales
├── Learning rate: 1e-3 for positions
├── Densification every 100 iterations
├── Pruning Gaussians with opacity < 0.005
└── Loss: L1 + SSIM

Phase 3: Fine Training (60 minutes, 25K iterations)
├── Reduce learning rate to 1e-4
├── Optimize SH coefficients
├── Disable densification
├── Add LPIPS perceptual loss
└── Total Gaussians: ~100K

Phase 4: Refinement (30 minutes, 10K iterations)
├── Learning rate: 1e-5
├── Focus on fine details
├── Export checkpoint
└── Validate on held-out views

Training Configuration#

yaml
training:
  iterations: 40000
  batch_size: 1

  learning_rates:
    position: 0.00016
    feature_dc: 0.0025
    feature_rest: 0.000125
    opacity: 0.05
    scaling: 0.005
    rotation: 0.001

  densification:
    start: 500
    end: 15000
    interval: 100
    gradient_threshold: 0.0002

  pruning:
    opacity_threshold: 0.005
    screen_size_threshold: 20 # pixels

  loss:
    l1_weight: 0.8
    ssim_weight: 0.2
    lpips_weight: 0.1 # Added after 15K iterations

Hardware Requirements#

Component Minimum Recommended
GPU RTX 3080 (10G) A100 (40G)
CPU 8 cores 16 cores
RAM 32 GB 64 GB
Storage 100 GB SSD 500 GB NVMe
Train Time 2-4 hours 1-2 hours

Voice Model Fine-Tuning#

XTTS Voice Cloning#

python
from TTS.tts.configs.xtts_config import XttsConfig
from TTS.tts.models.xtts import Xtts

# Fine-tuning for specific voice
config = XttsConfig()
config.audio.sample_rate = 24000
config.model_args.gpt_cond_len = 6  # 6 seconds conditioning

# Training parameters
training_config = {
    "epochs": 100,
    "batch_size": 4,
    "learning_rate": 1e-5,
    "warmup_steps": 1000,
    "gradient_clip": 1.0,
    "eval_interval": 500,
}

# Dataset: 30+ minutes of target voice
# Format: audio files + transcriptions

Performance Specifications#

Latency Benchmarks#

Model Input Size GPU Latency (ms) Throughput
3DGS Render 1080p RTX 3080 22 45 FPS
3DGS Render 1080p A100 12 83 FPS
FLAME Forward Batch 1 RTX 3080 5 200 FPS
Whisper Large v3 10s audio RTX 3080 450 22x RT
Whisper Large v3 10s audio A100 180 55x RT
XTTS v2 50 words RTX 3080 850 20x RT
RetinaFace 640×640 RTX 3080 8 125 FPS
FER+ 48×48 RTX 3080 2 500 FPS

Memory Usage#

Model FP32 FP16 INT8
3DGS (100K) 8.0 GB 4.0 GB 2.5 GB
FLAME 0.5 GB 0.25 GB 0.15 GB
Whisper Large v3 6.0 GB 3.0 GB 1.5 GB
XTTS v2 4.0 GB 2.0 GB 1.0 GB
RetinaFace 0.2 GB 0.1 GB 0.05 GB

Quality Metrics#

Model Metric Value
3DGS PSNR 32-35 dB
3DGS SSIM 0.95+
3DGS LPIPS 0.05-0.10
Whisper WER 4.2%
XTTS v2 MOS 4.2/5.0
RetinaFace mAP 91.4%
FER+ Accuracy 87.4%

Model Optimization#

TensorRT Optimization#

python
import tensorrt as trt
import torch

def optimize_model_tensorrt(model, input_shape, fp16=True):
    """Convert PyTorch model to TensorRT engine."""

    # Export to ONNX
    dummy_input = torch.randn(*input_shape).cuda()
    torch.onnx.export(
        model,
        dummy_input,
        "model.onnx",
        opset_version=17,
        do_constant_folding=True
    )

    # Build TensorRT engine
    logger = trt.Logger(trt.Logger.WARNING)
    builder = trt.Builder(logger)
    network = builder.create_network(
        1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH)
    )
    parser = trt.OnnxParser(network, logger)

    with open("model.onnx", "rb") as f:
        parser.parse(f.read())

    config = builder.create_builder_config()
    config.set_memory_pool_limit(trt.MemoryPoolType.WORKSPACE, 1 << 30)

    if fp16:
        config.set_flag(trt.BuilderFlag.FP16)

    engine = builder.build_serialized_network(network, config)
    return engine

Quantization#

python
import torch
from torch.quantization import quantize_dynamic

# Dynamic quantization (CPU)
quantized_model = quantize_dynamic(
    model,
    {torch.nn.Linear, torch.nn.LSTM},
    dtype=torch.qint8
)

# Static quantization with calibration
from torch.ao.quantization import get_default_qconfig_mapping

qconfig_mapping = get_default_qconfig_mapping("x86")
prepared_model = prepare_fx(model, qconfig_mapping, example_inputs)

# Calibrate with representative data
for batch in calibration_loader:
    prepared_model(batch)

quantized_model = convert_fx(prepared_model)

Deployment Guidelines#

Model Loading Best Practices#

python
import asyncio
from concurrent.futures import ThreadPoolExecutor

class ModelLoader:
    """Async model loader with caching."""

    def __init__(self, cache_dir: str, max_cache_size_gb: float = 10.0):
        self.cache_dir = cache_dir
        self.max_cache_size = max_cache_size_gb * 1e9
        self.loaded_models = {}
        self.executor = ThreadPoolExecutor(max_workers=4)

    async def load_model(self, model_name: str, version: str):
        cache_key = f"{model_name}:{version}"

        if cache_key in self.loaded_models:
            return self.loaded_models[cache_key]

        # Load in background thread
        loop = asyncio.get_event_loop()
        model = await loop.run_in_executor(
            self.executor,
            self._load_model_sync,
            model_name,
            version
        )

        self.loaded_models[cache_key] = model
        return model

    def _load_model_sync(self, model_name: str, version: str):
        # Load from S3 or local cache
        pass

Health Checks#

python
from fastapi import FastAPI, Response

app = FastAPI()

@app.get("/health")
async def health_check():
    return {"status": "healthy"}

@app.get("/ready")
async def readiness_check():
    # Check all models are loaded
    checks = {
        "3dgs_model": model_loader.is_loaded("3dgs"),
        "whisper_model": model_loader.is_loaded("whisper"),
        "flame_model": model_loader.is_loaded("flame"),
    }

    all_ready = all(checks.values())
    status_code = 200 if all_ready else 503

    return Response(
        content=json.dumps({"checks": checks}),
        status_code=status_code
    )

GPU Memory Management#

python
import torch
import gc

def clear_gpu_memory():
    """Force GPU memory cleanup."""
    gc.collect()
    torch.cuda.empty_cache()
    torch.cuda.synchronize()

def get_gpu_memory_usage():
    """Get current GPU memory usage in GB."""
    return torch.cuda.memory_allocated() / 1e9

def get_gpu_memory_reserved():
    """Get reserved GPU memory in GB."""
    return torch.cuda.memory_reserved() / 1e9


Revision History#

Version Date Author Changes
1.0.0 2026-01-31 Claude Opus Initial ML documentation