# Psyche Domain: ML Model Documentation

This document provides comprehensive documentation for all machine learning
models used in the Psyche (Serwaa) AI Virtual Assistant Platform, including
architecture details, training procedures, and performance specifications.

## Table of Contents

1. [Model Overview](#model-overview)
2. [Avatar Models](#avatar-models)
   - [3D Gaussian Splatting](#3d-gaussian-splatting)
   - [FLAME Morphable Model](#flame-morphable-model)
   - [Lip Sync Model](#lip-sync-model)
3. [Voice Models](#voice-models)
   - [Whisper STT](#whisper-stt)
   - [XTTS v2 TTS](#xtts-v2-tts)
4. [Perception Models](#perception-models)
   - [RetinaFace Detection](#retinaface-detection)
   - [FER+ Emotion Recognition](#fer-emotion-recognition)
5. [Training Procedures](#training-procedures)
6. [Performance Specifications](#performance-specifications)
7. [Model Optimization](#model-optimization)
8. [Deployment Guidelines](#deployment-guidelines)

---

## Model Overview

The Psyche platform employs 50+ ML models organized into four primary
categories:

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                         ML MODEL HIERARCHY                                   │
└─────────────────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────────────────┐
│                             AVATAR MODELS                                    │
│  ┌──────────────────┐  ┌──────────────────┐  ┌──────────────────┐          │
│  │  3D Gaussian     │  │  FLAME           │  │  Lip Sync        │          │
│  │  Splatting       │  │  Morphable       │  │                  │          │
│  │                  │  │                  │  │                  │          │
│  │  • Rendering     │  │  • Face mesh     │  │  • Audio→Viseme  │          │
│  │  • View synth    │  │  • Expressions   │  │  • Synchronizer  │          │
│  │  • Appearance    │  │  • Identity      │  │  • Encoder/Dec   │          │
│  └──────────────────┘  └──────────────────┘  └──────────────────┘          │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                             VOICE MODELS                                     │
│  ┌──────────────────┐  ┌──────────────────┐  ┌──────────────────┐          │
│  │  Whisper         │  │  XTTS v2         │  │  Voice Cloning   │          │
│  │  (STT)           │  │  (TTS)           │  │                  │          │
│  │                  │  │                  │  │                  │          │
│  │  • Transcription │  │  • Synthesis     │  │  • Embedding     │          │
│  │  • 99 languages  │  │  • 19 languages  │  │  • Speaker ID    │          │
│  │  • 4.2% WER      │  │  • Voice clone   │  │  • Adaptation    │          │
│  └──────────────────┘  └──────────────────┘  └──────────────────┘          │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                           PERCEPTION MODELS                                  │
│  ┌──────────────────┐  ┌──────────────────┐  ┌──────────────────┐          │
│  │  RetinaFace      │  │  MediaPipe       │  │  FER+            │          │
│  │  (Detection)     │  │  (Landmarks)     │  │  (Emotion)       │          │
│  │                  │  │                  │  │                  │          │
│  │  • Face boxes    │  │  • 468 points    │  │  • 8 emotions    │          │
│  │  • 0.91 mAP      │  │  • Real-time     │  │  • Confidence    │          │
│  │  • Multi-face    │  │  • 3D coords     │  │  • Valence       │          │
│  └──────────────────┘  └──────────────────┘  └──────────────────┘          │
└─────────────────────────────────────────────────────────────────────────────┘
```

### Model Summary Table

| Model            | Category   | Size   | VRAM   | Latency | Framework  |
| ---------------- | ---------- | ------ | ------ | ------- | ---------- |
| 3DGS Avatar      | Avatar     | 1.5 GB | 8 GB   | 20 ms   | PyTorch    |
| FLAME 2023       | Avatar     | 500 MB | 4 GB   | 15 ms   | PyTorch    |
| Lip Sync         | Avatar     | 100 MB | 2 GB   | 10 ms   | PyTorch    |
| Whisper Large v3 | Voice      | 3 GB   | 6 GB   | 50 ms   | PyTorch    |
| XTTS v2          | Voice      | 2 GB   | 4 GB   | 80 ms   | PyTorch    |
| RetinaFace       | Perception | 100 MB | 2 GB   | 8 ms    | ONNX       |
| MediaPipe Mesh   | Perception | 4 MB   | 0.5 GB | 2 ms    | TensorFlow |
| FER+             | Perception | 50 MB  | 2 GB   | 10 ms   | PyTorch    |

---

## Avatar Models

### 3D Gaussian Splatting

3D Gaussian Splatting (3DGS) is the core rendering technology for real-time
avatar generation. It represents scenes as collections of 3D Gaussian primitives
that can be efficiently rasterized.

#### Architecture

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                    3DGS ARCHITECTURE                                         │
└─────────────────────────────────────────────────────────────────────────────┘

                         Input: Camera Pose (4×4)
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                        GAUSSIAN REPRESENTATION                               │
│                                                                              │
│  For each Gaussian g_i (i = 1 to N, typically N = 50,000 - 100,000):        │
│                                                                              │
│  ┌──────────────────┐  ┌──────────────────┐  ┌──────────────────┐          │
│  │  Position μ_i    │  │  Covariance Σ_i  │  │  Appearance      │          │
│  │  (3D center)     │  │  (3D ellipsoid)  │  │                  │          │
│  │                  │  │                  │  │                  │          │
│  │  x, y, z         │  │  Rotation (quat) │  │  Opacity α       │          │
│  │  ∈ ℝ³            │  │  Scale (3D)      │  │  SH coefficients │          │
│  └──────────────────┘  └──────────────────┘  └──────────────────┘          │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                    DIFFERENTIABLE RASTERIZATION                              │
│                                                                              │
│  1. PROJECT Gaussians to 2D screen space                                    │
│     μ'_i = K × [R|t] × μ_i        (perspective projection)                  │
│     Σ'_i = J × Σ_i × J^T          (covariance transformation)               │
│                                                                              │
│  2. TILE-BASED SORTING (16×16 pixel tiles)                                  │
│     • Assign Gaussians to overlapping tiles                                 │
│     • Sort by depth within each tile                                        │
│                                                                              │
│  3. ALPHA COMPOSITING (front-to-back)                                       │
│     C(p) = Σ c_i × α_i × Π(1 - α_j)   for j < i                            │
│                                                                              │
│  4. SPHERICAL HARMONICS for view-dependent color                            │
│     c_i(d) = SH(d) · coefficients_i                                         │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
                         Output: RGB Image (H × W × 3)
```

#### Gaussian Parameters

| Parameter   | Shape   | Type    | Description                    |
| ----------- | ------- | ------- | ------------------------------ |
| `positions` | (N, 3)  | float32 | 3D center coordinates (x,y,z)  |
| `rotations` | (N, 4)  | float32 | Quaternion (w,x,y,z)           |
| `scales`    | (N, 3)  | float32 | Anisotropic scaling (sx,sy,sz) |
| `opacities` | (N, 1)  | float32 | Alpha value [0, 1]             |
| `sh_dc`     | (N, 3)  | float32 | 0th-order SH (base color)      |
| `sh_rest`   | (N, 45) | float32 | 1st-3rd order SH coefficients  |

#### Spherical Harmonics Degrees

| Degree | Coefficients | Visual Quality    | Use Case       |
| ------ | ------------ | ----------------- | -------------- |
| 0      | 3            | Flat shading      | Preview/Debug  |
| 1      | 12           | Basic directional | Low quality    |
| 2      | 27           | Soft specular     | Medium quality |
| 3      | 48           | Full appearance   | Production     |

#### Model Files

```
s3://serwaa-models/avatar/3dgs/v1.2.0/
├── core.pt.gz                    # 800 MB - Positions, rotations, scales, opacities
├── sh_coefficients.pt.gz         # 600 MB - Full SH coefficients
├── metadata.json                 # Model metadata and config
└── camera_calibration.json       # Reference camera parameters
```

#### Memory Requirements

| Resolution | Gaussians | VRAM (FP16) | VRAM (FP32) | Target FPS |
| ---------- | --------- | ----------- | ----------- | ---------- |
| 512×512    | 50,000    | 1.5 GB      | 3.0 GB      | 200+       |
| 720p       | 75,000    | 2.5 GB      | 5.0 GB      | 100+       |
| 1080p      | 100,000   | 4.0 GB      | 8.0 GB      | 45+        |
| 4K         | 150,000   | 8.0 GB      | 16.0 GB     | 15+        |

#### Code Example

```python
import torch
from avatar_engine.models.gaussian import GaussianModel

# Initialize model
model = GaussianModel(
    sh_degree=3,
    device="cuda:0"
)

# Load checkpoint
await model.load_checkpoint("s3://serwaa-models/avatar/3dgs/v1.2.0/")

# Render frame
camera_pose = torch.eye(4, device="cuda:0")  # Identity pose
camera_intrinsics = {
    "fx": 500, "fy": 500,
    "cx": 512, "cy": 512,
    "width": 1024, "height": 1024
}

rgb_image = model.render(
    camera_pose=camera_pose,
    intrinsics=camera_intrinsics
)  # Returns (1024, 1024, 3) tensor
```

---

### FLAME Morphable Model

FLAME (Faces Learned with an Articulated Model and Expressions) is a statistical
3D head model that enables identity-independent expression animation.

#### Architecture

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                       FLAME MODEL ARCHITECTURE                               │
└─────────────────────────────────────────────────────────────────────────────┘

      ┌────────────────┐   ┌────────────────┐   ┌────────────────┐
      │  Shape β       │   │ Expression ψ   │   │   Pose θ       │
      │  (100 dims)    │   │  (50 dims)     │   │  (15 dims)     │
      └───────┬────────┘   └───────┬────────┘   └───────┬────────┘
              │                    │                    │
              ▼                    ▼                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                         BLENDSHAPE DEFORMATION                               │
│                                                                              │
│  V = T + B_S × β + B_E × ψ + B_P × Σ(R_n(θ_n) - I)                          │
│                                                                              │
│  Where:                                                                      │
│    T     = Template vertices (5,023 × 3)                                    │
│    B_S   = Shape blendshapes (5,023 × 3 × 100)                              │
│    B_E   = Expression blendshapes (5,023 × 3 × 50)                          │
│    B_P   = Pose blendshapes (5,023 × 3 × 36)                                │
│    R_n   = Rotation matrix for joint n                                       │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                      LINEAR BLEND SKINNING (LBS)                             │
│                                                                              │
│  V' = Σ w_k × G_k × V                                                       │
│                                                                              │
│  5 Joints:                                                                   │
│    0 = Root (head orientation)                                              │
│    1 = Neck                                                                  │
│    2 = Jaw                                                                   │
│    3 = Left eye                                                              │
│    4 = Right eye                                                             │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
                         Output: Mesh (5,023 vertices)
```

#### Parameter Specifications

| Parameter    | Shape  | Range      | Description                |
| ------------ | ------ | ---------- | -------------------------- |
| `shape`      | (100,) | [-3, 3]    | Identity PCA coefficients  |
| `expression` | (50,)  | [-3, 3]    | Expression blendshapes     |
| `pose`       | (6,)   | axis-angle | Global head + jaw rotation |
| `neck_pose`  | (3,)   | axis-angle | Neck articulation          |
| `eye_pose`   | (6,)   | axis-angle | Left and right eye gaze    |

#### Expression Blendshapes Mapping

| Index Range | Description       | Examples                   |
| ----------- | ----------------- | -------------------------- |
| 0-9         | Eyebrow movements | Raise, furrow, inner raise |
| 10-19       | Eye movements     | Blink, squint, widen       |
| 20-29       | Mouth shapes      | Open, smile, frown         |
| 30-39       | Lip movements     | Pucker, stretch, roll      |
| 40-49       | Jaw and cheek     | Jaw open, cheek puff/suck  |

#### Model Files

```
s3://serwaa-models/flame/v2023.1.0/
├── flame2023.pkl                 # 200 MB - Main FLAME model
├── FLAME_masks.pkl               # 5 MB - Region segmentation masks
├── flame_static_embedding.pkl    # 1 MB - Static pose embedding
├── landmark_embedding.npy        # 10 KB - 68-point landmark positions
└── texture_mean.npy              # 50 MB - Mean texture (optional)
```

#### FLAME to FACS Mapping

The platform maps FLAME expression coefficients to FACS Action Units:

| FLAME Expression | FACS AU | Description         |
| ---------------- | ------- | ------------------- |
| exp[0-2]         | AU1+2   | Inner/outer brow    |
| exp[3-5]         | AU4     | Brow lowerer        |
| exp[10-12]       | AU5+7   | Upper lid / Squint  |
| exp[20-22]       | AU12+15 | Lip corner pull/dep |
| exp[25-27]       | AU17+23 | Chin raise / Pucker |
| exp[40-42]       | AU26+27 | Jaw drop / Stretch  |

---

### Lip Sync Model

The lip sync model generates accurate mouth movements from audio input,
synchronized with speech phonemes.

#### Architecture

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                      LIP SYNC PIPELINE                                       │
└─────────────────────────────────────────────────────────────────────────────┘

     Audio Input (16kHz)
            │
            ▼
┌──────────────────────────────────────┐
│        MEL SPECTROGRAM               │
│  window: 25ms, hop: 10ms, bins: 80   │
└─────────────────┬────────────────────┘
                  │
                  ▼
┌──────────────────────────────────────┐       ┌──────────────────────────────┐
│        AUDIO ENCODER                 │       │    REFERENCE FACE            │
│  2D Conv + ResBlocks + LSTM          │       │    (Identity encoder)        │
│  Output: (T, 512) embeddings         │       │    Output: (512,) embedding  │
└─────────────────┬────────────────────┘       └─────────────────┬────────────┘
                  │                                              │
                  └──────────────────┬───────────────────────────┘
                                     │
                                     ▼
                     ┌───────────────────────────────────────┐
                     │          FUSION LAYER                 │
                     │  Concatenate + MLP                    │
                     │  Output: (T, 512)                     │
                     └───────────────────┬───────────────────┘
                                         │
                                         ▼
                     ┌───────────────────────────────────────┐
                     │          LIP DECODER                  │
                     │  Deconv layers → lip region           │
                     │  Output: FLAME expression deltas      │
                     │          or direct vertex offsets     │
                     └───────────────────┬───────────────────┘
                                         │
                                         ▼
                            Lip Motion Parameters
```

#### Viseme Output

The model outputs 15 viseme classes with timing:

| Viseme | Phonemes   | Mouth Shape           |
| ------ | ---------- | --------------------- |
| `sil`  | silence    | Closed, neutral       |
| `PP`   | p, b, m    | Lips pressed          |
| `FF`   | f, v       | Lower lip under teeth |
| `TH`   | th (θ, ð)  | Tongue between teeth  |
| `DD`   | t, d, n, l | Tongue tip up         |
| `kk`   | k, g, ng   | Back tongue raised    |
| `CH`   | ch, j, sh  | Lips rounded, slight  |
| `SS`   | s, z       | Teeth together        |
| `nn`   | n          | Mouth slightly open   |
| `RR`   | r          | Lips slightly rounded |
| `aa`   | a, ah, ar  | Mouth wide open       |
| `E`    | e, eh      | Mouth medium, teeth   |
| `I`    | i, ih      | Lips spread           |
| `O`    | o, oh, aw  | Lips rounded, open    |
| `U`    | u, oo      | Lips pursed           |

#### Audio Processing Parameters

| Parameter     | Value     | Description              |
| ------------- | --------- | ------------------------ |
| Sample Rate   | 16,000 Hz | Standard for speech      |
| Mel Bins      | 80        | Frequency resolution     |
| Window Size   | 25 ms     | 400 samples              |
| Hop Length    | 10 ms     | 160 samples              |
| Window Type   | Hann      | Smooth windowing         |
| Pre-emphasis  | 0.97      | High-frequency boost     |
| Normalization | Per-batch | Zero mean, unit variance |

---

## Voice Models

### Whisper STT

OpenAI Whisper Large v3 provides state-of-the-art speech-to-text transcription.

#### Model Specifications

| Property      | Value             |
| ------------- | ----------------- |
| Model         | Whisper Large v3  |
| Parameters    | 1.5 billion       |
| Size          | 3 GB (FP16)       |
| Languages     | 99                |
| WER (English) | 4.2%              |
| Latency       | 50 ms / utterance |
| Framework     | PyTorch           |

#### Architecture

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                    WHISPER ARCHITECTURE                                      │
└─────────────────────────────────────────────────────────────────────────────┘

Audio (16kHz, 30s max)
        │
        ▼
┌───────────────────────────────────────┐
│      LOG-MEL SPECTROGRAM              │
│  80 mel bins, 25ms window, 10ms hop   │
│  Output: (3000, 80)                   │
└───────────────────┬───────────────────┘
                    │
                    ▼
┌───────────────────────────────────────┐
│      AUDIO ENCODER                    │
│  2 Conv1D layers + 32 Transformer     │
│  blocks with 20 attention heads       │
│  Hidden dim: 1280                     │
│  Output: (1500, 1280)                 │
└───────────────────┬───────────────────┘
                    │
                    ▼
┌───────────────────────────────────────┐
│      TEXT DECODER (Autoregressive)    │
│  32 Transformer decoder blocks        │
│  Cross-attention to encoder output    │
│  Hidden dim: 1280                     │
│  Vocab size: 51,864                   │
└───────────────────┬───────────────────┘
                    │
                    ▼
           Transcribed Text
```

#### Quantization Options

| Variant | Size   | Speedup | Quality Impact |
| ------- | ------ | ------- | -------------- |
| FP32    | 6 GB   | 1.0x    | Baseline       |
| FP16    | 3 GB   | 1.5x    | None           |
| INT8    | 1.5 GB | 2.5x    | Minimal (<1%)  |
| INT4    | 0.8 GB | 3.5x    | Small (~2%)    |

#### Usage

```python
from transformers import WhisperProcessor, WhisperForConditionalGeneration
import torch

processor = WhisperProcessor.from_pretrained("openai/whisper-large-v3")
model = WhisperForConditionalGeneration.from_pretrained(
    "openai/whisper-large-v3",
    torch_dtype=torch.float16,
    device_map="cuda"
)

def transcribe(audio_array, sample_rate=16000):
    inputs = processor(
        audio_array,
        sampling_rate=sample_rate,
        return_tensors="pt"
    ).input_features.to("cuda", torch.float16)

    generated_ids = model.generate(inputs, max_new_tokens=256)
    transcription = processor.batch_decode(
        generated_ids,
        skip_special_tokens=True
    )[0]
    return transcription
```

---

### XTTS v2 TTS

XTTS v2 provides high-quality text-to-speech with voice cloning capabilities.

#### Model Specifications

| Property      | Value                |
| ------------- | -------------------- |
| Model         | Coqui XTTS v2        |
| Size          | 2 GB                 |
| Languages     | 19                   |
| Voice Cloning | Yes (6s reference)   |
| RTF           | 0.05 (20x real-time) |
| Latency       | 80 ms                |
| Sample Rate   | 24,000 Hz            |

#### Supported Languages

| Code | Language   | Code | Language  |
| ---- | ---------- | ---- | --------- |
| en   | English    | ko   | Korean    |
| es   | Spanish    | tr   | Turkish   |
| fr   | French     | ru   | Russian   |
| de   | German     | nl   | Dutch     |
| it   | Italian    | cs   | Czech     |
| pt   | Portuguese | ar   | Arabic    |
| pl   | Polish     | zh   | Chinese   |
| ja   | Japanese   | hu   | Hungarian |
| hi   | Hindi      |      |           |

#### Voice Cloning

```python
from TTS.tts.configs.xtts_config import XttsConfig
from TTS.tts.models.xtts import Xtts

config = XttsConfig()
config.load_json("config.json")

model = Xtts.init_from_config(config)
model.load_checkpoint(config, checkpoint_dir="models/xtts-v2/")
model.cuda()

# Clone voice from reference audio
gpt_cond_latent, speaker_embedding = model.get_conditioning_latents(
    audio_path="reference_voice.wav"
)

# Synthesize with cloned voice
outputs = model.synthesize(
    text="Hello, this is my cloned voice!",
    gpt_cond_latent=gpt_cond_latent,
    speaker_embedding=speaker_embedding,
    language="en"
)

audio = outputs["wav"]  # (N,) numpy array, 24kHz
```

---

## Perception Models

### RetinaFace Detection

High-accuracy face detection with 5-point landmark localization.

#### Model Specifications

| Property         | Value                |
| ---------------- | -------------------- |
| Model            | RetinaFace ResNet-50 |
| Size             | 100 MB               |
| mAP (WIDER FACE) | 91.4%                |
| Latency          | 8 ms @ 640×640       |
| Output           | Boxes + 5 landmarks  |

#### Architecture

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                    RETINAFACE ARCHITECTURE                                   │
└─────────────────────────────────────────────────────────────────────────────┘

           Input Image (H × W × 3)
                    │
                    ▼
┌───────────────────────────────────────┐
│      BACKBONE (ResNet-50)             │
│  C2: 1/4, C3: 1/8, C4: 1/16, C5: 1/32│
└───────────────────┬───────────────────┘
                    │
                    ▼
┌───────────────────────────────────────┐
│      FEATURE PYRAMID NETWORK          │
│  P2, P3, P4, P5 with lateral conn.   │
└───────────────────┬───────────────────┘
                    │
        ┌───────────┼───────────┐
        │           │           │
        ▼           ▼           ▼
   ┌─────────┐ ┌─────────┐ ┌─────────┐
   │  Class  │ │  BBox   │ │Landmark │
   │  Head   │ │  Head   │ │  Head   │
   │ (face/bg)│ │ (4 coords)│ │(10 coords)│
   └─────────┘ └─────────┘ └─────────┘
```

#### Output Format

```python
{
    "boxes": [[x1, y1, x2, y2], ...],      # Face bounding boxes
    "scores": [0.98, 0.95, ...],            # Confidence scores
    "landmarks": [                           # 5-point landmarks
        [[left_eye_x, left_eye_y],
         [right_eye_x, right_eye_y],
         [nose_x, nose_y],
         [left_mouth_x, left_mouth_y],
         [right_mouth_x, right_mouth_y]],
        ...
    ]
}
```

---

### FER+ Emotion Recognition

Facial Emotion Recognition with 8 emotion classes.

#### Model Specifications

| Property   | Value               |
| ---------- | ------------------- |
| Model      | FER+ (ResNet-based) |
| Size       | 50 MB               |
| Classes    | 8 emotions          |
| Accuracy   | 87.4% (FER2013+)    |
| Latency    | 10 ms               |
| Input Size | 48×48 grayscale     |

#### Emotion Classes

| Index | Emotion   | Valence | Arousal |
| ----- | --------- | ------- | ------- |
| 0     | Neutral   | 0.0     | 0.0     |
| 1     | Happiness | +1.0    | +0.5    |
| 2     | Surprise  | +0.5    | +1.0    |
| 3     | Sadness   | -0.8    | -0.3    |
| 4     | Anger     | -0.8    | +0.8    |
| 5     | Disgust   | -0.7    | +0.3    |
| 6     | Fear      | -0.5    | +0.9    |
| 7     | Contempt  | -0.3    | +0.2    |

---

## Training Procedures

### 3DGS Avatar Training

#### Data Requirements

| Dataset Component  | Size        | Description                  |
| ------------------ | ----------- | ---------------------------- |
| Multi-view images  | 100-500     | Images from different angles |
| Camera calibration | JSON        | Intrinsics + extrinsics      |
| Point cloud init   | 10K-50K pts | Initial Gaussian positions   |
| Reference FLAME    | PKL         | Identity-specific FLAME      |

#### Training Pipeline

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                    3DGS TRAINING PIPELINE                                    │
└─────────────────────────────────────────────────────────────────────────────┘

Phase 1: Initialization (5 minutes)
├── Load point cloud from COLMAP/structure-from-motion
├── Initialize Gaussians at point positions
├── Set initial scales, rotations, opacities
└── Initialize SH coefficients from colors

Phase 2: Coarse Training (30 minutes, 5K iterations)
├── Optimize positions, rotations, scales
├── Learning rate: 1e-3 for positions
├── Densification every 100 iterations
├── Pruning Gaussians with opacity < 0.005
└── Loss: L1 + SSIM

Phase 3: Fine Training (60 minutes, 25K iterations)
├── Reduce learning rate to 1e-4
├── Optimize SH coefficients
├── Disable densification
├── Add LPIPS perceptual loss
└── Total Gaussians: ~100K

Phase 4: Refinement (30 minutes, 10K iterations)
├── Learning rate: 1e-5
├── Focus on fine details
├── Export checkpoint
└── Validate on held-out views
```

#### Training Configuration

```yaml
training:
  iterations: 40000
  batch_size: 1

  learning_rates:
    position: 0.00016
    feature_dc: 0.0025
    feature_rest: 0.000125
    opacity: 0.05
    scaling: 0.005
    rotation: 0.001

  densification:
    start: 500
    end: 15000
    interval: 100
    gradient_threshold: 0.0002

  pruning:
    opacity_threshold: 0.005
    screen_size_threshold: 20 # pixels

  loss:
    l1_weight: 0.8
    ssim_weight: 0.2
    lpips_weight: 0.1 # Added after 15K iterations
```

#### Hardware Requirements

| Component  | Minimum        | Recommended |
| ---------- | -------------- | ----------- |
| GPU        | RTX 3080 (10G) | A100 (40G)  |
| CPU        | 8 cores        | 16 cores    |
| RAM        | 32 GB          | 64 GB       |
| Storage    | 100 GB SSD     | 500 GB NVMe |
| Train Time | 2-4 hours      | 1-2 hours   |

---

### Voice Model Fine-Tuning

#### XTTS Voice Cloning

```python
from TTS.tts.configs.xtts_config import XttsConfig
from TTS.tts.models.xtts import Xtts

# Fine-tuning for specific voice
config = XttsConfig()
config.audio.sample_rate = 24000
config.model_args.gpt_cond_len = 6  # 6 seconds conditioning

# Training parameters
training_config = {
    "epochs": 100,
    "batch_size": 4,
    "learning_rate": 1e-5,
    "warmup_steps": 1000,
    "gradient_clip": 1.0,
    "eval_interval": 500,
}

# Dataset: 30+ minutes of target voice
# Format: audio files + transcriptions
```

---

## Performance Specifications

### Latency Benchmarks

| Model            | Input Size | GPU      | Latency (ms) | Throughput |
| ---------------- | ---------- | -------- | ------------ | ---------- |
| 3DGS Render      | 1080p      | RTX 3080 | 22           | 45 FPS     |
| 3DGS Render      | 1080p      | A100     | 12           | 83 FPS     |
| FLAME Forward    | Batch 1    | RTX 3080 | 5            | 200 FPS    |
| Whisper Large v3 | 10s audio  | RTX 3080 | 450          | 22x RT     |
| Whisper Large v3 | 10s audio  | A100     | 180          | 55x RT     |
| XTTS v2          | 50 words   | RTX 3080 | 850          | 20x RT     |
| RetinaFace       | 640×640    | RTX 3080 | 8            | 125 FPS    |
| FER+             | 48×48      | RTX 3080 | 2            | 500 FPS    |

### Memory Usage

| Model            | FP32   | FP16    | INT8    |
| ---------------- | ------ | ------- | ------- |
| 3DGS (100K)      | 8.0 GB | 4.0 GB  | 2.5 GB  |
| FLAME            | 0.5 GB | 0.25 GB | 0.15 GB |
| Whisper Large v3 | 6.0 GB | 3.0 GB  | 1.5 GB  |
| XTTS v2          | 4.0 GB | 2.0 GB  | 1.0 GB  |
| RetinaFace       | 0.2 GB | 0.1 GB  | 0.05 GB |

### Quality Metrics

| Model      | Metric   | Value     |
| ---------- | -------- | --------- |
| 3DGS       | PSNR     | 32-35 dB  |
| 3DGS       | SSIM     | 0.95+     |
| 3DGS       | LPIPS    | 0.05-0.10 |
| Whisper    | WER      | 4.2%      |
| XTTS v2    | MOS      | 4.2/5.0   |
| RetinaFace | mAP      | 91.4%     |
| FER+       | Accuracy | 87.4%     |

---

## Model Optimization

### TensorRT Optimization

```python
import tensorrt as trt
import torch

def optimize_model_tensorrt(model, input_shape, fp16=True):
    """Convert PyTorch model to TensorRT engine."""

    # Export to ONNX
    dummy_input = torch.randn(*input_shape).cuda()
    torch.onnx.export(
        model,
        dummy_input,
        "model.onnx",
        opset_version=17,
        do_constant_folding=True
    )

    # Build TensorRT engine
    logger = trt.Logger(trt.Logger.WARNING)
    builder = trt.Builder(logger)
    network = builder.create_network(
        1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH)
    )
    parser = trt.OnnxParser(network, logger)

    with open("model.onnx", "rb") as f:
        parser.parse(f.read())

    config = builder.create_builder_config()
    config.set_memory_pool_limit(trt.MemoryPoolType.WORKSPACE, 1 << 30)

    if fp16:
        config.set_flag(trt.BuilderFlag.FP16)

    engine = builder.build_serialized_network(network, config)
    return engine
```

### Quantization

```python
import torch
from torch.quantization import quantize_dynamic

# Dynamic quantization (CPU)
quantized_model = quantize_dynamic(
    model,
    {torch.nn.Linear, torch.nn.LSTM},
    dtype=torch.qint8
)

# Static quantization with calibration
from torch.ao.quantization import get_default_qconfig_mapping

qconfig_mapping = get_default_qconfig_mapping("x86")
prepared_model = prepare_fx(model, qconfig_mapping, example_inputs)

# Calibrate with representative data
for batch in calibration_loader:
    prepared_model(batch)

quantized_model = convert_fx(prepared_model)
```

---

## Deployment Guidelines

### Model Loading Best Practices

```python
import asyncio
from concurrent.futures import ThreadPoolExecutor

class ModelLoader:
    """Async model loader with caching."""

    def __init__(self, cache_dir: str, max_cache_size_gb: float = 10.0):
        self.cache_dir = cache_dir
        self.max_cache_size = max_cache_size_gb * 1e9
        self.loaded_models = {}
        self.executor = ThreadPoolExecutor(max_workers=4)

    async def load_model(self, model_name: str, version: str):
        cache_key = f"{model_name}:{version}"

        if cache_key in self.loaded_models:
            return self.loaded_models[cache_key]

        # Load in background thread
        loop = asyncio.get_event_loop()
        model = await loop.run_in_executor(
            self.executor,
            self._load_model_sync,
            model_name,
            version
        )

        self.loaded_models[cache_key] = model
        return model

    def _load_model_sync(self, model_name: str, version: str):
        # Load from S3 or local cache
        pass
```

### Health Checks

```python
from fastapi import FastAPI, Response

app = FastAPI()

@app.get("/health")
async def health_check():
    return {"status": "healthy"}

@app.get("/ready")
async def readiness_check():
    # Check all models are loaded
    checks = {
        "3dgs_model": model_loader.is_loaded("3dgs"),
        "whisper_model": model_loader.is_loaded("whisper"),
        "flame_model": model_loader.is_loaded("flame"),
    }

    all_ready = all(checks.values())
    status_code = 200 if all_ready else 503

    return Response(
        content=json.dumps({"checks": checks}),
        status_code=status_code
    )
```

### GPU Memory Management

```python
import torch
import gc

def clear_gpu_memory():
    """Force GPU memory cleanup."""
    gc.collect()
    torch.cuda.empty_cache()
    torch.cuda.synchronize()

def get_gpu_memory_usage():
    """Get current GPU memory usage in GB."""
    return torch.cuda.memory_allocated() / 1e9

def get_gpu_memory_reserved():
    """Get reserved GPU memory in GB."""
    return torch.cuda.memory_reserved() / 1e9
```

---

## Related Documentation

- [ML Model Dependencies](./ml-model-dependencies.md) - Complete model inventory
- [Architecture Overview](./architecture.md) - System architecture
- [Model Registry](../../../serwaa/infrastructure/docker/model-registry.yaml) -
  Version control
- [Performance Baseline](./performance-baseline.md) - Performance benchmarks

---

## Revision History

| Version | Date       | Author      | Changes                  |
| ------- | ---------- | ----------- | ------------------------ |
| 1.0.0   | 2026-01-31 | Claude Opus | Initial ML documentation |
