# 25.1.1.5 — Minerva Multimedia Production Pipeline

> Source: `oshun/minerva/minerva/multimedia/` (23 files, 9,764 LOC), `video/`
> (38 files, 19,137 LOC), `audio/` (36 files, 11,687 LOC) Analysis date:
> 2026-02-28 Total: 97 files, 40,588 LOC

---

## Pipeline Overview

```
Course Content
  ↓
┌─── DIAGRAMS ───────────────────────────────────┐
│ Intent Detection → Style Resolution → Render   │
│ → D3/SVG/HTML → Accessibility → Cache          │
└────────────────────────────────────────────────┘
  ↓
┌─── AUDIO ──────────────────────────────────────┐
│ Content Analysis → Persona Selection → SSML    │
│ → TTS (ChatterboxTTS/GPU) → Quality Pipeline   │
│ → Learner Adaptation                            │
└────────────────────────────────────────────────┘
  ↓
┌─── PRESENTATIONS ──────────────────────────────┐
│ Keynote Compile → Slide Render → Audio Sync    │
│ → HTML/CSS/JS Export                            │
└────────────────────────────────────────────────┘
  ↓
┌─── VIDEO ──────────────────────────────────────┐
│ Script → Storyboard → Visuals (GPU) + Audio    │
│ → Timeline Composition → FFmpeg Render → QA    │
└────────────────────────────────────────────────┘
  ↓
┌─── SYNCHRONIZATION ────────────────────────────┐
│ AV Synchronizer → Cross-media timing           │
│ → Timeline validation → Final output           │
└────────────────────────────────────────────────┘
```

---

## Multimedia Module (23 files, 9,764 LOC)

### Key Components

| File                          | LOC   | Purpose                                    |
| ----------------------------- | ----- | ------------------------------------------ |
| diagram_renderer.py           | 1,240 | Diagram config → SVG/HTML animations       |
| av_synchronizer.py            | 1,184 | Audio-visual sync, TTS retry (exp backoff) |
| d3_renderer.py                | 1,079 | D3.js interactive diagrams                 |
| diagram_pipeline.py           | 987   | End-to-end diagram generation              |
| diagram_runtime.py            | 888   | Runtime execution, CSS/JS injection        |
| presentation_renderer.py      | 812   | Keynote HTML slide generation              |
| educational_podcast_engine.py | 662   | Podcast with voice characters              |
| keynote_compiler.py           | 438   | Lesson → keynote with audio sync           |
| diagram_versioning.py         | 227   | Version control across refinement          |
| diagram_validator.py          | 217   | Data integrity, accessibility              |

### Diagram Types (D3Renderer)

Gantt, Networks, Timelines, Hierarchies, Sankey, Treemaps, Heatmaps, Mind Maps,
Flowcharts

### Diagram Pipeline Stages

1. Intent Detection → concept/diagram type inference
2. Style Resolution → educational style profiles
3. Data Processing → sanitize, downsample, constrain
4. Rendering → SVG/HTML (D3, PlotlyJS, custom)
5. Accessibility → ARIA labels, data tables, alt text
6. Caching → SHA256 digest for cache hits
7. Versioning → track across refinement rounds

---

## Audio Module (36 files, 11,687 LOC)

### Key Components

| File                        | LOC | Purpose                                |
| --------------------------- | --- | -------------------------------------- |
| tts_adapter.py              | 838 | ChatterboxTTS integration, GPU offload |
| tts/ssml_builder.py         | 788 | Emotion-aware SSML generation          |
| tts/personas.py             | 676 | 20+ voice personality profiles         |
| emotion/learner_detector.py | 604 | Learner state detection                |
| prosody/pitch_generator.py  | 577 | Pitch curves for emphasis              |
| emotion/content_analyzer.py | 545 | Emotion/emphasis analysis              |
| tts/voice_cloning.py        | 535 | Voice cloning/style transfer           |
| audio_quality_pipeline.py   | 414 | QA (intelligibility, pacing)           |

### Audio Pipeline Stages

1. Content Analysis → EmotionScores, EmphasisPoints, ToneRecommendations
2. Persona Selection → voice personality + prosody settings
3. SSML Generation → prosody, breaks, emphasis tags
4. TTS Synthesis → ChatterboxTTS (GPU-optimized)
5. Quality Assurance → intelligibility (RMS, speech ratio), pacing (WPM)
6. Learner Adaptation → adjusted prosody for next narration

### Voice System

- **20+ personas** with: voice_id, gender, accent, style, age,
  pitch/rate/expressiveness
- **Emotion types:** EXCITEMENT, CURIOSITY, SERIOUSNESS, ENCOURAGEMENT, EMPATHY,
  NEUTRAL
- **Emphasis levels:** NONE, REDUCED, MODERATE, STRONG
- **Break strengths:** X_WEAK, WEAK, MEDIUM, STRONG, X_STRONG

---

## Video Module (38 files, 19,137 LOC)

### Key Components

| File                            | LOC   | Purpose                            |
| ------------------------------- | ----- | ---------------------------------- |
| visuals/text_renderer.py        | 1,068 | Text overlays for video            |
| audio/video_tts.py              | 1,013 | TTS with timing                    |
| visuals/avatar_generator.py     | 1,003 | Animated avatars                   |
| rendering/ffmpeg_renderer.py    | 916   | FFmpeg composition                 |
| animation/diagram_animator.py   | 874   | Diagram keyframe animation         |
| composition/timeline.py         | 849   | Multi-track timeline sync          |
| composition/transitions.py      | 827   | Transition effects                 |
| rendering/subtitle_generator.py | 823   | SRT/VTT subtitles                  |
| qa/content_validator.py         | 799   | Content validation                 |
| audio/mixer.py                  | 776   | Audio mixing (narration+music+SFX) |
| pipeline.py                     | 832   | Main orchestration                 |

### Video Pipeline Phases

```
INITIALIZATION → SCRIPT_GENERATION → STORYBOARD_GENERATION
→ VISUAL_GENERATION (GPU) → AUDIO_GENERATION
→ COMPOSITION → RENDERING (FFmpeg) → QA_VALIDATION → FINALIZATION
```

### Timeline System

- **Media types:** VIDEO, AUDIO, IMAGE, TEXT, SUBTITLE
- **Track types:** VIDEO, NARRATION, MUSIC, SFX, SUBTITLE, OVERLAY
- **Sync strategies:** AUDIO_MASTER, VIDEO_MASTER, MANUAL, TIMECODE
- **Resolution presets:** HD (1280x720), FULL_HD (1920x1080), UHD_4K (3840x2160)

---

## External Dependencies

| Dependency           | Used For                         |
| -------------------- | -------------------------------- |
| ChatterboxTTS        | Primary TTS engine               |
| ESpeak               | Fallback TTS                     |
| RunPod               | GPU serverless for TTS/image     |
| D3.js                | Interactive diagrams             |
| PlotlyJS             | Chart rendering                  |
| FFmpeg               | Video encoding                   |
| ComfyUI              | Image/video generation workflows |
| Hunyuan              | Avatar/video generation          |
| Wav2Lip / LatentSync | Lip sync                         |
| librosa / scipy      | Audio signal processing          |

---

## Output Formats

- **Diagrams:** SVG, self-contained HTML, interactive D3
- **Audio:** WAV, MP3, M4A
- **Presentations:** HTML slides, PDF, PPTX
- **Podcasts:** MP3/M4A with multiple characters
- **Video:** MP4 (1080p default)
- **Subtitles:** SRT, VTT
- **Textbooks:** JSON, HTML, PDF, EPUB

---

## Features to KEEP

1. **AVSynchronizer** — Production-grade TTS retry with jitter/backoff
2. **DiagramRenderer** — SHA256 caching, educational style profiles
3. **D3Renderer** — 9+ diagram types, self-contained HTML
4. **SSMLBuilder** — Emotion-aware prosody, educational emphasis detection
5. **20+ voice personas** — Rich personality system
6. **Audio quality pipeline** — Intelligibility (RMS), pacing (WPM) metrics
7. **Timeline composition** — Multi-track sync (AUDIO_MASTER/VIDEO_MASTER)
8. **Video pipeline** — Clear phase separation, status tracking
9. **Accessibility** — ARIA labels, data tables, alt text injection
10. **Learner adaptation** — Dynamic prosody based on engagement signals

## Technical Debt to DISCARD

1. **ComfyUI custom nodes** — Standardize to ComfyUI API instead
2. **Scattered diagram files** — 6+ locations (diagram\_\*.py, diagram/,
   multimedia/) → consolidate
3. **LipSync modules** — Too specialized, make optional ComfyUI workflow
4. **Duplicate GPU utils** — Audio and video both have GPU client code
5. **Hardcoded podcast styles** — Move to configuration
6. **Avatar generators** — Extract to separate @metis/avatar library
7. **Music licensing logic** — Extract to separate service
