Source:
oshun/minerva/minerva/multimedia/(23 files, 9,764 LOC),video/(38 files, 19,137 LOC),audio/(36 files, 11,687 LOC) Analysis date: 2026-02-28 Total: 97 files, 40,588 LOC
Pipeline Overview#
text
Course Content
↓
┌─── DIAGRAMS ───────────────────────────────────┐
│ Intent Detection → Style Resolution → Render │
│ → D3/SVG/HTML → Accessibility → Cache │
└────────────────────────────────────────────────┘
↓
┌─── AUDIO ──────────────────────────────────────┐
│ Content Analysis → Persona Selection → SSML │
│ → TTS (ChatterboxTTS/GPU) → Quality Pipeline │
│ → Learner Adaptation │
└────────────────────────────────────────────────┘
↓
┌─── PRESENTATIONS ──────────────────────────────┐
│ Keynote Compile → Slide Render → Audio Sync │
│ → HTML/CSS/JS Export │
└────────────────────────────────────────────────┘
↓
┌─── VIDEO ──────────────────────────────────────┐
│ Script → Storyboard → Visuals (GPU) + Audio │
│ → Timeline Composition → FFmpeg Render → QA │
└────────────────────────────────────────────────┘
↓
┌─── SYNCHRONIZATION ────────────────────────────┐
│ AV Synchronizer → Cross-media timing │
│ → Timeline validation → Final output │
└────────────────────────────────────────────────┘
Multimedia Module (23 files, 9,764 LOC)#
Key Components#
| File | LOC | Purpose |
|---|---|---|
| diagram_renderer.py | 1,240 | Diagram config → SVG/HTML animations |
| av_synchronizer.py | 1,184 | Audio-visual sync, TTS retry (exp backoff) |
| d3_renderer.py | 1,079 | D3.js interactive diagrams |
| diagram_pipeline.py | 987 | End-to-end diagram generation |
| diagram_runtime.py | 888 | Runtime execution, CSS/JS injection |
| presentation_renderer.py | 812 | Keynote HTML slide generation |
| educational_podcast_engine.py | 662 | Podcast with voice characters |
| keynote_compiler.py | 438 | Lesson → keynote with audio sync |
| diagram_versioning.py | 227 | Version control across refinement |
| diagram_validator.py | 217 | Data integrity, accessibility |
Diagram Types (D3Renderer)#
Gantt, Networks, Timelines, Hierarchies, Sankey, Treemaps, Heatmaps, Mind Maps, Flowcharts
Diagram Pipeline Stages#
- Intent Detection → concept/diagram type inference
- Style Resolution → educational style profiles
- Data Processing → sanitize, downsample, constrain
- Rendering → SVG/HTML (D3, PlotlyJS, custom)
- Accessibility → ARIA labels, data tables, alt text
- Caching → SHA256 digest for cache hits
- Versioning → track across refinement rounds
Audio Module (36 files, 11,687 LOC)#
Key Components#
| File | LOC | Purpose |
|---|---|---|
| tts_adapter.py | 838 | ChatterboxTTS integration, GPU offload |
| tts/ssml_builder.py | 788 | Emotion-aware SSML generation |
| tts/personas.py | 676 | 20+ voice personality profiles |
| emotion/learner_detector.py | 604 | Learner state detection |
| prosody/pitch_generator.py | 577 | Pitch curves for emphasis |
| emotion/content_analyzer.py | 545 | Emotion/emphasis analysis |
| tts/voice_cloning.py | 535 | Voice cloning/style transfer |
| audio_quality_pipeline.py | 414 | QA (intelligibility, pacing) |
Audio Pipeline Stages#
- Content Analysis → EmotionScores, EmphasisPoints, ToneRecommendations
- Persona Selection → voice personality + prosody settings
- SSML Generation → prosody, breaks, emphasis tags
- TTS Synthesis → ChatterboxTTS (GPU-optimized)
- Quality Assurance → intelligibility (RMS, speech ratio), pacing (WPM)
- Learner Adaptation → adjusted prosody for next narration
Voice System#
- 20+ personas with: voice_id, gender, accent, style, age, pitch/rate/expressiveness
- Emotion types: EXCITEMENT, CURIOSITY, SERIOUSNESS, ENCOURAGEMENT, EMPATHY, NEUTRAL
- Emphasis levels: NONE, REDUCED, MODERATE, STRONG
- Break strengths: X_WEAK, WEAK, MEDIUM, STRONG, X_STRONG
Video Module (38 files, 19,137 LOC)#
Key Components#
| File | LOC | Purpose |
|---|---|---|
| visuals/text_renderer.py | 1,068 | Text overlays for video |
| audio/video_tts.py | 1,013 | TTS with timing |
| visuals/avatar_generator.py | 1,003 | Animated avatars |
| rendering/ffmpeg_renderer.py | 916 | FFmpeg composition |
| animation/diagram_animator.py | 874 | Diagram keyframe animation |
| composition/timeline.py | 849 | Multi-track timeline sync |
| composition/transitions.py | 827 | Transition effects |
| rendering/subtitle_generator.py | 823 | SRT/VTT subtitles |
| qa/content_validator.py | 799 | Content validation |
| audio/mixer.py | 776 | Audio mixing (narration+music+SFX) |
| pipeline.py | 832 | Main orchestration |
Video Pipeline Phases#
text
INITIALIZATION → SCRIPT_GENERATION → STORYBOARD_GENERATION
→ VISUAL_GENERATION (GPU) → AUDIO_GENERATION
→ COMPOSITION → RENDERING (FFmpeg) → QA_VALIDATION → FINALIZATION
Timeline System#
- Media types: VIDEO, AUDIO, IMAGE, TEXT, SUBTITLE
- Track types: VIDEO, NARRATION, MUSIC, SFX, SUBTITLE, OVERLAY
- Sync strategies: AUDIO_MASTER, VIDEO_MASTER, MANUAL, TIMECODE
- Resolution presets: HD (1280x720), FULL_HD (1920x1080), UHD_4K (3840x2160)
External Dependencies#
| Dependency | Used For |
|---|---|
| ChatterboxTTS | Primary TTS engine |
| ESpeak | Fallback TTS |
| RunPod | GPU serverless for TTS/image |
| D3.js | Interactive diagrams |
| PlotlyJS | Chart rendering |
| FFmpeg | Video encoding |
| ComfyUI | Image/video generation workflows |
| Hunyuan | Avatar/video generation |
| Wav2Lip / LatentSync | Lip sync |
| librosa / scipy | Audio signal processing |
Output Formats#
- Diagrams: SVG, self-contained HTML, interactive D3
- Audio: WAV, MP3, M4A
- Presentations: HTML slides, PDF, PPTX
- Podcasts: MP3/M4A with multiple characters
- Video: MP4 (1080p default)
- Subtitles: SRT, VTT
- Textbooks: JSON, HTML, PDF, EPUB
Features to KEEP#
- AVSynchronizer — Production-grade TTS retry with jitter/backoff
- DiagramRenderer — SHA256 caching, educational style profiles
- D3Renderer — 9+ diagram types, self-contained HTML
- SSMLBuilder — Emotion-aware prosody, educational emphasis detection
- 20+ voice personas — Rich personality system
- Audio quality pipeline — Intelligibility (RMS), pacing (WPM) metrics
- Timeline composition — Multi-track sync (AUDIO_MASTER/VIDEO_MASTER)
- Video pipeline — Clear phase separation, status tracking
- Accessibility — ARIA labels, data tables, alt text injection
- Learner adaptation — Dynamic prosody based on engagement signals
Technical Debt to DISCARD#
- ComfyUI custom nodes — Standardize to ComfyUI API instead
- Scattered diagram files — 6+ locations (diagram_*.py, diagram/, multimedia/) → consolidate
- LipSync modules — Too specialized, make optional ComfyUI workflow
- Duplicate GPU utils — Audio and video both have GPU client code
- Hardcoded podcast styles — Move to configuration
- Avatar generators — Extract to separate @metis/avatar library
- Music licensing logic — Extract to separate service