Eve library

25.1.1.5 — Minerva Multimedia Production Pipeline

Guide

Revision 1

On this page

Source: oshun/minerva/minerva/multimedia/ (23 files, 9,764 LOC), video/ (38 files, 19,137 LOC), audio/ (36 files, 11,687 LOC) Analysis date: 2026-02-28 Total: 97 files, 40,588 LOC


Pipeline Overview#

text
Course Content
  ↓
┌─── DIAGRAMS ───────────────────────────────────┐
│ Intent Detection → Style Resolution → Render   │
│ → D3/SVG/HTML → Accessibility → Cache          │
└────────────────────────────────────────────────┘
  ↓
┌─── AUDIO ──────────────────────────────────────┐
│ Content Analysis → Persona Selection → SSML    │
│ → TTS (ChatterboxTTS/GPU) → Quality Pipeline   │
│ → Learner Adaptation                            │
└────────────────────────────────────────────────┘
  ↓
┌─── PRESENTATIONS ──────────────────────────────┐
│ Keynote Compile → Slide Render → Audio Sync    │
│ → HTML/CSS/JS Export                            │
└────────────────────────────────────────────────┘
  ↓
┌─── VIDEO ──────────────────────────────────────┐
│ Script → Storyboard → Visuals (GPU) + Audio    │
│ → Timeline Composition → FFmpeg Render → QA    │
└────────────────────────────────────────────────┘
  ↓
┌─── SYNCHRONIZATION ────────────────────────────┐
│ AV Synchronizer → Cross-media timing           │
│ → Timeline validation → Final output           │
└────────────────────────────────────────────────┘

Multimedia Module (23 files, 9,764 LOC)#

Key Components#

File LOC Purpose
diagram_renderer.py 1,240 Diagram config → SVG/HTML animations
av_synchronizer.py 1,184 Audio-visual sync, TTS retry (exp backoff)
d3_renderer.py 1,079 D3.js interactive diagrams
diagram_pipeline.py 987 End-to-end diagram generation
diagram_runtime.py 888 Runtime execution, CSS/JS injection
presentation_renderer.py 812 Keynote HTML slide generation
educational_podcast_engine.py 662 Podcast with voice characters
keynote_compiler.py 438 Lesson → keynote with audio sync
diagram_versioning.py 227 Version control across refinement
diagram_validator.py 217 Data integrity, accessibility

Diagram Types (D3Renderer)#

Gantt, Networks, Timelines, Hierarchies, Sankey, Treemaps, Heatmaps, Mind Maps, Flowcharts

Diagram Pipeline Stages#

  1. Intent Detection → concept/diagram type inference
  2. Style Resolution → educational style profiles
  3. Data Processing → sanitize, downsample, constrain
  4. Rendering → SVG/HTML (D3, PlotlyJS, custom)
  5. Accessibility → ARIA labels, data tables, alt text
  6. Caching → SHA256 digest for cache hits
  7. Versioning → track across refinement rounds

Audio Module (36 files, 11,687 LOC)#

Key Components#

File LOC Purpose
tts_adapter.py 838 ChatterboxTTS integration, GPU offload
tts/ssml_builder.py 788 Emotion-aware SSML generation
tts/personas.py 676 20+ voice personality profiles
emotion/learner_detector.py 604 Learner state detection
prosody/pitch_generator.py 577 Pitch curves for emphasis
emotion/content_analyzer.py 545 Emotion/emphasis analysis
tts/voice_cloning.py 535 Voice cloning/style transfer
audio_quality_pipeline.py 414 QA (intelligibility, pacing)

Audio Pipeline Stages#

  1. Content Analysis → EmotionScores, EmphasisPoints, ToneRecommendations
  2. Persona Selection → voice personality + prosody settings
  3. SSML Generation → prosody, breaks, emphasis tags
  4. TTS Synthesis → ChatterboxTTS (GPU-optimized)
  5. Quality Assurance → intelligibility (RMS, speech ratio), pacing (WPM)
  6. Learner Adaptation → adjusted prosody for next narration

Voice System#

  • 20+ personas with: voice_id, gender, accent, style, age, pitch/rate/expressiveness
  • Emotion types: EXCITEMENT, CURIOSITY, SERIOUSNESS, ENCOURAGEMENT, EMPATHY, NEUTRAL
  • Emphasis levels: NONE, REDUCED, MODERATE, STRONG
  • Break strengths: X_WEAK, WEAK, MEDIUM, STRONG, X_STRONG

Video Module (38 files, 19,137 LOC)#

Key Components#

File LOC Purpose
visuals/text_renderer.py 1,068 Text overlays for video
audio/video_tts.py 1,013 TTS with timing
visuals/avatar_generator.py 1,003 Animated avatars
rendering/ffmpeg_renderer.py 916 FFmpeg composition
animation/diagram_animator.py 874 Diagram keyframe animation
composition/timeline.py 849 Multi-track timeline sync
composition/transitions.py 827 Transition effects
rendering/subtitle_generator.py 823 SRT/VTT subtitles
qa/content_validator.py 799 Content validation
audio/mixer.py 776 Audio mixing (narration+music+SFX)
pipeline.py 832 Main orchestration

Video Pipeline Phases#

text
INITIALIZATION → SCRIPT_GENERATION → STORYBOARD_GENERATION
→ VISUAL_GENERATION (GPU) → AUDIO_GENERATION
→ COMPOSITION → RENDERING (FFmpeg) → QA_VALIDATION → FINALIZATION

Timeline System#

  • Media types: VIDEO, AUDIO, IMAGE, TEXT, SUBTITLE
  • Track types: VIDEO, NARRATION, MUSIC, SFX, SUBTITLE, OVERLAY
  • Sync strategies: AUDIO_MASTER, VIDEO_MASTER, MANUAL, TIMECODE
  • Resolution presets: HD (1280x720), FULL_HD (1920x1080), UHD_4K (3840x2160)

External Dependencies#

Dependency Used For
ChatterboxTTS Primary TTS engine
ESpeak Fallback TTS
RunPod GPU serverless for TTS/image
D3.js Interactive diagrams
PlotlyJS Chart rendering
FFmpeg Video encoding
ComfyUI Image/video generation workflows
Hunyuan Avatar/video generation
Wav2Lip / LatentSync Lip sync
librosa / scipy Audio signal processing

Output Formats#

  • Diagrams: SVG, self-contained HTML, interactive D3
  • Audio: WAV, MP3, M4A
  • Presentations: HTML slides, PDF, PPTX
  • Podcasts: MP3/M4A with multiple characters
  • Video: MP4 (1080p default)
  • Subtitles: SRT, VTT
  • Textbooks: JSON, HTML, PDF, EPUB

Features to KEEP#

  1. AVSynchronizer — Production-grade TTS retry with jitter/backoff
  2. DiagramRenderer — SHA256 caching, educational style profiles
  3. D3Renderer — 9+ diagram types, self-contained HTML
  4. SSMLBuilder — Emotion-aware prosody, educational emphasis detection
  5. 20+ voice personas — Rich personality system
  6. Audio quality pipeline — Intelligibility (RMS), pacing (WPM) metrics
  7. Timeline composition — Multi-track sync (AUDIO_MASTER/VIDEO_MASTER)
  8. Video pipeline — Clear phase separation, status tracking
  9. Accessibility — ARIA labels, data tables, alt text injection
  10. Learner adaptation — Dynamic prosody based on engagement signals

Technical Debt to DISCARD#

  1. ComfyUI custom nodes — Standardize to ComfyUI API instead
  2. Scattered diagram files — 6+ locations (diagram_*.py, diagram/, multimedia/) → consolidate
  3. LipSync modules — Too specialized, make optional ComfyUI workflow
  4. Duplicate GPU utils — Audio and video both have GPU client code
  5. Hardcoded podcast styles — Move to configuration
  6. Avatar generators — Extract to separate @metis/avatar library
  7. Music licensing logic — Extract to separate service