Historical task source. The Eve task board holds current task state.
Phase 70: Production & Filmmaking SOTA Gap Closure#
This phase closes every identified gap from the comprehensive industry gap analysis (see
docs/releases/v1/reports/oshun-vs-industry-gap-analysis.md). It covers capabilities possessed by ILM, Pixar, Weta, Disney, Netflix, and 30+ AI filmmaking companies that are not yet represented in the Oshun roadmap. All tasks target SOTA parity or leadership.
Dependency: Phases 1-26 (foundation), Phase 42-44 (Neith engine/studio) for some tasks. Many tasks can proceed independently.
70.1 Per-Production AI Post-Production (@isis/post-production-ai)#
InterPositive (Netflix, $600M) equivalent. The single most critical gap. Train AI models on a production's own dailies for relighting, reframing, BG replacement, and VFX — all without traditional post-production pipelines.
70.1.1 Dailies Ingestion & Cinematic Analysis Pipeline#
- 70.1.1.1 Create @isis/post-production-ai library with project.json (scope:isis, layer:domain)
- 70.1.1.2 Implement dailies ingestion pipeline supporting ProRes, RAW, ARRIRAW, R3D, BRAW, CinemaDNG, XAVC, MXF codecs
- 70.1.1.3 Create per-frame metadata extraction engine (EXIF, sidecar XML, camera metadata)
- 70.1.1.4 Implement focal length detection from frame analysis (wide, normal, telephoto, macro classification)
- 70.1.1.5 Create camera movement classifier (static, pan, tilt, dolly, tracking, crane, handheld, Steadicam, drone)
- 70.1.1.6 Implement depth-of-field estimator (shallow/medium/deep, bokeh analysis, focus plane detection)
- 70.1.1.7 Create camera angle classifier (eye-level, low-angle, high-angle, bird's-eye, dutch, over-the-shoulder)
- 70.1.1.8 Implement occlusion detection (foreground/midground/background layer segmentation)
- 70.1.1.9 Create parallax analysis (object depth displacement across frames, motion parallax estimation)
- 70.1.1.10 Implement shot boundary detection with automatic scene segmentation
- 70.1.1.11 Create shot-type classifier (extreme wide, wide, medium, close-up, extreme close-up, insert)
- 70.1.1.12 Implement lighting analysis per frame (key/fill/back light ratios, color temperature, direction)
- 70.1.1.13 Create lens distortion profiling (barrel, pincushion, chromatic aberration detection)
- 70.1.1.14 Implement color palette extraction per shot (dominant colors, color harmony analysis)
- 70.1.1.15 Create cinematic element captioner model combining all analyses into unified metadata (inspired by InterPositive patent US 12,511,904 B1)
70.1.2 Per-Production Model Training#
- 70.1.2.1 Design per-production model architecture (lightweight diffusion or LoRA fine-tune on base model)
- 70.1.2.2 Implement automatic training data preparation from dailies (frame selection, augmentation, pairing)
- 70.1.2.3 Create production visual vocabulary builder (learn lighting signatures, lens characteristics, color palette)
- 70.1.2.4 Implement editorial consistency learning (shot-to-shot relationship modeling, match continuity)
- 70.1.2.5 Create actor appearance model (face/body/wardrobe consistency across shots without deepfake generation)
- 70.1.2.6 Implement set/environment model (location appearance learning for background consistency)
- 70.1.2.7 Create prop and vehicle tracking model
- 70.1.2.8 Implement training progress monitoring with quality metrics (FID, LPIPS against validation set)
- 70.1.2.9 Create model versioning with A/B comparison interface
- 70.1.2.10 Implement incremental training as new dailies arrive
70.1.3 AI Relighting Engine#
- 70.1.3.1 Implement inverse rendering from production footage (estimate scene geometry, materials, lighting)
- 70.1.3.2 Create light source addition in already-shot footage (synthetic key/fill/back/rim lights)
- 70.1.3.3 Implement light source removal/attenuation from footage
- 70.1.3.4 Create shot-to-shot lighting match (source shot → target shot lighting transfer)
- 70.1.3.5 Implement time-of-day modification (day-to-night, night-to-day, golden hour)
- 70.1.3.6 Create color temperature adjustment with physically-based color science
- 70.1.3.7 Implement shadow modification (soften, harden, redirect, remove)
- 70.1.3.8 Create specular/reflection modification
- 70.1.3.9 Implement volumetric lighting addition (god rays, atmosphere, haze)
- 70.1.3.10 Create HDR light probe estimation from footage for VFX integration
70.1.4 AI Shot Reframing & Coverage Synthesis#
- 70.1.4.1 Implement AI-powered shot reframing (extract different compositions from existing footage)
- 70.1.4.2 Create virtual camera repositioning (synthesize new angle from existing footage using production model)
- 70.1.4.3 Implement missing coverage synthesis (generate reverse shot, reaction shot, insert from existing footage context)
- 70.1.4.4 Create resolution-independent reframing (AI upscale cropped regions to maintain quality)
- 70.1.4.5 Implement aspect ratio conversion (16:9 to 2.39:1, IMAX to standard) with AI fill
- 70.1.4.6 Create stabilization with AI boundary fill (remove shake without losing frame edges)
- 70.1.4.7 Implement speed ramp with AI frame generation (smooth slow-motion from 24fps footage)
- 70.1.4.8 Create clean plate generation from multiple frames
- 70.1.4.9 Implement wire/rig removal using production model context
- 70.1.4.10 Create continuity error detection and suggestion system
70.1.5 AI Background Replacement & Enhancement#
- 70.1.5.1 Implement foreground/background segmentation using production model (actor, set elements, background)
- 70.1.5.2 Create AI background replacement maintaining lighting, perspective, and lens characteristics
- 70.1.5.3 Implement background extension (extend frame beyond original boundaries)
- 70.1.5.4 Create set extension from partial practical sets
- 70.1.5.5 Implement sky replacement with physically-correct atmospheric scattering
- 70.1.5.6 Create crowd augmentation (add/remove background extras)
- 70.1.5.7 Implement environmental VFX insertion (rain, snow, fog, dust with correct lighting interaction)
- 70.1.5.8 Create signage/branding replacement (product placement, localization)
- 70.1.5.9 Implement window/screen replacement with correct perspective and reflections
- 70.1.5.10 Create seasonal/weather modification for exterior shots
70.2 AI Video Enhancement & Restoration (@isis/video-enhancement)#
Topaz Labs Video AI equivalent plus Netflix-grade encoding optimization.
70.2.1 AI Super-Resolution#
- 70.2.1.1 Create @isis/video-enhancement library with project.json (scope:isis, layer:domain)
- 70.2.1.2 Implement 2x AI upscaling using Real-ESRGAN or equivalent (720p → 1440p)
- 70.2.1.3 Create 4x AI upscaling (480p → 1920p, 1080p → 4K)
- 70.2.1.4 Implement 8x AI upscaling for archival/restoration (SD → 4K+)
- 70.2.1.5 Create film-aware upscaling that preserves grain structure
- 70.2.1.6 Implement face-aware upscaling with detail enhancement for facial regions
- 70.2.1.7 Create text-aware upscaling preserving legibility of on-screen text and signage
- 70.2.1.8 Implement batch upscaling pipeline with GPU queue management
- 70.2.1.9 Create quality metrics dashboard (VMAF, SSIM, PSNR before/ after comparison)
- 70.2.1.10 Implement model selection heuristic (choose optimal model based on content analysis)
70.2.2 AI Frame Interpolation#
- 70.2.2.1 Implement optical-flow-based frame interpolation (24→48, 24→60, 30→60, 30→120 fps)
- 70.2.2.2 Create RIFE or equivalent neural frame interpolation
- 70.2.2.3 Implement scene-change-aware interpolation (no ghosting at cuts)
- 70.2.2.4 Create variable speed retiming with smooth interpolation
- 70.2.2.5 Implement artifact detection and fallback (detect and avoid hallucinated frames)
- 70.2.2.6 Create slow-motion synthesis from 24fps source (2x, 4x, 8x)
- 70.2.2.7 Implement cadence correction (remove 3:2 pulldown, convert interlaced to progressive)
- 70.2.2.8 Create high-speed footage interpolation (120fps → 240fps for extreme slow-mo)
70.2.3 AI Denoising & Restoration#
- 70.2.3.1 Implement AI denoising for low-light footage (preserve detail, remove noise)
- 70.2.3.2 Create AI denoising for high-ISO digital footage
- 70.2.3.3 Implement film grain management (removal, synthesis, matching between shots)
- 70.2.3.4 Create film grain synthesis engine (Netflix AV1 FGS equivalent — strip grain before encoding, resynthesize at decoder)
- 70.2.3.5 Implement codec artifact removal (deblocking, deringing for compressed sources)
- 70.2.3.6 Create AI sharpening with artifact-free detail enhancement
- 70.2.3.7 Implement AI dejudder (remove motion judder from panned shots)
- 70.2.3.8 Create flicker removal for archival/timelapse footage
- 70.2.3.9 Implement scratch and damage repair for film scans
- 70.2.3.10 Create AI color restoration for faded/degraded footage
70.2.4 ONNX Model Runner#
- 70.2.4.1 Implement ONNX runtime integration for custom ML models
- 70.2.4.2 Create model loading interface (file, URL, registry)
- 70.2.4.3 Implement GPU inference (CUDA, DirectML, CoreML backends)
- 70.2.4.4 Create model chaining (pipe output of one model to input of another)
- 70.2.4.5 Implement per-shot model application with Nuke CopyCat-style training-on-production-footage workflow
70.3 AI Face Synthesis & De-Aging (@isis/face-synthesis)#
Metaphysic/Brahma (DNEG), ILM FaceSwap, MARZ Vanity AI equivalent.
70.3.1 Face De-Aging & Aging#
- 70.3.1.1 Create @isis/face-synthesis library with project.json (scope:isis, layer:domain)
- 70.3.1.2 Implement age regression model (de-age actor to younger appearance: -5, -10, -20, -30, -40 years)
- 70.3.1.3 Create age progression model (age actor: +5, +10, +20, +30 years)
- 70.3.1.4 Implement reference-guided de-aging (use archival photos/ footage of actor at target age)
- 70.3.1.5 Create temporal consistency enforcement (no flickering between frames)
- 70.3.1.6 Implement expression preservation during age transformation
- 70.3.1.7 Create lighting-consistent age transformation (match scene lighting on de-aged face)
- 70.3.1.8 Implement multi-angle consistency (de-aged face consistent across camera angles)
- 70.3.1.9 Create body-age consistency (match neck, hands, posture to facial age)
- 70.3.1.10 Implement quality validation against reference footage
70.3.2 Creative Face Synthesis#
- 70.3.2.1 Implement actor-to-character face transfer (map actor performance to different character face)
- 70.3.2.2 Create volumetric 3D face reconstruction from 2D video (monocular 3D face estimation)
- 70.3.2.3 Implement expression transfer across faces (actor A's expression on actor B's face)
- 70.3.2.4 Create face shape modification (subtle structural changes while preserving identity)
- 70.3.2.5 Implement injury/scar/makeup synthesis (add/remove facial features in post)
- 70.3.2.6 Create digital cosmetic adjustment (skin smoothing, blemish removal, subtle enhancement)
- 70.3.2.7 Implement eye-line correction (redirect gaze in post)
- 70.3.2.8 Create stunt face replacement (replace stunt double's face with actor's face)
- 70.3.2.9 Implement crowd face generation (unique faces for background extras)
- 70.3.2.10 Create ethical safeguards (consent verification, usage logging, watermarking of synthesized faces)
70.4 Visual Dubbing & Dialogue Editing (@isis/visual-dubbing)#
Flawless TrueSync, Flawless DeepEditor, MARZ LipDub equivalent.
70.4.1 Visual Dubbing (Cross-Language Lip Sync)#
- 70.4.1.1 Create @isis/visual-dubbing library with project.json (scope:isis, layer:domain)
- 70.4.1.2 Implement phoneme extraction from dubbed audio track
- 70.4.1.3 Create viseme mapping for target language (phoneme → mouth shape)
- 70.4.1.4 Implement volumetric face representation for natural mouth movement modification
- 70.4.1.5 Create lip movement adaptation engine (modify mouth movements to match dubbed dialogue)
- 70.4.1.6 Implement emotional delivery preservation (maintain original performance intensity in adapted visuals)
- 70.4.1.7 Create jaw/chin movement adaptation for dubbed timing
- 70.4.1.8 Implement tongue and teeth visibility handling during speech
- 70.4.1.9 Create multi-language batch processing (dub to 10+ languages from single source)
- 70.4.1.10 Implement quality validation (lip-sync accuracy scoring per phoneme)
70.4.2 Dialogue Editing in Post (DeepEditor Equivalent)#
- 70.4.2.1 Implement same-language dialogue replacement (change spoken words while maintaining lip sync)
- 70.4.2.2 Create word-level surgical editing (replace individual words without re-rendering full sentence)
- 70.4.2.3 Implement ADR-free pickup system (synthesize corrected line delivery with matching lip movement)
- 70.4.2.4 Create profanity replacement for broadcast versions (visual + audio replacement)
- 70.4.2.5 Implement whisper/shout modification (change vocal intensity with matching visual)
- 70.4.2.6 Create timing adjustment (stretch/compress dialogue timing with visual adaptation)
- 70.4.2.7 Implement emotional regrade (same words, different emotional delivery)
- 70.4.2.8 Create integration with @isis/voice-cloning for voice + visual synchronization
70.5 Professional Voice Cloning & Synthesis (@isis/voice-cloning)#
ElevenLabs Professional, Respeecher equivalent for film/TV production.
70.5.1 Voice Cloning Pipeline#
- 70.5.1.1 Create @isis/voice-cloning library with project.json (scope:isis, layer:domain)
- 70.5.1.2 Implement voice profile creation from reference audio (minimum 30 minutes clean audio)
- 70.5.1.3 Create rapid voice profile from limited samples (5-10 minutes for lower fidelity)
- 70.5.1.4 Implement speech-to-speech conversion (preserve intonation, pacing, emotion while changing voice)
- 70.5.1.5 Create text-to-speech with cloned voice (generate new dialogue in actor's voice)
- 70.5.1.6 Implement age-modified voice synthesis (younger/older version of cloned voice)
- 70.5.1.7 Create emotional control tags (whisper, shout, cry, laugh, sarcasm, anger, tenderness)
- 70.5.1.8 Implement archival voice restoration (reconstruct degraded historical recordings)
- 70.5.1.9 Create multilingual voice preservation (same voice in multiple languages)
- 70.5.1.10 Implement multi-provider routing (ElevenLabs, Respeecher, local models) with quality-based selection
70.5.2 Production Integration#
- 70.5.2.1 Implement ADR replacement workflow (actor records guide → AI generates final in character voice)
- 70.5.2.2 Create walla/background dialogue generation (crowd murmur, background conversation)
- 70.5.2.3 Implement voice consistency validation across scenes
- 70.5.2.4 Create consent and rights management system for cloned voices
- 70.5.2.5 Implement voice watermarking (embed provenance in synthesized audio)
70.6 AI Music Generation for Film & Game Scoring (@euterpe/ai-scoring)#
Suno, Udio, AIVA, Google Lyria integration with production-aware scoring.
70.6.1 Multi-Provider Music Generation#
- 70.6.1.1 Create @euterpe/ai-scoring library with project.json (scope:euterpe, layer:domain)
- 70.6.1.2 Implement Suno API provider adapter (text-to-song, style control, vocal generation)
- 70.6.1.3 Create Udio API provider adapter (text-to-song, custom mode, lyric timing)
- 70.6.1.4 Implement AIVA API provider adapter (250+ styles, MIDI export, orchestral focus)
- 70.6.1.5 Create Google Lyria API provider adapter (vocals, lyrics, cover art)
- 70.6.1.6 Implement Boomy API provider adapter (genre/mood selection, quick generation)
- 70.6.1.7 Create multi-provider ensemble routing (generate with N providers, score, select best)
- 70.6.1.8 Implement generation cost optimizer (route to cheapest provider meeting quality threshold)
70.6.2 Scene-Aware Film/Game Scoring#
- 70.6.2.1 Implement scene mood analyzer (analyze video/script for emotional beats)
- 70.6.2.2 Create prompt-to-score pipeline (scene description → musical score)
- 70.6.2.3 Implement style-consistent generation across scenes (maintain musical identity throughout production)
- 70.6.2.4 Create leitmotif system (assign and maintain character/theme musical signatures)
- 70.6.2.5 Implement tempo/intensity matching to scene pacing
- 70.6.2.6 Create stem separation for generated music (vocals, drums, bass, melody, strings)
- 70.6.2.7 Implement MIDI export for DAW refinement by human composers
- 70.6.2.8 Create adaptive game music system (generate variations for combat, exploration, cutscene, menu)
- 70.6.2.9 Implement music-to-video sync (hit points, tempo mapping, beat alignment to cuts)
- 70.6.2.10 Create commercial licensing verification (ensure generated music is cleared for commercial use)
70.7 Color Science & HDR Pipeline (@oshun/color-science)#
ACES, OpenColorIO, Dolby Vision, HDR10+ — industry-standard color management.
70.7.1 ACES Integration#
- 70.7.1.1 Create @oshun/color-science shared library with project.json (scope:shared, layer:foundation)
- 70.7.1.2 Implement ACES 2065-1 (linear, AP0) as interchange color space
- 70.7.1.3 Create ACEScg (AP1) as working color space for rendering and compositing
- 70.7.1.4 Implement ACES output transforms (RRT + ODT) for SDR, HDR, and P3 displays
- 70.7.1.5 Create ACES Metadata File (AMF) reader/writer
- 70.7.1.6 Implement camera-specific IDTs (Input Device Transforms) for ARRI, RED, Sony, Canon, Blackmagic
- 70.7.1.7 Create look modification transforms (LMT) support
- 70.7.1.8 Implement ACES reference gamut compression (RGC) for out-of- gamut color handling
70.7.2 OpenColorIO Integration#
- 70.7.2.1 Integrate OpenColorIO (OCIO) v2 for color transform management
- 70.7.2.2 Create OCIO config generation for production pipelines
- 70.7.2.3 Implement color space conversion between all major spaces (sRGB, Rec.709, Rec.2020, P3-D65, ACEScg, ACES2065-1)
- 70.7.2.4 Create display transform management (map scene-referred to display-referred per monitor)
- 70.7.2.5 Implement GPU-accelerated color transforms (OCIO GPU path)
70.7.3 HDR Mastering Pipeline#
- 70.7.3.1 Implement Dolby Vision metadata generation (frame-by-frame dynamic metadata)
- 70.7.3.2 Create HDR10+ metadata generation (scene-by-scene dynamic metadata)
- 70.7.3.3 Implement HDR-to-SDR trim pass automation
- 70.7.3.4 Create HDR hero grade workflow (grade HDR first, derive SDR)
- 70.7.3.5 Implement monitor calibration profile management (P3-D65, Rec.2020, sRGB with per-display profiles)
- 70.7.3.6 Create HDR quality validation (highlight clipping, color volume coverage, peak brightness verification)
- 70.7.3.7 Implement wide color gamut visualization (gamut boundary, out-of-gamut indicators)
70.8 Video Encoding & Delivery Pipeline (@oshun/encoding)#
Netflix VMAF, Dynamic Optimizer, AV1 Film Grain Synthesis equivalent.
70.8.1 Perceptual Quality Metrics#
- 70.8.1.1 Create @oshun/encoding shared library with project.json (scope:shared, layer:foundation)
- 70.8.1.2 Integrate Netflix VMAF (open-source) for perceptual video quality measurement
- 70.8.1.3 Implement SSIM and MS-SSIM quality metrics
- 70.8.1.4 Create PSNR measurement with per-channel analysis
- 70.8.1.5 Implement quality comparison dashboard (source vs encoded, side-by-side, wipe, diff)
70.8.2 Shot-Based Encoding Optimization#
- 70.8.2.1 Implement scene detection for encoding segmentation (IDR frames at shot boundaries)
- 70.8.2.2 Create per-shot complexity analysis (spatial and temporal complexity scoring)
- 70.8.2.3 Implement per-shot bitrate allocation optimizer (allocate bits based on VMAF quality target)
- 70.8.2.4 Create adaptive bitrate ladder generation (per-title encoding — custom resolution/bitrate pairs per content)
- 70.8.2.5 Implement convex hull optimization for encoding parameter selection
70.8.3 Advanced Codec Support#
- 70.8.3.1 Implement AV1 encoding with film grain synthesis (strip grain → encode → resynthesize at decoder)
- 70.8.3.2 Create HEVC encoding with Dolby Vision metadata injection
- 70.8.3.3 Implement VP9 encoding for web delivery
- 70.8.3.4 Create AVIF image encoding for thumbnails and artwork (significant savings over JPEG)
- 70.8.3.5 Implement ProRes encoding for editorial/finishing delivery
70.8.4 IMF Delivery Pipeline#
- 70.8.4.1 Implement IMF (Interoperable Master Format, SMPTE st2067-2) package creation
- 70.8.4.2 Integrate Netflix Photon (open-source) for IMF validation
- 70.8.4.3 Create Dolby Vision IMF packaging (derive DV, HDR10, SDR from single master)
- 70.8.4.4 Implement Dolby Atmos IMF packaging (IAB track files)
- 70.8.4.5 Create delivery QC pipeline (encoding artifacts, A/V sync, subtitle timing, metadata validation)
70.9 Content Security & Provenance (@oshun/content-security)#
Netflix forensic watermarking, Google SynthID, C2PA — cross-domain shared library.
70.9.1 Forensic Watermarking#
- 70.9.1.1 Create @oshun/content-security shared library with project.json (scope:shared, layer:foundation)
- 70.9.1.2 Implement invisible video watermarking (per-session unique identifier embedded in video stream)
- 70.9.1.3 Create invisible audio watermarking (survives transcoding, compression, speed changes)
- 70.9.1.4 Implement image watermarking (invisible, survives screenshots, cropping, resizing)
- 70.9.1.5 Create document watermarking for scripts, storyboards, call sheets (dynamic, per-recipient)
- 70.9.1.6 Implement watermark extraction and source identification (trace leaked content to source)
- 70.9.1.7 Create watermark robustness testing suite (verify survival across transcoding, compression, screen capture)
70.9.2 AI Content Provenance#
- 70.9.2.1 Integrate C2PA (Coalition for Content Provenance and Authenticity) standard for content credentials
- 70.9.2.2 Implement Google SynthID integration for watermarking AI-generated content (text, image, audio, video)
- 70.9.2.3 Create provenance chain tracking (record generation history from AI model to final output)
- 70.9.2.4 Implement AI-generated content declaration metadata
- 70.9.2.5 Create content authenticity verification API (verify if content is human-created, AI-assisted, or AI-generated)
70.9.3 DRM Integration#
- 70.9.3.1 Implement Google Widevine DRM integration (Android, Chrome, Smart TVs)
- 70.9.3.2 Create Apple FairPlay DRM integration (iOS, macOS, Apple TV)
- 70.9.3.3 Implement Microsoft PlayReady DRM integration (Windows, Xbox)
- 70.9.3.4 Create multi-DRM license server abstraction (unified API across all DRM providers)
- 70.9.3.5 Implement hardware DRM enforcement for UHD/4K content per MovieLabs specifications
70.10 Dailies Review & Annotation Pipeline (@yemaya/dailies-review)#
Netflix MPS dailies, Autodesk Flow Capture, Frame.io equivalent.
70.10.1 Dailies Ingestion#
- 70.10.1.1 Create @yemaya/dailies-review library with project.json (scope:yemaya, layer:domain)
- 70.10.1.2 Implement dailies ingest from camera cards (OCF — original camera footage)
- 70.10.1.3 Create automatic sound sync (match audio recorder timecode to camera timecode)
- 70.10.1.4 Implement automatic color application (apply show LUT or CDL for viewing)
- 70.10.1.5 Create proxy generation for review (H.264 1080p for fast streaming review)
- 70.10.1.6 Implement metadata preservation (camera metadata, scene/take numbers, circle takes)
- 70.10.1.7 Create automatic delivery to editorial (synced, color- applied dailies in editorial codec)
70.10.2 Frame-Accurate Review & Annotation#
- 70.10.2.1 Implement frame-accurate video playback with timecode overlay
- 70.10.2.2 Create drawing annotation tools (freehand, shapes, arrows, text on video frames)
- 70.10.2.3 Implement per-frame notes system with threaded comments
- 70.10.2.4 Create version comparison tools (side-by-side, wipe, overlay, onion skin, A/B toggle)
- 70.10.2.5 Implement review session management (create session, invite reviewers, collect feedback)
- 70.10.2.6 Create approval workflow (approve, request changes, reject per shot/take)
- 70.10.2.7 Implement annotation export (PDF contact sheets with notes, EDL with comments)
- 70.10.2.8 Create mobile review support (review dailies on tablet/ phone with annotation)
70.10.3 VFX Pulls & Conform#
- 70.10.3.1 Implement VFX pull automation (extract plates, handles, reference, framing charts per VFX shot)
- 70.10.3.2 Create conform pipeline (consolidate, trim, and deliver OCF to finishing)
- 70.10.3.3 Implement ASC MHL (Media Hash List) for file verification integrity
- 70.10.3.4 Create ASC FDL (Framing Decision List) for reframing decisions
- 70.10.3.5 Implement OpenTimelineIO (OTIO) integration for editorial timeline interchange
70.11 AI Video Provider Expansion (@isis/video-generation extension)#
Expand beyond Hunyuan/WAN to all major providers with ensemble routing.
70.11.1 New Provider Adapters#
- 70.11.1.1 Implement Luma AI Ray3 provider adapter (native HDR, 16-bit EXR output, 1080p)
- 70.11.1.2 Create MiniMax/Hailuo provider adapter (cost-effective, Director Mode, anime/game CG styles)
- 70.11.1.3 Implement Higgsfield Cinema Studio provider adapter (optical physics simulation, Director Panel)
- 70.11.1.4 Create Stability AI Stable Video Diffusion provider adapter (self-hosted option)
- 70.11.1.5 Implement Haiper provider adapter (video repainting)
70.11.2 Multi-Model Ensemble & HDR#
- 70.11.2.1 Implement multi-model ensemble generation (submit to N providers, collect results)
- 70.11.2.2 Create perceptual quality scoring for ensemble selection (VMAF-based ranking of results)
- 70.11.2.3 Implement style consistency scoring (select result most consistent with production's visual language)
- 70.11.2.4 Create cost-aware routing (quality target → cheapest provider meeting threshold)
- 70.11.2.5 Implement HDR video generation pipeline (generate → tonemap → validate → EXR export)
70.11.3 Camera & Lens Physics Simulation#
- 70.11.3.1 Implement virtual lens library (prime 24mm, 35mm, 50mm, 85mm, 135mm; zoom 24-70, 70-200 with accurate optical characteristics)
- 70.11.3.2 Create depth-of-field simulation with physically accurate bokeh (circle of confusion, aperture blade count)
- 70.11.3.3 Implement motion blur simulation based on shutter angle
- 70.11.3.4 Create lens breathing simulation (focal length shift during focus pull)
- 70.11.3.5 Implement lens distortion and chromatic aberration simulation
- 70.11.3.6 Create anamorphic lens simulation (horizontal squeeze, oval bokeh, lens flares)
70.12 CG Character Replacement from Video (@aja/cg-replacement)#
Wonder Dynamics / Autodesk Wonder Studio equivalent.
70.12.1 Video-to-CG Pipeline#
- 70.12.1.1 Create @aja/cg-replacement library with project.json (scope:aja, layer:domain)
- 70.12.1.2 Implement actor segmentation from video (isolate person from background per frame)
- 70.12.1.3 Create 3D pose estimation from video for full skeleton extraction
- 70.12.1.4 Implement performance transfer to CG character rig (retarget human skeleton to character skeleton)
- 70.12.1.5 Create automatic lighting estimation from video (estimate light positions, colors, intensities from footage)
- 70.12.1.6 Implement scene decomposition (separate foreground actor, midground elements, background)
- 70.12.1.7 Create CG character rendering with matched lighting (render CG character under estimated scene lighting)
- 70.12.1.8 Implement CG-to-scene compositing (insert rendered character back into original footage with correct occlusion)
- 70.12.1.9 Create shadow and reflection synthesis (CG character casts correct shadows and reflections in scene)
- 70.12.1.10 Implement temporal consistency enforcement (no flickering or jitter across frames)
70.12.2 Facial Performance Transfer#
- 70.12.2.1 Implement facial expression extraction from video (FACS-based action unit detection)
- 70.12.2.2 Create facial retargeting to CG character (transfer actor expressions to different face proportions)
- 70.12.2.3 Implement eye tracking and gaze transfer
- 70.12.2.4 Create lip-sync transfer from video to CG character
- 70.12.2.5 Implement micro-expression preservation during transfer
70.13 AI Text-to-Animation & Inbetweening (@aja/generative-animation)#
DeepMotion SayMotion, Cascadeur AI Inbetweening equivalent.
70.13.1 Text-to-Animation#
- 70.13.1.1 Create @aja/generative-animation library with project.json (scope:aja, layer:domain)
- 70.13.1.2 Implement text-to-motion generation using motion diffusion models (text prompt → 3D skeleton animation)
- 70.13.1.3 Create physics-informed generation (gravity, momentum, balance, ground contact)
- 70.13.1.4 Implement style-specific generation (walk styles, fight styles, dance styles, emote styles)
- 70.13.1.5 Create multi-character interaction generation (two-person fight, handshake, conversation)
- 70.13.1.6 Implement character-specific motion adaptation (body proportions affect motion style)
- 70.13.1.7 Create motion editing from natural language ("make the walk more confident", "slow down the run")
- 70.13.1.8 Implement export to BVH, FBX, USD for DCC consumption
70.13.2 AI Inbetweening & Motion Enhancement#
- 70.13.2.1 Implement physics-aware inbetweening (given 2 keyframes, generate physically correct in-between motion)
- 70.13.2.2 Create ballistic trajectory calculation for jumping and falling
- 70.13.2.3 Implement center-of-mass awareness for balance during transitions
- 70.13.2.4 Create contact-aware interpolation (foot contacts, hand grabs, sitting)
- 70.13.2.5 Implement motion style transfer (apply motion style of reference clip to different motion content)
- 70.13.2.6 Create motion quality scoring (rate generated motion for naturalness, physics plausibility, smoothness)
- 70.13.2.7 Implement iterative refinement loop (generate → score → regenerate until quality threshold met)
- 70.13.2.8 Create motion looping tools (generate seamless loopable animations for games)
70.14 Render Farm Management (@oshun/render-farm)#
Pixar Tractor, Sony Imageworks OpenCue equivalent.
70.14.1 Job Distribution System#
- 70.14.1.1 Create @oshun/render-farm shared library with project.json (scope:shared, layer:infrastructure)
- 70.14.1.2 Implement job submission API (submit render jobs with priority, dependencies, resource requirements)
- 70.14.1.3 Create job queue with priority scheduling (FIFO, priority, deadline-based, fair-share)
- 70.14.1.4 Implement blade/worker node management (register, health check, capabilities, load)
- 70.14.1.5 Create automatic job distribution based on node capabilities (GPU type, memory, storage)
- 70.14.1.6 Implement job preemption for priority escalation
- 70.14.1.7 Create dependency graph execution (task A must complete before task B starts)
- 70.14.1.8 Implement checkpoint-based resume (resume failed jobs from last completed frame)
70.14.2 Resource Management & Cloud Bursting#
- 70.14.2.1 Implement resource usage tracking per job (CPU, GPU, memory, storage, wall time)
- 70.14.2.2 Create resource allocation limits per project/user
- 70.14.2.3 Implement cloud bursting to AWS EC2/RunPod for overflow capacity
- 70.14.2.4 Create cost estimation before job submission
- 70.14.2.5 Implement progress tracking with ETA estimation per job
- 70.14.2.6 Create web dashboard for job monitoring (status, progress, logs, frame previews)
- 70.14.2.7 Implement alerting for failed jobs, stuck jobs, and quota warnings
70.15 AI Video Object & Interaction Removal (@isis/video-object-removal)#
Netflix VOID production integration. While @nous/video-removal (Phase 47.7) provides the model inference layer, this section covers the production-facing workflows, UI, orchestration, and domain-specific pipelines that make physics-aware video object removal usable in professional film/TV production.
70.15.1 Production Object Removal Workflow#
- 70.15.1.1 Create @isis/video-object-removal library with project.json (scope:isis, layer:domain) integrating @nous/video-removal inference with ISIS workflow orchestration
- 70.15.1.2 Implement interactive object selection UI in Yemaya studio: point-click, bounding box, and lasso tools for marking objects to remove with real-time SAM2 mask preview overlay
- 70.15.1.3 Create VLM quadmask visualization overlay showing primary object (red), overlap zones (yellow), physics-affected regions (blue), and background (transparent) for director review before committing to removal
- 70.15.1.4 Implement text-prompt-guided object selection ("remove the red car", "remove all background pedestrians") using VLM + SAM2 for hands-free object identification
- 70.15.1.5 Create quality tier selector in removal UI: Preview (STTN, seconds, low quality for quick check), Standard (DiffuEraser, minutes, good quality), Premium (VOID two-pass, longer, physics-aware SOTA quality)
- 70.15.1.6 Implement before/after comparison viewer with synchronized playback, wipe transition, overlay diff, and frame-by-frame stepping for QC review
- 70.15.1.7 Create removal job submission to ISIS generation API with progress tracking, GPU worker assignment, and estimated completion time
- 70.15.1.8 Implement removal undo/versioning allowing directors to revert specific removals while keeping others in a multi-object edit session
- 70.15.1.9 Create removal batch queue for submitting multiple shots/scenes for overnight processing with priority ordering
- 70.15.1.10 Implement removal approval workflow: artist submits → supervisor reviews in comparison viewer → approves or requests revision with annotated notes
70.15.2 Cross-Shot & Episodic Consistency#
- 70.15.2.1 Implement cross-shot object removal consistency ensuring the same object (character, vehicle, prop) is removed identically across all shots in a scene regardless of camera angle
- 70.15.2.2 Create character removal pipeline for removing an actor from an entire episode/film: automatic detection across all shots, batch mask generation, consistent background reconstruction
- 70.15.2.3 Implement background consistency enforcement across cuts — when an object is removed in shot A and shot B of the same scene, the revealed background must match between shots
- 70.15.2.4 Create continuity validation system comparing removal results across adjacent shots in the edit timeline and flagging inconsistencies
- 70.15.2.5 Implement shadow and reflection consistency — when removing an object, ensure its shadow and reflections are also removed consistently across all shots where they appear
70.15.3 Domain-Specific Removal Pipelines#
- 70.15.3.1 Implement regional content adaptation pipeline: remove or replace culturally sensitive elements (alcohol, tobacco, religious symbols, branding) for market-specific versions while maintaining scene coherence
- 70.15.3.2 Create product placement insertion/removal pipeline: remove existing brands and optionally insert replacement brands with correct perspective, lighting, and surface interaction
- 70.15.3.3 Implement safety wire and rig removal pipeline extending 70.1.4.9 with VOID physics-awareness — remove wires AND the subtle body deformation they cause on harness-wearing actors
- 70.15.3.4 Create crowd reduction/augmentation pipeline: selectively remove or thin background extras while maintaining natural crowd dynamics and preserving physics (no floating shadows, correct reflections)
- 70.15.3.5 Implement vehicle and prop removal with physics-correct ground contact — remove parked cars while reconstructing the ground surface, curb lines, and shadows they occluded
- 70.15.3.6 Create boom mic and crew removal pipeline for documentary and live event footage where crew inadvertently appears in frame
- 70.15.3.7 Implement anachronism removal for period productions — remove modern elements (antennas, power lines, aircraft contrails, contemporary signage) that break period authenticity
- 70.15.3.8 Create weather element removal/addition using VOID's physics awareness — remove rain and its surface splash effects, or remove snow and its accumulation on surfaces
70.15.4 DaVinci Resolve & NLE Integration#
- 70.15.4.1 Implement DaVinci Resolve plugin (via @bellona/davinci-adapter) for in-timeline object removal: select object in Resolve viewer → generate mask → send to VOID pipeline → receive clean plate → auto-insert on timeline
- 70.15.4.2 Create OpenFX plugin interface for object removal accessible from any OFX-compatible host (Resolve, Nuke, Flame, Fusion)
- 70.15.4.3 Implement round-trip metadata preservation: export removal job from NLE with timeline position, handles, and grade → process → import result at exact timeline position with matching metadata
- 70.15.4.4 Create EXR multi-layer output with separate clean plate, original, mask, and difference layers for compositor flexibility
- 70.15.4.5 Implement EDL/XML/OTIO-based batch export allowing editors to mark removal points in timeline and batch-submit all removal jobs
70.15.5 Content Provenance & Safety#
- 70.15.5.1 Implement removal provenance tracking recording what was removed, from which frames, by whom, when, and using which model version — linked to @isis/outputs provenance chain
- 70.15.5.2 Create C2PA content credential integration (link to 70.9.2) embedding machine-readable record of AI object removal in output file metadata per Coalition for Content Provenance and Authenticity standard
- 70.15.5.3 Implement removal audit log with frame-accurate before/after snapshots stored immutably for legal and compliance review
- 70.15.5.4 Create deepfake/manipulation detection resistance — ensure removal outputs include SynthID watermarks (link to 70.9.2.2) identifying them as AI-modified content
- 70.15.5.5 Implement ethical guardrails: block removal of faces for identity concealment unless authorized, block removal of evidence markers in forensic contexts, require consent documentation for actor removal
70.16 Generative Relighting Studio (@isis/relight-studio)#
While Phase 70.1.3 covers relighting of already-shot footage via inverse rendering, this section covers DIFFUSION-BASED generative relighting using IC-Light, GenLit, and LightLab integrated into production workflows. Enables relighting of generated content, composited scenes, and virtual environments with artistic lighting control.
70.16.1 Interactive Relighting Workflow#
- 70.16.1.1 Create @isis/relight-studio library with project.json (scope:isis, layer:domain) integrating IC-Light, GenLit, and LightLab inference with ISIS workflow orchestration
- 70.16.1.2 Implement interactive light placement UI in Yemaya studio: click to place point, directional, and area lights on image or video with real-time position and orientation gizmos
- 70.16.1.3 Create HDRI environment map browser with thumbnail preview, category filtering, and support for loading from library or capturing from 360-degree footage
- 70.16.1.4 Implement real-time relighting preview using IC-Light at reduced resolution for interactive feedback during light placement and parameter adjustment
- 70.16.1.5 Create lighting preset library with production-standard setups: golden hour, moonlight, studio key+fill, neon, dramatic side-light, Rembrandt, butterfly, split, and loop lighting patterns
- 70.16.1.6 Implement light parameter adjustment controls: color (with color picker and Kelvin temperature), intensity, direction, size, falloff curve, and cone angle with real-time visual feedback
- 70.16.1.7 Create batch relighting pipeline: apply same lighting setup across multiple frames, shots, or entire sequences for consistent look development across a scene
- 70.16.1.8 Implement lighting reference matcher: analyze reference photograph or film still to extract lighting parameters (direction, color temperature, ratio, quality) and apply extracted setup to target content
- 70.16.1.9 Create relighting comparison view: before/after toggle, A/B split between lighting setups, synchronized playback for video, and onion-skin overlay for subtle difference inspection
- 70.16.1.10 Implement relighting export with alpha, depth, and normal passes for downstream compositor integration in Nuke, Fusion, or After Effects
70.16.2 Shot-to-Shot Relighting Consistency & NLE Integration#
- 70.16.2.1 Implement cross-shot lighting consistency enforcement: lock lighting parameters across all shots in a scene so relighting results match regardless of original on-set lighting differences
- 70.16.2.2 Create lighting continuity validation comparing relighting results between adjacent shots in the edit timeline and flagging color temperature, intensity, or direction discontinuities
- 70.16.2.3 Implement DaVinci Resolve plugin (via @bellona/davinci-adapter) for in-timeline relighting: select shot in Resolve viewer, adjust lights in panel, render relighting in background, auto-insert result on timeline
- 70.16.2.4 Create OpenFX plugin interface for relighting accessible from any OFX-compatible host (Resolve, Nuke, Flame, Fusion)
- 70.16.2.5 Implement EXR multi-layer relighting output with separate lit result, individual light passes, normals, depth, and albedo layers for compositor flexibility
- 70.16.2.6 Create temporal lighting animation: keyframe light parameters over time for effects like sunrise progression, moving spotlight, flickering candle, or lightning flash with smooth interpolation
- 70.16.2.7 Implement video relighting via GenLit: relight full video sequences using video diffusion model's implicit 3D understanding for temporally consistent results without per-frame flickering
- 70.16.2.8 Create relighting quality metrics comparing generated lighting against physically-based rendering reference including PSNR, SSIM, and perceptual light direction accuracy scoring
70.17 Unified Video Editing Suite (@isis/video-edit-studio)#
Production integration for VACE (Wan2.1-VACE), Motion Prompting, Wan-Move, MotionStream, PostCam, and TokenFlow. While Phase 47.10 provides the model inference layer, this section covers the production-facing UI, workflows, and batch processing that make advanced video control usable for editors and directors.
70.17.1 VACE-Based Production Editing Workflow#
- 70.17.1.1 Create @isis/video-edit-studio library with project.json (scope:isis, layer:domain) integrating VACE, Motion Prompting, and PostCam inference with ISIS workflow orchestration
- 70.17.1.2 Implement unified video editing UI in Yemaya studio with task selector panel: inpaint, outpaint, restyle, extend, and reference-to-video modes with contextual tool palettes per mode
- 70.17.1.3 Create drag-and-drop reference image panel for VACE reference-to-video: place reference image of subject, character, or object and generate video maintaining subject identity and appearance
- 70.17.1.4 Implement interactive mask painting for video inpainting: paint spatiotemporal mask on timeline with frame-by-frame or interpolated keyframe masking, then fill masked region with AI-generated content guided by prompt
- 70.17.1.5 Create video outpainting controls: extend canvas in any direction (left, right, top, bottom) with aspect ratio selection presets (16:9, 2.39:1, 9:16, 1:1) and content guidance prompt for generated regions
- 70.17.1.6 Implement text-guided video restyling: type style description, preview restyle on representative keyframe for approval, then apply to full video with temporal consistency enforcement across all frames
- 70.17.1.7 Create VACE quality tier selection: Fast mode (1.3B model, preview quality, seconds per clip) vs Premium mode (14B model, production quality, minutes per clip) with automatic tier recommendation based on output requirements
- 70.17.1.8 Implement VACE batch editing pipeline: apply same edit operation (inpaint, restyle, outpaint) across multiple shots or scenes for episodic production with consistent parameters and style
- 70.17.1.9 Create edit comparison viewer: original vs edited side-by-side with synchronized playback, frame stepping, horizontal/vertical wipe, and difference overlay visualization
- 70.17.1.10 Implement edit versioning: save multiple edit variants per shot, compare variants in grid view, select preferred version, and track version history with rollback capability
70.17.2 Interactive Motion & Camera Control UI#
- 70.17.2.1 Implement motion trajectory painting UI: draw object motion paths directly on first frame with bezier curve handles, speed control via point spacing, and real-time trajectory preview overlay
- 70.17.2.2 Create camera trajectory designer: 3D viewport with interactive camera path editor for PostCam and DepthDirector control, showing camera frustum preview along the path
- 70.17.2.3 Implement cinematographic camera preset library: orbit, dolly, crane, tracking, reveal, push-in, pull-out, and whip-pan as one-click trajectory presets with adjustable magnitude and timing
- 70.17.2.4 Create motion intensity slider: control magnitude of object/camera movement from subtle (micro-movement) to dramatic (full traversal) with proportional scaling of all trajectory keypoints
- 70.17.2.5 Implement multi-object motion control: draw independent trajectories for multiple objects in same video with per-object timing, speed, and easing curves
- 70.17.2.6 Create motion transfer UI: select source video, extract motion signature (camera movement and object dynamics), apply extracted motion to target content with preview and intensity adjustment
- 70.17.2.7 Implement real-time motion preview using MotionStream for sub-second visual feedback during trajectory editing, updating preview as user adjusts control points
- 70.17.2.8 Create motion quality validation: detect physically implausible motions including teleportation, object interpenetration, impossible acceleration, and gravity violations with frame-level annotation
- 70.17.2.9 Implement DaVinci/NLE round-trip: export motion-controlled video with timeline metadata, handles, and grade preservation for seamless integration into existing editorial workflow
- 70.17.2.10 Create EDL/XML/OTIO batch motion editing: mark shots in timeline, assign motion templates per shot, batch-process all marked shots, and re-import results at correct timeline positions
70.18 Portrait Animation & Virtual Performance Studio (@isis/portrait-studio)#
Production integration for Hallo3, LivePortrait, MuseTalk, and audio-driven portrait generation. Enables virtual presenter creation, cross-language dubbing with lip replacement, character animation from reference photos, and performance consistency across episodic content.
70.18.1 Virtual Presenter & Performance Pipeline#
- 70.18.1.1 Create @isis/portrait-studio library with project.json (scope:isis, layer:domain) integrating Hallo3, LivePortrait, MuseTalk, and RAP inference with ISIS workflow orchestration
- 70.18.1.2 Implement virtual presenter creation wizard: upload reference photo, select voice (from ElevenLabs, F5-TTS, or CosyVoice integration), input script text, and generate full presenter video with synchronized lip-sync and natural gestures
- 70.18.1.3 Create expression direction controls: specify emotion and energy level per line of script (calm, excited, serious, warm, emphatic, concerned) with smooth transitions between emotional states
- 70.18.1.4 Implement multi-angle presenter generation: generate same presenter from different camera angles (front, three-quarter left/right, profile) for editing flexibility and shot variety
- 70.18.1.5 Create presenter wardrobe and background control: specify or change outfit and background via text prompt while preserving identity consistency and natural clothing physics
- 70.18.1.6 Implement real-time avatar mode using RAP: live microphone input drives real-time portrait animation for interactive presentations, live streams, and virtual meetings with sub-200ms latency
- 70.18.1.7 Create character animation from reference photo: single photo of fictional, historical, or illustrated character produces animated performance driven by script audio with style-appropriate movement
- 70.18.1.8 Implement performance consistency enforcement across episodes: same character must look, move, and sound consistent across all generated segments with identity drift detection and correction
- 70.18.1.9 Create performance quality gates: lip-sync accuracy check (SyncNet score threshold), identity consistency check (face embedding distance), and expression naturalness check before delivery approval
- 70.18.1.10 Implement batch presenter generation: generate an entire video series with consistent presenter across all episodes, applying uniform visual style, voice, and mannerisms
70.18.2 Cross-Language Dubbing with Portrait Animation#
- 70.18.2.1 Implement cross-language dubbing pipeline: original video, translate script to target language, generate target language audio via TTS, apply MuseTalk lip replacement to match new audio with original actor's face
- 70.18.2.2 Create multi-language dubbing batch processor: dub single source video into N target languages in one job with parallel processing and per-language quality validation
- 70.18.2.3 Implement dubbing quality validation per language: lip-sync accuracy scoring for each target language accounting for phoneme differences (e.g., wider mouth shapes for Romance languages, subtler for East Asian)
- 70.18.2.4 Create actor approval workflow for AI dubbing: actor reviews AI-dubbed version in secure viewer, approves or requests adjustments with frame-level annotation, and signs off per language
- 70.18.2.5 Implement emotion preservation across languages: ensure emotional delivery (intensity, pacing, emphasis) matches original performance regardless of target language prosodic differences
- 70.18.2.6 Create dubbing A/B comparison viewer: original language vs each dubbed version with synchronized playback, frame stepping, and audio waveform overlay for timing comparison
- 70.18.2.7 Implement subtitle and closed-caption generation from dubbing transcript with accurate timing derived from the generated audio track
- 70.18.2.8 Create dubbing provenance tracking: record original actor, translator, TTS model, portrait animation model, and approval chain for each dubbed version linked to @isis/outputs provenance system
70.19 Automated Foley & Sound Design Studio (@isis/foley-studio)#
Production integration for AV-Link, FoleyCrafter, and Any2Audio. While Phase 47.12 provides the model inference layer, this section covers the production-facing workflows for generating, reviewing, and delivering AI-generated foley and sound effects for film/TV post-production.
70.19.1 Production Foley Generation Pipeline#
- 70.19.1.1 Create @isis/foley-studio library with project.json (scope:isis, layer:domain) integrating FoleyCrafter, AV-Link, and Any2Audio inference with ISIS workflow orchestration
- 70.19.1.2 Implement automatic foley detection: analyze video to identify moments requiring foley (footsteps, impacts, doors, cloth rustling, object interactions) with timecoded event list and material classification
- 70.19.1.3 Create foley generation queue: detected events are batch-generated as foley sounds via FoleyCrafter/AV-Link with material-aware synthesis matching surface type and impact force
- 70.19.1.4 Implement foley review interface: video playback with generated foley layered on timeline, per-event approve/reject/regenerate controls, and waveform display synchronized to video
- 70.19.1.5 Create foley sweetening controls: adjust volume, reverb (room size and decay), EQ (low/mid/high bands), and spatial panning per generated foley event to match production mix requirements
- 70.19.1.6 Implement foley material override: when auto-detected material is incorrect (e.g., metal detected but surface is wood), override material classification and regenerate with correct surface properties
- 70.19.1.7 Create ambient soundscape generation from scene analysis: detect environment type (forest, city street, office, beach, cave) and generate continuous ambient bed via Any2Audio matching visual context
- 70.19.1.8 Implement foley export in industry formats: AAF, OMF, and BWF-P with timecoded metadata for direct import into Pro Tools, Logic Pro, and DaVinci Fairlight
- 70.19.1.9 Create foley consistency across cuts: ensure same surface and material produces consistent sound characteristics across different shots in the same scene with automatic gain and timbre matching
- 70.19.1.10 Implement foley library builder: save generated foley sounds to searchable library with material tags, intensity levels, duration metadata, and source video reference for future reuse
70.19.2 Sound Design from Storyboards & Quality Validation#
- 70.19.2.1 Implement storyboard-to-sound pipeline: storyboard frames plus text descriptions produce preliminary sound design for pre-production previsualization and director review before principal photography
- 70.19.2.2 Create text-to-sound-effect generation for editors: type natural language description (e.g., "heavy wooden door creaking open slowly") and generate matching sound effect with multiple variants to choose from
- 70.19.2.3 Implement audio-visual sync validation: automated check that generated foley onsets align within plus-or-minus 2 frames of visual contact points with per-event sync accuracy reporting
- 70.19.2.4 Create A/B comparison between AI-generated foley and traditional library foley for quality evaluation with blind listening test workflow and preference scoring
- 70.19.2.5 Implement foley budgeting: estimate number of foley events per reel from video analysis and generate cost/time comparison versus traditional foley stage recording
- 70.19.2.6 Create sound design versioning: save and compare multiple foley and sound design passes for same edit with diff view showing changed, added, and removed sound events between versions
- 70.19.2.7 Implement integration with existing sound editing pipeline: generated foley arrives in editor's DAW (Pro Tools, Fairlight, Logic Pro) at correct timecode positions via AAF/OMF round-trip
- 70.19.2.8 Create sound design quality report: per-event sync accuracy, material classification confidence, spectral quality metrics, and overall mix balance assessment for QC review before final delivery
70.20 Joint Audio-Video Production & Multi-Shot Narrative Studio (@isis/av-narrative-studio)#
Production integration for Seedance 2.0's unique capabilities that have no equivalent elsewhere in the pipeline: joint audio-video generation (synchronized audio+video in one pass), multi-shot narrative from single prompt, multi-asset reference workflows, one-click video recreation, and prompt-driven camera planning. While Phase 17.7 provides the API client and Phase 47.16 covers the model-agnostic paradigm, this section covers production-facing workflows.
70.20.1 Joint Audio-Video Production Workflow#
- 70.20.1.1 Create @isis/av-narrative-studio library with project.json, package.json, tsconfig.json, and barrel exports for joint audio-video production workflows, multi-shot narrative pipelines, and one-click recreation interfaces
- 70.20.1.2 Implement joint A/V generation UI in Yemaya studio: generate video WITH synchronized audio in single operation — director hears soundtrack as they preview video, with real-time progress for both streams
- 70.20.1.3 Create audio stem separation from joint output: auto-split generated audio into dialogue, music, effects, ambient layers for mixer control with per-stem gain, pan, and mute/solo toggles
- 70.20.1.4 Implement selective audio override: keep Seedance's generated video but replace specific audio stems (e.g., replace generated music with licensed track, keep generated SFX) with crossfade and level matching
- 70.20.1.5 Create A/V quality review interface: synchronized playback with per-stem volume controls, mute/solo per layer, waveform visualization alongside video, and frame-accurate scrubbing
- 70.20.1.6 Implement joint A/V batch generation: generate multiple video+audio variations from same prompt for director selection with side-by-side comparison and preference ranking
- 70.20.1.7 Create A/V consistency across scenes: when generating multiple scenes for same project, ensure audio style (music theme, ambient tone, voice character) remains consistent via style reference conditioning
- 70.20.1.8 Implement audio-first generation mode: provide reference audio/music → generate video that matches audio pacing, energy, and mood with beat-aligned cut suggestions
- 70.20.1.9 Create video-first generation mode: generate video → use as reference for joint A/V regeneration with improved audio sync and director-specified audio direction notes
- 70.20.1.10 Implement joint A/V export in professional formats: separate video (ProRes/DNxHR) and audio (WAV stems) tracks with embedded timecode for NLE import into Premiere, Resolve, and Final Cut Pro
70.20.2 Multi-Shot Narrative Production Pipeline#
- 70.20.2.1 Implement narrative prompt interface: director writes scene description → system shows proposed shot breakdown (shot type, duration, camera, transition) before generation with editable parameters per shot
- 70.20.2.2 Create shot plan editor: drag-and-drop reordering, per-shot camera override, duration adjustment, transition type selection, and preview thumbnail generation before committing to full render
- 70.20.2.3 Implement character reference integration: upload character photos → maintain consistency across all generated shots in the narrative with automatic face and wardrobe matching verification
- 70.20.2.4 Create location reference integration: upload location photos → maintain environmental consistency across shots with lighting direction and time-of-day continuity enforcement
- 70.20.2.5 Implement multi-shot timeline view: generated shots appear on timeline with handles, allowing trimming, reordering, and re-generation of individual shots without affecting adjacent shots
- 70.20.2.6 Create narrative template library: pre-built shot sequences for common production needs (interview setup, product reveal, chase sequence, dialogue scene, establishing shot → close-up) with customizable parameters
- 70.20.2.7 Implement episodic consistency: for series production, maintain character, location, and audio style consistency across multiple generated episodes with per-episode and per-season style locks
- 70.20.2.8 Create multi-shot approval workflow: director reviews shot-by-shot with approve/reject/regenerate per shot, with annotated feedback and revision history tracking
- 70.20.2.9 Implement shot replacement: regenerate single shot within multi-shot sequence while preserving continuity with adjacent shots using boundary frame conditioning and transition blending
- 70.20.2.10 Create narrative export: export final multi-shot sequence as single video or individual shots with EDL/XML metadata for NLE import and AAF for audio round-trip
70.20.3 One-Click Recreation & Reference Workflows#
- 70.20.3.1 Implement one-click video recreation interface: upload reference video → specify what to change (different model/product/branding) → generate new video preserving original pacing, cuts, and camera work
- 70.20.3.2 Create brand template system: save successful recreations as templates (original video structure + what to swap) for repeatable ad production with per-brand asset presets and approval gates
- 70.20.3.3 Implement reference asset library: organize and tag reference images/videos/audio for quick retrieval during multi-asset generation with search, filtering, and usage tracking
- 70.20.3.4 Create reference combination builder: visual interface for assembling 12-asset reference sets with preview of each asset's intended contribution (style, motion, character, environment, audio) and conflict detection
- 70.20.3.5 Implement motion reference extraction: select specific motion patterns from reference video (just the camera movement, just the subject motion) for targeted conditioning with visual preview of extracted motion path
- 70.20.3.6 Create style consistency across recreations: when producing multiple recreated videos for same campaign, enforce consistent visual style and audio tone via shared style reference and per-campaign guardrails
- 70.20.3.7 Implement recreation A/B testing: generate multiple recreation variants → compare in split-screen viewer → select best version with frame-by-frame diff and quality metric overlays
- 70.20.3.8 Create recreation audit trail: record original reference, recreation parameters, and generated output for content provenance and licensing compliance with exportable chain-of-custody report
Phase 70 Summary Statistics#
| Section | Task Count |
|---|---|
| 70.1 Per-Production AI Post-Production | 55 |
| 70.2 AI Video Enhancement & Restoration | 33 |
| 70.3 AI Face Synthesis & De-Aging | 20 |
| 70.4 Visual Dubbing & Dialogue Editing | 18 |
| 70.5 Professional Voice Cloning & Synthesis | 15 |
| 70.6 AI Music Generation for Film/Game Scoring | 18 |
| 70.7 Color Science & HDR Pipeline | 20 |
| 70.8 Video Encoding & Delivery Pipeline | 20 |
| 70.9 Content Security & Provenance | 17 |
| 70.10 Dailies Review & Annotation Pipeline | 20 |
| 70.11 AI Video Provider Expansion | 16 |
| 70.12 CG Character Replacement from Video | 15 |
| 70.13 AI Text-to-Animation & Inbetweening | 16 |
| 70.14 Render Farm Management | 15 |
| 70.15 AI Video Object & Interaction Removal | 33 |
| 70.16 Generative Relighting Studio | 18 |
| 70.17 Unified Video Editing Suite | 20 |
| 70.18 Portrait Animation & Virtual Performance | 18 |
| 70.19 Automated Foley & Sound Design | 18 |
| 70.20 Joint A/V & Multi-Shot Narrative | 28 |
| TOTAL Phase 70 | 433 |