Eve library

phase-70.md

Tracker

Revision 1

On this page

Historical task source. The Eve task board holds current task state.

Phase 70: Production & Filmmaking SOTA Gap Closure#

This phase closes every identified gap from the comprehensive industry gap analysis (see docs/releases/v1/reports/oshun-vs-industry-gap-analysis.md). It covers capabilities possessed by ILM, Pixar, Weta, Disney, Netflix, and 30+ AI filmmaking companies that are not yet represented in the Oshun roadmap. All tasks target SOTA parity or leadership.

Dependency: Phases 1-26 (foundation), Phase 42-44 (Neith engine/studio) for some tasks. Many tasks can proceed independently.


70.1 Per-Production AI Post-Production (@isis/post-production-ai)#

InterPositive (Netflix, $600M) equivalent. The single most critical gap. Train AI models on a production's own dailies for relighting, reframing, BG replacement, and VFX — all without traditional post-production pipelines.

70.1.1 Dailies Ingestion & Cinematic Analysis Pipeline#

  • 70.1.1.1 Create @isis/post-production-ai library with project.json (scope:isis, layer:domain)
  • 70.1.1.2 Implement dailies ingestion pipeline supporting ProRes, RAW, ARRIRAW, R3D, BRAW, CinemaDNG, XAVC, MXF codecs
  • 70.1.1.3 Create per-frame metadata extraction engine (EXIF, sidecar XML, camera metadata)
  • 70.1.1.4 Implement focal length detection from frame analysis (wide, normal, telephoto, macro classification)
  • 70.1.1.5 Create camera movement classifier (static, pan, tilt, dolly, tracking, crane, handheld, Steadicam, drone)
  • 70.1.1.6 Implement depth-of-field estimator (shallow/medium/deep, bokeh analysis, focus plane detection)
  • 70.1.1.7 Create camera angle classifier (eye-level, low-angle, high-angle, bird's-eye, dutch, over-the-shoulder)
  • 70.1.1.8 Implement occlusion detection (foreground/midground/background layer segmentation)
  • 70.1.1.9 Create parallax analysis (object depth displacement across frames, motion parallax estimation)
  • 70.1.1.10 Implement shot boundary detection with automatic scene segmentation
  • 70.1.1.11 Create shot-type classifier (extreme wide, wide, medium, close-up, extreme close-up, insert)
  • 70.1.1.12 Implement lighting analysis per frame (key/fill/back light ratios, color temperature, direction)
  • 70.1.1.13 Create lens distortion profiling (barrel, pincushion, chromatic aberration detection)
  • 70.1.1.14 Implement color palette extraction per shot (dominant colors, color harmony analysis)
  • 70.1.1.15 Create cinematic element captioner model combining all analyses into unified metadata (inspired by InterPositive patent US 12,511,904 B1)

70.1.2 Per-Production Model Training#

  • 70.1.2.1 Design per-production model architecture (lightweight diffusion or LoRA fine-tune on base model)
  • 70.1.2.2 Implement automatic training data preparation from dailies (frame selection, augmentation, pairing)
  • 70.1.2.3 Create production visual vocabulary builder (learn lighting signatures, lens characteristics, color palette)
  • 70.1.2.4 Implement editorial consistency learning (shot-to-shot relationship modeling, match continuity)
  • 70.1.2.5 Create actor appearance model (face/body/wardrobe consistency across shots without deepfake generation)
  • 70.1.2.6 Implement set/environment model (location appearance learning for background consistency)
  • 70.1.2.7 Create prop and vehicle tracking model
  • 70.1.2.8 Implement training progress monitoring with quality metrics (FID, LPIPS against validation set)
  • 70.1.2.9 Create model versioning with A/B comparison interface
  • 70.1.2.10 Implement incremental training as new dailies arrive

70.1.3 AI Relighting Engine#

  • 70.1.3.1 Implement inverse rendering from production footage (estimate scene geometry, materials, lighting)
  • 70.1.3.2 Create light source addition in already-shot footage (synthetic key/fill/back/rim lights)
  • 70.1.3.3 Implement light source removal/attenuation from footage
  • 70.1.3.4 Create shot-to-shot lighting match (source shot → target shot lighting transfer)
  • 70.1.3.5 Implement time-of-day modification (day-to-night, night-to-day, golden hour)
  • 70.1.3.6 Create color temperature adjustment with physically-based color science
  • 70.1.3.7 Implement shadow modification (soften, harden, redirect, remove)
  • 70.1.3.8 Create specular/reflection modification
  • 70.1.3.9 Implement volumetric lighting addition (god rays, atmosphere, haze)
  • 70.1.3.10 Create HDR light probe estimation from footage for VFX integration

70.1.4 AI Shot Reframing & Coverage Synthesis#

  • 70.1.4.1 Implement AI-powered shot reframing (extract different compositions from existing footage)
  • 70.1.4.2 Create virtual camera repositioning (synthesize new angle from existing footage using production model)
  • 70.1.4.3 Implement missing coverage synthesis (generate reverse shot, reaction shot, insert from existing footage context)
  • 70.1.4.4 Create resolution-independent reframing (AI upscale cropped regions to maintain quality)
  • 70.1.4.5 Implement aspect ratio conversion (16:9 to 2.39:1, IMAX to standard) with AI fill
  • 70.1.4.6 Create stabilization with AI boundary fill (remove shake without losing frame edges)
  • 70.1.4.7 Implement speed ramp with AI frame generation (smooth slow-motion from 24fps footage)
  • 70.1.4.8 Create clean plate generation from multiple frames
  • 70.1.4.9 Implement wire/rig removal using production model context
  • 70.1.4.10 Create continuity error detection and suggestion system

70.1.5 AI Background Replacement & Enhancement#

  • 70.1.5.1 Implement foreground/background segmentation using production model (actor, set elements, background)
  • 70.1.5.2 Create AI background replacement maintaining lighting, perspective, and lens characteristics
  • 70.1.5.3 Implement background extension (extend frame beyond original boundaries)
  • 70.1.5.4 Create set extension from partial practical sets
  • 70.1.5.5 Implement sky replacement with physically-correct atmospheric scattering
  • 70.1.5.6 Create crowd augmentation (add/remove background extras)
  • 70.1.5.7 Implement environmental VFX insertion (rain, snow, fog, dust with correct lighting interaction)
  • 70.1.5.8 Create signage/branding replacement (product placement, localization)
  • 70.1.5.9 Implement window/screen replacement with correct perspective and reflections
  • 70.1.5.10 Create seasonal/weather modification for exterior shots

70.2 AI Video Enhancement & Restoration (@isis/video-enhancement)#

Topaz Labs Video AI equivalent plus Netflix-grade encoding optimization.

70.2.1 AI Super-Resolution#

  • 70.2.1.1 Create @isis/video-enhancement library with project.json (scope:isis, layer:domain)
  • 70.2.1.2 Implement 2x AI upscaling using Real-ESRGAN or equivalent (720p → 1440p)
  • 70.2.1.3 Create 4x AI upscaling (480p → 1920p, 1080p → 4K)
  • 70.2.1.4 Implement 8x AI upscaling for archival/restoration (SD → 4K+)
  • 70.2.1.5 Create film-aware upscaling that preserves grain structure
  • 70.2.1.6 Implement face-aware upscaling with detail enhancement for facial regions
  • 70.2.1.7 Create text-aware upscaling preserving legibility of on-screen text and signage
  • 70.2.1.8 Implement batch upscaling pipeline with GPU queue management
  • 70.2.1.9 Create quality metrics dashboard (VMAF, SSIM, PSNR before/ after comparison)
  • 70.2.1.10 Implement model selection heuristic (choose optimal model based on content analysis)

70.2.2 AI Frame Interpolation#

  • 70.2.2.1 Implement optical-flow-based frame interpolation (24→48, 24→60, 30→60, 30→120 fps)
  • 70.2.2.2 Create RIFE or equivalent neural frame interpolation
  • 70.2.2.3 Implement scene-change-aware interpolation (no ghosting at cuts)
  • 70.2.2.4 Create variable speed retiming with smooth interpolation
  • 70.2.2.5 Implement artifact detection and fallback (detect and avoid hallucinated frames)
  • 70.2.2.6 Create slow-motion synthesis from 24fps source (2x, 4x, 8x)
  • 70.2.2.7 Implement cadence correction (remove 3:2 pulldown, convert interlaced to progressive)
  • 70.2.2.8 Create high-speed footage interpolation (120fps → 240fps for extreme slow-mo)

70.2.3 AI Denoising & Restoration#

  • 70.2.3.1 Implement AI denoising for low-light footage (preserve detail, remove noise)
  • 70.2.3.2 Create AI denoising for high-ISO digital footage
  • 70.2.3.3 Implement film grain management (removal, synthesis, matching between shots)
  • 70.2.3.4 Create film grain synthesis engine (Netflix AV1 FGS equivalent — strip grain before encoding, resynthesize at decoder)
  • 70.2.3.5 Implement codec artifact removal (deblocking, deringing for compressed sources)
  • 70.2.3.6 Create AI sharpening with artifact-free detail enhancement
  • 70.2.3.7 Implement AI dejudder (remove motion judder from panned shots)
  • 70.2.3.8 Create flicker removal for archival/timelapse footage
  • 70.2.3.9 Implement scratch and damage repair for film scans
  • 70.2.3.10 Create AI color restoration for faded/degraded footage

70.2.4 ONNX Model Runner#

  • 70.2.4.1 Implement ONNX runtime integration for custom ML models
  • 70.2.4.2 Create model loading interface (file, URL, registry)
  • 70.2.4.3 Implement GPU inference (CUDA, DirectML, CoreML backends)
  • 70.2.4.4 Create model chaining (pipe output of one model to input of another)
  • 70.2.4.5 Implement per-shot model application with Nuke CopyCat-style training-on-production-footage workflow

70.3 AI Face Synthesis & De-Aging (@isis/face-synthesis)#

Metaphysic/Brahma (DNEG), ILM FaceSwap, MARZ Vanity AI equivalent.

70.3.1 Face De-Aging & Aging#

  • 70.3.1.1 Create @isis/face-synthesis library with project.json (scope:isis, layer:domain)
  • 70.3.1.2 Implement age regression model (de-age actor to younger appearance: -5, -10, -20, -30, -40 years)
  • 70.3.1.3 Create age progression model (age actor: +5, +10, +20, +30 years)
  • 70.3.1.4 Implement reference-guided de-aging (use archival photos/ footage of actor at target age)
  • 70.3.1.5 Create temporal consistency enforcement (no flickering between frames)
  • 70.3.1.6 Implement expression preservation during age transformation
  • 70.3.1.7 Create lighting-consistent age transformation (match scene lighting on de-aged face)
  • 70.3.1.8 Implement multi-angle consistency (de-aged face consistent across camera angles)
  • 70.3.1.9 Create body-age consistency (match neck, hands, posture to facial age)
  • 70.3.1.10 Implement quality validation against reference footage

70.3.2 Creative Face Synthesis#

  • 70.3.2.1 Implement actor-to-character face transfer (map actor performance to different character face)
  • 70.3.2.2 Create volumetric 3D face reconstruction from 2D video (monocular 3D face estimation)
  • 70.3.2.3 Implement expression transfer across faces (actor A's expression on actor B's face)
  • 70.3.2.4 Create face shape modification (subtle structural changes while preserving identity)
  • 70.3.2.5 Implement injury/scar/makeup synthesis (add/remove facial features in post)
  • 70.3.2.6 Create digital cosmetic adjustment (skin smoothing, blemish removal, subtle enhancement)
  • 70.3.2.7 Implement eye-line correction (redirect gaze in post)
  • 70.3.2.8 Create stunt face replacement (replace stunt double's face with actor's face)
  • 70.3.2.9 Implement crowd face generation (unique faces for background extras)
  • 70.3.2.10 Create ethical safeguards (consent verification, usage logging, watermarking of synthesized faces)

70.4 Visual Dubbing & Dialogue Editing (@isis/visual-dubbing)#

Flawless TrueSync, Flawless DeepEditor, MARZ LipDub equivalent.

70.4.1 Visual Dubbing (Cross-Language Lip Sync)#

  • 70.4.1.1 Create @isis/visual-dubbing library with project.json (scope:isis, layer:domain)
  • 70.4.1.2 Implement phoneme extraction from dubbed audio track
  • 70.4.1.3 Create viseme mapping for target language (phoneme → mouth shape)
  • 70.4.1.4 Implement volumetric face representation for natural mouth movement modification
  • 70.4.1.5 Create lip movement adaptation engine (modify mouth movements to match dubbed dialogue)
  • 70.4.1.6 Implement emotional delivery preservation (maintain original performance intensity in adapted visuals)
  • 70.4.1.7 Create jaw/chin movement adaptation for dubbed timing
  • 70.4.1.8 Implement tongue and teeth visibility handling during speech
  • 70.4.1.9 Create multi-language batch processing (dub to 10+ languages from single source)
  • 70.4.1.10 Implement quality validation (lip-sync accuracy scoring per phoneme)

70.4.2 Dialogue Editing in Post (DeepEditor Equivalent)#

  • 70.4.2.1 Implement same-language dialogue replacement (change spoken words while maintaining lip sync)
  • 70.4.2.2 Create word-level surgical editing (replace individual words without re-rendering full sentence)
  • 70.4.2.3 Implement ADR-free pickup system (synthesize corrected line delivery with matching lip movement)
  • 70.4.2.4 Create profanity replacement for broadcast versions (visual + audio replacement)
  • 70.4.2.5 Implement whisper/shout modification (change vocal intensity with matching visual)
  • 70.4.2.6 Create timing adjustment (stretch/compress dialogue timing with visual adaptation)
  • 70.4.2.7 Implement emotional regrade (same words, different emotional delivery)
  • 70.4.2.8 Create integration with @isis/voice-cloning for voice + visual synchronization

70.5 Professional Voice Cloning & Synthesis (@isis/voice-cloning)#

ElevenLabs Professional, Respeecher equivalent for film/TV production.

70.5.1 Voice Cloning Pipeline#

  • 70.5.1.1 Create @isis/voice-cloning library with project.json (scope:isis, layer:domain)
  • 70.5.1.2 Implement voice profile creation from reference audio (minimum 30 minutes clean audio)
  • 70.5.1.3 Create rapid voice profile from limited samples (5-10 minutes for lower fidelity)
  • 70.5.1.4 Implement speech-to-speech conversion (preserve intonation, pacing, emotion while changing voice)
  • 70.5.1.5 Create text-to-speech with cloned voice (generate new dialogue in actor's voice)
  • 70.5.1.6 Implement age-modified voice synthesis (younger/older version of cloned voice)
  • 70.5.1.7 Create emotional control tags (whisper, shout, cry, laugh, sarcasm, anger, tenderness)
  • 70.5.1.8 Implement archival voice restoration (reconstruct degraded historical recordings)
  • 70.5.1.9 Create multilingual voice preservation (same voice in multiple languages)
  • 70.5.1.10 Implement multi-provider routing (ElevenLabs, Respeecher, local models) with quality-based selection

70.5.2 Production Integration#

  • 70.5.2.1 Implement ADR replacement workflow (actor records guide → AI generates final in character voice)
  • 70.5.2.2 Create walla/background dialogue generation (crowd murmur, background conversation)
  • 70.5.2.3 Implement voice consistency validation across scenes
  • 70.5.2.4 Create consent and rights management system for cloned voices
  • 70.5.2.5 Implement voice watermarking (embed provenance in synthesized audio)

70.6 AI Music Generation for Film & Game Scoring (@euterpe/ai-scoring)#

Suno, Udio, AIVA, Google Lyria integration with production-aware scoring.

70.6.1 Multi-Provider Music Generation#

  • 70.6.1.1 Create @euterpe/ai-scoring library with project.json (scope:euterpe, layer:domain)
  • 70.6.1.2 Implement Suno API provider adapter (text-to-song, style control, vocal generation)
  • 70.6.1.3 Create Udio API provider adapter (text-to-song, custom mode, lyric timing)
  • 70.6.1.4 Implement AIVA API provider adapter (250+ styles, MIDI export, orchestral focus)
  • 70.6.1.5 Create Google Lyria API provider adapter (vocals, lyrics, cover art)
  • 70.6.1.6 Implement Boomy API provider adapter (genre/mood selection, quick generation)
  • 70.6.1.7 Create multi-provider ensemble routing (generate with N providers, score, select best)
  • 70.6.1.8 Implement generation cost optimizer (route to cheapest provider meeting quality threshold)

70.6.2 Scene-Aware Film/Game Scoring#

  • 70.6.2.1 Implement scene mood analyzer (analyze video/script for emotional beats)
  • 70.6.2.2 Create prompt-to-score pipeline (scene description → musical score)
  • 70.6.2.3 Implement style-consistent generation across scenes (maintain musical identity throughout production)
  • 70.6.2.4 Create leitmotif system (assign and maintain character/theme musical signatures)
  • 70.6.2.5 Implement tempo/intensity matching to scene pacing
  • 70.6.2.6 Create stem separation for generated music (vocals, drums, bass, melody, strings)
  • 70.6.2.7 Implement MIDI export for DAW refinement by human composers
  • 70.6.2.8 Create adaptive game music system (generate variations for combat, exploration, cutscene, menu)
  • 70.6.2.9 Implement music-to-video sync (hit points, tempo mapping, beat alignment to cuts)
  • 70.6.2.10 Create commercial licensing verification (ensure generated music is cleared for commercial use)

70.7 Color Science & HDR Pipeline (@oshun/color-science)#

ACES, OpenColorIO, Dolby Vision, HDR10+ — industry-standard color management.

70.7.1 ACES Integration#

  • 70.7.1.1 Create @oshun/color-science shared library with project.json (scope:shared, layer:foundation)
  • 70.7.1.2 Implement ACES 2065-1 (linear, AP0) as interchange color space
  • 70.7.1.3 Create ACEScg (AP1) as working color space for rendering and compositing
  • 70.7.1.4 Implement ACES output transforms (RRT + ODT) for SDR, HDR, and P3 displays
  • 70.7.1.5 Create ACES Metadata File (AMF) reader/writer
  • 70.7.1.6 Implement camera-specific IDTs (Input Device Transforms) for ARRI, RED, Sony, Canon, Blackmagic
  • 70.7.1.7 Create look modification transforms (LMT) support
  • 70.7.1.8 Implement ACES reference gamut compression (RGC) for out-of- gamut color handling

70.7.2 OpenColorIO Integration#

  • 70.7.2.1 Integrate OpenColorIO (OCIO) v2 for color transform management
  • 70.7.2.2 Create OCIO config generation for production pipelines
  • 70.7.2.3 Implement color space conversion between all major spaces (sRGB, Rec.709, Rec.2020, P3-D65, ACEScg, ACES2065-1)
  • 70.7.2.4 Create display transform management (map scene-referred to display-referred per monitor)
  • 70.7.2.5 Implement GPU-accelerated color transforms (OCIO GPU path)

70.7.3 HDR Mastering Pipeline#

  • 70.7.3.1 Implement Dolby Vision metadata generation (frame-by-frame dynamic metadata)
  • 70.7.3.2 Create HDR10+ metadata generation (scene-by-scene dynamic metadata)
  • 70.7.3.3 Implement HDR-to-SDR trim pass automation
  • 70.7.3.4 Create HDR hero grade workflow (grade HDR first, derive SDR)
  • 70.7.3.5 Implement monitor calibration profile management (P3-D65, Rec.2020, sRGB with per-display profiles)
  • 70.7.3.6 Create HDR quality validation (highlight clipping, color volume coverage, peak brightness verification)
  • 70.7.3.7 Implement wide color gamut visualization (gamut boundary, out-of-gamut indicators)

70.8 Video Encoding & Delivery Pipeline (@oshun/encoding)#

Netflix VMAF, Dynamic Optimizer, AV1 Film Grain Synthesis equivalent.

70.8.1 Perceptual Quality Metrics#

  • 70.8.1.1 Create @oshun/encoding shared library with project.json (scope:shared, layer:foundation)
  • 70.8.1.2 Integrate Netflix VMAF (open-source) for perceptual video quality measurement
  • 70.8.1.3 Implement SSIM and MS-SSIM quality metrics
  • 70.8.1.4 Create PSNR measurement with per-channel analysis
  • 70.8.1.5 Implement quality comparison dashboard (source vs encoded, side-by-side, wipe, diff)

70.8.2 Shot-Based Encoding Optimization#

  • 70.8.2.1 Implement scene detection for encoding segmentation (IDR frames at shot boundaries)
  • 70.8.2.2 Create per-shot complexity analysis (spatial and temporal complexity scoring)
  • 70.8.2.3 Implement per-shot bitrate allocation optimizer (allocate bits based on VMAF quality target)
  • 70.8.2.4 Create adaptive bitrate ladder generation (per-title encoding — custom resolution/bitrate pairs per content)
  • 70.8.2.5 Implement convex hull optimization for encoding parameter selection

70.8.3 Advanced Codec Support#

  • 70.8.3.1 Implement AV1 encoding with film grain synthesis (strip grain → encode → resynthesize at decoder)
  • 70.8.3.2 Create HEVC encoding with Dolby Vision metadata injection
  • 70.8.3.3 Implement VP9 encoding for web delivery
  • 70.8.3.4 Create AVIF image encoding for thumbnails and artwork (significant savings over JPEG)
  • 70.8.3.5 Implement ProRes encoding for editorial/finishing delivery

70.8.4 IMF Delivery Pipeline#

  • 70.8.4.1 Implement IMF (Interoperable Master Format, SMPTE st2067-2) package creation
  • 70.8.4.2 Integrate Netflix Photon (open-source) for IMF validation
  • 70.8.4.3 Create Dolby Vision IMF packaging (derive DV, HDR10, SDR from single master)
  • 70.8.4.4 Implement Dolby Atmos IMF packaging (IAB track files)
  • 70.8.4.5 Create delivery QC pipeline (encoding artifacts, A/V sync, subtitle timing, metadata validation)

70.9 Content Security & Provenance (@oshun/content-security)#

Netflix forensic watermarking, Google SynthID, C2PA — cross-domain shared library.

70.9.1 Forensic Watermarking#

  • 70.9.1.1 Create @oshun/content-security shared library with project.json (scope:shared, layer:foundation)
  • 70.9.1.2 Implement invisible video watermarking (per-session unique identifier embedded in video stream)
  • 70.9.1.3 Create invisible audio watermarking (survives transcoding, compression, speed changes)
  • 70.9.1.4 Implement image watermarking (invisible, survives screenshots, cropping, resizing)
  • 70.9.1.5 Create document watermarking for scripts, storyboards, call sheets (dynamic, per-recipient)
  • 70.9.1.6 Implement watermark extraction and source identification (trace leaked content to source)
  • 70.9.1.7 Create watermark robustness testing suite (verify survival across transcoding, compression, screen capture)

70.9.2 AI Content Provenance#

  • 70.9.2.1 Integrate C2PA (Coalition for Content Provenance and Authenticity) standard for content credentials
  • 70.9.2.2 Implement Google SynthID integration for watermarking AI-generated content (text, image, audio, video)
  • 70.9.2.3 Create provenance chain tracking (record generation history from AI model to final output)
  • 70.9.2.4 Implement AI-generated content declaration metadata
  • 70.9.2.5 Create content authenticity verification API (verify if content is human-created, AI-assisted, or AI-generated)

70.9.3 DRM Integration#

  • 70.9.3.1 Implement Google Widevine DRM integration (Android, Chrome, Smart TVs)
  • 70.9.3.2 Create Apple FairPlay DRM integration (iOS, macOS, Apple TV)
  • 70.9.3.3 Implement Microsoft PlayReady DRM integration (Windows, Xbox)
  • 70.9.3.4 Create multi-DRM license server abstraction (unified API across all DRM providers)
  • 70.9.3.5 Implement hardware DRM enforcement for UHD/4K content per MovieLabs specifications

70.10 Dailies Review & Annotation Pipeline (@yemaya/dailies-review)#

Netflix MPS dailies, Autodesk Flow Capture, Frame.io equivalent.

70.10.1 Dailies Ingestion#

  • 70.10.1.1 Create @yemaya/dailies-review library with project.json (scope:yemaya, layer:domain)
  • 70.10.1.2 Implement dailies ingest from camera cards (OCF — original camera footage)
  • 70.10.1.3 Create automatic sound sync (match audio recorder timecode to camera timecode)
  • 70.10.1.4 Implement automatic color application (apply show LUT or CDL for viewing)
  • 70.10.1.5 Create proxy generation for review (H.264 1080p for fast streaming review)
  • 70.10.1.6 Implement metadata preservation (camera metadata, scene/take numbers, circle takes)
  • 70.10.1.7 Create automatic delivery to editorial (synced, color- applied dailies in editorial codec)

70.10.2 Frame-Accurate Review & Annotation#

  • 70.10.2.1 Implement frame-accurate video playback with timecode overlay
  • 70.10.2.2 Create drawing annotation tools (freehand, shapes, arrows, text on video frames)
  • 70.10.2.3 Implement per-frame notes system with threaded comments
  • 70.10.2.4 Create version comparison tools (side-by-side, wipe, overlay, onion skin, A/B toggle)
  • 70.10.2.5 Implement review session management (create session, invite reviewers, collect feedback)
  • 70.10.2.6 Create approval workflow (approve, request changes, reject per shot/take)
  • 70.10.2.7 Implement annotation export (PDF contact sheets with notes, EDL with comments)
  • 70.10.2.8 Create mobile review support (review dailies on tablet/ phone with annotation)

70.10.3 VFX Pulls & Conform#

  • 70.10.3.1 Implement VFX pull automation (extract plates, handles, reference, framing charts per VFX shot)
  • 70.10.3.2 Create conform pipeline (consolidate, trim, and deliver OCF to finishing)
  • 70.10.3.3 Implement ASC MHL (Media Hash List) for file verification integrity
  • 70.10.3.4 Create ASC FDL (Framing Decision List) for reframing decisions
  • 70.10.3.5 Implement OpenTimelineIO (OTIO) integration for editorial timeline interchange

70.11 AI Video Provider Expansion (@isis/video-generation extension)#

Expand beyond Hunyuan/WAN to all major providers with ensemble routing.

70.11.1 New Provider Adapters#

  • 70.11.1.1 Implement Luma AI Ray3 provider adapter (native HDR, 16-bit EXR output, 1080p)
  • 70.11.1.2 Create MiniMax/Hailuo provider adapter (cost-effective, Director Mode, anime/game CG styles)
  • 70.11.1.3 Implement Higgsfield Cinema Studio provider adapter (optical physics simulation, Director Panel)
  • 70.11.1.4 Create Stability AI Stable Video Diffusion provider adapter (self-hosted option)
  • 70.11.1.5 Implement Haiper provider adapter (video repainting)

70.11.2 Multi-Model Ensemble & HDR#

  • 70.11.2.1 Implement multi-model ensemble generation (submit to N providers, collect results)
  • 70.11.2.2 Create perceptual quality scoring for ensemble selection (VMAF-based ranking of results)
  • 70.11.2.3 Implement style consistency scoring (select result most consistent with production's visual language)
  • 70.11.2.4 Create cost-aware routing (quality target → cheapest provider meeting threshold)
  • 70.11.2.5 Implement HDR video generation pipeline (generate → tonemap → validate → EXR export)

70.11.3 Camera & Lens Physics Simulation#

  • 70.11.3.1 Implement virtual lens library (prime 24mm, 35mm, 50mm, 85mm, 135mm; zoom 24-70, 70-200 with accurate optical characteristics)
  • 70.11.3.2 Create depth-of-field simulation with physically accurate bokeh (circle of confusion, aperture blade count)
  • 70.11.3.3 Implement motion blur simulation based on shutter angle
  • 70.11.3.4 Create lens breathing simulation (focal length shift during focus pull)
  • 70.11.3.5 Implement lens distortion and chromatic aberration simulation
  • 70.11.3.6 Create anamorphic lens simulation (horizontal squeeze, oval bokeh, lens flares)

70.12 CG Character Replacement from Video (@aja/cg-replacement)#

Wonder Dynamics / Autodesk Wonder Studio equivalent.

70.12.1 Video-to-CG Pipeline#

  • 70.12.1.1 Create @aja/cg-replacement library with project.json (scope:aja, layer:domain)
  • 70.12.1.2 Implement actor segmentation from video (isolate person from background per frame)
  • 70.12.1.3 Create 3D pose estimation from video for full skeleton extraction
  • 70.12.1.4 Implement performance transfer to CG character rig (retarget human skeleton to character skeleton)
  • 70.12.1.5 Create automatic lighting estimation from video (estimate light positions, colors, intensities from footage)
  • 70.12.1.6 Implement scene decomposition (separate foreground actor, midground elements, background)
  • 70.12.1.7 Create CG character rendering with matched lighting (render CG character under estimated scene lighting)
  • 70.12.1.8 Implement CG-to-scene compositing (insert rendered character back into original footage with correct occlusion)
  • 70.12.1.9 Create shadow and reflection synthesis (CG character casts correct shadows and reflections in scene)
  • 70.12.1.10 Implement temporal consistency enforcement (no flickering or jitter across frames)

70.12.2 Facial Performance Transfer#

  • 70.12.2.1 Implement facial expression extraction from video (FACS-based action unit detection)
  • 70.12.2.2 Create facial retargeting to CG character (transfer actor expressions to different face proportions)
  • 70.12.2.3 Implement eye tracking and gaze transfer
  • 70.12.2.4 Create lip-sync transfer from video to CG character
  • 70.12.2.5 Implement micro-expression preservation during transfer

70.13 AI Text-to-Animation & Inbetweening (@aja/generative-animation)#

DeepMotion SayMotion, Cascadeur AI Inbetweening equivalent.

70.13.1 Text-to-Animation#

  • 70.13.1.1 Create @aja/generative-animation library with project.json (scope:aja, layer:domain)
  • 70.13.1.2 Implement text-to-motion generation using motion diffusion models (text prompt → 3D skeleton animation)
  • 70.13.1.3 Create physics-informed generation (gravity, momentum, balance, ground contact)
  • 70.13.1.4 Implement style-specific generation (walk styles, fight styles, dance styles, emote styles)
  • 70.13.1.5 Create multi-character interaction generation (two-person fight, handshake, conversation)
  • 70.13.1.6 Implement character-specific motion adaptation (body proportions affect motion style)
  • 70.13.1.7 Create motion editing from natural language ("make the walk more confident", "slow down the run")
  • 70.13.1.8 Implement export to BVH, FBX, USD for DCC consumption

70.13.2 AI Inbetweening & Motion Enhancement#

  • 70.13.2.1 Implement physics-aware inbetweening (given 2 keyframes, generate physically correct in-between motion)
  • 70.13.2.2 Create ballistic trajectory calculation for jumping and falling
  • 70.13.2.3 Implement center-of-mass awareness for balance during transitions
  • 70.13.2.4 Create contact-aware interpolation (foot contacts, hand grabs, sitting)
  • 70.13.2.5 Implement motion style transfer (apply motion style of reference clip to different motion content)
  • 70.13.2.6 Create motion quality scoring (rate generated motion for naturalness, physics plausibility, smoothness)
  • 70.13.2.7 Implement iterative refinement loop (generate → score → regenerate until quality threshold met)
  • 70.13.2.8 Create motion looping tools (generate seamless loopable animations for games)

70.14 Render Farm Management (@oshun/render-farm)#

Pixar Tractor, Sony Imageworks OpenCue equivalent.

70.14.1 Job Distribution System#

  • 70.14.1.1 Create @oshun/render-farm shared library with project.json (scope:shared, layer:infrastructure)
  • 70.14.1.2 Implement job submission API (submit render jobs with priority, dependencies, resource requirements)
  • 70.14.1.3 Create job queue with priority scheduling (FIFO, priority, deadline-based, fair-share)
  • 70.14.1.4 Implement blade/worker node management (register, health check, capabilities, load)
  • 70.14.1.5 Create automatic job distribution based on node capabilities (GPU type, memory, storage)
  • 70.14.1.6 Implement job preemption for priority escalation
  • 70.14.1.7 Create dependency graph execution (task A must complete before task B starts)
  • 70.14.1.8 Implement checkpoint-based resume (resume failed jobs from last completed frame)

70.14.2 Resource Management & Cloud Bursting#

  • 70.14.2.1 Implement resource usage tracking per job (CPU, GPU, memory, storage, wall time)
  • 70.14.2.2 Create resource allocation limits per project/user
  • 70.14.2.3 Implement cloud bursting to AWS EC2/RunPod for overflow capacity
  • 70.14.2.4 Create cost estimation before job submission
  • 70.14.2.5 Implement progress tracking with ETA estimation per job
  • 70.14.2.6 Create web dashboard for job monitoring (status, progress, logs, frame previews)
  • 70.14.2.7 Implement alerting for failed jobs, stuck jobs, and quota warnings

70.15 AI Video Object & Interaction Removal (@isis/video-object-removal)#

Netflix VOID production integration. While @nous/video-removal (Phase 47.7) provides the model inference layer, this section covers the production-facing workflows, UI, orchestration, and domain-specific pipelines that make physics-aware video object removal usable in professional film/TV production.

70.15.1 Production Object Removal Workflow#

  • 70.15.1.1 Create @isis/video-object-removal library with project.json (scope:isis, layer:domain) integrating @nous/video-removal inference with ISIS workflow orchestration
  • 70.15.1.2 Implement interactive object selection UI in Yemaya studio: point-click, bounding box, and lasso tools for marking objects to remove with real-time SAM2 mask preview overlay
  • 70.15.1.3 Create VLM quadmask visualization overlay showing primary object (red), overlap zones (yellow), physics-affected regions (blue), and background (transparent) for director review before committing to removal
  • 70.15.1.4 Implement text-prompt-guided object selection ("remove the red car", "remove all background pedestrians") using VLM + SAM2 for hands-free object identification
  • 70.15.1.5 Create quality tier selector in removal UI: Preview (STTN, seconds, low quality for quick check), Standard (DiffuEraser, minutes, good quality), Premium (VOID two-pass, longer, physics-aware SOTA quality)
  • 70.15.1.6 Implement before/after comparison viewer with synchronized playback, wipe transition, overlay diff, and frame-by-frame stepping for QC review
  • 70.15.1.7 Create removal job submission to ISIS generation API with progress tracking, GPU worker assignment, and estimated completion time
  • 70.15.1.8 Implement removal undo/versioning allowing directors to revert specific removals while keeping others in a multi-object edit session
  • 70.15.1.9 Create removal batch queue for submitting multiple shots/scenes for overnight processing with priority ordering
  • 70.15.1.10 Implement removal approval workflow: artist submits → supervisor reviews in comparison viewer → approves or requests revision with annotated notes

70.15.2 Cross-Shot & Episodic Consistency#

  • 70.15.2.1 Implement cross-shot object removal consistency ensuring the same object (character, vehicle, prop) is removed identically across all shots in a scene regardless of camera angle
  • 70.15.2.2 Create character removal pipeline for removing an actor from an entire episode/film: automatic detection across all shots, batch mask generation, consistent background reconstruction
  • 70.15.2.3 Implement background consistency enforcement across cuts — when an object is removed in shot A and shot B of the same scene, the revealed background must match between shots
  • 70.15.2.4 Create continuity validation system comparing removal results across adjacent shots in the edit timeline and flagging inconsistencies
  • 70.15.2.5 Implement shadow and reflection consistency — when removing an object, ensure its shadow and reflections are also removed consistently across all shots where they appear

70.15.3 Domain-Specific Removal Pipelines#

  • 70.15.3.1 Implement regional content adaptation pipeline: remove or replace culturally sensitive elements (alcohol, tobacco, religious symbols, branding) for market-specific versions while maintaining scene coherence
  • 70.15.3.2 Create product placement insertion/removal pipeline: remove existing brands and optionally insert replacement brands with correct perspective, lighting, and surface interaction
  • 70.15.3.3 Implement safety wire and rig removal pipeline extending 70.1.4.9 with VOID physics-awareness — remove wires AND the subtle body deformation they cause on harness-wearing actors
  • 70.15.3.4 Create crowd reduction/augmentation pipeline: selectively remove or thin background extras while maintaining natural crowd dynamics and preserving physics (no floating shadows, correct reflections)
  • 70.15.3.5 Implement vehicle and prop removal with physics-correct ground contact — remove parked cars while reconstructing the ground surface, curb lines, and shadows they occluded
  • 70.15.3.6 Create boom mic and crew removal pipeline for documentary and live event footage where crew inadvertently appears in frame
  • 70.15.3.7 Implement anachronism removal for period productions — remove modern elements (antennas, power lines, aircraft contrails, contemporary signage) that break period authenticity
  • 70.15.3.8 Create weather element removal/addition using VOID's physics awareness — remove rain and its surface splash effects, or remove snow and its accumulation on surfaces

70.15.4 DaVinci Resolve & NLE Integration#

  • 70.15.4.1 Implement DaVinci Resolve plugin (via @bellona/davinci-adapter) for in-timeline object removal: select object in Resolve viewer → generate mask → send to VOID pipeline → receive clean plate → auto-insert on timeline
  • 70.15.4.2 Create OpenFX plugin interface for object removal accessible from any OFX-compatible host (Resolve, Nuke, Flame, Fusion)
  • 70.15.4.3 Implement round-trip metadata preservation: export removal job from NLE with timeline position, handles, and grade → process → import result at exact timeline position with matching metadata
  • 70.15.4.4 Create EXR multi-layer output with separate clean plate, original, mask, and difference layers for compositor flexibility
  • 70.15.4.5 Implement EDL/XML/OTIO-based batch export allowing editors to mark removal points in timeline and batch-submit all removal jobs

70.15.5 Content Provenance & Safety#

  • 70.15.5.1 Implement removal provenance tracking recording what was removed, from which frames, by whom, when, and using which model version — linked to @isis/outputs provenance chain
  • 70.15.5.2 Create C2PA content credential integration (link to 70.9.2) embedding machine-readable record of AI object removal in output file metadata per Coalition for Content Provenance and Authenticity standard
  • 70.15.5.3 Implement removal audit log with frame-accurate before/after snapshots stored immutably for legal and compliance review
  • 70.15.5.4 Create deepfake/manipulation detection resistance — ensure removal outputs include SynthID watermarks (link to 70.9.2.2) identifying them as AI-modified content
  • 70.15.5.5 Implement ethical guardrails: block removal of faces for identity concealment unless authorized, block removal of evidence markers in forensic contexts, require consent documentation for actor removal

70.16 Generative Relighting Studio (@isis/relight-studio)#

While Phase 70.1.3 covers relighting of already-shot footage via inverse rendering, this section covers DIFFUSION-BASED generative relighting using IC-Light, GenLit, and LightLab integrated into production workflows. Enables relighting of generated content, composited scenes, and virtual environments with artistic lighting control.

70.16.1 Interactive Relighting Workflow#

  • 70.16.1.1 Create @isis/relight-studio library with project.json (scope:isis, layer:domain) integrating IC-Light, GenLit, and LightLab inference with ISIS workflow orchestration
  • 70.16.1.2 Implement interactive light placement UI in Yemaya studio: click to place point, directional, and area lights on image or video with real-time position and orientation gizmos
  • 70.16.1.3 Create HDRI environment map browser with thumbnail preview, category filtering, and support for loading from library or capturing from 360-degree footage
  • 70.16.1.4 Implement real-time relighting preview using IC-Light at reduced resolution for interactive feedback during light placement and parameter adjustment
  • 70.16.1.5 Create lighting preset library with production-standard setups: golden hour, moonlight, studio key+fill, neon, dramatic side-light, Rembrandt, butterfly, split, and loop lighting patterns
  • 70.16.1.6 Implement light parameter adjustment controls: color (with color picker and Kelvin temperature), intensity, direction, size, falloff curve, and cone angle with real-time visual feedback
  • 70.16.1.7 Create batch relighting pipeline: apply same lighting setup across multiple frames, shots, or entire sequences for consistent look development across a scene
  • 70.16.1.8 Implement lighting reference matcher: analyze reference photograph or film still to extract lighting parameters (direction, color temperature, ratio, quality) and apply extracted setup to target content
  • 70.16.1.9 Create relighting comparison view: before/after toggle, A/B split between lighting setups, synchronized playback for video, and onion-skin overlay for subtle difference inspection
  • 70.16.1.10 Implement relighting export with alpha, depth, and normal passes for downstream compositor integration in Nuke, Fusion, or After Effects

70.16.2 Shot-to-Shot Relighting Consistency & NLE Integration#

  • 70.16.2.1 Implement cross-shot lighting consistency enforcement: lock lighting parameters across all shots in a scene so relighting results match regardless of original on-set lighting differences
  • 70.16.2.2 Create lighting continuity validation comparing relighting results between adjacent shots in the edit timeline and flagging color temperature, intensity, or direction discontinuities
  • 70.16.2.3 Implement DaVinci Resolve plugin (via @bellona/davinci-adapter) for in-timeline relighting: select shot in Resolve viewer, adjust lights in panel, render relighting in background, auto-insert result on timeline
  • 70.16.2.4 Create OpenFX plugin interface for relighting accessible from any OFX-compatible host (Resolve, Nuke, Flame, Fusion)
  • 70.16.2.5 Implement EXR multi-layer relighting output with separate lit result, individual light passes, normals, depth, and albedo layers for compositor flexibility
  • 70.16.2.6 Create temporal lighting animation: keyframe light parameters over time for effects like sunrise progression, moving spotlight, flickering candle, or lightning flash with smooth interpolation
  • 70.16.2.7 Implement video relighting via GenLit: relight full video sequences using video diffusion model's implicit 3D understanding for temporally consistent results without per-frame flickering
  • 70.16.2.8 Create relighting quality metrics comparing generated lighting against physically-based rendering reference including PSNR, SSIM, and perceptual light direction accuracy scoring

70.17 Unified Video Editing Suite (@isis/video-edit-studio)#

Production integration for VACE (Wan2.1-VACE), Motion Prompting, Wan-Move, MotionStream, PostCam, and TokenFlow. While Phase 47.10 provides the model inference layer, this section covers the production-facing UI, workflows, and batch processing that make advanced video control usable for editors and directors.

70.17.1 VACE-Based Production Editing Workflow#

  • 70.17.1.1 Create @isis/video-edit-studio library with project.json (scope:isis, layer:domain) integrating VACE, Motion Prompting, and PostCam inference with ISIS workflow orchestration
  • 70.17.1.2 Implement unified video editing UI in Yemaya studio with task selector panel: inpaint, outpaint, restyle, extend, and reference-to-video modes with contextual tool palettes per mode
  • 70.17.1.3 Create drag-and-drop reference image panel for VACE reference-to-video: place reference image of subject, character, or object and generate video maintaining subject identity and appearance
  • 70.17.1.4 Implement interactive mask painting for video inpainting: paint spatiotemporal mask on timeline with frame-by-frame or interpolated keyframe masking, then fill masked region with AI-generated content guided by prompt
  • 70.17.1.5 Create video outpainting controls: extend canvas in any direction (left, right, top, bottom) with aspect ratio selection presets (16:9, 2.39:1, 9:16, 1:1) and content guidance prompt for generated regions
  • 70.17.1.6 Implement text-guided video restyling: type style description, preview restyle on representative keyframe for approval, then apply to full video with temporal consistency enforcement across all frames
  • 70.17.1.7 Create VACE quality tier selection: Fast mode (1.3B model, preview quality, seconds per clip) vs Premium mode (14B model, production quality, minutes per clip) with automatic tier recommendation based on output requirements
  • 70.17.1.8 Implement VACE batch editing pipeline: apply same edit operation (inpaint, restyle, outpaint) across multiple shots or scenes for episodic production with consistent parameters and style
  • 70.17.1.9 Create edit comparison viewer: original vs edited side-by-side with synchronized playback, frame stepping, horizontal/vertical wipe, and difference overlay visualization
  • 70.17.1.10 Implement edit versioning: save multiple edit variants per shot, compare variants in grid view, select preferred version, and track version history with rollback capability

70.17.2 Interactive Motion & Camera Control UI#

  • 70.17.2.1 Implement motion trajectory painting UI: draw object motion paths directly on first frame with bezier curve handles, speed control via point spacing, and real-time trajectory preview overlay
  • 70.17.2.2 Create camera trajectory designer: 3D viewport with interactive camera path editor for PostCam and DepthDirector control, showing camera frustum preview along the path
  • 70.17.2.3 Implement cinematographic camera preset library: orbit, dolly, crane, tracking, reveal, push-in, pull-out, and whip-pan as one-click trajectory presets with adjustable magnitude and timing
  • 70.17.2.4 Create motion intensity slider: control magnitude of object/camera movement from subtle (micro-movement) to dramatic (full traversal) with proportional scaling of all trajectory keypoints
  • 70.17.2.5 Implement multi-object motion control: draw independent trajectories for multiple objects in same video with per-object timing, speed, and easing curves
  • 70.17.2.6 Create motion transfer UI: select source video, extract motion signature (camera movement and object dynamics), apply extracted motion to target content with preview and intensity adjustment
  • 70.17.2.7 Implement real-time motion preview using MotionStream for sub-second visual feedback during trajectory editing, updating preview as user adjusts control points
  • 70.17.2.8 Create motion quality validation: detect physically implausible motions including teleportation, object interpenetration, impossible acceleration, and gravity violations with frame-level annotation
  • 70.17.2.9 Implement DaVinci/NLE round-trip: export motion-controlled video with timeline metadata, handles, and grade preservation for seamless integration into existing editorial workflow
  • 70.17.2.10 Create EDL/XML/OTIO batch motion editing: mark shots in timeline, assign motion templates per shot, batch-process all marked shots, and re-import results at correct timeline positions

70.18 Portrait Animation & Virtual Performance Studio (@isis/portrait-studio)#

Production integration for Hallo3, LivePortrait, MuseTalk, and audio-driven portrait generation. Enables virtual presenter creation, cross-language dubbing with lip replacement, character animation from reference photos, and performance consistency across episodic content.

70.18.1 Virtual Presenter & Performance Pipeline#

  • 70.18.1.1 Create @isis/portrait-studio library with project.json (scope:isis, layer:domain) integrating Hallo3, LivePortrait, MuseTalk, and RAP inference with ISIS workflow orchestration
  • 70.18.1.2 Implement virtual presenter creation wizard: upload reference photo, select voice (from ElevenLabs, F5-TTS, or CosyVoice integration), input script text, and generate full presenter video with synchronized lip-sync and natural gestures
  • 70.18.1.3 Create expression direction controls: specify emotion and energy level per line of script (calm, excited, serious, warm, emphatic, concerned) with smooth transitions between emotional states
  • 70.18.1.4 Implement multi-angle presenter generation: generate same presenter from different camera angles (front, three-quarter left/right, profile) for editing flexibility and shot variety
  • 70.18.1.5 Create presenter wardrobe and background control: specify or change outfit and background via text prompt while preserving identity consistency and natural clothing physics
  • 70.18.1.6 Implement real-time avatar mode using RAP: live microphone input drives real-time portrait animation for interactive presentations, live streams, and virtual meetings with sub-200ms latency
  • 70.18.1.7 Create character animation from reference photo: single photo of fictional, historical, or illustrated character produces animated performance driven by script audio with style-appropriate movement
  • 70.18.1.8 Implement performance consistency enforcement across episodes: same character must look, move, and sound consistent across all generated segments with identity drift detection and correction
  • 70.18.1.9 Create performance quality gates: lip-sync accuracy check (SyncNet score threshold), identity consistency check (face embedding distance), and expression naturalness check before delivery approval
  • 70.18.1.10 Implement batch presenter generation: generate an entire video series with consistent presenter across all episodes, applying uniform visual style, voice, and mannerisms

70.18.2 Cross-Language Dubbing with Portrait Animation#

  • 70.18.2.1 Implement cross-language dubbing pipeline: original video, translate script to target language, generate target language audio via TTS, apply MuseTalk lip replacement to match new audio with original actor's face
  • 70.18.2.2 Create multi-language dubbing batch processor: dub single source video into N target languages in one job with parallel processing and per-language quality validation
  • 70.18.2.3 Implement dubbing quality validation per language: lip-sync accuracy scoring for each target language accounting for phoneme differences (e.g., wider mouth shapes for Romance languages, subtler for East Asian)
  • 70.18.2.4 Create actor approval workflow for AI dubbing: actor reviews AI-dubbed version in secure viewer, approves or requests adjustments with frame-level annotation, and signs off per language
  • 70.18.2.5 Implement emotion preservation across languages: ensure emotional delivery (intensity, pacing, emphasis) matches original performance regardless of target language prosodic differences
  • 70.18.2.6 Create dubbing A/B comparison viewer: original language vs each dubbed version with synchronized playback, frame stepping, and audio waveform overlay for timing comparison
  • 70.18.2.7 Implement subtitle and closed-caption generation from dubbing transcript with accurate timing derived from the generated audio track
  • 70.18.2.8 Create dubbing provenance tracking: record original actor, translator, TTS model, portrait animation model, and approval chain for each dubbed version linked to @isis/outputs provenance system

70.19 Automated Foley & Sound Design Studio (@isis/foley-studio)#

Production integration for AV-Link, FoleyCrafter, and Any2Audio. While Phase 47.12 provides the model inference layer, this section covers the production-facing workflows for generating, reviewing, and delivering AI-generated foley and sound effects for film/TV post-production.

70.19.1 Production Foley Generation Pipeline#

  • 70.19.1.1 Create @isis/foley-studio library with project.json (scope:isis, layer:domain) integrating FoleyCrafter, AV-Link, and Any2Audio inference with ISIS workflow orchestration
  • 70.19.1.2 Implement automatic foley detection: analyze video to identify moments requiring foley (footsteps, impacts, doors, cloth rustling, object interactions) with timecoded event list and material classification
  • 70.19.1.3 Create foley generation queue: detected events are batch-generated as foley sounds via FoleyCrafter/AV-Link with material-aware synthesis matching surface type and impact force
  • 70.19.1.4 Implement foley review interface: video playback with generated foley layered on timeline, per-event approve/reject/regenerate controls, and waveform display synchronized to video
  • 70.19.1.5 Create foley sweetening controls: adjust volume, reverb (room size and decay), EQ (low/mid/high bands), and spatial panning per generated foley event to match production mix requirements
  • 70.19.1.6 Implement foley material override: when auto-detected material is incorrect (e.g., metal detected but surface is wood), override material classification and regenerate with correct surface properties
  • 70.19.1.7 Create ambient soundscape generation from scene analysis: detect environment type (forest, city street, office, beach, cave) and generate continuous ambient bed via Any2Audio matching visual context
  • 70.19.1.8 Implement foley export in industry formats: AAF, OMF, and BWF-P with timecoded metadata for direct import into Pro Tools, Logic Pro, and DaVinci Fairlight
  • 70.19.1.9 Create foley consistency across cuts: ensure same surface and material produces consistent sound characteristics across different shots in the same scene with automatic gain and timbre matching
  • 70.19.1.10 Implement foley library builder: save generated foley sounds to searchable library with material tags, intensity levels, duration metadata, and source video reference for future reuse

70.19.2 Sound Design from Storyboards & Quality Validation#

  • 70.19.2.1 Implement storyboard-to-sound pipeline: storyboard frames plus text descriptions produce preliminary sound design for pre-production previsualization and director review before principal photography
  • 70.19.2.2 Create text-to-sound-effect generation for editors: type natural language description (e.g., "heavy wooden door creaking open slowly") and generate matching sound effect with multiple variants to choose from
  • 70.19.2.3 Implement audio-visual sync validation: automated check that generated foley onsets align within plus-or-minus 2 frames of visual contact points with per-event sync accuracy reporting
  • 70.19.2.4 Create A/B comparison between AI-generated foley and traditional library foley for quality evaluation with blind listening test workflow and preference scoring
  • 70.19.2.5 Implement foley budgeting: estimate number of foley events per reel from video analysis and generate cost/time comparison versus traditional foley stage recording
  • 70.19.2.6 Create sound design versioning: save and compare multiple foley and sound design passes for same edit with diff view showing changed, added, and removed sound events between versions
  • 70.19.2.7 Implement integration with existing sound editing pipeline: generated foley arrives in editor's DAW (Pro Tools, Fairlight, Logic Pro) at correct timecode positions via AAF/OMF round-trip
  • 70.19.2.8 Create sound design quality report: per-event sync accuracy, material classification confidence, spectral quality metrics, and overall mix balance assessment for QC review before final delivery

70.20 Joint Audio-Video Production & Multi-Shot Narrative Studio (@isis/av-narrative-studio)#

Production integration for Seedance 2.0's unique capabilities that have no equivalent elsewhere in the pipeline: joint audio-video generation (synchronized audio+video in one pass), multi-shot narrative from single prompt, multi-asset reference workflows, one-click video recreation, and prompt-driven camera planning. While Phase 17.7 provides the API client and Phase 47.16 covers the model-agnostic paradigm, this section covers production-facing workflows.

70.20.1 Joint Audio-Video Production Workflow#

  • 70.20.1.1 Create @isis/av-narrative-studio library with project.json, package.json, tsconfig.json, and barrel exports for joint audio-video production workflows, multi-shot narrative pipelines, and one-click recreation interfaces
  • 70.20.1.2 Implement joint A/V generation UI in Yemaya studio: generate video WITH synchronized audio in single operation — director hears soundtrack as they preview video, with real-time progress for both streams
  • 70.20.1.3 Create audio stem separation from joint output: auto-split generated audio into dialogue, music, effects, ambient layers for mixer control with per-stem gain, pan, and mute/solo toggles
  • 70.20.1.4 Implement selective audio override: keep Seedance's generated video but replace specific audio stems (e.g., replace generated music with licensed track, keep generated SFX) with crossfade and level matching
  • 70.20.1.5 Create A/V quality review interface: synchronized playback with per-stem volume controls, mute/solo per layer, waveform visualization alongside video, and frame-accurate scrubbing
  • 70.20.1.6 Implement joint A/V batch generation: generate multiple video+audio variations from same prompt for director selection with side-by-side comparison and preference ranking
  • 70.20.1.7 Create A/V consistency across scenes: when generating multiple scenes for same project, ensure audio style (music theme, ambient tone, voice character) remains consistent via style reference conditioning
  • 70.20.1.8 Implement audio-first generation mode: provide reference audio/music → generate video that matches audio pacing, energy, and mood with beat-aligned cut suggestions
  • 70.20.1.9 Create video-first generation mode: generate video → use as reference for joint A/V regeneration with improved audio sync and director-specified audio direction notes
  • 70.20.1.10 Implement joint A/V export in professional formats: separate video (ProRes/DNxHR) and audio (WAV stems) tracks with embedded timecode for NLE import into Premiere, Resolve, and Final Cut Pro

70.20.2 Multi-Shot Narrative Production Pipeline#

  • 70.20.2.1 Implement narrative prompt interface: director writes scene description → system shows proposed shot breakdown (shot type, duration, camera, transition) before generation with editable parameters per shot
  • 70.20.2.2 Create shot plan editor: drag-and-drop reordering, per-shot camera override, duration adjustment, transition type selection, and preview thumbnail generation before committing to full render
  • 70.20.2.3 Implement character reference integration: upload character photos → maintain consistency across all generated shots in the narrative with automatic face and wardrobe matching verification
  • 70.20.2.4 Create location reference integration: upload location photos → maintain environmental consistency across shots with lighting direction and time-of-day continuity enforcement
  • 70.20.2.5 Implement multi-shot timeline view: generated shots appear on timeline with handles, allowing trimming, reordering, and re-generation of individual shots without affecting adjacent shots
  • 70.20.2.6 Create narrative template library: pre-built shot sequences for common production needs (interview setup, product reveal, chase sequence, dialogue scene, establishing shot → close-up) with customizable parameters
  • 70.20.2.7 Implement episodic consistency: for series production, maintain character, location, and audio style consistency across multiple generated episodes with per-episode and per-season style locks
  • 70.20.2.8 Create multi-shot approval workflow: director reviews shot-by-shot with approve/reject/regenerate per shot, with annotated feedback and revision history tracking
  • 70.20.2.9 Implement shot replacement: regenerate single shot within multi-shot sequence while preserving continuity with adjacent shots using boundary frame conditioning and transition blending
  • 70.20.2.10 Create narrative export: export final multi-shot sequence as single video or individual shots with EDL/XML metadata for NLE import and AAF for audio round-trip

70.20.3 One-Click Recreation & Reference Workflows#

  • 70.20.3.1 Implement one-click video recreation interface: upload reference video → specify what to change (different model/product/branding) → generate new video preserving original pacing, cuts, and camera work
  • 70.20.3.2 Create brand template system: save successful recreations as templates (original video structure + what to swap) for repeatable ad production with per-brand asset presets and approval gates
  • 70.20.3.3 Implement reference asset library: organize and tag reference images/videos/audio for quick retrieval during multi-asset generation with search, filtering, and usage tracking
  • 70.20.3.4 Create reference combination builder: visual interface for assembling 12-asset reference sets with preview of each asset's intended contribution (style, motion, character, environment, audio) and conflict detection
  • 70.20.3.5 Implement motion reference extraction: select specific motion patterns from reference video (just the camera movement, just the subject motion) for targeted conditioning with visual preview of extracted motion path
  • 70.20.3.6 Create style consistency across recreations: when producing multiple recreated videos for same campaign, enforce consistent visual style and audio tone via shared style reference and per-campaign guardrails
  • 70.20.3.7 Implement recreation A/B testing: generate multiple recreation variants → compare in split-screen viewer → select best version with frame-by-frame diff and quality metric overlays
  • 70.20.3.8 Create recreation audit trail: record original reference, recreation parameters, and generated output for content provenance and licensing compliance with exportable chain-of-custody report

Phase 70 Summary Statistics#

Section Task Count
70.1 Per-Production AI Post-Production 55
70.2 AI Video Enhancement & Restoration 33
70.3 AI Face Synthesis & De-Aging 20
70.4 Visual Dubbing & Dialogue Editing 18
70.5 Professional Voice Cloning & Synthesis 15
70.6 AI Music Generation for Film/Game Scoring 18
70.7 Color Science & HDR Pipeline 20
70.8 Video Encoding & Delivery Pipeline 20
70.9 Content Security & Provenance 17
70.10 Dailies Review & Annotation Pipeline 20
70.11 AI Video Provider Expansion 16
70.12 CG Character Replacement from Video 15
70.13 AI Text-to-Animation & Inbetweening 16
70.14 Render Farm Management 15
70.15 AI Video Object & Interaction Removal 33
70.16 Generative Relighting Studio 18
70.17 Unified Video Editing Suite 20
70.18 Portrait Animation & Virtual Performance 18
70.19 Automated Foley & Sound Design 18
70.20 Joint A/V & Multi-Shot Narrative 28
TOTAL Phase 70 433