Agentic Long-Form Video — Research & Lessons for Oshun (2026-07-04)

Scope: the user-flagged cluster of recent approaches — long video composed from smaller pieces with granular per-piece evaluation and re-generation, storyboarding and the full creative pipeline as agent workflow, and structured JSON scene descriptions as a compression that lets agents navigate an emerging media structure. Method: one deep web-research pass (~280 primary-source fetches: arXiv, GitHub, vendor docs; UNVERIFIED items flagged; 3 highest-risk claims independently re-verified) + a full inventory of our own adjacent systems, then a mapping: what the field converged on, what we already hold, and exactly what to take.

Companion artifacts: V1_COMFYUI_SOTA_AUDIT_2026-07-03.md (the execution substrate this report builds on).

1. What the field converged on (condensed, with receipts)#

1.1 Long video from pieces is now engineering, not research#

The 2024–2026 arc solved chunked rollout stability in three moves: per-token/per-chunk noise causality (Diffusion Forcing, Jul 2024 → Rolling Diffusion → MAGI-1's 24-frame chunk pipeline), distilling bidirectional teachers into KV-cached causal students (CausVid, Dec 2024), and — the decisive one — training the student on its own rollouts (Self-Forcing, Jun 2025; Self-Forcing++ reaches 4min15s; BAgger removes the teacher entirely). FramePack (Apr 2025) made context cost constant in duration via importance-weighted frame packing; LongCat-Video (Oct 2025) pretrained natively on continuation and reports "minutes-long without color drift". By 1H-2026 the open problems moved up a level: KV-cache and anchor-frame ("attention sink") engineering for identity persistence at minute-plus horizons (LongLive frame sinks, Rolling Forcing, the 2026 DySink/AdaState cluster).

What practitioners actually run (ComfyUI land, all verified):

• Native context windows in core since 2025-08-14 (PR #9238 — ContextWindowsManual/WanContextWindowsManual, 81-frame window / 30 overlap, pyramid fusing) — our pinned v0.27.0 already ships these.
• First/last-frame (FLF) chaining via official Wan FLF2V templates, with a documented failure mode: color/contrast drift per segment (Wan2.2 #172).
• VACE (ali-vilab, ICCV 2025) as the workhorse: firstclip/lastclip/ firstlastclip temporal extension conditions each new segment on a clip, not one frame (drifts least), plus masked spatio-temporal V2V.
• The economic enabler: lightx2v Self-Forcing distill LoRAs (4-step, CFG-free) made every segment ~20x cheaper — this LoRA, not any model, made hobbyist long video practical.
• Finishing stack: per-segment ColorMatch against a canonical reference frame → seam joining → RIFE/GIMM interpolation → upscale.

Products: LTX-2 (open weights Jan 2026, Apache; joint audio-video; ships KeyframeInterpolationPipeline and RetakePipeline as first-class pipelines), Veo 3.1 scene-extension (conditions on the last ~1s) + Ingredients, Kling 3.0 Director Mode (Feb 2026: up to 6 shots in one generation with per-shot camera/size/content), Wan 2.6 R2V (Dec 2025: reference video → new scenes with that character's face + voice). Notable: Sora 2 was discontinued (app 2026-04-26; API retires 2026-09-24) — its storyboard-cards pattern survives in others.

1.2 Granular evaluation + surgical re-generation#

• Zero-shot LMM judging is the measured weak link (VideoJudge: 0.56 Spearman zero-shot → 0.73 trained; UVE: MLLMs beat old metrics but lag humans, worst on fine temporal dynamics). Production loops therefore use trained multi-dimension reward models (VideoScore2, Kling's VideoReward, VisionReward's hierarchical checklist) or pairwise tournaments with reasoning scaffolds (VISTA's three adversarial judge agents; Video-Bench).
• Localization is its own primitive: Spotlight (Nov 2025) annotates errors with spatial+temporal extents (physics errors span long segments; pose glitches are short); VBench-2.0 mixes VLM judges with specialist artifact detectors.
• The published regen loops all converge on regenerate-the-node: VideoRepair (mask and regenerate only misaligned regions, preserving grounded entities — beats full re-rolls at equal compute), CoAgent (Dec 2025: verifier agent triggers selective regen of only inconsistent shots), AniMaker (MCTS-Gen spends candidate budget where value is low; AniEval scores each clip in the context of its neighbors), NeuS-E (temporal-logic model checking finds spec-violating segments), Video-T1 (tree-of-frames search beats Best-of-N).
• Productized: LTX Retake API (regenerate a 2–16s time window inside a generation — video, audio, or both — everything outside bit-preserved), Runway Aleph 2.0 (edit-instead-of-reroll, propagate one edit across shots), Kling Multi-Elements (masked swap/add/delete with tracking).

1.3 Storyboarding: the load-bearing artifact and the human gate#

FilmAgent (Jan 2025) has the best role ablation in the literature: single-pass GPT-4o 2.63 → solo agent 3.04 → full multi-agent (director/ screenwriter/actors/cinematographer with Critique-Correct-Verify and Debate-Judge protocols) 3.98/5. Camera Artist (Apr 2026) shows the same additivity for cinematography-specific stages. FilMaster (Jun 2025) emits an editable industry-standard editing project, not just an MP4, and drives Rough-Cut→Fine-Cut from simulated audience feedback.

Products converge on one placement of the human gate: the shot list / storyboard (Sora cards, Kling Custom mode, LTX scene/shot view, Saga), with script as the second gate, and assembly/grading remaining human-owned even in the most automated pipelines (Hailuo's roadmap: templates → semi-customizable → fully autonomous — automate-then-optionally-gate).

1.4 JSON scene structure as agent-navigable compression#

• No cross-tool shot-JSON standard exists. The de-facto ledger is vendor request bodies + ComfyUI's embedded workflow graph (the only deployed full recipe-graph provenance — and it leaks at the MP4 boundary: VideoHelperSuite drops the workflow blob from MP4). OTIO is the technically right timeline JSON that no major AI video tool emits; agentic editors ship FCPXML/EDL or bespoke JSON. LTX first-party endorses JSON prompting (hold scene+subject constant, vary camera); the ubiquitous "Veo JSON prompting" convention is folk practice, not in Google's docs.
• Index-not-pixels navigation is uniform across long-video agents: Deep Video Discovery (Microsoft, May 2025 — multi-granular DB of clip captions/embeddings/subtitles, o3-class agent with search tools reading "textual description rather than original video pixels", SOTA 74.2% LVBench), VideoTree (query-adaptive keyframe tree), VideoRAG (entity/event knowledge graph grounded across clips), Stanford VideoAgent (answers hour-long questions from ~8.4 frames). Even Gemini's native video is a 1fps timestamped token stream. The evaluation-side index and the generation-side plan converge on the same tree.
• Provenance is a dual ledger: recipe (seeds/models/prompts per node → deterministic regen; ComfyUI graphs, Runway/fal/Replicate seeds) vs authenticity (C2PA manifests + SynthID watermarks). Only ComfyUI unifies them, imperfectly.

1.5 The ten cross-cutting patterns#

1. Keyframe-anchored hierarchy: plan top-down in cheap stills, fill bottom-up with expensive motion; the anchor set IS the plan.
2. Causal student + bidirectional teacher + train-on-own-rollouts.
3. Anchor frames / attention sinks + cache engineering for identity.
4. Reference-bank identity locking (character banks everywhere in 2025).
5. Critic-gated per-shot retry — regenerate the node, not the film.
6. Train the judge; decompose dimensions; judge in neighbor context.
7. The storyboard is the load-bearing artifact and the human gate.
8. JSON shot ledger as source of truth — vendor-shaped, define your own.
9. Index-not-pixels navigation; one tree serves generation AND review.
10. Dual provenance: recipe ledger vs authenticity marking.

2. What we already hold (honest inventory)#

We are much closer to this architecture than a cold read would suggest — most primitives exist; the missing part is the connective tissue.

The five real gaps (each is connective tissue, not a missing primitive):

• G1 — No scene ledger. No canonical hierarchical JSON (film→scene→shot→segment) that carries intent + recipe + artifacts + measurements + lineage per node and is shared by the compiler, the approval workflow, the edit studio, and the agents.
• G2 — No chunk-chaining orchestration or assembly. No last-frame → next-segment conditioning chain, no concat/transition stitcher, no cross-segment color-lock pass (the nodes exist; nothing composes them).
• G3 — No closed per-shot critic→regen loop. Scoring (temporal QA), the regenerate status, and the critique→revise machinery exist in three different libraries and never call each other.
• G4 — Shot recipes aren't addressable. Seeds/params are resolved at compile time but not persisted as regen-capable ledger nodes.
• G5 — No media index for agent review. BLIP + ExtractKeyframes + sophia embeddings exist; nothing builds the captions+keyframes+embeddings tree over produced long-form output.

3. The lessons — what to take, mapped to our systems#

L1 (keystone): Define the Scene Ledger — one tree for plan, recipe, artifacts, scores, and navigation#

The field's deepest convergence (patterns 8+9) is that the generation-side plan and the review-side index are the same hierarchical JSON. Nobody ships a standard; everyone serious defines their own and treats vendor payloads as compilation targets. We should define ours once, in @isis/workflows (zod, versioned like workflow.schema.ts), with per-node:

• intent: beat/action, dialogue, camera language, style/character refs;
• recipe: workflow_id + resolved inputs + seed + model/LoRA versions + reference-image URIs — sufficient for deterministic re-render of THIS node (ComfyUI-graph-in-PNG is the precedent; keep the ledger external — the MP4-metadata route is known-lossy);
• artifacts: clip URI, keyframe thumbnails, audio stems;
• measurements: temporal-QA scores, judge verdicts, approval status — the same fields av-narrative-studio approvals and the BFF release gate already emit;
• index: caption (BLIP node exists), embedding ref, timestamp range;
• lineage: parent node, derived-from retake chain, C2PA at assembly.

Existing consumers line up almost perfectly: storyboard-shot-templates compiles INTO it, sequence-orchestration reads/writes continuity state ON it, av-narrative-studio approvals become node fields, video-edit-studio operations become node mutations with version history, EDL/XML/AAF export becomes a projection of it. This is mostly schema + adapter work, not new algorithms — and it unblocks every other lesson.

L2: Keyframe-first — plan in stills, spend in motion#

Pattern 1 (MovieDreamer → Captain Cinema → LTX KeyframeInterpolation; gate cheap-to-audit anchors before paying for video). We are unusually well positioned: our stills tier already does structural control (union ControlNet), style locking (IPAdapter set), identity (FaceID), and has calibrated image gates. The orchestrated flow to build: beats → keyframe batch (style/identity-locked stills, one per shot) → judge/human gate on stills → i2v per approved keyframe (Wan start_image — already a single conditional graph) → stitch (L4). Rough cost intuition: a rejected still costs ~1/40th of a rejected 5s clip on the same pool — the gate placement is nearly free quality leverage.

L3: Close the surgical-regen loop — regenerate the shot, not the film#

Pattern 5, with published ablations (VideoRepair, AniMaker, CoAgent) that localized regen beats full re-rolls at equal compute, and LTX Retake as the product proof. Wire what we have into one loop:

1. score every shot node — deterministic first (temporal-quality-qa, probe luma consistency, artifact detection store), judge second;
2. select worst node(s) below threshold — scored in neighbor context (AniEval's lesson: action completion and consistency are properties of a shot given its neighbors, not in isolation);
3. regen ONLY that node from its ledger recipe with a bounded delta (seed bump, prompt repair from the critique, neighbor last-frame as anchor) — this is exactly critique-revise.ts pointed at a shot node instead of a generic artifact, honoring its existing budget/best-draft semantics;
4. re-splice + re-run boundary checks (color delta across the seam).

av-narrative-studio's regenerate approval status becomes the human override on the same path, and video-edit-studio's inpaint/extend ops are the finer-grained fallback when only a region (not the whole shot) is bad — which is VACE territory (P2 below).

L4: Chunk-chaining + assembly — compose the primitives we already bake#

The verified practical recipe is anchor-conditioned chaining + per-segment color-lock + interpolated seams — all of which we hold as parts:

• chain: ExtractKeyframes/last-frame → next segment's Wan start_image (catalog workflow addition: video/segment-chain taking prior-segment tail frames as conditioning; the conditional-input machinery from wan-video t2v/i2v carries over directly);
• color-lock: our ColorTransfer (Reinhard/Lab) applied per segment against a canonical reference frame — same role as KJNodes ColorMatch in the community stack, already in our pack; FLF drift is a documented failure mode, so this is not optional polish;
• seams: RIFE (baked) across boundaries; temporal smoother nodes exist;
• assembly: the genuinely missing piece — an ffmpeg concat/transition stitcher (@shared/encoding has shot segmentation but no assembly). One real module: ordered clips + per-boundary transition (cut/dissolve — av-narrative-studio already enumerates the vocabulary) + audio-stem mux → master; probed and gated by the existing video measurement path.
• Also worth noting: ComfyUI-core context windows (v0.27.0) mean our AnimateDiff family can render longer sequences windowed today; ADE context options are already in the video graphs.

L5: Identity as a bank, not a parameter#

Pattern 4 — every serious 2025 product grew a character bank (Sora Characters, Veo Ingredients, Vidu 7-ref, Kling Elements; Wan 2.6 R2V takes a reference video, face + voice). Our taxonomy already names reference-ensemble as a control modality and identity-profiles carries state — but references live per-request today. Make the bank a stored artifact (N curated images + optional voice ref per character/style), addressed from ledger nodes and auto-injected into per-shot recipes (FaceID/IPAdapter today; VACE reference-to-video when adopted). This is the single highest-leverage consistency mechanism available to us short of model work.

L6: Judge design — decompose, scaffold, tournament; never one zero-shot score#

The measured finding (VideoJudge, UVE): zero-shot LMM judging is the weakest link, especially on temporal dynamics — precisely what matters for long-form. Our stack should keep the hierarchy we already believe in: deterministic detectors (temporal QA, artifact detection, probe metrics) for localization and hard floors; LLM/VLM judges only with reasoning scaffolds and pairwise tournaments — and we already own an ε-BT tournament engine in creative-autonomy — plus neighbor-context scoring for shots (L3). If/when volume justifies it, a trained lightweight reward model (VideoScore2/VideoReward pattern) can replace the scaffolded judge; until then, tournaments + detectors are the honest configuration.

L7: The storyboard is the human gate — put it on the ledger, ride autonomy levels#

Product convergence: gate at shot list/storyboard (and script), automate downstream, keep assembly human-owned initially (Hailuo's automate-then-optionally-gate roadmap is the sane maturity curve). We already have autonomy levels + review queue in creative-autonomy; the lesson is placement: the review surface should edit ledger nodes (shot prompts, order, camera fields) pre-generation — cheap text/still review — with per-stage autonomy thresholds (auto-approve keyframes at high trust; always gate first assemblies). FilmAgent's ablation (2.63 → 3.98/5) is the evidence that structured multi-role critique at the plan stage pays; our crew/critique machinery covers this if pointed at storyboard drafts.

L8: Budget-aware test-time search at the shot level#

AniMaker's MCTS-Gen and Video-T1: spend candidates where value is low, not uniformly. Our Thompson bandit + ConcurrencyGovernor already allocate at the idea/variant level; extend the same policy to per-shot candidate counts driven by judge uncertainty and shot importance (establishing shots and identity-critical closeups earn more candidates than cutaways). No new infrastructure — a policy change in how the orchestrator requests variants.

L9: Dual provenance, ledger-first#

Keep the two ledgers distinct and both honest: the recipe ledger lives in scene-ledger nodes (deterministic regen; already our norm at the workflow layer), and authenticity (C2PA at assembly + watermark when a video watermarker lands) stays on the release-gate floors that currently block — correctly — as honest deploy-bound gaps. Do not rely on embedded media metadata for either (the ComfyUI MP4 lesson).

L10: Index-not-pixels review for agents#

Deep Video Discovery's result (74.2% LVBench reading descriptions, not pixels) is the permission slip: our agents should review long-form output through the ledger index — keyframe thumbnails + captions + timestamps + scores — drilling into actual frames only on demand. Builder job: ExtractKeyframes → BLIP captions → sophia embeddings → ledger index fields. This same index is what makes L3's "find the worst segment" cheap, and it is how a critique agent can hold a 3-minute film in a few thousand tokens — the "compression" the user pointed at, exactly.

4. Anti-lessons (what NOT to do)#

• Don't build on Sora API — retiring 2026-09-24 (we don't; our OpenRouter video default is alibaba/wan-2.7, and the durable storyboard products are LTX Studio, Kling 3.0, Hailuo).
• Don't wait for a JSON standard. OTIO is right-in-theory and adopted by zero AI video tools; keep our EDL/XML/AAF export (already in av-narrative-studio) and own the internal schema (L1).
• Don't chase real-time streaming AR models (LongLive, Krea Realtime) on our fleet — wrong fit for contemplative curated content; the value for us is their anchor/cache lessons, not their latency.
• Don't trust single zero-shot judge scores as gates (L6 evidence).
• Don't treat Wan 2.7 open-weights claims as real — API-only verified; the durable open stack is Wan 2.2 + VACE + context windows + distill LoRAs, with LTX-2 as the open audio-video flagship.
• Don't regenerate the film when the shot is bad — and don't regenerate the shot when a region is bad (escalate downward: region → shot → scene; the compute ordering the ablations support).

5. Suggested sequencing (if/when we act on this)#

• P0 — connective tissue, no new models: scene-ledger schema + compiler emit + recipe capture (L1/L9); ffmpeg assembly module with transitions + color-lock stitcher (L4); close the per-shot critic→regen loop over existing scorers/tournaments (L3/L6).
• P1 — orchestrated flows on existing primitives: keyframe-first pipeline (L2); character/style bank artifact (L5); video/segment-chain catalog workflow (L4); media index builder (L10); shot-level candidate budgeting (L8); storyboard gate on autonomy levels (L7).
• P2 — deliberate endpoint additions (same policy as the flagship tier): LTX-2 endpoint (open joint audio-video; native keyframe-interp + retake pipelines map 1:1 onto L2/L3), VACE(-Fun) for masked video edit + clip joining (gives video-edit-studio a real execution backend), lightx2v distill LoRAs for segment economics.

The through-line of every lesson: we own nearly all the organs; the field is telling us the skeleton. The scene ledger is the skeleton, keyframes are the joints, the critic-regen loop is the reflex arc — and all three are composition work over systems that already pass their own gates.

Research honesty ledger (inherited from the research pass): Wan 2.7 open weights, Kling official seed support, Sora storyboard-JSON export, Hailuo internal critique loop, OTIO adoption by any major AI tool, and the 1H-2026 arXiv cache-engineering cluster details are UNVERIFIED; Sora discontinuation, Kling 3.0 Director Mode, LTX Retake API, VACE mechanics, ComfyUI context-window merge, and the FilmAgent/VideoRepair/CoAgent ablations are primary-source verified.

Implementation addendum (same day): all P0/P1/P2 items landed#

Every actionable item in §5 is implemented, tested, and gated. Where things live and what was verified:

P0 — connective tissue

• Scene ledger (libs/isis/workflows/src/ledger/scene-ledger.ts + storyboard-compile.ts): versioned zod tree (film→scene→shot with intent/recipe/artifacts/measurements/index/lineage), pure fail-loud mutators, retake archival (regenerate-the-node, remember-the-node), seed-conflict protection (recordResolvedSeed refuses silent overwrite), timeline computation, edit-list projection that refuses holes, and compilation from the REAL storyboard compiler (explicit per-shot seeds from the baseSeed cascade). 13 spec tests.
• Assembly (libs/shared/encoding/src/timeline-assembly.ts): plan/ execute split, concat-filter cuts + xfade/acrossfade dissolves with correct cumulative offsets, stem mixing (explicit volumes, no loudness claims), duration-mismatch gate, and REAL seam measurement — sRGB→Lab CIE76 deltaE unit-tested against reference values; 23 real-ffmpeg tests (measured: red→blue seam ΔE 176.09 vs 176.31 theoretical).
• Critic→regen loop (ledger/shot-regen-loop.ts): deterministic detectors first (temporal-QA summary is a hard floor + score component), injectable neighbor-context judge, seam drift attributed to the later shot, worst-N budget, per-shot lifetime retake cap, deterministic delta planner (seed+1, anchor-on-drift), stale-seam invalidation, failures reported never disguised. End-to-end spec drives REAL assessTemporalQuality output through the loop.

P1 — orchestrated flows

• Keyframe-first (ledger/keyframe-first.ts): stage A stills with budget-allocated candidates and deterministic per-candidate seed streams; stage B i2v jobs hard-blocked by the keyframe gate.
• Reference bank (ledger/reference-bank.ts): validated character/ style entries (face-image invariant), explicit per-workflow binding table (verified input names), explicit-input-wins injection, kind-checked resolution.
• Candidate budgeting (ledger/candidate-allocation.ts): water-filling with largest remainders, floors/caps, exact totals, unspent reported; ledger-derived importance/uncertainty signals.
• Stage gates (ledger/gate-policy.ts): the automate-then-optionally- gate matrix over the creative-autonomy vocabulary; first assemblies always reviewed.
• Media index (ledger/media-index.ts): sampler/captioner/embedder seams, idempotent build, lexical shot search, and the budget-bounded hierarchical digest (structure never dropped; description shrinks first).
• Segment chain (flagship/wan-segment-chain): previous tail frame → Wan i2v conditioning, Reinhard color-lock vs a canonical reference frame, core ImageFromBatch(-1) tail extraction saved as a first-class output, fast_mode on by default.

P2 — endpoint adoptions (all wire-verified at ComfyUI v0.27.0 tag + HF)

• Distill economics: TI2V-5B Turbo LoRA (quanhaol Self-Forcing, Kijai rank-64 extraction — the only real 5B distill; lightx2v is A14B-only) baked on flagship-video and wired as fast_mode (4 steps, cfg 1) on wan-video + wan-segment-chain.
• VACE (video/vace-video-edit, video/vace-clip-join + new comfyui-vace endpoint, 1.3B baked/14B on-demand): core WanVaceToVideo masked spatio-temporal edit and firstlastclip joining, with new IsisVaceExtendControl/IsisVaceJoinControl pack nodes building the exact gray-0.5/mask-1 control convention (6 pytest cases pin it); CausVid 0.3 + 4-step cfg-1 shift-5 per the official template. VACE is Wan2.1-era — it does NOT run on our baked 5B (honest constraint, documented in-graph).
• LTX-2.3 (flagship/ltx2-av-scene, flagship/ltx2-keyframe-interp + new H100 comfyui-ltx2 endpoint, 22B distilled fp8 + Gemma-3 fp4 + spatial upscaler baked): joint audio-video in one nested-latent sampling pass (blueprint 8-step distilled sigmas, cfg 1, euler_cfg_pp), and the keyframe-first motion engine — first/last/optional-middle stills pinned via chained LTXVAddGuide (strength 0.7, guides cropped before decode); 24fps keeps integer durations on the 8n+1 frame rule.

Renderer contracts hardened for the above (both engines + unit tests): same-name consumer routing for multi-output conditional nodes (the dropped middle-guide reroutes positive/negative/latent index-correctly), and the _condition_drop_consumers schema key (a dropped audio decoder DELETES its consumers' inputs instead of bypassing a type-wrong latent).

Gates at completion: 16 endpoints / 77 workflow references / 0 violations; 63/63 workflows valid; Python render contract 21 tests / 2,680 subtests; TS suite 208 tests; encoding suite 52 tests (real ffmpeg); pack pytest green; tsc -p tsconfig.lib.json clean on both touched libs.

Honest boundaries: contract tests prove graph validity, wire shape, and supply-chain resolvability — not pixels; first deploy should smoke one workflow per endpoint including the two new ones. The 5B-Turbo sampler/ shift beyond "4 steps, no CFG" is not primary-source pinned (base-graph values kept, noted in-graph). LTX-2.3 second-stage latent-upsampling is deliberately not in the catalog graphs yet (the upscaler weight ships in the ltx2 flavor; a two-pass workflow is a follow-on, not a blocker). BFF/persistence bindings for the ledger and reference bank (stores, routes) are the natural next slice — the schemas and seams they bind to are done.