Live Direction, Conductor Runtime, and Blend Kernel

This page documents the three runtime layers that turn a validated Scene Score into a streaming, continuous, viewer-steerable experience: the Live Direction Channel (how a viewer nudges a scene mid-flight), the Conductor Runtime (how segments are scheduled, pre-warmed, and handed off), and the Blend Kernel (how Segment N is stitched perceptually to Segment N+1). It serves the contemplative customer watching a Living Scene and the AAA creator authoring one in the Scene Score Editor; it sits between the Scene Score Schema that defines the artifact and the Composition Surface and Cinematographic Technique Catalog that decides which transition plays where. For the surrounding promise and concepts, start at Living Scenes — Concept and Customer Promise.

Real vs. aspirational — read this first. Everything described here that is decision logic is real, pure, deterministic TypeScript with tests: the Conductor's segment-slot state machine, the Blend Kernel's typed transition contracts and continuity gate, the compatibility scorer, the technique catalog, tone gating, compose-assist budgets, and the scene-event telemetry. Everything pixel- or frame-level — actual latent video generation, learned optical-flow warps, FVD computation, GPU determinism — is not computed in these modules. The Conductor's own header is explicit: "the conductor models scheduling decisions in deterministic state. Actual GPU dispatch lives outside this module." The Blend Kernel header likewise: "Implementations of the actual DSP / shader code live downstream; this module owns the contracts + parameter validation." Continuity metrics (FVD, flicker, color jump, motion vector, audio RMS) are consumed as score inputs by pure gate functions, not measured here. Where a thing is provider-gated or downstream, this page says so plainly.

The two runtimes are real, versioned packages with exactly the names the docs cite: @yemaya/living-scenes-runtime (v0.1.0, libs/yemaya/living-scenes-runtime/package.json) and @yemaya/blend-kernel (v0.1.0, libs/yemaya/blend-kernel/package.json). The shared Score contract lives in @oshun/contracts at libs/contracts/src/living-scene/score.ts and is mirrored into the runtime's score/score-schema so the BFF, queues, and persistence validate against the same schema the runtime trusts. Backlog references: §25.1 (Score / Conductor), §25.2 (cue vocabulary), §25.8 (scene events), §25.18 (continuity release gate), §25.19 (technique catalog), §25.20 (compatibility scorer), §25.23 (tone gating).

Live Direction Channel#

Voice, tap, and text cues are supported on every Living Scene template. Cues are typed events that map onto a constrained verb vocabulary; free text never reaches the segment generator unwrapped. This is the central safety design: the constrained vocabulary is the prompt-injection defense, because raw text is never propagated to the model. The channel enum is voice | tap | text (CueChannelSchema, score.ts:75; CUE_CHANNELS, cue-parser.ts:24).

The verb vocabulary#

The real cue verb set is nine verbs, defined identically in the contract (ScoreCueVerbSchema, score.ts:62-72) and in the runtime parser (CUE_VERBS_VOCABULARY, cue-parser.ts:11-21):

Workflow classes restrict the available verb set per template. For example, tara-contemplative-arc allows only {linger, advance, save-moment, abandon, kill} (template-catalog.ts:50) — morph and shift-style are excluded so a contemplative arc cannot be visually wrenched off-course; Veritas excludes morph outright (morphForbidden: true on the template) to prevent visual claim drift. The Tara sub-variants tighten this further to the same five-verb set (tara.ts:44).

The three input paths#

• Tap path — single-tap = linger, double-tap = advance, long-press = palette of context-sensitive cues. The constrained vocabulary means tap requires no policy pass and takes effect at the next frame boundary.
• Voice path — ASR partial → ASR final → cue parser → Lilith pre-screen → cue queue. Identity-bound by default (the speaker's voiceprint must match the session owner's); shared-viewing mode is opt-in per session and disables identity-binding with explicit notice.
• Text path — typed input → sanitize → cue parser → Lilith pre-screen → cue queue. The sanitizer (sanitizeForCueParse, cue-parser.ts:111-132) is real: it strips control characters (U+0000–U+001F and U+007F), removes quotes and code-fence markers (` " ' and ```), strips <...> tags, collapses whitespace, and truncates to 240 characters before the parser ever sees the input.

Parse outcomes and confidence#

parseCue (cue-parser.ts:49) returns one of three verdicts: parsed (head token matched a vocabulary verb, with extracted args), unrecognised (falls through to the Lilith pre-screen / soft-reject), or low-confidence. For the voice channel, ASR confidence below LOW_CONFIDENCE_THRESHOLD = 0.6 (cue-parser.ts:47) short-circuits to low-confidence rather than risking a misheard directive; tap and text carry confidence 1.

Latency budgets, reconciliation, and crisis handling#

• Cue latency budgets — tap effect ≤ 1 frame at the next segment boundary; voice effect P95 ≤ 800 ms (ASR, cue-policy, and mapping); text effect P95 ≤ 300 ms. Above budget a cue queues for the next segment boundary rather than being dropped. (These are the §25.2 product budgets; the pure parser does not itself enforce wall-clock latency.)
• Sensitive-intention classifier — every voice/text cue is screened against the existing Lilith crisis taxonomy before queueing. A crisis-signal cue triggers an immediate scene crisis frame: fade-to-still, plain operator voice, suspend Iris memory writes, open a safety-incident record. See Scene Safety, Determinism, Provenance, and Cue Privacy and Lilith Persona Policy.
• Cue reconciliation — tap cues are immediate; voice/text cues queue behind tap; conflicts resolve by recency with policy precedence: kill > abandon > advance > linger > the rest.
• Locale parity — cue-policy classifiers reach parity with Lilith conversational classifiers across V1 launch locales; a cue rejected in any supported locale must reject with the same semantic reach in every supported locale.
• Rejected-cue UX — the scene continues without a break; the user gets a soft notice ("let's stay with what's unfolding," "that direction isn't in this lineage") rather than an error modal.

CueSpec on the Score#

A cue that is baked into a Score (as opposed to one issued live) is a CueSpec (score.ts:78-86): {cueId, verb, args, appliesToSegmentId, channels}. channels is serialized as an array because a Set is not JSON-serializable. deepParseScore enforces that every cue either targets the wildcard * or an existing segment id (score.ts:160-163) — a cue that points at a segment that does not exist in the Score is rejected at validation time.

Conductor Runtime#

The Conductor (@yemaya/living-scenes-runtime, conductor/conductor.ts) is a Yemaya-owned runtime that streams a Score through the Psyche session envelope, advances segment-by-segment, plans segment generations for GPU workers, applies Blend Kernel transitions at boundaries, processes the cue queue, and emits a unified frame stream over Psyche events. What lives in this module is the scheduling decision logic as a pure state machine — actual GPU dispatch is outside it.

Segment-slot state machine#

Each segment becomes a SegmentSlot (conductor.ts:24-32) tracking a SegmentRenderState and the carry-state on each side:

SegmentRenderState = pending | pre-warming | ready | streaming | done | aborted

The lifecycle (real exported functions, conductor.ts:16-209):

• initialiseConductor({ score }) — builds one slot per segment, all pending, playhead at index 0, backpressure off. Inbound carry-state is seeded from each segment's inboundCarryState.
• planPreWarm({ state, nowUnixSeconds }) — marks pending slots in the pre-warm window as pre-warming. The window is [playheadIdx + 1, playheadIdx + minLookaheadSegments] (conductor.ts:85-102). This is the lookahead that makes streaming feel seamless: the next segment(s) begin generation before the current segment reaches its blend point.
• completePreWarm({ state, segmentId }) — transitions one pre-warming slot to ready. Completing from any other state throws a ConductorError with code invalid-transition (conductor.ts:116).
• advancePlayhead({ state, carryStateOutbound }) — marks the current slot done, records its outbound carry-state and completion time, then promotes the next slot to streaming — but only if that next slot is already ready or streaming (conductor.ts:150). Crucially, it propagates carry-state across the boundary: carryStateInbound: input.carryStateOutbound ?? next.carryStateInbound (conductor.ts:159). This is the in-memory handoff that the Blend Kernel later consumes.
• setBackpressure({ state, active }) — toggles the backpressure flag.
• planReconnect({ state, atSegmentId }) — returns { resumable, resumeCarryState, reason }. A segment is resumable only if it is streaming or ready; otherwise resumable: false with a reason like "segment in non-resumable state pending". The resume payload is the slot's inbound carry-state — exactly what a reconnecting client needs to pick up mid-segment.

Lookahead and the ≥2 release-gate invariant#

The pre-warm lookahead is governed by RenderEnvelope.minLookaheadSegments, which is a literal union 2 | 3 | 4 (score.ts:105) — not an open integer. deepParseScore enforces a hard floor: a Score whose envelope requires fewer than 2 lookahead segments throws "renderEnvelope must require ≥ 2 lookahead segments" (score.ts:165-167). So the product promise "render lookahead of ≥ 2 segments" is a validated invariant, not a hope. Because Live Direction cues alter the next enqueue rather than interrupting in-flight generation, apparent responsiveness is bounded to roughly one segment-length.

Backpressure: truncation to lookahead 1#

When downstream GPU latency or quality degrades, the caller flips backpressureActive. planPreWarm then sets lookahead = 1 (conductor.ts:86), so the Conductor pre-warms only the immediate next slot instead of the full 2–4 window. This is the deterministic core of the product's backpressure story (extend the current blend window, fall back to longer pre-rendered durations, or degrade to still-image plus narration before failing the session). The pure module models the scheduling truncation; the heavier degradation modes are reported via scene events (see scene.fallback-engaged below).

Reconnect#

The Conductor's reconnect support is durable-session-oriented: on disconnect, a client can call planReconnect with the segment it was watching and resume mid-segment from the carry-state of that slot, within Psyche's reconnect rules. Because carry-state is portable (see next section), the resume does not depend on node-bound latents. See Psyche Real-Time Runtime.

Provider failover#

Per-segment generator failover (e.g. Hunyuan / WAN and other approved engines) is transparent to the user; the failover event is recorded in the Render Envelope's audit log. The choice and execution of the backing model are out-of-module — see External Model Intelligence and Execution Providers and Isis Generation Control.

The CarryState protocol#

The "carry-state protocol" the docs describe in prose is a concrete typed schema (CarryStateSchema, score.ts:38-44). Each segment hand-off carries:

This maps almost 1:1 onto the prose ("CLIP-style style anchor, last-frame conditioning, motion-vector descriptor, color-LUT alignment, audio tail") but the schema pins the exact field names and constraints. Because it is an asset-id, hash, and descriptor bundle — not an opaque node-bound tensor — the same Score reproduces on any approved worker. A segment's inbound carry-state is nullable (inboundCarryState on SegmentSpec); the opening segment of a Score has none.

The RenderEnvelope schema#

The Render Envelope is far richer than "engine version + model hashes" prose. The real RenderEnvelopeSchema (score.ts:91-107) pins concrete literal-typed fields, which is what makes determinism and worker-portability claims auditable:

Blend Kernel and Continuity Evals#

The Blend Kernel (@yemaya/blend-kernel, transitions.ts) is the algorithmic core of perceived continuity. It composes typed transitions and (downstream) emits the boundary frames that bridge Segment N to Segment N+1. The module owns the transition contracts, parameter-range validation, and the continuity scorecard gate — the DSP/shader implementations live downstream.

Nine transition kinds (not five)#

The catalog of cinematographic techniques (next section) composes lower-level transition kinds. The real TRANSITION_KINDS array has nine entries (transitions.ts:13-23), not the five the older prose listed:

Out-of-range parameters throw a TransitionValidationError with code out-of-range, unknown-kind, or missing-required. In product terms: latent warm-start eliminates style discontinuity by conditioning Segment N+1 on Segment N's terminal style anchor and last-frame embedding; optical-flow morph warps the last frames toward the next opening frames to kill pixel snap-cut; color-LUT match normalizes palette/brightness/contrast across the boundary; audio-crossfade runs the narration, ambient, and music buses; and motion-descriptor handoff carries camera motion (pan/dolly/parallax) so the next segment opens consistent with the prior segment's outgoing camera motion.

The continuity scorecard and §25.18 release gate#

isBlendContinuityAcceptable (transitions.ts:239-284) is the deterministic gate that the §25.18 release process calls. It takes a ContinuityScorecard (fvd, flickerScore, colorJump, motionVectorContinuity, audioRmsDeltaDb) and a thresholds object, and returns { acceptable, reasons }. Every metric is first checked for finiteness, then compared — fvd, flickerScore, colorJump must be ≤ their max; motionVectorContinuity must be ≥ its min; and |audioRmsDeltaDb| must be ≤ its max. The metrics themselves are inputs. The kernel does not compute FVD or measure flicker; those numbers are produced downstream and fed in. A below-threshold transition blocks promotion of the workflow class — that is the real meaning of "continuity evals block release."

Segment-pair compatibility scorer (§25.20)#

Before a boundary is even a candidate, scoreSegmentPair (scorer.ts:175-207) decides whether two segments can sit adjacent. It is a pure function over seven dimensions (COMPATIBILITY_DIMENSIONS, scorer.ts:21-29): style-anchor, motion-descriptor, audio-role, narration, tone-band, grounding, persona. The composite is the mean of the seven (compositeOf, scorer.ts:167-173), compared against compositeThresholdSoft and compositeThresholdCompatible. The verdict is hard-incompatible, soft-incompatible, or compatible.

Accuracy note on style/motion scoring. Earlier prose claimed style-anchor distance was a CLIP-embedding distance between outgoing and incoming anchors. That is not what this pure module does. styleScore (scorer.ts:75-77) returns 1 when a.styleAnchorId === b.styleAnchorId and 0.35 otherwise — exact-id equality, not an embedding metric. motionScore (scorer.ts:79-81) is likewise exact-equality (1 vs 0.5). The only dimension using a set-overlap metric is grounding, which uses the Dice coefficient over the segments' groundingSourcePinHashes (diceCoefficient, scorer.ts:68-73). Audio-role, narration, tone-band, and persona are graded comparisons (e.g. narration↔ambient scores 0.85 but music-bed↔silence scores 0.3). Any embedding-based perceptual distance is downstream of this module, not in it.

Hard-incompatible policy gate. policyGate (scorer.ts:118-165) runs before any mechanical scoring and blocks four pairs regardless of numbers:

1. Crisis ↔ non-crisis without a recovery transition — a pair where one side's crisisLabel !== 'safe' and the other is safe is hard-incompatible unless the transition is fade-to-black.
2. Pedagogical ↔ entertainment tone bands intra-Score — forbidden outright.
3. Tara contemplative ↔ forbidden technique — on tara-contemplative-arc, smash-cut, whip-pan, and jump-cut are blocked.
4. Veritas mutually-retracted Sophia pins — on veritas-grounded-explainer, a pair where both segments cite only retracted Sophia source pins is blocked.

Live Direction telemetry: the eight scene events#

The real Live Direction Channel telemetry surface is a set of typed events on the Psyche envelope (libs/oshun/embodiment-psyche/src/events/scene-events.ts, §25.8). There are eight SCENE_EVENT_KINDS (scene-events.ts:13-22):

verifyEventStream (scene-events.ts:180-211) enforces a monotonic, gap-free sequence (each event's sequence must be exactly the previous plus 1) under a single trace-id for the whole stream; a sequence that goes backward, skips a number, or changes trace-id mid-stream throws a SceneEventError (sequence-out-of-order, sequence-gap, or trace-id-missing). This is what makes the scene viewer's feed auditable end-to-end.

Technique catalog, allowlists, and tone gating#

The Blend Kernel transitions are composed into the customer-facing cinematographic techniques in cinematographic-catalog.ts (§25.19). The catalog has 12 entries (TECHNIQUE_IDS, cinematographic-catalog.ts:16-29): hard-cut, match-cut, l-cut, j-cut, dissolve, smash-cut, whip-pan, jump-cut, match-action, cross-cut, montage, fade-to-black. fade-to-black is the universal fallback — it is allow-listed on every template, sits in the crisis-collapse safe set, and is the recovery transition the compatibility scorer demands for crisis↔non-crisis pairs. Every entry ships a provenanceTag matching technique:[a-z0-9-]+ (e.g. technique:match-cut).

Each technique pins concrete evalThresholds (maxFvd, maxFlicker, maxColorJump, maxMotionVectorDelta, minNarrationAlignment). The numbers are real and differ per technique — for example match-cut holds maxFvd 0.3 / minNarrationAlignment 0.92, dissolve maxFvd 0.3 / maxFlicker 0.03, while the energetic smash-cut relaxes to maxFvd 0.5 and jump-cut to maxFvd 0.6. These per-entry numbers are what the continuity gate is configured against.

Crisis-collapse and operator-curated sets#

Two safety sets are real (tone-gating.ts, §25.23):

• CRISIS_COLLAPSE_ALLOWLIST (tone-gating.ts:28-32) = {hard-cut, dissolve, fade-to-black}. Within one frame of crisis-frame activation, the entire technique allowlist collapses to this safe trio regardless of template — assertTechniqueAllowed checks this first, before any other gate (tone-gating.ts:62).
• OPERATOR_CURATED_ARETE_TECHNIQUES (cinematographic-catalog.ts:91-95) = {smash-cut, whip-pan, jump-cut}. On arete-living-offering these require an operatorCuratedTechniqueGrant and are always blocked under Lilith's strictest tone band (tone-gating.ts:92-110).

A third gate is the Themis interlock for Metis: on metis-lesson-visualizer, every technique is blocked on a segment whose node is in an in-progress assessment (tone-gating.ts:69-80). The per-template allowlist itself is enforced at three points (ENFORCEMENT_POINTS, tone-gating.ts:25): the customer technique picker, compose-assist candidate generation, and score promotion.

Duplicated source of truth — reconcile. There are two per-template technique allowlists in code: the per-technique templateAllowlist[] in cinematographic-catalog.ts, and TEMPLATE_TECHNIQUE_ALLOWLISTS in compose-assist.ts. They currently agree for all five templates, but they are duplicated and can drift; treat them as one conceptual allowlist that must be kept in sync. See Composition Surface and Cinematographic Technique Catalog.

Compose-assist: budgets, tier caps, and the promotion gate#

Compose Assist runs as an Isis AgentRun with budget caps and gold-set evaluation, all real in compose-assist.ts. An AgentRunBudget (compose-assist.ts:17-21) caps maxTokens, maxCostCents, and maxLatencyMs; AgentRunGrants (:23-26) constrains toolIds and allowEgress. The gold-set evaluation harness (evaluateGoldSet, :307) returns {precision, recall, refusalRate, correctRefusals, falseRefusals} — refusal quality is graded, not just hit rate, so the assistant is held to correctly refusing rather than merely refusing.

Tier caps and the AAA routing flag#

TIER_CAPS (compose-assist.ts:447-452) and validateCompositionDraft enforce:

The AAA path is uncapped and sets routesToStudioEditor: true, sending the draft to the full Scene Score Editor rather than the inline composer. A crisis-frame draft is blocked entirely (crisis-frame-composition-lock, :498).

The 0.85 continuity promotion gate#

promoteCompositionDraft (:566-611) is the real gate from draft to Score. Beyond rights checks on every segment source, it throws "continuity gate failed" unless continuityPreScore ≥ 0.85 (:585), and re-checks the same ≥ 0.85 floor per boundary in assertPromotableBoundary (:688). So the product's continuity story has a concrete promotion threshold: a draft cannot be promoted unless its pre-score clears 0.85 both overall and at every boundary, and every boundary technique passes the template allowlist and tone gate.

Templates and their eval gates#

The five canonical template machine IDs are tara-contemplative-arc, nyx-sky-briefing, veritas-grounded-explainer, metis-lesson-visualizer, arete-living-offering (LIVING_SCENE_TEMPLATES, template-catalog.ts:9-15) — these, not the friendly names ("Tara Contemplative Arcs", etc.), are what the code keys on. Tara has five sub-variants (TARA_SUBVARIANTS, template-catalog.ts:18-24): loving-kindness, body-scan, breath-anchor, nature-immersion, gratitude. Each is paced to a concrete BreathCycle (tara.ts) — e.g. loving-kindness runs inhale 4 / hold-in 1 / exhale 6 / hold-out 1 seconds, breath-anchor is a square 4/4/4/4.

Every template carries a release-gate eval (TEMPLATE_EVAL_GATES, template-catalog.ts:147-199): all five require minimumFixtures: 30, and each pins per-suite pass-rate thresholds for golden, adversarialCue, crisisTrigger (always 1.0 — crisis handling must never regress), localeParity, accessibilityParity, and groundingCorrectness (null for Tara and Arete, which have no grounding requirement). Veritas is the strictest: golden ≥ 0.97, adversarialCue ≥ 0.99, groundingCorrectness ≥ 0.97. See Domain Templates and the Scene Score Editor and Latency, Accessibility, Eval Sets, and Tests.

Where these surfaces appear#

• Customer viewer (contemplative product) — public scene routes live at apps/oshun/web/src/app/scene/[id]/page.tsx, plus /scene/[id]/embed and /scene/[id]/report.
• Inline compose — apps/oshun/web/src/app/studio/compose (ComposeClient.tsx) on web and apps/oshun/mobile/app/compose.tsx on mobile.
• AAA / operator Scene Score Editor — apps/yemaya/studio-web/src/score-editor/ScoreEditorPage.tsx (and a desktop sibling under apps/yemaya/studio-desktop). Note that a second Scene/Tara editor also lives on the Oshun web app at apps/oshun/web/src/app/lilith-studio/scenes/TaraSceneEditor.tsx (and /lilith-studio/scene/new) — a "Lilith Studio" surface on the contemplative product, which complicates any claim that the editor "does not appear on the contemplative product under any entitlement."
• BFF — apps/oshun/bff/src/routes/living-scenes.ts is ~1,871 lines of real keep / share / idempotency logic, e2e-tested via scene-viewer-deepening.spec.ts, living-scene-card.spec.ts, and public-scene-abuse-report.spec.ts.

Related#

• Living Scenes — Concept and Customer Promise
• Scene Score Schema
• Composition Surface and Cinematographic Technique Catalog
• Domain Templates and the Scene Score Editor
• Scene Safety, Determinism, Provenance, and Cue Privacy
• Latency, Accessibility, Eval Sets, and Tests
• Psyche Real-Time Runtime
• Isis Generation Control
• Lilith Persona Policy
• ../features.md