Owner library: libs/isis/operation-orchestrator Estimate: ~2 weeks
Blocks: every other block. Hard gate. Dependencies: none Status: not
started
Overview#
Add real PhaseType.CHECKPOINT runtime semantics to operation-orchestrator.
Today the type exists (types.ts:134) and injectCheckpointPhases
synthetically inserts CHECKPOINT phases at a configured frequency
(execution-orchestrator.ts:1056-1057, function at 1362-1389), but the
runtime executes them as ordinary phases. There is no approveCheckpoint
method, no AWAITING_APPROVAL chain status, no event indicating "operator
decision required".
This block makes CHECKPOINT phases first-class citizens: a chain can pause, emit a typed "awaiting approval" event with the intermediate artifact, accept an approval/rejection/modification decision, persist that decision to state, and either continue or halt with a typed reason.
Why this is the foundation#
- Block B (composer library) needs checkpoints between every phase.
- Blocks G, H, I all build on the orchestrator surface.
- Three V2 services (
isis-commentary-lipsync-pipeline,isis-cosmetic-pipeline,isis-venue-capture-pipeline) need the same primitive.
Files to create#
libs/isis/operation-orchestrator/src/checkpoint/checkpoint-types.ts— typed envelopes (CheckpointDecision,CheckpointArtifact,CheckpointPolicy,CheckpointTimeoutAction).libs/isis/operation-orchestrator/src/checkpoint/checkpoint-coordinator.ts— the runtime: registers awaiting checkpoints, accepts decisions, fires timeouts, emits events.libs/isis/operation-orchestrator/src/checkpoint/checkpoint-coordinator.test.tslibs/isis/operation-orchestrator/src/checkpoint/auto-decision-policies.ts— built-in policies:auto-approve,auto-reject,escalate-to-handler.libs/isis/operation-orchestrator/src/checkpoint/auto-decision-policies.test.tslibs/isis/operation-orchestrator/src/checkpoint/index.ts
Files to modify#
libs/isis/operation-orchestrator/src/types.ts- Add
AWAITING_APPROVALtoChainStatus(line 118 enum). - Add
CheckpointDecisionStatusenum:pending | approved | rejected | modified | timed-out. - Add typed event variants to the existing event schema:
pipeline_awaiting_approval{ pipelineId, chainId, phaseId, artifact, deadlineAt }pipeline_checkpoint_decided{ pipelineId, chainId, phaseId, decision }pipeline_checkpoint_timed_out{ pipelineId, chainId, phaseId, autoDecision }
- Add
libs/isis/operation-orchestrator/src/execution-orchestrator/execution-orchestrator.ts- Updated wording after A-002 read. The orchestrator is imperative —
callers explicitly call
startPhaseandcompletePhase. There is no internal phase loop. Hook intoExecutionOrchestrator.startPhase(line 1104): when the target phase hasphaseType === CHECKPOINT, instead of running it inline, mark the chainAWAITING_APPROVAL, emitpipeline_awaiting_approval, and callcheckpointCoordinator.awaitDecision({...}). The returned Promise drives the resolution: approved → callcompletePhase(chainId, phaseId, true); rejected or timed-out-with-reject-policy → callcompletePhase(chainId, phaseId, false)(which downstream propagatesFAILEDto the chain via the existing dependent-blocked logic at line 614+); modified → stashdecision.modificationsin a coordinator- side overlay map keyed by(chainId, nextPhaseId)for callers to consume on their nextstartPhasefor the affected phase. - Extend
CHAIN_STATUS_TRANSITIONSat line 52: addAWAITING_APPROVALtoRUNNING's allowed targets, add a new[ChainStatus.AWAITING_APPROVAL]: [RUNNING, FAILED, CANCELLED]entry. - Backward compatibility:
createStrictExecutionOrchestrator(line 1410) already opts intoenableCheckpoints: true, checkpointFrequency: 2. The newcheckpointPolicyconfig default ({ timeoutMs: 30_000, onTimeout: 'approve' }) preserves existing strict-mode behaviour — no pause unless the caller opts in.
- Updated wording after A-002 read. The orchestrator is imperative —
callers explicitly call
libs/isis/operation-orchestrator/src/pipeline-orchestrator/pipeline-orchestrator.ts- Updated after A-003 read. The two orchestrators are decoupled today —
PipelineOrchestratorhas no reference toExecutionOrchestrator.PipelineExecution.chainsstores onlyChainId[], not chain objects. Therefore theCheckpointCoordinatormust be a shared instance injected into both orchestrators by the caller (the composer constructs one and passes it to both). - Constructor gains optional
coordinator?: CheckpointCoordinatorparameter. Default: each orchestrator constructs its own if not supplied, but cross-orchestrator coordination then doesn't work; tests and the composer always supply a shared one. - Add
approveCheckpoint(pipelineId, phaseId, decision: CheckpointDecision): Promise<void>method. Delegates tocoordinator.recordDecision({...}). - Surface
getAwaitingCheckpoints(pipelineId?): readonly AwaitingCheckpoint[]for UI/observability. Delegates tocoordinator.getAwaiting(pipelineId). - Pipeline-level status does NOT auto-transition to
AWAITING_APPROVALin v1 (matches existing pattern — pipeline status is caller-driven, not derived from chain state). Thepipeline_awaiting_approvalevent +getAwaitingCheckpoints()is the v1 surface. UI consumers wanting "is this pipeline waiting?" call the getter, notpipeline.status. v2 may extendPIPELINE_STATUS_TRANSITIONS(line 50) and addnotifyChainAwaitingApproval(pipelineId)if a richer surface is needed. - Extend
pruneHistory(line 1004) to also callcoordinator.purgeForPipeline(pipelineId)when pruning terminal pipelines, so the coordinator's maps don't leak. - Update
createPipelineOrchestratorfactory at line 1037 to acceptcoordinatorandcheckpointPolicyin the config.
- Updated after A-003 read. The two orchestrators are decoupled today —
libs/isis/operation-orchestrator/src/execution-orchestrator/execution-orchestrator.ts(already covered above, plus): constructor gains the same optionalcoordinator?: CheckpointCoordinatorparameter for symmetry; same default behaviour.createExecutionOrchestratorfactory (line 1399) andcreateStrictExecutionOrchestrator(line 1410) pass the coordinator through.libs/isis/operation-orchestrator/src/index.ts- Re-export the checkpoint surface.
Public API surface#
// checkpoint-types.ts
export interface CheckpointArtifact {
readonly schemaVersion: '1.0.0';
readonly key: string; // 'environment-output', 'grounding-evaluation', etc.
readonly mimeType: string; // 'application/json' | 'image/png' | 'application/octet-stream'
readonly inlineData?: unknown; // small artifacts inline
readonly artifactUri?: string; // large artifacts via storage URI
}
export interface CheckpointDecision {
readonly status: 'approved' | 'rejected' | 'modified' | 'timed-out';
readonly reviewer: string; // 'human:elikem', 'auto:timeout-5m', 'consensus:abc123'
readonly reviewedAt: string; // ISO
readonly notes?: string;
readonly modifications?: Readonly<Record<string, unknown>>;
}
export type CheckpointTimeoutAction = 'approve' | 'reject' | 'escalate';
export interface CheckpointPolicy {
readonly timeoutMs: number;
readonly onTimeout: CheckpointTimeoutAction;
readonly escalationHandler?: (
artifact: CheckpointArtifact
) => Promise<CheckpointDecision>;
}
// checkpoint-coordinator.ts (selected)
export class CheckpointCoordinator {
awaitDecision(args: {
pipelineId: PipelineId;
chainId: ChainId;
phaseId: PhaseId;
artifact: CheckpointArtifact;
policy: CheckpointPolicy;
}): Promise<CheckpointDecision>;
recordDecision(args: {
pipelineId: PipelineId;
phaseId: PhaseId;
decision: CheckpointDecision;
}): void;
getAwaiting(pipelineId: PipelineId): readonly AwaitingCheckpoint[];
}
// pipeline-orchestrator additions
class PipelineOrchestrator {
approveCheckpoint(
pipelineId: PipelineId,
phaseId: PhaseId,
decision: CheckpointDecision
): Promise<void>;
getAwaitingCheckpoints(pipelineId: PipelineId): readonly AwaitingCheckpoint[];
}
Algorithm#
Updated after A-002 read. The orchestrator is imperative; the flow below sits
inside ExecutionOrchestrator.startPhase and the public approveCheckpoint API
on PipelineOrchestrator.
- Caller invokes
executionOrchestrator.startPhase(chainId, phaseId).- If the target phase has
phaseType !== CHECKPOINT: existing behaviour. Mark phaseRUNNING, emitphase_started, return. - If the target phase has
phaseType === CHECKPOINT:- Build a
CheckpointArtifactfrom the caller-suppliedpendingArtifact(a new optional parameter onstartPhase; if omitted, the coordinator synthesises a placeholder artifact with only{ schemaVersion, key: phaseId, mimeType: 'application/json', inlineData: null }). - Resolve effective policy: phase-level
CheckpointPolicyif set, else orchestrator-level default. - Transition chain status
RUNNING → AWAITING_APPROVAL(via the extendedCHAIN_STATUS_TRANSITIONStable). - Emit
pipeline_awaiting_approvalwith{ pipelineId, chainId, phaseId, artifact, deadlineAt: Date.now() + policy.timeoutMs }. - Call
coordinator.awaitDecision({...}). The returned Promise resolves when eitherrecordDecisionis invoked or the timer fires. Do not mark the phase RUNNING — leave it PENDING so the chain-state stays consistent if the caller queries it during the wait.
- Build a
- If the target phase has
- Caller invokes
pipelineOrchestrator.approveCheckpoint(pipelineId, phaseId, decision), or the timeout fires (which the coordinator handles internally by constructing an auto-decision perpolicy.onTimeout).- Coordinator's
recordDecisionresolves the awaiting Promise. - Emit
pipeline_checkpoint_decidedwith{ pipelineId, chainId, phaseId, decision }.
- Coordinator's
- Back in the orchestrator, branch on
decision.status:approved→ callexecutionOrchestrator.completePhase(chainId, phaseId, true). The chain status transitions backAWAITING_APPROVAL → RUNNING(so downstream phases can resume), then normalphase_completedlogic runs.modified→ stashdecision.modificationsincoordinator.modificationOverlay[chainId][nextNonCheckpointPhaseId]. Then callcompletePhase(chainId, phaseId, true). The nextstartPhasefor the affected phase reads and consumes the overlay.rejected, ortimed-outwithpolicy.onTimeout === 'reject'→ callcompletePhase(chainId, phaseId, false). The existing dependent-blocked logic (completePhaseline 614+) propagatesBLOCKEDto dependent phases and ultimatelyfailChainruns once all phases settle. Chain endsFAILED; PR emits the standardchain_completedevent withfailedPhases > 0.timed-outwithpolicy.onTimeout === 'escalate'→ invokepolicy.escalationHandler(artifact); its returned decision is treated as authoritative (recurse into this same branch).
- The coordinator persists every awaiting checkpoint and decision via the
pipeline-state repository (block H). For block A standalone, persistence is
in-memory; block H replaces with Postgres via the
CheckpointRepositorydependency-injection seam introduced here.
Public API impact summary: startPhase gains an optional
pendingArtifact?: CheckpointArtifact parameter. approveCheckpoint,
getAwaitingCheckpoints are new. CHAIN_STATUS_TRANSITIONS table grows.
Otherwise the existing surface is unchanged.
Test plan#
Unit (checkpoint-coordinator.test.ts):
awaitDecisionresolves whenrecordDecisionis called.awaitDecisionresolves withstatus: 'timed-out'andreviewer: 'auto:timeout-{policy.onTimeout}'afterpolicy.timeoutMs.getAwaitingreturns currently-awaiting checkpoints; clears after decision.- Concurrent
recordDecisioncalls for the same phase: first wins, subsequent throwCheckpointAlreadyDecidedError. escalationHandleris invoked whenpolicy.onTimeout === 'escalate'; its returned decision is treated as authoritative.
Unit (auto-decision-policies.test.ts):
auto-approveproduces a decision withstatus: 'approved',reviewer: 'auto:approve',notesderived from policy.auto-rejectproduces a decision withstatus: 'rejected',reviewer: 'auto:reject'.escalate-to-handlerinvokes the handler and propagates its decision.
Integration (execution-orchestrator.checkpoint.test.ts — new file):
- A 3-phase chain (EXECUTION → CHECKPOINT → EXECUTION) pauses at the checkpoint,
fires
pipeline_awaiting_approval, accepts an approval, resumes, completes. - Same chain with rejection: chain ends
FAILED, second EXECUTION phase never runs. - Same chain with
modifications: second EXECUTION phase receives the modification overlay (fixture-tagged so we can assert). - Chain with two checkpoints: each independently fires + decides; events arrive in order.
- Chain with
enableCheckpointsAND a manually-defined CHECKPOINT phase: the manual one is honoured even if not at a frequency boundary.
Adversarial (checkpoint-coordinator.adversarial.test.ts):
- Decision recorded for unknown
phaseId: throws. - Decision with malformed
modifications(not a record): throws via schema validation. - Two timeouts firing simultaneously for the same phase: idempotent (first wins).
Acceptance criteria#
- All new tests pass:
npx vitest run src/checkpoint/andnpx vitest run src/execution-orchestrator/. npx tsc --noEmitclean.- Stub-indicator scan clean on all new files.
- A CHECKPOINT phase, run with the default
auto-approvepolicy and noapproveCheckpointcall, completes exactly when its timeout fires (not before). - The composer (block B) can build on this surface without touching operation-orchestrator internals.
Open questions#
- Persistence in block A: ship in-memory now and let block H upgrade to
Postgres? Or punt block A's tests through an in-memory adapter and require
block H first? Recommended: ship in-memory; block H upgrades via dependency
injection of a
CheckpointRepositoryinterface defined in block A. - Event ordering guarantees: do we need strict FIFO ordering of
pipeline_awaiting_approvalevents across pipelines? Default: no ordering across pipelines, FIFO within a pipeline. - Modification schema: free-form
Record<string, unknown>is loose. Should each phase declare aModificationOverlayschema it accepts and the coordinator validate? Recommended: yes, but defer to block B which defines its own per-phase modification shapes.