Eve library

Block A · Orchestrator Checkpoint Primitive

Other

Revision 1

On this page

Owner library: libs/isis/operation-orchestrator Estimate: ~2 weeks Blocks: every other block. Hard gate. Dependencies: none Status: not started

Overview#

Add real PhaseType.CHECKPOINT runtime semantics to operation-orchestrator. Today the type exists (types.ts:134) and injectCheckpointPhases synthetically inserts CHECKPOINT phases at a configured frequency (execution-orchestrator.ts:1056-1057, function at 1362-1389), but the runtime executes them as ordinary phases. There is no approveCheckpoint method, no AWAITING_APPROVAL chain status, no event indicating "operator decision required".

This block makes CHECKPOINT phases first-class citizens: a chain can pause, emit a typed "awaiting approval" event with the intermediate artifact, accept an approval/rejection/modification decision, persist that decision to state, and either continue or halt with a typed reason.

Why this is the foundation#

  • Block B (composer library) needs checkpoints between every phase.
  • Blocks G, H, I all build on the orchestrator surface.
  • Three V2 services (isis-commentary-lipsync-pipeline, isis-cosmetic-pipeline, isis-venue-capture-pipeline) need the same primitive.

Files to create#

  • libs/isis/operation-orchestrator/src/checkpoint/checkpoint-types.ts — typed envelopes (CheckpointDecision, CheckpointArtifact, CheckpointPolicy, CheckpointTimeoutAction).
  • libs/isis/operation-orchestrator/src/checkpoint/checkpoint-coordinator.ts — the runtime: registers awaiting checkpoints, accepts decisions, fires timeouts, emits events.
  • libs/isis/operation-orchestrator/src/checkpoint/checkpoint-coordinator.test.ts
  • libs/isis/operation-orchestrator/src/checkpoint/auto-decision-policies.ts — built-in policies: auto-approve, auto-reject, escalate-to-handler.
  • libs/isis/operation-orchestrator/src/checkpoint/auto-decision-policies.test.ts
  • libs/isis/operation-orchestrator/src/checkpoint/index.ts

Files to modify#

  • libs/isis/operation-orchestrator/src/types.ts
    • Add AWAITING_APPROVAL to ChainStatus (line 118 enum).
    • Add CheckpointDecisionStatus enum: pending | approved | rejected | modified | timed-out.
    • Add typed event variants to the existing event schema:
      • pipeline_awaiting_approval { pipelineId, chainId, phaseId, artifact, deadlineAt }
      • pipeline_checkpoint_decided { pipelineId, chainId, phaseId, decision }
      • pipeline_checkpoint_timed_out { pipelineId, chainId, phaseId, autoDecision }
  • libs/isis/operation-orchestrator/src/execution-orchestrator/execution-orchestrator.ts
    • Updated wording after A-002 read. The orchestrator is imperative — callers explicitly call startPhase and completePhase. There is no internal phase loop. Hook into ExecutionOrchestrator.startPhase (line 1104): when the target phase has phaseType === CHECKPOINT, instead of running it inline, mark the chain AWAITING_APPROVAL, emit pipeline_awaiting_approval, and call checkpointCoordinator.awaitDecision({...}). The returned Promise drives the resolution: approved → call completePhase(chainId, phaseId, true); rejected or timed-out-with-reject-policy → call completePhase(chainId, phaseId, false) (which downstream propagates FAILED to the chain via the existing dependent-blocked logic at line 614+); modified → stash decision.modifications in a coordinator- side overlay map keyed by (chainId, nextPhaseId) for callers to consume on their next startPhase for the affected phase.
    • Extend CHAIN_STATUS_TRANSITIONS at line 52: add AWAITING_APPROVAL to RUNNING's allowed targets, add a new [ChainStatus.AWAITING_APPROVAL]: [RUNNING, FAILED, CANCELLED] entry.
    • Backward compatibility: createStrictExecutionOrchestrator (line 1410) already opts into enableCheckpoints: true, checkpointFrequency: 2. The new checkpointPolicy config default ({ timeoutMs: 30_000, onTimeout: 'approve' }) preserves existing strict-mode behaviour — no pause unless the caller opts in.
  • libs/isis/operation-orchestrator/src/pipeline-orchestrator/pipeline-orchestrator.ts
    • Updated after A-003 read. The two orchestrators are decoupled today — PipelineOrchestrator has no reference to ExecutionOrchestrator. PipelineExecution.chains stores only ChainId[], not chain objects. Therefore the CheckpointCoordinator must be a shared instance injected into both orchestrators by the caller (the composer constructs one and passes it to both).
    • Constructor gains optional coordinator?: CheckpointCoordinator parameter. Default: each orchestrator constructs its own if not supplied, but cross-orchestrator coordination then doesn't work; tests and the composer always supply a shared one.
    • Add approveCheckpoint(pipelineId, phaseId, decision: CheckpointDecision): Promise<void> method. Delegates to coordinator.recordDecision({...}).
    • Surface getAwaitingCheckpoints(pipelineId?): readonly AwaitingCheckpoint[] for UI/observability. Delegates to coordinator.getAwaiting(pipelineId).
    • Pipeline-level status does NOT auto-transition to AWAITING_APPROVAL in v1 (matches existing pattern — pipeline status is caller-driven, not derived from chain state). The pipeline_awaiting_approval event + getAwaitingCheckpoints() is the v1 surface. UI consumers wanting "is this pipeline waiting?" call the getter, not pipeline.status. v2 may extend PIPELINE_STATUS_TRANSITIONS (line 50) and add notifyChainAwaitingApproval(pipelineId) if a richer surface is needed.
    • Extend pruneHistory (line 1004) to also call coordinator.purgeForPipeline(pipelineId) when pruning terminal pipelines, so the coordinator's maps don't leak.
    • Update createPipelineOrchestrator factory at line 1037 to accept coordinator and checkpointPolicy in the config.
  • libs/isis/operation-orchestrator/src/execution-orchestrator/execution-orchestrator.ts (already covered above, plus): constructor gains the same optional coordinator?: CheckpointCoordinator parameter for symmetry; same default behaviour. createExecutionOrchestrator factory (line 1399) and createStrictExecutionOrchestrator (line 1410) pass the coordinator through.
  • libs/isis/operation-orchestrator/src/index.ts
    • Re-export the checkpoint surface.

Public API surface#

ts
// checkpoint-types.ts
export interface CheckpointArtifact {
  readonly schemaVersion: '1.0.0';
  readonly key: string; // 'environment-output', 'grounding-evaluation', etc.
  readonly mimeType: string; // 'application/json' | 'image/png' | 'application/octet-stream'
  readonly inlineData?: unknown; // small artifacts inline
  readonly artifactUri?: string; // large artifacts via storage URI
}

export interface CheckpointDecision {
  readonly status: 'approved' | 'rejected' | 'modified' | 'timed-out';
  readonly reviewer: string; // 'human:elikem', 'auto:timeout-5m', 'consensus:abc123'
  readonly reviewedAt: string; // ISO
  readonly notes?: string;
  readonly modifications?: Readonly<Record<string, unknown>>;
}

export type CheckpointTimeoutAction = 'approve' | 'reject' | 'escalate';

export interface CheckpointPolicy {
  readonly timeoutMs: number;
  readonly onTimeout: CheckpointTimeoutAction;
  readonly escalationHandler?: (
    artifact: CheckpointArtifact
  ) => Promise<CheckpointDecision>;
}

// checkpoint-coordinator.ts (selected)
export class CheckpointCoordinator {
  awaitDecision(args: {
    pipelineId: PipelineId;
    chainId: ChainId;
    phaseId: PhaseId;
    artifact: CheckpointArtifact;
    policy: CheckpointPolicy;
  }): Promise<CheckpointDecision>;

  recordDecision(args: {
    pipelineId: PipelineId;
    phaseId: PhaseId;
    decision: CheckpointDecision;
  }): void;

  getAwaiting(pipelineId: PipelineId): readonly AwaitingCheckpoint[];
}

// pipeline-orchestrator additions
class PipelineOrchestrator {
  approveCheckpoint(
    pipelineId: PipelineId,
    phaseId: PhaseId,
    decision: CheckpointDecision
  ): Promise<void>;

  getAwaitingCheckpoints(pipelineId: PipelineId): readonly AwaitingCheckpoint[];
}

Algorithm#

Updated after A-002 read. The orchestrator is imperative; the flow below sits inside ExecutionOrchestrator.startPhase and the public approveCheckpoint API on PipelineOrchestrator.

  1. Caller invokes executionOrchestrator.startPhase(chainId, phaseId).
    • If the target phase has phaseType !== CHECKPOINT: existing behaviour. Mark phase RUNNING, emit phase_started, return.
    • If the target phase has phaseType === CHECKPOINT:
      • Build a CheckpointArtifact from the caller-supplied pendingArtifact (a new optional parameter on startPhase; if omitted, the coordinator synthesises a placeholder artifact with only { schemaVersion, key: phaseId, mimeType: 'application/json', inlineData: null }).
      • Resolve effective policy: phase-level CheckpointPolicy if set, else orchestrator-level default.
      • Transition chain status RUNNING → AWAITING_APPROVAL (via the extended CHAIN_STATUS_TRANSITIONS table).
      • Emit pipeline_awaiting_approval with { pipelineId, chainId, phaseId, artifact, deadlineAt: Date.now() + policy.timeoutMs }.
      • Call coordinator.awaitDecision({...}). The returned Promise resolves when either recordDecision is invoked or the timer fires. Do not mark the phase RUNNING — leave it PENDING so the chain-state stays consistent if the caller queries it during the wait.
  2. Caller invokes pipelineOrchestrator.approveCheckpoint(pipelineId, phaseId, decision), or the timeout fires (which the coordinator handles internally by constructing an auto-decision per policy.onTimeout).
    • Coordinator's recordDecision resolves the awaiting Promise.
    • Emit pipeline_checkpoint_decided with { pipelineId, chainId, phaseId, decision }.
  3. Back in the orchestrator, branch on decision.status:
    • approved → call executionOrchestrator.completePhase(chainId, phaseId, true). The chain status transitions back AWAITING_APPROVAL → RUNNING (so downstream phases can resume), then normal phase_completed logic runs.
    • modified → stash decision.modifications in coordinator.modificationOverlay[chainId][nextNonCheckpointPhaseId]. Then call completePhase(chainId, phaseId, true). The next startPhase for the affected phase reads and consumes the overlay.
    • rejected, or timed-out with policy.onTimeout === 'reject' → call completePhase(chainId, phaseId, false). The existing dependent-blocked logic (completePhase line 614+) propagates BLOCKED to dependent phases and ultimately failChain runs once all phases settle. Chain ends FAILED; PR emits the standard chain_completed event with failedPhases > 0.
    • timed-out with policy.onTimeout === 'escalate' → invoke policy.escalationHandler(artifact); its returned decision is treated as authoritative (recurse into this same branch).
  4. The coordinator persists every awaiting checkpoint and decision via the pipeline-state repository (block H). For block A standalone, persistence is in-memory; block H replaces with Postgres via the CheckpointRepository dependency-injection seam introduced here.

Public API impact summary: startPhase gains an optional pendingArtifact?: CheckpointArtifact parameter. approveCheckpoint, getAwaitingCheckpoints are new. CHAIN_STATUS_TRANSITIONS table grows. Otherwise the existing surface is unchanged.

Test plan#

Unit (checkpoint-coordinator.test.ts):

  • awaitDecision resolves when recordDecision is called.
  • awaitDecision resolves with status: 'timed-out' and reviewer: 'auto:timeout-{policy.onTimeout}' after policy.timeoutMs.
  • getAwaiting returns currently-awaiting checkpoints; clears after decision.
  • Concurrent recordDecision calls for the same phase: first wins, subsequent throw CheckpointAlreadyDecidedError.
  • escalationHandler is invoked when policy.onTimeout === 'escalate'; its returned decision is treated as authoritative.

Unit (auto-decision-policies.test.ts):

  • auto-approve produces a decision with status: 'approved', reviewer: 'auto:approve', notes derived from policy.
  • auto-reject produces a decision with status: 'rejected', reviewer: 'auto:reject'.
  • escalate-to-handler invokes the handler and propagates its decision.

Integration (execution-orchestrator.checkpoint.test.ts — new file):

  • A 3-phase chain (EXECUTION → CHECKPOINT → EXECUTION) pauses at the checkpoint, fires pipeline_awaiting_approval, accepts an approval, resumes, completes.
  • Same chain with rejection: chain ends FAILED, second EXECUTION phase never runs.
  • Same chain with modifications: second EXECUTION phase receives the modification overlay (fixture-tagged so we can assert).
  • Chain with two checkpoints: each independently fires + decides; events arrive in order.
  • Chain with enableCheckpoints AND a manually-defined CHECKPOINT phase: the manual one is honoured even if not at a frequency boundary.

Adversarial (checkpoint-coordinator.adversarial.test.ts):

  • Decision recorded for unknown phaseId: throws.
  • Decision with malformed modifications (not a record): throws via schema validation.
  • Two timeouts firing simultaneously for the same phase: idempotent (first wins).

Acceptance criteria#

  • All new tests pass: npx vitest run src/checkpoint/ and npx vitest run src/execution-orchestrator/.
  • npx tsc --noEmit clean.
  • Stub-indicator scan clean on all new files.
  • A CHECKPOINT phase, run with the default auto-approve policy and no approveCheckpoint call, completes exactly when its timeout fires (not before).
  • The composer (block B) can build on this surface without touching operation-orchestrator internals.

Open questions#

  • Persistence in block A: ship in-memory now and let block H upgrade to Postgres? Or punt block A's tests through an in-memory adapter and require block H first? Recommended: ship in-memory; block H upgrades via dependency injection of a CheckpointRepository interface defined in block A.
  • Event ordering guarantees: do we need strict FIFO ordering of pipeline_awaiting_approval events across pipelines? Default: no ordering across pipelines, FIFO within a pipeline.
  • Modification schema: free-form Record<string, unknown> is loose. Should each phase declare a ModificationOverlay schema it accepts and the coordinator validate? Recommended: yes, but defer to block B which defines its own per-phase modification shapes.