# Block A · Orchestrator Checkpoint Primitive

**Owner library:** `libs/isis/operation-orchestrator` **Estimate:** ~2 weeks
**Blocks:** every other block. Hard gate. **Dependencies:** none **Status:** not
started

## Overview

Add real `PhaseType.CHECKPOINT` runtime semantics to `operation-orchestrator`.
Today the type exists (`types.ts:134`) and `injectCheckpointPhases`
synthetically inserts CHECKPOINT phases at a configured frequency
(`execution-orchestrator.ts:1056-1057`, function at `1362-1389`), but the
runtime executes them as ordinary phases. There is no `approveCheckpoint`
method, no `AWAITING_APPROVAL` chain status, no event indicating "operator
decision required".

This block makes CHECKPOINT phases first-class citizens: a chain can pause, emit
a typed "awaiting approval" event with the intermediate artifact, accept an
approval/rejection/modification decision, persist that decision to state, and
either continue or halt with a typed reason.

## Why this is the foundation

- Block B (composer library) needs checkpoints between every phase.
- Blocks G, H, I all build on the orchestrator surface.
- Three V2 services (`isis-commentary-lipsync-pipeline`,
  `isis-cosmetic-pipeline`, `isis-venue-capture-pipeline`) need the same
  primitive.

## Files to create

- `libs/isis/operation-orchestrator/src/checkpoint/checkpoint-types.ts` — typed
  envelopes (`CheckpointDecision`, `CheckpointArtifact`, `CheckpointPolicy`,
  `CheckpointTimeoutAction`).
- `libs/isis/operation-orchestrator/src/checkpoint/checkpoint-coordinator.ts` —
  the runtime: registers awaiting checkpoints, accepts decisions, fires
  timeouts, emits events.
- `libs/isis/operation-orchestrator/src/checkpoint/checkpoint-coordinator.test.ts`
- `libs/isis/operation-orchestrator/src/checkpoint/auto-decision-policies.ts` —
  built-in policies: `auto-approve`, `auto-reject`, `escalate-to-handler`.
- `libs/isis/operation-orchestrator/src/checkpoint/auto-decision-policies.test.ts`
- `libs/isis/operation-orchestrator/src/checkpoint/index.ts`

## Files to modify

- `libs/isis/operation-orchestrator/src/types.ts`
  - Add `AWAITING_APPROVAL` to `ChainStatus` (line 118 enum).
  - Add `CheckpointDecisionStatus` enum:
    `pending | approved | rejected | modified | timed-out`.
  - Add typed event variants to the existing event schema:
    - `pipeline_awaiting_approval`
      `{ pipelineId, chainId, phaseId, artifact, deadlineAt }`
    - `pipeline_checkpoint_decided` `{ pipelineId, chainId, phaseId, decision }`
    - `pipeline_checkpoint_timed_out`
      `{ pipelineId, chainId, phaseId, autoDecision }`
- `libs/isis/operation-orchestrator/src/execution-orchestrator/execution-orchestrator.ts`
  - **Updated wording after A-002 read.** The orchestrator is imperative —
    callers explicitly call `startPhase` and `completePhase`. There is no
    internal phase loop. Hook into `ExecutionOrchestrator.startPhase` (line
    1104): when the target phase has `phaseType === CHECKPOINT`, instead of
    running it inline, mark the chain `AWAITING_APPROVAL`, emit
    `pipeline_awaiting_approval`, and call
    `checkpointCoordinator.awaitDecision({...})`. The returned Promise drives
    the resolution: approved → call `completePhase(chainId, phaseId, true)`;
    rejected or timed-out-with-reject-policy → call
    `completePhase(chainId, phaseId, false)` (which downstream propagates
    `FAILED` to the chain via the existing dependent-blocked logic at line
    614+); modified → stash `decision.modifications` in a coordinator- side
    overlay map keyed by `(chainId, nextPhaseId)` for callers to consume on
    their next `startPhase` for the affected phase.
  - Extend `CHAIN_STATUS_TRANSITIONS` at line 52: add `AWAITING_APPROVAL` to
    `RUNNING`'s allowed targets, add a new
    `[ChainStatus.AWAITING_APPROVAL]: [RUNNING, FAILED, CANCELLED]` entry.
  - Backward compatibility: `createStrictExecutionOrchestrator` (line 1410)
    already opts into `enableCheckpoints: true, checkpointFrequency: 2`. The new
    `checkpointPolicy` config default
    (`{ timeoutMs: 30_000, onTimeout: 'approve' }`) preserves existing
    strict-mode behaviour — no pause unless the caller opts in.
- `libs/isis/operation-orchestrator/src/pipeline-orchestrator/pipeline-orchestrator.ts`
  - **Updated after A-003 read.** The two orchestrators are decoupled today —
    `PipelineOrchestrator` has no reference to `ExecutionOrchestrator`.
    `PipelineExecution.chains` stores only `ChainId[]`, not chain objects.
    Therefore the `CheckpointCoordinator` must be a **shared instance** injected
    into both orchestrators by the caller (the composer constructs one and
    passes it to both).
  - Constructor gains optional `coordinator?: CheckpointCoordinator` parameter.
    Default: each orchestrator constructs its own if not supplied, but
    cross-orchestrator coordination then doesn't work; tests and the composer
    always supply a shared one.
  - Add
    `approveCheckpoint(pipelineId, phaseId, decision: CheckpointDecision): Promise<void>`
    method. Delegates to `coordinator.recordDecision({...})`.
  - Surface `getAwaitingCheckpoints(pipelineId?): readonly AwaitingCheckpoint[]`
    for UI/observability. Delegates to `coordinator.getAwaiting(pipelineId)`.
  - **Pipeline-level status does NOT auto-transition to `AWAITING_APPROVAL` in
    v1** (matches existing pattern — pipeline status is caller-driven, not
    derived from chain state). The `pipeline_awaiting_approval` event +
    `getAwaitingCheckpoints()` is the v1 surface. UI consumers wanting "is this
    pipeline waiting?" call the getter, not `pipeline.status`. v2 may extend
    `PIPELINE_STATUS_TRANSITIONS` (line 50) and add
    `notifyChainAwaitingApproval(pipelineId)` if a richer surface is needed.
  - Extend `pruneHistory` (line 1004) to also call
    `coordinator.purgeForPipeline(pipelineId)` when pruning terminal pipelines,
    so the coordinator's maps don't leak.
  - Update `createPipelineOrchestrator` factory at line 1037 to accept
    `coordinator` and `checkpointPolicy` in the config.
- `libs/isis/operation-orchestrator/src/execution-orchestrator/execution-orchestrator.ts`
  (already covered above, plus): constructor gains the same optional
  `coordinator?: CheckpointCoordinator` parameter for symmetry; same default
  behaviour. `createExecutionOrchestrator` factory (line 1399) and
  `createStrictExecutionOrchestrator` (line 1410) pass the coordinator through.
- `libs/isis/operation-orchestrator/src/index.ts`
  - Re-export the checkpoint surface.

## Public API surface

```ts
// checkpoint-types.ts
export interface CheckpointArtifact {
  readonly schemaVersion: '1.0.0';
  readonly key: string; // 'environment-output', 'grounding-evaluation', etc.
  readonly mimeType: string; // 'application/json' | 'image/png' | 'application/octet-stream'
  readonly inlineData?: unknown; // small artifacts inline
  readonly artifactUri?: string; // large artifacts via storage URI
}

export interface CheckpointDecision {
  readonly status: 'approved' | 'rejected' | 'modified' | 'timed-out';
  readonly reviewer: string; // 'human:elikem', 'auto:timeout-5m', 'consensus:abc123'
  readonly reviewedAt: string; // ISO
  readonly notes?: string;
  readonly modifications?: Readonly<Record<string, unknown>>;
}

export type CheckpointTimeoutAction = 'approve' | 'reject' | 'escalate';

export interface CheckpointPolicy {
  readonly timeoutMs: number;
  readonly onTimeout: CheckpointTimeoutAction;
  readonly escalationHandler?: (
    artifact: CheckpointArtifact
  ) => Promise<CheckpointDecision>;
}

// checkpoint-coordinator.ts (selected)
export class CheckpointCoordinator {
  awaitDecision(args: {
    pipelineId: PipelineId;
    chainId: ChainId;
    phaseId: PhaseId;
    artifact: CheckpointArtifact;
    policy: CheckpointPolicy;
  }): Promise<CheckpointDecision>;

  recordDecision(args: {
    pipelineId: PipelineId;
    phaseId: PhaseId;
    decision: CheckpointDecision;
  }): void;

  getAwaiting(pipelineId: PipelineId): readonly AwaitingCheckpoint[];
}

// pipeline-orchestrator additions
class PipelineOrchestrator {
  approveCheckpoint(
    pipelineId: PipelineId,
    phaseId: PhaseId,
    decision: CheckpointDecision
  ): Promise<void>;

  getAwaitingCheckpoints(pipelineId: PipelineId): readonly AwaitingCheckpoint[];
}
```

## Algorithm

Updated after A-002 read. The orchestrator is imperative; the flow below sits
inside `ExecutionOrchestrator.startPhase` and the public `approveCheckpoint` API
on `PipelineOrchestrator`.

1. **Caller invokes `executionOrchestrator.startPhase(chainId, phaseId)`.**
   - If the target phase has `phaseType !== CHECKPOINT`: existing behaviour.
     Mark phase `RUNNING`, emit `phase_started`, return.
   - If the target phase has `phaseType === CHECKPOINT`:
     - Build a `CheckpointArtifact` from the caller-supplied `pendingArtifact`
       (a new optional parameter on `startPhase`; if omitted, the coordinator
       synthesises a placeholder artifact with only
       `{ schemaVersion, key: phaseId, mimeType: 'application/json', inlineData: null }`).
     - Resolve effective policy: phase-level `CheckpointPolicy` if set, else
       orchestrator-level default.
     - Transition chain status `RUNNING → AWAITING_APPROVAL` (via the extended
       `CHAIN_STATUS_TRANSITIONS` table).
     - Emit `pipeline_awaiting_approval` with
       `{ pipelineId, chainId, phaseId, artifact, deadlineAt: Date.now() + policy.timeoutMs }`.
     - Call `coordinator.awaitDecision({...})`. The returned Promise resolves
       when either `recordDecision` is invoked or the timer fires. Do **not**
       mark the phase RUNNING — leave it PENDING so the chain-state stays
       consistent if the caller queries it during the wait.
2. **Caller invokes
   `pipelineOrchestrator.approveCheckpoint(pipelineId, phaseId, decision)`,** or
   the timeout fires (which the coordinator handles internally by constructing
   an auto-decision per `policy.onTimeout`).
   - Coordinator's `recordDecision` resolves the awaiting Promise.
   - Emit `pipeline_checkpoint_decided` with
     `{ pipelineId, chainId, phaseId, decision }`.
3. **Back in the orchestrator**, branch on `decision.status`:
   - `approved` → call
     `executionOrchestrator.completePhase(chainId, phaseId, true)`. The chain
     status transitions back `AWAITING_APPROVAL → RUNNING` (so downstream phases
     can resume), then normal `phase_completed` logic runs.
   - `modified` → stash `decision.modifications` in
     `coordinator.modificationOverlay[chainId][nextNonCheckpointPhaseId]`. Then
     call `completePhase(chainId, phaseId, true)`. The next `startPhase` for the
     affected phase reads and consumes the overlay.
   - `rejected`, or `timed-out` with `policy.onTimeout === 'reject'` → call
     `completePhase(chainId, phaseId, false)`. The existing dependent-blocked
     logic (`completePhase` line 614+) propagates `BLOCKED` to dependent phases
     and ultimately `failChain` runs once all phases settle. Chain ends
     `FAILED`; PR emits the standard `chain_completed` event with
     `failedPhases > 0`.
   - `timed-out` with `policy.onTimeout === 'escalate'` → invoke
     `policy.escalationHandler(artifact)`; its returned decision is treated as
     authoritative (recurse into this same branch).
4. The coordinator persists every awaiting checkpoint and decision via the
   pipeline-state repository (block H). For block A standalone, persistence is
   in-memory; block H replaces with Postgres via the `CheckpointRepository`
   dependency-injection seam introduced here.

**Public API impact summary:** `startPhase` gains an optional
`pendingArtifact?: CheckpointArtifact` parameter. `approveCheckpoint`,
`getAwaitingCheckpoints` are new. `CHAIN_STATUS_TRANSITIONS` table grows.
Otherwise the existing surface is unchanged.

## Test plan

**Unit (`checkpoint-coordinator.test.ts`):**

- `awaitDecision` resolves when `recordDecision` is called.
- `awaitDecision` resolves with `status: 'timed-out'` and
  `reviewer: 'auto:timeout-{policy.onTimeout}'` after `policy.timeoutMs`.
- `getAwaiting` returns currently-awaiting checkpoints; clears after decision.
- Concurrent `recordDecision` calls for the same phase: first wins, subsequent
  throw `CheckpointAlreadyDecidedError`.
- `escalationHandler` is invoked when `policy.onTimeout === 'escalate'`; its
  returned decision is treated as authoritative.

**Unit (`auto-decision-policies.test.ts`):**

- `auto-approve` produces a decision with `status: 'approved'`,
  `reviewer: 'auto:approve'`, `notes` derived from policy.
- `auto-reject` produces a decision with `status: 'rejected'`,
  `reviewer: 'auto:reject'`.
- `escalate-to-handler` invokes the handler and propagates its decision.

**Integration (`execution-orchestrator.checkpoint.test.ts` — new file):**

- A 3-phase chain (EXECUTION → CHECKPOINT → EXECUTION) pauses at the checkpoint,
  fires `pipeline_awaiting_approval`, accepts an approval, resumes, completes.
- Same chain with rejection: chain ends `FAILED`, second EXECUTION phase never
  runs.
- Same chain with `modifications`: second EXECUTION phase receives the
  modification overlay (fixture-tagged so we can assert).
- Chain with two checkpoints: each independently fires + decides; events arrive
  in order.
- Chain with `enableCheckpoints` AND a manually-defined CHECKPOINT phase: the
  manual one is honoured even if not at a frequency boundary.

**Adversarial (`checkpoint-coordinator.adversarial.test.ts`):**

- Decision recorded for unknown `phaseId`: throws.
- Decision with malformed `modifications` (not a record): throws via schema
  validation.
- Two timeouts firing simultaneously for the same phase: idempotent (first
  wins).

## Acceptance criteria

- All new tests pass: `npx vitest run src/checkpoint/` and
  `npx vitest run src/execution-orchestrator/`.
- `npx tsc --noEmit` clean.
- Stub-indicator scan clean on all new files.
- A CHECKPOINT phase, run with the default `auto-approve` policy and no
  `approveCheckpoint` call, completes exactly when its timeout fires (not
  before).
- The composer (block B) can build on this surface without touching
  operation-orchestrator internals.

## Open questions

- **Persistence in block A**: ship in-memory now and let block H upgrade to
  Postgres? Or punt block A's tests through an in-memory adapter and require
  block H first? Recommended: ship in-memory; block H upgrades via dependency
  injection of a `CheckpointRepository` interface defined in block A.
- **Event ordering guarantees**: do we need strict FIFO ordering of
  `pipeline_awaiting_approval` events across pipelines? Default: no ordering
  across pipelines, FIFO within a pipeline.
- **Modification schema**: free-form `Record<string, unknown>` is loose. Should
  each phase declare a `ModificationOverlay` schema it accepts and the
  coordinator validate? Recommended: yes, but defer to block B which defines its
  own per-phase modification shapes.
