# Block H · State Persistence and Resume

**Owner library:** `libs/isis/operation-orchestrator` (extends);
`libs/isis/database` (Prisma schema additions). **Estimate:** ~1–2 weeks
**Blocks:** running composer in production (deploys interrupt in-flight runs).
**Dependencies:** block A (defines the `Pipeline`/`Chain`/`Phase`/`Checkpoint`
shapes that need persisting); block G (dispatched jobs need to be resumable).
**Status:** not started

## Overview

`PipelineOrchestrator` keeps chains in an in-memory `Map`
(`pipeline-orchestrator.ts:1088`). If the BFF restarts (deploy, OOM, host
reschedule) every in-flight pipeline is lost. Composer runs are 5–30 minutes;
restarts happen.

Block H adds Postgres-persisted pipeline state via `@isis/database`'s Prisma
layer, plus a `resumePipeline(pipelineId, fromPhaseId, modifications)` API on
`PipelineOrchestrator`.

## Why this is needed

- Composer runs survive process restarts.
- Operator can review a checkpoint hours after it was reached, not just while
  the BFF is up.
- Block G's job dispatch goes through Redis → if the BFF restarts mid-bake, the
  bake job continues on the worker; on restart, the BFF reconnects to the
  pipeline state and reads the bake's eventual response.
- Audit trail: who approved which checkpoint, with what modifications, when.

## Files to create

### Prisma schema additions

- `libs/isis/database/prisma/migrations/<timestamp>_add_pipeline_state/migration.sql`
- `libs/isis/database/prisma/schema.prisma` — extended (modified, see below)

### Orchestrator persistence layer

- `libs/isis/operation-orchestrator/src/persistence/pipeline-repository.ts` —
  the `PipelineRepository` interface.
- `libs/isis/operation-orchestrator/src/persistence/postgres-pipeline-repository.ts`
  — Prisma-backed implementation.
- `libs/isis/operation-orchestrator/src/persistence/postgres-pipeline-repository.test.ts`
- `libs/isis/operation-orchestrator/src/persistence/in-memory-pipeline-repository.ts`
  — in-memory implementation (the existing behavior, formalised).
- `libs/isis/operation-orchestrator/src/persistence/in-memory-pipeline-repository.test.ts`
- `libs/isis/operation-orchestrator/src/persistence/index.ts`

### Resume API

- `libs/isis/operation-orchestrator/src/resume/resume-pipeline.ts` — the resume
  orchestration logic.
- `libs/isis/operation-orchestrator/src/resume/resume-pipeline.test.ts`

## Files to modify

- `libs/isis/database/prisma/schema.prisma` — add models: `Pipeline`, `Chain`,
  `Phase`, `PhaseArtifact`, `Checkpoint`, `CheckpointDecision`.
- `libs/isis/operation-orchestrator/src/pipeline-orchestrator/pipeline-orchestrator.ts`:
  - Constructor accepts `repository: PipelineRepository` (defaults to in-memory
    if none supplied — preserves block A behavior).
  - All state mutations route through the repository.
  - New method: `resumePipeline(pipelineId, fromPhaseId, modifications)`.
- `libs/isis/operation-orchestrator/src/checkpoint/checkpoint-coordinator.ts`:
  - Awaiting checkpoints persist to repository on `awaitDecision`.
  - Decisions persist on `recordDecision`.
- `libs/isis/scene-from-image-composer/src/types.ts`:
  - Add `repositoryConfig?: PipelineRepositoryConfig` to `SceneFromImageRequest`
    (composer creates the orchestrator internally with this repository).

## Prisma schema additions

```prisma
model Pipeline {
  id               String   @id @default(uuid())
  name             String
  status           String   // 'pending' | 'running' | 'completed' | 'failed' | 'cancelled' | 'awaiting-approval'
  startedAt        DateTime
  completedAt      DateTime?
  metadata         Json?
  chains           Chain[]
  tenantId         String?
  userId           String?

  @@index([tenantId, status])
  @@index([userId, status])
}

model Chain {
  id          String  @id @default(uuid())
  pipelineId  String
  pipeline    Pipeline @relation(fields: [pipelineId], references: [id], onDelete: Cascade)
  name        String
  status      String  // ChainStatus
  startedAt   DateTime?
  completedAt DateTime?
  phases      Phase[]

  @@index([pipelineId, status])
}

model Phase {
  id          String  @id @default(uuid())
  chainId     String
  chain       Chain   @relation(fields: [chainId], references: [id], onDelete: Cascade)
  name        String
  phaseType   String  // PhaseType
  status      String
  startedAt   DateTime?
  completedAt DateTime?
  durationMs  Int?
  artifacts   PhaseArtifact[]
  checkpoints Checkpoint[]

  @@index([chainId, status])
}

model PhaseArtifact {
  id           String   @id @default(uuid())
  phaseId      String
  phase        Phase    @relation(fields: [phaseId], references: [id], onDelete: Cascade)
  key          String   // 'environment-output', 'grounding-evaluation', etc.
  mimeType     String
  inlineData   Json?    // small artifacts inline
  artifactUri  String?  // large artifacts via storage URI
  createdAt    DateTime @default(now())

  @@unique([phaseId, key])
}

model Checkpoint {
  id           String              @id @default(uuid())
  phaseId      String
  phase        Phase               @relation(fields: [phaseId], references: [id], onDelete: Cascade)
  status       String              // 'pending' | 'approved' | 'rejected' | 'modified' | 'timed-out'
  awaitedAt    DateTime
  deadlineAt   DateTime
  decisions    CheckpointDecision[]
}

model CheckpointDecision {
  id              String   @id @default(uuid())
  checkpointId    String
  checkpoint      Checkpoint @relation(fields: [checkpointId], references: [id], onDelete: Cascade)
  status          String   // 'approved' | 'rejected' | 'modified' | 'timed-out'
  reviewer        String
  reviewedAt      DateTime
  notes           String?
  modifications   Json?

  @@index([checkpointId, reviewedAt])
}
```

## Public API surface

```ts
// persistence/pipeline-repository.ts

export interface PipelineRepository {
  // CRUD
  createPipeline(
    pipeline: Omit<PersistedPipeline, 'id'>
  ): Promise<PersistedPipeline>;
  loadPipeline(id: PipelineId): Promise<PersistedPipeline | null>;
  updatePipelineStatus(id: PipelineId, status: ChainStatus): Promise<void>;

  // Phase artifact storage
  storeArtifact(phaseId: PhaseId, artifact: CheckpointArtifact): Promise<void>;
  loadArtifact(
    phaseId: PhaseId,
    key: string
  ): Promise<CheckpointArtifact | null>;

  // Checkpoint state
  recordAwaitingCheckpoint(args: {
    phaseId: PhaseId;
    artifact: CheckpointArtifact;
    deadlineAt: Date;
  }): Promise<void>;
  recordCheckpointDecision(args: {
    phaseId: PhaseId;
    decision: CheckpointDecision;
  }): Promise<void>;
  loadAwaitingCheckpoints(
    pipelineId?: PipelineId
  ): Promise<readonly AwaitingCheckpoint[]>;

  // For resume
  loadPipelineSnapshot(id: PipelineId): Promise<PipelineSnapshot | null>;
}

// resume/resume-pipeline.ts

export interface ResumeOptions {
  readonly pipelineId: PipelineId;
  readonly fromPhaseId?: PhaseId; // defaults to first incomplete
  readonly modifications?: Readonly<Record<string, unknown>>;
}

export async function resumePipeline(
  orchestrator: PipelineOrchestrator,
  options: ResumeOptions
): Promise<void>;
```

## Algorithm

### Pipeline mutations route through repository

```
PipelineOrchestrator.runPipeline(definition):
  // Old: pipelines.set(id, runtimePipeline)
  // New:
  persistedPipeline = await repository.createPipeline({
    name: definition.name,
    status: 'pending',
    startedAt: new Date(),
    chains: definition.chains.map(toPersistedChain),
  })
  pipelines.set(persistedPipeline.id, runtimePipeline)
  // ... existing logic with mutations also writing through repository
```

### Resume

```
resumePipeline({ pipelineId, fromPhaseId, modifications }):
  snapshot = await repository.loadPipelineSnapshot(pipelineId)
  if !snapshot: throw PipelineNotFoundError
  if snapshot.status === 'completed' || snapshot.status === 'cancelled':
    throw PipelineAlreadyClosedError(snapshot.status)

  startPhase = fromPhaseId ?? findFirstIncompletePhase(snapshot)
  if !startPhase: throw NoIncompletePhaseError

  // Reconstruct pipeline runtime state from snapshot.
  runtimePipeline = reconstructFromSnapshot(snapshot, startFromPhase: startPhase)

  // Apply modifications to the start phase's input artifact.
  if modifications:
    runtimePipeline.applyModifications(startPhase, modifications)

  // Resume execution.
  await runtimePipeline.executeFrom(startPhase)
```

## Test plan

**Unit (`postgres-pipeline-repository.test.ts`):**

Use a Postgres test instance (per CLAUDE.md, infrastructure starts via
docker-compose; the test runs against the existing `oshun_dev` database or a
dedicated `oshun_test`). Tests:

- Create pipeline → load → assert all fields.
- Update status → load → assert new status.
- Store artifact, load by `(phaseId, key)` → assert deep equality.
- Record awaiting checkpoint → `loadAwaitingCheckpoints` returns it.
- Record decision → checkpoint status updates; awaiting list shrinks.
- Cascade delete: deleting a pipeline removes its chains, phases, artifacts,
  checkpoints (verify via direct query).
- Concurrent update: two `recordCheckpointDecision` calls for the same
  checkpoint → second throws `CheckpointAlreadyDecidedError` (enforced by unique
  constraint).

**Unit (`in-memory-pipeline-repository.test.ts`):**

Same suite as Postgres, runs against in-memory implementation. Behavioural
parity verified by sharing the test cases (parametrised over implementations).

**Unit (`resume-pipeline.test.ts`):**

- Pipeline halted at phase 3 of 5 → `resumePipeline` resumes and completes
  phases 3–5; phases 1–2 not re-executed.
- Pipeline halted at checkpoint → `resumePipeline` with `modifications` applies
  them and continues.
- `pipelineId` not found → throws.
- Pipeline already completed → throws.
- Resume after BFF restart simulation: snapshot persisted; new
  `PipelineOrchestrator` instance loads snapshot and resumes correctly.

**Integration (`composer.persistence.test.ts`):**

- Run composer with persistence enabled → assert Postgres rows for pipeline +
  chains + phases + artifacts + checkpoints.
- Kill composer mid-pipeline (simulate by throwing in a phase handler) → state
  in DB reflects the partial completion.
- Resume from the killed state → composer completes the remaining phases; final
  manifest produced.

**Adversarial:**

- Resume into a phase whose artifact is missing → throws `MissingArtifactError`.
- Repository throws on write → orchestrator surfaces error; pipeline marked
  `failed` with `cause: 'persistence-error'`.

## Acceptance criteria

- All new tests pass with both in-memory and Postgres repositories.
- Prisma migration applies cleanly to a fresh database.
- `npx tsc --noEmit` clean.
- Stub-indicator scan clean.
- A composer run can be killed via SIGKILL after the env phase, restarted in a
  new BFF process, resumed via `resumePipeline`, and produces the same final
  manifest as a non-interrupted run (golden assertion on manifest contents).

## Open questions

- **Artifact size handling.** Inline JSON is fine for grounding-evaluations and
  decisions. Big artifacts (splat scenes, baked Gaussians, mesh buffers) need to
  live in MinIO/S3 with the DB row holding only the URI. Recommended threshold:
  ≥ 1MB → URI; else inline.
- **Snapshot freshness.** Each phase completion writes a snapshot; if the BFF
  dies mid-phase, the in-progress phase is lost and must restart. v1 acceptable;
  v1.5 may add intra-phase checkpoints for very long bakes.
- **Multi-tenant isolation.** Repository accepts `tenantId` for indexing; no
  row-level security beyond this v1. Production deployments may layer RLS via
  Postgres policies (out of scope).
- **Migration vs schema evolution.** When the `metadata.sceneFromImage` schema
  evolves (new field), do we migrate existing rows? Recommended: no — schema is
  forward-compatible (new fields optional), old runs stay readable.
