Eve library

Block H · State Persistence and Resume

Other

Revision 1

On this page

Owner library: libs/isis/operation-orchestrator (extends); libs/isis/database (Prisma schema additions). Estimate: ~1–2 weeks Blocks: running composer in production (deploys interrupt in-flight runs). Dependencies: block A (defines the Pipeline/Chain/Phase/Checkpoint shapes that need persisting); block G (dispatched jobs need to be resumable). Status: not started

Overview#

PipelineOrchestrator keeps chains in an in-memory Map (pipeline-orchestrator.ts:1088). If the BFF restarts (deploy, OOM, host reschedule) every in-flight pipeline is lost. Composer runs are 5–30 minutes; restarts happen.

Block H adds Postgres-persisted pipeline state via @isis/database's Prisma layer, plus a resumePipeline(pipelineId, fromPhaseId, modifications) API on PipelineOrchestrator.

Why this is needed#

  • Composer runs survive process restarts.
  • Operator can review a checkpoint hours after it was reached, not just while the BFF is up.
  • Block G's job dispatch goes through Redis → if the BFF restarts mid-bake, the bake job continues on the worker; on restart, the BFF reconnects to the pipeline state and reads the bake's eventual response.
  • Audit trail: who approved which checkpoint, with what modifications, when.

Files to create#

Prisma schema additions#

  • libs/isis/database/prisma/migrations/<timestamp>_add_pipeline_state/migration.sql
  • libs/isis/database/prisma/schema.prisma — extended (modified, see below)

Orchestrator persistence layer#

  • libs/isis/operation-orchestrator/src/persistence/pipeline-repository.ts — the PipelineRepository interface.
  • libs/isis/operation-orchestrator/src/persistence/postgres-pipeline-repository.ts — Prisma-backed implementation.
  • libs/isis/operation-orchestrator/src/persistence/postgres-pipeline-repository.test.ts
  • libs/isis/operation-orchestrator/src/persistence/in-memory-pipeline-repository.ts — in-memory implementation (the existing behavior, formalised).
  • libs/isis/operation-orchestrator/src/persistence/in-memory-pipeline-repository.test.ts
  • libs/isis/operation-orchestrator/src/persistence/index.ts

Resume API#

  • libs/isis/operation-orchestrator/src/resume/resume-pipeline.ts — the resume orchestration logic.
  • libs/isis/operation-orchestrator/src/resume/resume-pipeline.test.ts

Files to modify#

  • libs/isis/database/prisma/schema.prisma — add models: Pipeline, Chain, Phase, PhaseArtifact, Checkpoint, CheckpointDecision.
  • libs/isis/operation-orchestrator/src/pipeline-orchestrator/pipeline-orchestrator.ts:
    • Constructor accepts repository: PipelineRepository (defaults to in-memory if none supplied — preserves block A behavior).
    • All state mutations route through the repository.
    • New method: resumePipeline(pipelineId, fromPhaseId, modifications).
  • libs/isis/operation-orchestrator/src/checkpoint/checkpoint-coordinator.ts:
    • Awaiting checkpoints persist to repository on awaitDecision.
    • Decisions persist on recordDecision.
  • libs/isis/scene-from-image-composer/src/types.ts:
    • Add repositoryConfig?: PipelineRepositoryConfig to SceneFromImageRequest (composer creates the orchestrator internally with this repository).

Prisma schema additions#

prisma
model Pipeline {
  id               String   @id @default(uuid())
  name             String
  status           String   // 'pending' | 'running' | 'completed' | 'failed' | 'cancelled' | 'awaiting-approval'
  startedAt        DateTime
  completedAt      DateTime?
  metadata         Json?
  chains           Chain[]
  tenantId         String?
  userId           String?

  @@index([tenantId, status])
  @@index([userId, status])
}

model Chain {
  id          String  @id @default(uuid())
  pipelineId  String
  pipeline    Pipeline @relation(fields: [pipelineId], references: [id], onDelete: Cascade)
  name        String
  status      String  // ChainStatus
  startedAt   DateTime?
  completedAt DateTime?
  phases      Phase[]

  @@index([pipelineId, status])
}

model Phase {
  id          String  @id @default(uuid())
  chainId     String
  chain       Chain   @relation(fields: [chainId], references: [id], onDelete: Cascade)
  name        String
  phaseType   String  // PhaseType
  status      String
  startedAt   DateTime?
  completedAt DateTime?
  durationMs  Int?
  artifacts   PhaseArtifact[]
  checkpoints Checkpoint[]

  @@index([chainId, status])
}

model PhaseArtifact {
  id           String   @id @default(uuid())
  phaseId      String
  phase        Phase    @relation(fields: [phaseId], references: [id], onDelete: Cascade)
  key          String   // 'environment-output', 'grounding-evaluation', etc.
  mimeType     String
  inlineData   Json?    // small artifacts inline
  artifactUri  String?  // large artifacts via storage URI
  createdAt    DateTime @default(now())

  @@unique([phaseId, key])
}

model Checkpoint {
  id           String              @id @default(uuid())
  phaseId      String
  phase        Phase               @relation(fields: [phaseId], references: [id], onDelete: Cascade)
  status       String              // 'pending' | 'approved' | 'rejected' | 'modified' | 'timed-out'
  awaitedAt    DateTime
  deadlineAt   DateTime
  decisions    CheckpointDecision[]
}

model CheckpointDecision {
  id              String   @id @default(uuid())
  checkpointId    String
  checkpoint      Checkpoint @relation(fields: [checkpointId], references: [id], onDelete: Cascade)
  status          String   // 'approved' | 'rejected' | 'modified' | 'timed-out'
  reviewer        String
  reviewedAt      DateTime
  notes           String?
  modifications   Json?

  @@index([checkpointId, reviewedAt])
}

Public API surface#

ts
// persistence/pipeline-repository.ts

export interface PipelineRepository {
  // CRUD
  createPipeline(
    pipeline: Omit<PersistedPipeline, 'id'>
  ): Promise<PersistedPipeline>;
  loadPipeline(id: PipelineId): Promise<PersistedPipeline | null>;
  updatePipelineStatus(id: PipelineId, status: ChainStatus): Promise<void>;

  // Phase artifact storage
  storeArtifact(phaseId: PhaseId, artifact: CheckpointArtifact): Promise<void>;
  loadArtifact(
    phaseId: PhaseId,
    key: string
  ): Promise<CheckpointArtifact | null>;

  // Checkpoint state
  recordAwaitingCheckpoint(args: {
    phaseId: PhaseId;
    artifact: CheckpointArtifact;
    deadlineAt: Date;
  }): Promise<void>;
  recordCheckpointDecision(args: {
    phaseId: PhaseId;
    decision: CheckpointDecision;
  }): Promise<void>;
  loadAwaitingCheckpoints(
    pipelineId?: PipelineId
  ): Promise<readonly AwaitingCheckpoint[]>;

  // For resume
  loadPipelineSnapshot(id: PipelineId): Promise<PipelineSnapshot | null>;
}

// resume/resume-pipeline.ts

export interface ResumeOptions {
  readonly pipelineId: PipelineId;
  readonly fromPhaseId?: PhaseId; // defaults to first incomplete
  readonly modifications?: Readonly<Record<string, unknown>>;
}

export async function resumePipeline(
  orchestrator: PipelineOrchestrator,
  options: ResumeOptions
): Promise<void>;

Algorithm#

Pipeline mutations route through repository#

text
PipelineOrchestrator.runPipeline(definition):
  // Old: pipelines.set(id, runtimePipeline)
  // New:
  persistedPipeline = await repository.createPipeline({
    name: definition.name,
    status: 'pending',
    startedAt: new Date(),
    chains: definition.chains.map(toPersistedChain),
  })
  pipelines.set(persistedPipeline.id, runtimePipeline)
  // ... existing logic with mutations also writing through repository

Resume#

text
resumePipeline({ pipelineId, fromPhaseId, modifications }):
  snapshot = await repository.loadPipelineSnapshot(pipelineId)
  if !snapshot: throw PipelineNotFoundError
  if snapshot.status === 'completed' || snapshot.status === 'cancelled':
    throw PipelineAlreadyClosedError(snapshot.status)

  startPhase = fromPhaseId ?? findFirstIncompletePhase(snapshot)
  if !startPhase: throw NoIncompletePhaseError

  // Reconstruct pipeline runtime state from snapshot.
  runtimePipeline = reconstructFromSnapshot(snapshot, startFromPhase: startPhase)

  // Apply modifications to the start phase's input artifact.
  if modifications:
    runtimePipeline.applyModifications(startPhase, modifications)

  // Resume execution.
  await runtimePipeline.executeFrom(startPhase)

Test plan#

Unit (postgres-pipeline-repository.test.ts):

Use a Postgres test instance (per CLAUDE.md, infrastructure starts via docker-compose; the test runs against the existing oshun_dev database or a dedicated oshun_test). Tests:

  • Create pipeline → load → assert all fields.
  • Update status → load → assert new status.
  • Store artifact, load by (phaseId, key) → assert deep equality.
  • Record awaiting checkpoint → loadAwaitingCheckpoints returns it.
  • Record decision → checkpoint status updates; awaiting list shrinks.
  • Cascade delete: deleting a pipeline removes its chains, phases, artifacts, checkpoints (verify via direct query).
  • Concurrent update: two recordCheckpointDecision calls for the same checkpoint → second throws CheckpointAlreadyDecidedError (enforced by unique constraint).

Unit (in-memory-pipeline-repository.test.ts):

Same suite as Postgres, runs against in-memory implementation. Behavioural parity verified by sharing the test cases (parametrised over implementations).

Unit (resume-pipeline.test.ts):

  • Pipeline halted at phase 3 of 5 → resumePipeline resumes and completes phases 3–5; phases 1–2 not re-executed.
  • Pipeline halted at checkpoint → resumePipeline with modifications applies them and continues.
  • pipelineId not found → throws.
  • Pipeline already completed → throws.
  • Resume after BFF restart simulation: snapshot persisted; new PipelineOrchestrator instance loads snapshot and resumes correctly.

Integration (composer.persistence.test.ts):

  • Run composer with persistence enabled → assert Postgres rows for pipeline + chains + phases + artifacts + checkpoints.
  • Kill composer mid-pipeline (simulate by throwing in a phase handler) → state in DB reflects the partial completion.
  • Resume from the killed state → composer completes the remaining phases; final manifest produced.

Adversarial:

  • Resume into a phase whose artifact is missing → throws MissingArtifactError.
  • Repository throws on write → orchestrator surfaces error; pipeline marked failed with cause: 'persistence-error'.

Acceptance criteria#

  • All new tests pass with both in-memory and Postgres repositories.
  • Prisma migration applies cleanly to a fresh database.
  • npx tsc --noEmit clean.
  • Stub-indicator scan clean.
  • A composer run can be killed via SIGKILL after the env phase, restarted in a new BFF process, resumed via resumePipeline, and produces the same final manifest as a non-interrupted run (golden assertion on manifest contents).

Open questions#

  • Artifact size handling. Inline JSON is fine for grounding-evaluations and decisions. Big artifacts (splat scenes, baked Gaussians, mesh buffers) need to live in MinIO/S3 with the DB row holding only the URI. Recommended threshold: ≥ 1MB → URI; else inline.
  • Snapshot freshness. Each phase completion writes a snapshot; if the BFF dies mid-phase, the in-progress phase is lost and must restart. v1 acceptable; v1.5 may add intra-phase checkpoints for very long bakes.
  • Multi-tenant isolation. Repository accepts tenantId for indexing; no row-level security beyond this v1. Production deployments may layer RLS via Postgres policies (out of scope).
  • Migration vs schema evolution. When the metadata.sceneFromImage schema evolves (new field), do we migrate existing rows? Recommended: no — schema is forward-compatible (new fields optional), old runs stay readable.