Owner library: libs/isis/operation-orchestrator (extends);
libs/isis/database (Prisma schema additions). Estimate: ~1–2 weeks
Blocks: running composer in production (deploys interrupt in-flight runs).
Dependencies: block A (defines the Pipeline/Chain/Phase/Checkpoint
shapes that need persisting); block G (dispatched jobs need to be resumable).
Status: not started
Overview#
PipelineOrchestrator keeps chains in an in-memory Map
(pipeline-orchestrator.ts:1088). If the BFF restarts (deploy, OOM, host
reschedule) every in-flight pipeline is lost. Composer runs are 5–30 minutes;
restarts happen.
Block H adds Postgres-persisted pipeline state via @isis/database's Prisma
layer, plus a resumePipeline(pipelineId, fromPhaseId, modifications) API on
PipelineOrchestrator.
Why this is needed#
- Composer runs survive process restarts.
- Operator can review a checkpoint hours after it was reached, not just while the BFF is up.
- Block G's job dispatch goes through Redis → if the BFF restarts mid-bake, the bake job continues on the worker; on restart, the BFF reconnects to the pipeline state and reads the bake's eventual response.
- Audit trail: who approved which checkpoint, with what modifications, when.
Files to create#
Prisma schema additions#
libs/isis/database/prisma/migrations/<timestamp>_add_pipeline_state/migration.sqllibs/isis/database/prisma/schema.prisma— extended (modified, see below)
Orchestrator persistence layer#
libs/isis/operation-orchestrator/src/persistence/pipeline-repository.ts— thePipelineRepositoryinterface.libs/isis/operation-orchestrator/src/persistence/postgres-pipeline-repository.ts— Prisma-backed implementation.libs/isis/operation-orchestrator/src/persistence/postgres-pipeline-repository.test.tslibs/isis/operation-orchestrator/src/persistence/in-memory-pipeline-repository.ts— in-memory implementation (the existing behavior, formalised).libs/isis/operation-orchestrator/src/persistence/in-memory-pipeline-repository.test.tslibs/isis/operation-orchestrator/src/persistence/index.ts
Resume API#
libs/isis/operation-orchestrator/src/resume/resume-pipeline.ts— the resume orchestration logic.libs/isis/operation-orchestrator/src/resume/resume-pipeline.test.ts
Files to modify#
libs/isis/database/prisma/schema.prisma— add models:Pipeline,Chain,Phase,PhaseArtifact,Checkpoint,CheckpointDecision.libs/isis/operation-orchestrator/src/pipeline-orchestrator/pipeline-orchestrator.ts:- Constructor accepts
repository: PipelineRepository(defaults to in-memory if none supplied — preserves block A behavior). - All state mutations route through the repository.
- New method:
resumePipeline(pipelineId, fromPhaseId, modifications).
- Constructor accepts
libs/isis/operation-orchestrator/src/checkpoint/checkpoint-coordinator.ts:- Awaiting checkpoints persist to repository on
awaitDecision. - Decisions persist on
recordDecision.
- Awaiting checkpoints persist to repository on
libs/isis/scene-from-image-composer/src/types.ts:- Add
repositoryConfig?: PipelineRepositoryConfigtoSceneFromImageRequest(composer creates the orchestrator internally with this repository).
- Add
Prisma schema additions#
model Pipeline {
id String @id @default(uuid())
name String
status String // 'pending' | 'running' | 'completed' | 'failed' | 'cancelled' | 'awaiting-approval'
startedAt DateTime
completedAt DateTime?
metadata Json?
chains Chain[]
tenantId String?
userId String?
@@index([tenantId, status])
@@index([userId, status])
}
model Chain {
id String @id @default(uuid())
pipelineId String
pipeline Pipeline @relation(fields: [pipelineId], references: [id], onDelete: Cascade)
name String
status String // ChainStatus
startedAt DateTime?
completedAt DateTime?
phases Phase[]
@@index([pipelineId, status])
}
model Phase {
id String @id @default(uuid())
chainId String
chain Chain @relation(fields: [chainId], references: [id], onDelete: Cascade)
name String
phaseType String // PhaseType
status String
startedAt DateTime?
completedAt DateTime?
durationMs Int?
artifacts PhaseArtifact[]
checkpoints Checkpoint[]
@@index([chainId, status])
}
model PhaseArtifact {
id String @id @default(uuid())
phaseId String
phase Phase @relation(fields: [phaseId], references: [id], onDelete: Cascade)
key String // 'environment-output', 'grounding-evaluation', etc.
mimeType String
inlineData Json? // small artifacts inline
artifactUri String? // large artifacts via storage URI
createdAt DateTime @default(now())
@@unique([phaseId, key])
}
model Checkpoint {
id String @id @default(uuid())
phaseId String
phase Phase @relation(fields: [phaseId], references: [id], onDelete: Cascade)
status String // 'pending' | 'approved' | 'rejected' | 'modified' | 'timed-out'
awaitedAt DateTime
deadlineAt DateTime
decisions CheckpointDecision[]
}
model CheckpointDecision {
id String @id @default(uuid())
checkpointId String
checkpoint Checkpoint @relation(fields: [checkpointId], references: [id], onDelete: Cascade)
status String // 'approved' | 'rejected' | 'modified' | 'timed-out'
reviewer String
reviewedAt DateTime
notes String?
modifications Json?
@@index([checkpointId, reviewedAt])
}
Public API surface#
// persistence/pipeline-repository.ts
export interface PipelineRepository {
// CRUD
createPipeline(
pipeline: Omit<PersistedPipeline, 'id'>
): Promise<PersistedPipeline>;
loadPipeline(id: PipelineId): Promise<PersistedPipeline | null>;
updatePipelineStatus(id: PipelineId, status: ChainStatus): Promise<void>;
// Phase artifact storage
storeArtifact(phaseId: PhaseId, artifact: CheckpointArtifact): Promise<void>;
loadArtifact(
phaseId: PhaseId,
key: string
): Promise<CheckpointArtifact | null>;
// Checkpoint state
recordAwaitingCheckpoint(args: {
phaseId: PhaseId;
artifact: CheckpointArtifact;
deadlineAt: Date;
}): Promise<void>;
recordCheckpointDecision(args: {
phaseId: PhaseId;
decision: CheckpointDecision;
}): Promise<void>;
loadAwaitingCheckpoints(
pipelineId?: PipelineId
): Promise<readonly AwaitingCheckpoint[]>;
// For resume
loadPipelineSnapshot(id: PipelineId): Promise<PipelineSnapshot | null>;
}
// resume/resume-pipeline.ts
export interface ResumeOptions {
readonly pipelineId: PipelineId;
readonly fromPhaseId?: PhaseId; // defaults to first incomplete
readonly modifications?: Readonly<Record<string, unknown>>;
}
export async function resumePipeline(
orchestrator: PipelineOrchestrator,
options: ResumeOptions
): Promise<void>;
Algorithm#
Pipeline mutations route through repository#
PipelineOrchestrator.runPipeline(definition):
// Old: pipelines.set(id, runtimePipeline)
// New:
persistedPipeline = await repository.createPipeline({
name: definition.name,
status: 'pending',
startedAt: new Date(),
chains: definition.chains.map(toPersistedChain),
})
pipelines.set(persistedPipeline.id, runtimePipeline)
// ... existing logic with mutations also writing through repository
Resume#
resumePipeline({ pipelineId, fromPhaseId, modifications }):
snapshot = await repository.loadPipelineSnapshot(pipelineId)
if !snapshot: throw PipelineNotFoundError
if snapshot.status === 'completed' || snapshot.status === 'cancelled':
throw PipelineAlreadyClosedError(snapshot.status)
startPhase = fromPhaseId ?? findFirstIncompletePhase(snapshot)
if !startPhase: throw NoIncompletePhaseError
// Reconstruct pipeline runtime state from snapshot.
runtimePipeline = reconstructFromSnapshot(snapshot, startFromPhase: startPhase)
// Apply modifications to the start phase's input artifact.
if modifications:
runtimePipeline.applyModifications(startPhase, modifications)
// Resume execution.
await runtimePipeline.executeFrom(startPhase)
Test plan#
Unit (postgres-pipeline-repository.test.ts):
Use a Postgres test instance (per CLAUDE.md, infrastructure starts via
docker-compose; the test runs against the existing oshun_dev database or a
dedicated oshun_test). Tests:
- Create pipeline → load → assert all fields.
- Update status → load → assert new status.
- Store artifact, load by
(phaseId, key)→ assert deep equality. - Record awaiting checkpoint →
loadAwaitingCheckpointsreturns it. - Record decision → checkpoint status updates; awaiting list shrinks.
- Cascade delete: deleting a pipeline removes its chains, phases, artifacts, checkpoints (verify via direct query).
- Concurrent update: two
recordCheckpointDecisioncalls for the same checkpoint → second throwsCheckpointAlreadyDecidedError(enforced by unique constraint).
Unit (in-memory-pipeline-repository.test.ts):
Same suite as Postgres, runs against in-memory implementation. Behavioural parity verified by sharing the test cases (parametrised over implementations).
Unit (resume-pipeline.test.ts):
- Pipeline halted at phase 3 of 5 →
resumePipelineresumes and completes phases 3–5; phases 1–2 not re-executed. - Pipeline halted at checkpoint →
resumePipelinewithmodificationsapplies them and continues. pipelineIdnot found → throws.- Pipeline already completed → throws.
- Resume after BFF restart simulation: snapshot persisted; new
PipelineOrchestratorinstance loads snapshot and resumes correctly.
Integration (composer.persistence.test.ts):
- Run composer with persistence enabled → assert Postgres rows for pipeline + chains + phases + artifacts + checkpoints.
- Kill composer mid-pipeline (simulate by throwing in a phase handler) → state in DB reflects the partial completion.
- Resume from the killed state → composer completes the remaining phases; final manifest produced.
Adversarial:
- Resume into a phase whose artifact is missing → throws
MissingArtifactError. - Repository throws on write → orchestrator surfaces error; pipeline marked
failedwithcause: 'persistence-error'.
Acceptance criteria#
- All new tests pass with both in-memory and Postgres repositories.
- Prisma migration applies cleanly to a fresh database.
npx tsc --noEmitclean.- Stub-indicator scan clean.
- A composer run can be killed via SIGKILL after the env phase, restarted in a
new BFF process, resumed via
resumePipeline, and produces the same final manifest as a non-interrupted run (golden assertion on manifest contents).
Open questions#
- Artifact size handling. Inline JSON is fine for grounding-evaluations and decisions. Big artifacts (splat scenes, baked Gaussians, mesh buffers) need to live in MinIO/S3 with the DB row holding only the URI. Recommended threshold: ≥ 1MB → URI; else inline.
- Snapshot freshness. Each phase completion writes a snapshot; if the BFF dies mid-phase, the in-progress phase is lost and must restart. v1 acceptable; v1.5 may add intra-phase checkpoints for very long bakes.
- Multi-tenant isolation. Repository accepts
tenantIdfor indexing; no row-level security beyond this v1. Production deployments may layer RLS via Postgres policies (out of scope). - Migration vs schema evolution. When the
metadata.sceneFromImageschema evolves (new field), do we migrate existing rows? Recommended: no — schema is forward-compatible (new fields optional), old runs stay readable.