Eve library

Composer production runbook

Runbook

Revision 1

On this page

Operational handbook for running the @isis/scene-from-image-composer pipeline against real upstream providers + persistence + observability. Pairs with manual-qa.md (acceptance recipes for the deferred ISIS_GAPS items) and phase-summaries.md (per-phase shipped work).

Closes P3-007.


1. Prerequisites#

  • Node 22.x, pnpm ≥9, TypeScript ≥5.
  • Postgres 16.x with a database the operator can prisma db push into. The composer uses one schema + pipeline_state_* table prefix.
  • Redis 7.x for dispatchMode: 'job-envelope' runs (in- process mode skips this).
  • MinIO/S3-compatible blob storage for large artifact off-loading (block H persistence currently inlines all artifacts; the artifactUri path is reserved for a future cleanup).
  • API keys for each upstream provider used:
    • World Labs Marble (MARBLE_API_KEY)
    • Hunyuan / Tripo / Meshy (per provider)
    • ElevenLabs SFX (ELEVENLABS_API_KEY)

2. Boot sequence#

bash
# 1. Bring up infra.
docker compose -f docker/docker-compose.dev.yml up -d postgres redis minio
docker compose -f docker/docker-compose.dev.yml ps  # verify all healthy

# 2. Install deps + sync schema.
pnpm install
cd libs/isis/database && \
  ISIS_DATABASE_URL='postgresql://oshun:oshun_dev@localhost:5432/isis' \
  npx prisma db push
cd -

# 3. Build packages the BFF will load.
pnpm nx run @isis/scene-from-image-composer:build
pnpm nx run @isis/operation-orchestrator:build

3. Composer call shape#

The convenience entry point:

ts
import {
  composeSceneFromImage,
  type SceneFromImageRequest,
} from '@isis/scene-from-image-composer';
import { createMarbleProvider } from '@isis/3d-generation/providers';
import { createHunyuanProvider } from '@isis/3d-generation/providers';
import { createElevenLabsSfxProvider } from '@isis/audio-generation/generation';

const result = await composeSceneFromImage({
  assetId: 'asset-12345',
  tenantId: 'tenant-abc',
  userId: 'user-xyz',
  sourceImage: readFileSync('input.png'),
  mimeType: 'image/png',
  providers: {
    marble: createMarbleProvider({
      apiKey: process.env.MARBLE_API_KEY,
      mode: 'environment',
      splatDensity: 'high',
      emitAmbientHint: true,
    }),
    mesh: createHunyuanProvider({ apiKey: process.env.HUNYUAN_API_KEY }),
    sfx: createElevenLabsSfxProvider({
      apiKey: process.env.ELEVENLABS_API_KEY,
    }),
  },
  objects: [
    {
      source: 'explicit',
      id: 'tree-1',
      prompt: 'cypress tree',
      anchor: { type: 'centre-ground' },
    },
  ],
  outputMode: 'baked', // produce composed .spz
  bakeOptions: { method: 'surface-sampling' }, // 'hybrid' for sharper edges
  ambient: { prompt: 'forest dawn ambient' },
  // Block H — persist run state to Postgres.
  repositoryConfig: { mode: 'postgres' },
  // Block I — wire observability + policy.
  policyContext: {
    budget: {
      caps: [{ maxCostCents: 5000 }],
      perPhaseEstimateCents: { environment: 300, objects: 500, bake: 200 },
      reportPhaseCostCents: (phase) => upstreamBillingClient.lastCost(phase),
    },
    license: {
      registry: orgLicenseRegistry,
      deploymentTopology: 'self-hosted-saas',
      commercialIntent: 'commercial-saas',
    },
    gallery: {
      tenantId: 'tenant-abc',
      userId: 'user-xyz',
      domain: 'sophia',
      entitlementTier: 'curated-creator',
      model: 'world-labs-marble',
      generatorKey: 'scene-from-image-composer/v1',
      locale: 'en-US',
      assetUrl: 'https://cdn.example.com/scenes/asset-12345.spz',
      thumbnailUrl: 'https://cdn.example.com/scenes/asset-12345.thumb.png',
      costCents: 0, // updated post-run from result.policyOutcomes.budget
    },
    anomaly: { zScoreThreshold: 3.0 },
  },
});

The result carries:

  • manifest — the canonical OutputManifest to persist.
  • phaseOutputs — per-phase outputs (env splat bytes, mesh GLBs, bake metrics, etc.).
  • policyOutcomes — gallery OutputRecord, lineage graph, budget report, license assessment, anomaly stats.

4. Dispatch mode (separate worker process)#

For BFF setups where the composer's heavy phases run on a GPU worker pool instead of inline:

ts
// In the BFF:
const result = await composeSceneFromImage({
  ...,
  dispatchMode: 'job-envelope',
  dispatchConfig: { redisUrl: 'redis://localhost:6379' },
});

// In the worker process (separate Node):
import { attachWorker } from '@isis/scene-from-image-composer/dispatch/attach-worker';
import { RedisJobChannel } from '@isis/scene-from-image-composer/dispatch/redis-job-channel';
import { createEnvironmentHandler, createMeshHandler, createBakeHandler } from '...';

const channel = new RedisJobChannel({ redisUrl: 'redis://localhost:6379' });
const envHandler = createEnvironmentHandler({ createMarbleProvider });
const meshHandler = createMeshHandler({ providers: { hunyuan, tripo, meshy } });
const bakeHandler = createBakeHandler();  // uses real bakeMesh by default

await attachWorker({ channel, handler: envHandler });
await attachWorker({ channel, handler: meshHandler });
await attachWorker({ channel, handler: bakeHandler });

console.log('composer worker ready');

The worker process is long-lived; the BFF dispatches per-job envelopes; the worker handlers process them; responses flow back via per-jobId response channels. See libs/isis/scene-from-image-composer/src/dispatch/ for the full wiring.

5. Resume after interruption#

Composer runs can be 5–30 minutes. If the BFF restarts:

ts
// 1. Look up unfinished pipelines from the repo:
const repo = new PostgresPipelineRepository({ prisma });
const orch = createPipelineOrchestrator(undefined, undefined, repo);
const awaiting = await repo.loadAwaitingCheckpoints();  // pending checkpoints
// (For RUNNING pipelines, the BFF should also track pipelineIds in its
// own state — the orchestrator doesn't surface a list-all method by design.)

// 2. Resume each:
const resumed = await composeSceneFromImage({
  ...,
  repositoryConfig: { mode: 'postgres' },
  resumeFromPipelineId: pipelineId,
});

Resume reads the snapshot, skips completed phases (hydrating their outputs from artifacts), and runs the remaining phases. End state matches a non-interrupted run; see composer.persistence.test.ts > SIGKILL-resume golden.

6. Observability hooks#

The composer's policyOutcomes surface is the primary hook. For per-phase events, subscribe to the orchestrator:

ts
import { createPipelineOrchestrator } from '@isis/operation-orchestrator';

const orch = createPipelineOrchestrator(undefined, undefined, repo);
orch.on('pipeline_started', (e) =>
  metricsClient.gauge('pipeline.started', 1, { pipelineId: e.pipelineId })
);
orch.on('pipeline_completed', (e) =>
  metricsClient.gauge('pipeline.completed', 1, {
    pipelineId: e.pipelineId,
    durationMs: e.durationMs,
  })
);
orch.on('pipeline_error', (e) =>
  metricsClient.gauge('pipeline.failed', 1, {
    pipelineId: e.pipelineId,
    error: e.error,
  })
);

policyOutcomes.anomaly.anomalies carries any z-score threshold violations across the run's tracked channels (object-count, grounding-recall, grounding-novelty, coherence-score) — wire to PagerDuty / Slack as appropriate.

7. Cost reporting + budget enforcement#

Pre-flight budget caps:

ts
policyContext: {
  budget: {
    caps: [
      { name: 'per-pipeline-cap', maxCostCents: 5000 },
      { name: 'env-phase-cap', maxCostCents: 1000, maxCostPerOpCents: 1000 },
    ],
    perPhaseEstimateCents: { environment: 300, objects: 500, bake: 200 },
    reportPhaseCostCents: (phase) => upstreamBillingClient.lastCost(phase),
  },
},

If a pre-flight check fails (estimate + spent > cap), the composer throws ComposerError({ code: 'invalid-input' }) BEFORE running the phase. Caller catches + reports.

8. License compliance#

Per-artifact decisions:

ts
policyContext: {
  license: {
    registry: orgLicenseRegistry,
    deploymentTopology: 'self-hosted-saas',
    commercialIntent: 'commercial-saas',
    failOnDeny: true,  // default; set false to audit without blocking
  },
},

A deny aggregate decision throws ComposerError({ code: 'invalid-input' }) after the objects phase (before bake- phase compute burns). Per-artifact assessments land in result.policyOutcomes.license.perArtifact for audit logs.

policyOutcomes.gallery is the OutputRecord the operator writes to the gallery's storage tier (the composer doesn't own that write — the BFF or a downstream consumer does).

policyOutcomes.lineage is the full LineageGraph — pipe into the gallery's lineage indexer or store as a JSON blob alongside the manifest.

10. Troubleshooting#

  • ComposerError({ code: 'phase-failed', phase: 'environment' }) — Marble API call failed. Check Marble API key + the upstream service's status page.
  • ComposerError({ code: 'provider-missing', phase: 'objects' }) — request had objects: [...] but no providers.mesh. Add a mesh provider.
  • PipelineAlreadyClosedError — tried to resume a pipeline already marked COMPLETED or CANCELLED. Check repo.loadPipeline(pipelineId).status first.
  • JobTimeoutError — dispatch mode hit response timeout. Increase dispatchConfig.responseTimeoutMs (default 30 min) or check worker process health.
  • DuplicateJobIdError — the same jobId was dispatched twice on the same JobDispatcher instance. Caller should use unique jobIds or wait for the first dispatch to settle before re-dispatching.

11. Health checks#

  • Postgres: SELECT 1 against the isis database.
  • Redis (when dispatching): PING returns PONG.
  • Composer wiring: run composer.test.ts against fixtures — if 13 tests pass, the composer's public API contract is green.