Eve library

ADR-0005: Slim Worker Image and Volume-Backed Models for RunPod Serverless

ADR

Revision 9

On this page

Status: Proposed. Written 2026-09-11 as accepted, with no approval record; awaiting approval in Eve. "Current state (2026-10-01)" below corrects what has changed since.
Date: 2026-09-11
Authors: Isis Runtime Engineering
Supersedes: the per-flavor baked-model images described in docker/runpod/comfyui/Dockerfile (MODEL_FLAVOR=sd15|sdxl|…|flagship-image) for every new endpoint; existing flavor entries are retained in infra/runpod/endpoints/inventory/ as records only.

Current state (2026-10-01)#

The sections after this one record the decision as made on 2026-09-11. This section describes the code on 2026-10-01 and governs wherever they differ.

  1. Cost (Decision 5, Consequences). The two development endpoints still run with workersMin: 0, workersMax: 1, idleTimeout: 10 and an execution timeout of 300 s or 900 s (infra/runpod/endpoints/desired-state.json). The generation API's daily ledger (runpod-cost-ledger.service.ts) is an estimate-based admission guard, not a strict budget. It refuses a job when today's recorded spend plus that job's estimate exceeds the daily budget, but it reserves nothing for jobs admitted and not yet finished. Concurrent submissions can therefore pass against the same remaining balance, and a job can cost more than its estimate. workersMax limits concurrency, not cumulative spend. "Idle costs nothing" holds only for GPU workers scaled to zero: the network volume is billed continuously (infra/runpod/volumes/desired-state.json: 400 GB, $28 a month). A strict budget remains unresolved. It would need reservations for outstanding jobs, execution ceilings, and reconciliation against actual cost.
  2. Build path (Decision 3, Alternatives). GitHub Actions has been billing-blocked on this account since 2026-09-11. The endpoints therefore build from the repository through RunPod's GitHub integration ("kind": "runpod-github" in the endpoint desired state), with the repository root as build context. That is the only working path. The GHCR route of Decision 3 is not a usable fallback today, even with billing restored. Its workflow (.github/workflows/runpod-comfyui-slim.yml) builds with docker/runpod/comfyui-slim as context, but the Dockerfile now copies the workflow golden fixtures (libs/isis/workflows/.../__golden__/) and the volume manifest from the repository root, so that build would fail. Using it again needs Actions billing, a repository-root build context, a passing smoke test and a published private image.
  3. Models (Decision 2, Consequences). The model library on Hetzner is the source of truth for weights. The RunPod network volume is a 400 GB least-recently-used cache of it (Isis ADR-0010, Eve document b3fb7aaa-7bec-83d8-b91c-a8cec377e1b2; infra/runpod/volumes/isis-chroma-models.manifest.json says the same). The pre-submit check compares a workflow's model names with the volume manifest; it does not prove a file is present. Residency is decided by cache admission (apps/isis/generation-api/src/services/runpod-model-cache/admission.ts). With ISIS_RUNPOD_CACHE_AUTOSYNC off (the default), a job whose model families are not cached is refused with ModelsNotCachedError. With it on, the job is held as awaiting_cache while the missing families sync, and the cached ones stay leased.
  4. Capabilities (Consequences). The vendored SAM 3 nodes were removed (A.01.09). SAM3Segment and its sibling classes must not appear in the image (docker/runpod/comfyui-slim/smoke-test.sh). Text-prompted segmentation uses ComfyUI's core SAM 3.1 classes (SAM3_Detect in libs/isis/workflows/src/workflows/chroma/sam3-segment.json). RMBG-2.0 was dropped from the matting catalog (A.02.08).
  5. References. The tracker is now the isis-chroma-runpod board family. The evidence ledger is Eve document fe1e2d43-646f-8c5c-be24-1ce5398263b4. Isis ADR-0003 and ADR-0004 are Eve documents 079a1759-7861-89b6-862b-5942e6843a53 and 06493ace-dc51-88ce-8be8-fa12a5a8dd55. None of these is approved yet, so this ADR names them without pinning them. docker/runpod/comfyui-slim/README.md, which held the token-rotation steps, is no longer in the repository.

Context and Problem Statement#

Isis needed real image generation on RunPod serverless with the full-precision Chroma stack plus SAM 3, BiRefNet and RMBG-2.0, usable from the Isis dashboard the same day and with spend bounded by configuration.

Findings on 2026-09-11:

  • The image lineage every existing endpoint assumes, oshunai/runpod-comfyui:* on top of oshunai/runpod-base, has never been published (Docker Hub returns 404 for both repositories). Nothing in desired-state.json has ever run.
  • That lineage bakes a per-flavor model set into the image. The comfyui-flagship-image inventory lists 47 baked artifacts. Every new model means a multi-hour, multi-hundred-GB image rebuild, and the build cannot run on the developer machine (16 GB RAM).
  • RunPod's stock runpod/worker-comfyui image already implements the serverless handler contract (API-format graph in, base64/S3 images out), reads models from a network volume at /runpod-volume/models/..., and supports custom node packs via a derived Dockerfile.
  • The user's constraints: no new third-party accounts, no public image, no quantized weights, cost strictly bounded.

Decision#

  1. Worker image = ComfyUI + node packs only. docker/runpod/comfyui-slim/ derives from runpod/worker-comfyui:5.10.0-base (pinned by tag and digest) and adds ComfyUI-RMBG at a pinned commit. No model file is ever copied into an image.
  2. Models live on a RunPod network volume (now a cache of the Hetzner library; Current state 3), seeded from a checked-in manifest (infra/runpod/volumes/isis-chroma-models.manifest.json: source, pinned revision or CivitAI file id, sha256, destination) by a stdlib-only loader running in a short-lived CPU pod. Adding a model is a manifest row plus a loader run. Only a new node pack rebuilds the image.
  3. The image is built by GitHub Actions and published to GHCR as a private package (not usable today; Current state 2) (ghcr.io/greychimp/oshun/runpod-comfyui-slim), which the repo already uses for its other services. RunPod pulls it with a Container Registry Auth built from a read-only read:packages token.
  4. Every build runs an in-image smoke test before push: ComfyUI boots on CPU and /object_info must list every node class the Isis Chroma catalog uses. The resulting object_info.json is the source of the generation API's pre-submit class check.
  5. Cost is bounded in desired state, not by attention (the ledger is an estimate-based guard; Current state 1): workersMax: 1, workersMin: 0, executionTimeout: 300 s, idleTimeout: 10 s, plus the generation API's daily USD ledger (fails closed).
  6. The generation API renders catalog workflows to bare graphs itself (ISIS_RUNPOD_WORKER_KIND=worker-comfyui), because the stock worker has no catalog renderer. The old handler contract remains selectable for the unpublished flavor images but is not the path forward.

Alternatives Considered#

  • Publish the baked flavor images. Rejected: 50+ GB images, rebuild on every model change, cannot be built on the developer machine, and none of the flavors matched the Chroma stack anyway.
  • Docker Hub. Rejected by the user (no new accounts). GHCR is already in use.
  • Public GHCR package to avoid a pull credential. Rejected by the user; a read-only token is a one-time setup and keeps the package private.
  • RunPod's GitHub-integration build. Would avoid a registry entirely, but requires connecting the GitHub account to RunPod and building the whole monorepo as context. Kept as a fallback here; it became the only working path (Current state 2).
  • Custom nodes on the volume instead of in the image. Fragile (Python dependencies would need to be installed at cold start, on billed GPU seconds). Rejected.
  • Quantized Chroma (fp8/GGUF) to fit smaller GPUs. Rejected by the user; the volume holds BF16/FP16 only and the manifest lint enforces it.

Consequences#

  • Cold start now includes loading ~27 GB (checkpoint + T5) from the network volume, roughly 60–120 billed seconds per worker start. Warm images cost cents; idle GPU workers cost nothing (the volume is billed; Current state 1). The dashboard shows cold vs warm per job.
  • A model referenced by a workflow but missing from the volume fails the pre-submit check (manifest cross-check), not at runtime. That check compares names only; cache admission decides residency (Current state 3).
  • groundingdino-py is excluded from the ComfyUI-RMBG install (needs a compiler the runtime base lacks); the GroundingDINO-prompted Segment nodes are therefore unavailable. Text-prompted segmentation used SAM3Segment; it now uses core SAM3_Detect (Current state 4).
  • RMBG-2.0 carries a non-commercial license (BRIA). It is on the volume for evaluation; commercial use needs a BRIA agreement or switching to BiRefNet. It has since been dropped from the matting catalog (Current state 4).
  • Rotating the pull token is a documented three-step swap (docker/runpod/comfyui-slim/README.md, since removed; Current state 5).

References#

  • Tracker: ISIS_CHROMA_RUNPOD_MVP_TODOS_2026-09-11.md (now the isis-chroma-runpod board family)
  • Evidence: docs/agents/isis-chroma-runpod-evidence.md (now Eve document fe1e2d43-646f-8c5c-be24-1ce5398263b4)
  • Provisioner: scripts/isis/runpod-provision.mjs
  • Loader: scripts/isis/runpod-volume-loader.py
  • ADR-0003 (serverless execution strategy), ADR-0004 (provisioning mechanism)