Status: Proposed. Written 2026-09-11 as accepted, with no approval record;
awaiting approval in Eve. "Current state (2026-10-01)" below corrects what has
changed since.
Date: 2026-09-11
Authors: Isis Runtime Engineering
Supersedes: the per-flavor baked-model images described in
docker/runpod/comfyui/Dockerfile (MODEL_FLAVOR=sd15|sdxl|…|flagship-image)
for every new endpoint; existing flavor entries are retained in
infra/runpod/endpoints/inventory/ as records only.
Current state (2026-10-01)#
The sections after this one record the decision as made on 2026-09-11. This section describes the code on 2026-10-01 and governs wherever they differ.
- Cost (Decision 5, Consequences). The two development endpoints still run
with
workersMin: 0,workersMax: 1,idleTimeout: 10and an execution timeout of 300 s or 900 s (infra/runpod/endpoints/desired-state.json). The generation API's daily ledger (runpod-cost-ledger.service.ts) is an estimate-based admission guard, not a strict budget. It refuses a job when today's recorded spend plus that job's estimate exceeds the daily budget, but it reserves nothing for jobs admitted and not yet finished. Concurrent submissions can therefore pass against the same remaining balance, and a job can cost more than its estimate.workersMaxlimits concurrency, not cumulative spend. "Idle costs nothing" holds only for GPU workers scaled to zero: the network volume is billed continuously (infra/runpod/volumes/desired-state.json: 400 GB, $28 a month). A strict budget remains unresolved. It would need reservations for outstanding jobs, execution ceilings, and reconciliation against actual cost. - Build path (Decision 3, Alternatives). GitHub Actions has been
billing-blocked on this account since 2026-09-11. The endpoints therefore
build from the repository through RunPod's GitHub integration
(
"kind": "runpod-github"in the endpoint desired state), with the repository root as build context. That is the only working path. The GHCR route of Decision 3 is not a usable fallback today, even with billing restored. Its workflow (.github/workflows/runpod-comfyui-slim.yml) builds withdocker/runpod/comfyui-slimas context, but the Dockerfile now copies the workflow golden fixtures (libs/isis/workflows/.../__golden__/) and the volume manifest from the repository root, so that build would fail. Using it again needs Actions billing, a repository-root build context, a passing smoke test and a published private image. - Models (Decision 2, Consequences). The model library on Hetzner is the
source of truth for weights. The RunPod network volume is a 400 GB
least-recently-used cache of it (Isis ADR-0010, Eve document
b3fb7aaa-7bec-83d8-b91c-a8cec377e1b2;
infra/runpod/volumes/isis-chroma-models.manifest.jsonsays the same). The pre-submit check compares a workflow's model names with the volume manifest; it does not prove a file is present. Residency is decided by cache admission (apps/isis/generation-api/src/services/runpod-model-cache/admission.ts). WithISIS_RUNPOD_CACHE_AUTOSYNCoff (the default), a job whose model families are not cached is refused withModelsNotCachedError. With it on, the job is held asawaiting_cachewhile the missing families sync, and the cached ones stay leased. - Capabilities (Consequences). The vendored SAM 3 nodes were removed
(A.01.09).
SAM3Segmentand its sibling classes must not appear in the image (docker/runpod/comfyui-slim/smoke-test.sh). Text-prompted segmentation uses ComfyUI's core SAM 3.1 classes (SAM3_Detectinlibs/isis/workflows/src/workflows/chroma/sam3-segment.json). RMBG-2.0 was dropped from the matting catalog (A.02.08). - References. The tracker is now the
isis-chroma-runpodboard family. The evidence ledger is Eve document fe1e2d43-646f-8c5c-be24-1ce5398263b4. Isis ADR-0003 and ADR-0004 are Eve documents 079a1759-7861-89b6-862b-5942e6843a53 and 06493ace-dc51-88ce-8be8-fa12a5a8dd55. None of these is approved yet, so this ADR names them without pinning them.docker/runpod/comfyui-slim/README.md, which held the token-rotation steps, is no longer in the repository.
Context and Problem Statement#
Isis needed real image generation on RunPod serverless with the full-precision Chroma stack plus SAM 3, BiRefNet and RMBG-2.0, usable from the Isis dashboard the same day and with spend bounded by configuration.
Findings on 2026-09-11:
- The image lineage every existing endpoint assumes,
oshunai/runpod-comfyui:*on top ofoshunai/runpod-base, has never been published (Docker Hub returns 404 for both repositories). Nothing indesired-state.jsonhas ever run. - That lineage bakes a per-flavor model set into the image. The
comfyui-flagship-imageinventory lists 47 baked artifacts. Every new model means a multi-hour, multi-hundred-GB image rebuild, and the build cannot run on the developer machine (16 GB RAM). - RunPod's stock
runpod/worker-comfyuiimage already implements the serverless handler contract (API-format graph in, base64/S3 images out), reads models from a network volume at/runpod-volume/models/..., and supports custom node packs via a derived Dockerfile. - The user's constraints: no new third-party accounts, no public image, no quantized weights, cost strictly bounded.
Decision#
- Worker image = ComfyUI + node packs only.
docker/runpod/comfyui-slim/derives fromrunpod/worker-comfyui:5.10.0-base(pinned by tag and digest) and adds ComfyUI-RMBG at a pinned commit. No model file is ever copied into an image. - Models live on a RunPod network volume (now a cache of the Hetzner
library; Current state 3), seeded from a checked-in manifest
(
infra/runpod/volumes/isis-chroma-models.manifest.json: source, pinned revision or CivitAI file id, sha256, destination) by a stdlib-only loader running in a short-lived CPU pod. Adding a model is a manifest row plus a loader run. Only a new node pack rebuilds the image. - The image is built by GitHub Actions and published to GHCR as a private
package (not usable today; Current state 2) (
ghcr.io/greychimp/oshun/runpod-comfyui-slim), which the repo already uses for its other services. RunPod pulls it with a Container Registry Auth built from a read-onlyread:packagestoken. - Every build runs an in-image smoke test before push: ComfyUI boots on CPU
and
/object_infomust list every node class the Isis Chroma catalog uses. The resultingobject_info.jsonis the source of the generation API's pre-submit class check. - Cost is bounded in desired state, not by attention (the ledger is an
estimate-based guard; Current state 1):
workersMax: 1,workersMin: 0,executionTimeout: 300 s,idleTimeout: 10 s, plus the generation API's daily USD ledger (fails closed). - The generation API renders catalog workflows to bare graphs itself
(
ISIS_RUNPOD_WORKER_KIND=worker-comfyui), because the stock worker has no catalog renderer. The old handler contract remains selectable for the unpublished flavor images but is not the path forward.
Alternatives Considered#
- Publish the baked flavor images. Rejected: 50+ GB images, rebuild on every model change, cannot be built on the developer machine, and none of the flavors matched the Chroma stack anyway.
- Docker Hub. Rejected by the user (no new accounts). GHCR is already in use.
- Public GHCR package to avoid a pull credential. Rejected by the user; a read-only token is a one-time setup and keeps the package private.
- RunPod's GitHub-integration build. Would avoid a registry entirely, but requires connecting the GitHub account to RunPod and building the whole monorepo as context. Kept as a fallback here; it became the only working path (Current state 2).
- Custom nodes on the volume instead of in the image. Fragile (Python dependencies would need to be installed at cold start, on billed GPU seconds). Rejected.
- Quantized Chroma (fp8/GGUF) to fit smaller GPUs. Rejected by the user; the volume holds BF16/FP16 only and the manifest lint enforces it.
Consequences#
- Cold start now includes loading ~27 GB (checkpoint + T5) from the network volume, roughly 60–120 billed seconds per worker start. Warm images cost cents; idle GPU workers cost nothing (the volume is billed; Current state 1). The dashboard shows cold vs warm per job.
- A model referenced by a workflow but missing from the volume fails the pre-submit check (manifest cross-check), not at runtime. That check compares names only; cache admission decides residency (Current state 3).
groundingdino-pyis excluded from the ComfyUI-RMBG install (needs a compiler the runtime base lacks); the GroundingDINO-promptedSegmentnodes are therefore unavailable. Text-prompted segmentation usedSAM3Segment; it now uses coreSAM3_Detect(Current state 4).- RMBG-2.0 carries a non-commercial license (BRIA). It is on the volume for evaluation; commercial use needs a BRIA agreement or switching to BiRefNet. It has since been dropped from the matting catalog (Current state 4).
- Rotating the pull token is a documented three-step swap
(
docker/runpod/comfyui-slim/README.md, since removed; Current state 5).
References#
- Tracker:
ISIS_CHROMA_RUNPOD_MVP_TODOS_2026-09-11.md(now theisis-chroma-runpodboard family) - Evidence:
docs/agents/isis-chroma-runpod-evidence.md(now Eve document fe1e2d43-646f-8c5c-be24-1ce5398263b4) - Provisioner:
scripts/isis/runpod-provision.mjs - Loader:
scripts/isis/runpod-volume-loader.py - ADR-0003 (serverless execution strategy), ADR-0004 (provisioning mechanism)