Eve library

Isis Chroma on RunPod — live evidence ledger

Evidence

Revision 1

On this page

Companion to ISIS_CHROMA_RUNPOD_MVP_TODOS_2026-09-11.md. Every [x] in tracker sections C.01 (build), C.02, C.03 and C.11 cites a dated row here. Rows record what RunPod, GitHub or the browser actually returned — ids, seconds, dollars — never estimates. A failed attempt is recorded as a failed attempt.

Conventions: times in UTC; money in USD to four decimals; RunPod job timings are the delayTime / executionTime fields of /status/<jobId> in ms.

Account baseline (2026-09-11, before any provisioning)#

Field Value
clientBalance 15.00
spendLimit 80
endpoints none
networkVolumes none
pods none
templates none
currentSpendPerHr 0

C.01 Worker image#

Date Item Result
2026-09-11 C.01.05 registry credential GHCR_PULL_TOKEN (classic, 40 chars) verified: authenticates as GreyChimp, GHCR token exchange OK, listed tags of the private package oshun/iris/voice. RunPod Container Registry Auth created 14:44Z: id cmtx2h80b001kiob5v4p8dtwc, name oshun-ghcr-pull, username GreyChimp (POST /containerregistryauth).
2026-09-11 C.01.09 GHCR CI build — CLOSED, superseded User decision 2026-09-11: no GHCR. RunPod is connected directly to the GitHub repo and builds the image itself, so there is no CI build to trigger and no registry digest to record. .github/workflows/runpod-comfyui-slim.yml is left unrun and is not the build path.
2026-09-11 C.01.09 RunPod git builds (replaces CI: Actions billing-blocked) Endpoint 6l1f3tku7z2zoy builds from branch runpod-comfyui-slim, Dockerfile docker/runpod/comfyui-slim/Dockerfile, repo-root context. Builds f0421816 / 5f3af663 CANCELLED (each push to the source branch restarts the build). Build 445787c8 FAILED at the build-manifest step: /bin/sh: 1: WORKER_COMFYUI_TAG: parameter not set — the ARG was declared before FROM and not re-declared in the stage; the base pull, node-pack install (rmbg deps ok, uv worked under the builder's nofile=1024 limit) and the volume symlinks all succeeded in that log. Fixed by re-declaring both ARGs after FROM (commit 95216491); build 234d08bd started 16:10Z.

| 2026-09-11 19:05Z | C.01.09 equivalent: RunPod git build green | Build d677f914-d4eb-4a6a-891f-86c94f9f0b24 COMPLETED from commit 105ad3cf (branch runpod-comfyui-slim); release cmtxbtcjo00hz1pmmka2539tw created 19:05:26Z. Two builds newer than 234d08bd, whose containers failed to create. git diff 105ad3cf -- docker/runpod/comfyui-slim/ is empty, so the release is the Dockerfile in the tree. No digest: /v2/serverless/<id>/builds and /releases expose none for git-integration builds — the commit SHA is the handle. Untested at runtime: the endpoint is at workersMax: 0. |

C.02 Network volume and models#

Date Item Result
2026-09-11 C.02.05 volume create Volume ax0oj250j3 isis-chroma-models, datacenter EU-RO-1, 100 GB at create, grown to 500 GB 15:12:27Z (V.02.01). Re-read live 2026-09-11 evening: GET /v1/networkvolumes/ax0oj250j3 → {dataCenterId: EU-RO-1, size: 500}, matching infra/runpod/volumes/desired-state.json including its sizeHistory. Datacenter chosen for serverless GPU stock; the later availability query (row C.03.07) showed EU-RO-1 had no free 4090 and no A5000/3090 at all, which is why both endpoints run GPU pools instead of GPU names — the datacenter choice is the one constraint of this plan that did not hold as written.
pending C.02.06 loader pod pod id, SKU, start/end UTC, cost
pending C.02.07 manifest sha256 rows verified / rows failed
2026-09-11 13:48Z C.02.08 loader pod deleted DELETE /pods/1k341jwuk1ci9q → 204; pods: []; currentSpendPerHr: 0.01 (= the 100 GB volume, $0.0096/h); balance 14.9395 (spent $0.0605 so far incl. volume accrual).

C.03 Endpoint#

Date Item Result
2026-09-11 15:47Z C.03.04 endpoint create Created in the console via Import Git Repository (GitHub Actions billing-blocked): endpoint 6l1f3tku7z2zoy isis-comfyui-chroma, template j1gzx4nccu, workersMin 0 / workersMax 1, idleTimeout 10 s, executionTimeoutMs 300000, flashboot true, gpuTypeIds [RTX 4090, RTX A5000, L4, RTX 3090], volume ax0oj250j3, CUDA ≥ 12.8, scaler QUEUE_DELAY 4 — read back via GET /endpoints/{id} and matching desired state.
2026-09-11 20:09Z C.03.05 health at rest GET https://api.runpod.ai/v2/6l1f3tku7z2zoy/health → workers {idle 0, initializing 0, ready 0, running 0, throttled 0, unhealthy 0}, jobs {completed 0, failed 0, inProgress 0, inQueue 0, retried 0}. Nothing running, nothing queued, nothing billed beyond the volume. completed: 0 is consistent with the first image having run on the since-deleted interim endpoint, not on this one.
2026-09-11 16:36Z C.03.06 kill switch (real) PATCH workersMax 0 on 6l1f3tku7z2zoy while a worker was cycling container creates: currentSpendPerHr 1.149 → 0.049 (volume only) within one poll; workers 0. Restored later per endpoint.
2026-09-11 16:48Z C.03.07 GPU fallback (real) GraphQL dataCenters.gpuAvailability for EU-RO-1: 4090 available=false, A5000/3090 not offered, L4 + RTX PRO 4000/4500 Blackwell available (Low). Both endpoints set to GPU pools ADA_24,AMPERE_24,BLACKWELL_32 via saveEndpoint (REST PATCH ignores gpuTypeIds silently). Renders landed on RTX PRO 4500 Blackwell 32 GB.
2026-09-11 16:42Z bisect: stock image boots Throwaway endpoint ac988l8is5axkf (template zxojew9tyo, runpod/worker-comfyui:5.10.0-base, same volume/CUDA/pools): container starts, GPU fitness passes, ComfyUI 0.34.0 up. The git-built slim image on 6l1f3tku7z2zoy fails at container create on the same host class → defect is in our image/its packaging, not the host. Probe pod bi32hrhaxwxsxx from the slim image never reached a runtime in 15 min (deleted).
2026-09-11 17:08Z interim runtime = stock image + start command Template zxojew9tyo dockerStartCmd writes our extra_model_paths.yaml into /comfyui then exec /start.sh; Isis RUNPOD_COMFYUI_ENDPOINT_ID=ac988l8is5axkf. Volume layout models/{diffusion_models,text_encoders,vae} resolves; RMBG/SAM 3 nodes are NOT in this image (matting/segment workflows wait on the slim image).

| 2026-09-11 20:08Z | C.03.04 drift check (real) | New scripts/isis/runpod-endpoint-drift-check.mjs compares infra/runpod/endpoints/desired-state.json (development/comfyui-chroma) against the live endpoint and exits 1 on drift. Run against 6l1f3tku7z2zoy: 7 fields matched (name, workersMin 0, idleTimeout 10, executionTimeoutMs 300000, networkVolumeId ax0oj250j3, templateId j1gzx4nccu, gpuIds ADA_24,AMPERE_24,BLACKWELL_32) and 1 real drift caught — workersMax: desired 1, live 0, the kill switch from the 16:36Z incident never lifted. Exit code 1. Source of truth is GraphQL myself.endpoints, not REST: REST still echoes the stale GPU-name list for this endpoint while GraphQL reports the pools actually in force, so a REST-based check would invent drift. 11 node --test cases cover the unit conversion (s vs ms), order-insensitive pools, the kill switch in both directions, and missing values as drift rather than silent passes. |

| 2026-09-11 20:46-20:54Z | C.12.12 retest attempt — endpoint will not schedule | User approved one paid probe. workersMax 0 -> 1 by REST PATCH (GPU pools survived: console still reads 24 GB Pro / 24 GB / 32 GB, so the PATCH did NOT clobber them). Submitted a model-free graph (EmptyImage 64x64 -> SaveImage) so the probe tests container create and ComfyUI boot without paying for a 17 GB model load: job 3a03294c-7387-4267-bd51-6283023f982a-e2. After 6+ minutes it was still IN_QUEUE with workers {idle 0, initializing 0, ready 0, running 0, throttled 0, unhealthy 0} and workers: [] — the endpoint placed no worker in any state. Not a stock problem: EU-RO-1 availability at the time had RTX PRO 4500 Blackwell (BLACKWELL_32) and L4 (AMPERE_24) both available=true, stock=Low; a starved endpoint would show throttled, not nothing. This is the recorded post-saveEndpoint breakage: 6l1f3tku7z2zoy stopped placing workers after the GraphQL saveEndpoint call and has never placed one since (jobs.completed is still 0 for its whole life). C.12.12 is therefore still unproven — the slim image has never been given a container to fail in. Queue purged (removed: 1), workersMax restored to 0. Cost of the attempt: $0.00 — balance 13.970325507 before and after, spend/hr back to 0.049 (volume only). Testing the image needs a NEW endpoint on the same git release; this one is unusable regardless of image. |

C.05 Adapter adversarial pass (C.05.05, 2026-09-11)#

Scope: runpod-adapter.ts plus the three specs added for C.05.03/C.05.04.

  1. Stub-vocabulary grep (the tracker's list, plus fake/dummy): zero hits across the adapter and the new specs.
  2. Silent-stub scan. One Math.random() in the whole adapter, at computeCanaryRoutingBucket (line ~4479): it is the fallback when a request carries no deterministic canary seed, where random assignment is the correct behaviour, and it is annotated as such. Not on the timing or status path. No catch block returns fabricated success; no fixed literal stands in for a measured value — runpod-adapter.timing.spec.ts pins that the dollar figure scales ~15x with 15x the GPU seconds, which a constant would fail.
  3. Delegation chain. submitJob → pollForCompletion → getJobStatus → mapRunPodStatus/mapRunPodOutput/normalizeDurationMs all read. normalizeDurationMs returns undefined for a missing value rather than 0, which is why an absent delayTime cannot be reported as a warm start.
  4. Test coherence. The specs assert domain facts, not shapes: all six RunPod states map individually, an unknown state falls to pending (never to success or failure), a FAILED job's worker text survives verbatim in errorDetails.providerMessage under a deliberately generic user message, and the cold-start flag is strictly above the 20 s threshold, not at it.

Result: 25 tests green across four adapter spec files; 4 more for the service-side runpod.cold_start event. No defects found in the adapter changes.

C.06 Executor/service adversarial pass (C.06.05, 2026-09-11)#

Scope: executor.ts, worker-comfyui-submission.ts, comfyui.service.ts, inline-job-runner.service.ts.

  1. Stub-vocabulary grep: one hit, executor.ts:3927 — the string "...for generation type '<type>' in production." inside the fail-closed guard's error message. That is an honest fail-loud seam (the guard refuses to run rather than guessing a workflow), not a stub; CLAUDE.md exempts exactly this. No other hits across the four files.
  2. Silent stubs: no Math.random() in the submission builder or executor — seed handling goes through normalizeSeedValue, and worker-comfyui-submission.spec.ts pins that rendering is deterministic for a fixed seed and rng. No catch block returns fabricated success.
  3. Delegation chain: prepareWorkerComfyUiSubmission → liftInlineImages → sniffBase64ImageType → the per-job caps read in full. Image lifting throws INVALID_IMAGE_INPUT on a non-image payload rather than passing bytes the worker would reject after billing; unknown node classes and off-manifest models are refused by name before submit.
  4. Test coherence: the 12 cases in worker-comfyui-submission.spec.ts assert domain behaviour (magic-byte detection per format, filename substitution, determinism, caps at the boundary, refusal naming the offending node), not shape. translateRequest is now pinned to pass a bare graph through unwrapped while still wrapping a string id.

Result: 7 tests green in comfyui.service.spec.ts, 14 in libs/shared/config/src/features.spec.ts. No defects found.

| 2026-09-11 21:25Z | executor.spec.ts baseline re-measured | Memory recorded 11 pre-existing failures on this branch; the file now fails 13. Bisected by git stash push -- executor.ts: 13 failed / 80 passed both with and without the C.07.02 change, so the drift came from other sessions' merges into main, not from this work. All 13 are catalog-bootstrap/resolver cases (registers every @isis/workflows catalog entry, the four certifies <workflow> cases, resolver counters, control-mode and voice-consistency); none touch outputs, storage or persistence. Treat 13 as the new baseline. |

| 2026-09-11 21:37-21:39Z | C.12.12 RESOLVED — the slim image works; the endpoint was the fault | Endpoint 6l1f3tku7z2zoy had placed zero workers for its whole life. Fix: Manage → Edit endpoint → Save Endpoint in the console, re-saving the same config (only workersMax 0 → 1 changed). Status went Idle → Initializing, a worker was placed within ~40 s, and the model-free probe (EmptyImage 64x64 → SaveImage) returned a real base64 PNG: job 2ae689ed-871b-4899-bd50-08eb135210b2-e2, delayTime 78.888 s, executionTime 6.403 s, worker bh3ek2iz6grdek, release d677f914 (commit 105ad3cf). Cost $0.026 at the 24 GB PRO rate; spend/hr back to 0.049 (volume only) and balance 13.9217 afterwards. jobs.completed went 0 → 1, the first ever for this endpoint. Two earlier diagnoses in this file and in memory were wrong: (a) "the git-built slim image fails at container create" — it does not, it boots and renders; (b) "EU-RO-1 has no capacity" — the edit dialog shows the two selected pools as Low Supply / Unavailable, but a worker was placed in ~40 s anyway, so the Low Supply label is not a blocker. The real rule: a RunPod endpoint wedged by a GraphQL saveEndpoint is revived by a console re-save, not by a REST PATCH and not by a rebuild. Matting and SAM 3 now have a runtime; the stock-image interim (template zxojew9tyo) is no longer required. Region migration was considered and rejected — the network volume cannot move datacenters, so it would mean a second 500 GB volume plus re-seeding ~70 GB, to fix a problem that was configuration, not capacity. |

C.08 Chroma catalog adversarial pass (C.08.10, 2026-09-11)#

Scope: all seven files in libs/isis/workflows/src/workflows/chroma/.

  1. Placeholder/stub vocabulary (TODO, FIXME, placeholder, lorem, your prompt here, example prompt, xxx, changeme, tbd): zero hits.
  2. Prompts and seeds are real. Every prompt default is a descriptive Chroma prompt, not filler; sam3-segment defaults text_prompt to the person, which is a usable subject rather than a placeholder. seed: -1 in five files is the renderer's documented randomise sentinel (normalised per class by normalizeSeedValue), not an unset value — chroma-txt2img-golden.spec.ts pins that an explicit seed survives verbatim.
  3. Every models_required entry exists on the volume manifest. Checked all seven against infra/runpod/volumes/isis-chroma-models.manifest.json (10 rows, 23 accepted names counting basenames): 35 requirements, 0 missing.
  4. Every node class is one the worker image guarantees. Checked against the 25 classes in docker/runpod/comfyui-slim/smoke-test.sh REQUIRED_CLASSES, resolving templated class_type values through each workflow's own class map (matte_node_class_map → RMBG / BiRefNetRMBG): 0 unguaranteed classes across all seven. A false alarm on the first pass was my parser, not the catalog — REQUIRED_CLASSES=(...) was read with a non-greedy regex that stopped at the ) inside the comment "(ComfyUI core)", reporting 0 guaranteed classes and therefore everything as unguaranteed. Re-parsed line-wise.

Defects found and fixed during this section (recorded against their items): matte-cutout classified as its own mask (C.08.04), and chroma-txt2img-then-matte offered under image-to-image despite having no image input (C.08.08). No defect remained at the end of the pass.

C.11 live proof matrix (2026-09-11/12, endpoint 6l1f3tku7z2zoy, slim image release d677f914)#

Every number below is RunPod's own delayTime/executionTime from /status, times the ledger rate 0.000306 $/s. Graphs were produced by the repository's own renderCatalogWorkflow via scripts/isis/runpod-catalog-run.mjs, so a rendering defect would surface here rather than being hidden by a hand-written graph. Outputs are written to docs/agents/evidence/isis-chroma-runpod/ and hashed.

Item Workflow Status RunPod job delay ms exec ms USD Output / error Worker
C.11.02 chroma-txt2img COMPLETED 5b7af111-275f-4c04… 10061 180871 $0.0584 chroma-txt2img-0-9d49c5b4.png (1450684 B, sha 9d49c5b4e08d) bh3ek2iz6grdek
C.11.03 chroma-txt2img COMPLETED e54a1b62-b36b-490f… 1074 104445 $0.0323 chroma-txt2img-0-f9090659.png (1310954 B, sha f9090659f114) bh3ek2iz6grdek
C.11.04 chroma-txt2img FAILED 738550da-ea77-4443… 95 98828 $0.0303 WebSocket communication error: ComfyUI HTTP unreachable during websocket reconnect bh3ek2iz6grdek
C.11.05 chroma-img2img COMPLETED 83a6f93b-3c38-4bf3… 108 218049 $0.0668 chroma-img2img-0-abbd2b61.png (1439568 B, sha abbd2b6134d4) qj0iqrwap0g38x
C.11.06 chroma-inpaint COMPLETED 8567b78c-c86b-4122… 122 24933 $0.0077 chroma-inpaint-0-2f55a5c9.png (1483361 B, sha 2f55a5c97761) qj0iqrwap0g38x
C.11.07a matting-remove-background FAILED 7aeaf7f1-865b-4072… 103042 13764 $0.0357 Job processing failed qj0iqrwap0g38x
C.11.07a-retry matting-remove-background FAILED cf2b6337-d871-48a7… 1237 15265 $0.0050 Job processing failed qj0iqrwap0g38x
C.11.07b matting-remove-background COMPLETED 1df8f050-cc39-4465… 110 10035 $0.0031 matting-remove-background-0-3a7fb590.png (1655339 B, sha 3a7fb59035ce); matting-remove-background-1-1afc9124.png (25446 B, sha 1afc9124e1df) qj0iqrwap0g38x
C.11.07c matting-remove-background COMPLETED c086428e-e395-4a6c… 112 9894 $0.0031 matting-remove-background-0-4e9d373c.png (1654269 B, sha 4e9d373cfc39); matting-remove-background-1-2e83d507.png (23325 B, sha 2e83d507d19b) qj0iqrwap0g38x
C.11.07d matting-remove-background COMPLETED 531c1700-257a-4e68… 106 7168 $0.0022 matting-remove-background-0-1a46a331.png (1661618 B, sha 1a46a3314328); matting-remove-background-1-5e8a380e.png (49787 B, sha 5e8a380e5f49) qj0iqrwap0g38x
C.11.08 sam3-segment FAILED 5e209aa2-8376-410c… 111 638 $0.0002 Node 'SAM3Segment' not found. The custom node may not be installed.. Raw response: {"error": {"type": "missing_node_type", "message": "Node 'SAM3Segme qj0iqrwap0g38x
C.11.10 chroma-txt2img-then-matte COMPLETED 32a2e96e-dc01-40de… 90 35071 $0.0108 chroma-txt2img-then-matte-0-c53d031e.png (1348726 B, sha c53d031e48df); chroma-txt2img-then-matte-1-f54fdd9b.png (1541465 B, sha f54fdd9b651f) qj0iqrwap0g38x

Measured total across the matrix: $0.2556. Balance 13.8074 → 13.5110 = $0.2964 spent, the difference being the 500 GB volume accruing at $0.049/h across the run.

What passed#

chroma-txt2img on both remaining checkpoints (uncanny-v13-bf16, photonic-fusion-debloated-bf16), chroma-img2img, chroma-inpaint, chroma-txt2img-then-matte, and three of the four matting models (birefnet-portrait, birefnet-matting, birefnet-hr-matting). Verified as real images, not just non-empty responses: every matting run returned a 1024×1024 RGBA cut-out (1.58 MB) plus a separate RGB mask (20–50 KB), and chroma-txt2img-then-matte returned an RGBA cut-out alongside the RGB original — the two-SaveImage output contract from C.08.04/C.08.05 working on live output.

Three defects the matrix found#

  1. SAM3Segment is not in the image (C.11.08, C.11.09 blocked). The worker answers Node 'SAM3Segment' not found. The custom node may not be installed. Root cause, traced to the pinned pack source: py/AILab_SAM3Segment.py does a module-level from sam3.model_builder import build_sam3_image_model, the pack's requirements.txt ships SAM 3's support deps (decord, ftfy, typing_extensions) but not the sam3 package itself, and __init__.py auto-discovers node files inside try/except — so the ImportError is swallowed and the node disappears silently. The Dockerfile's existing note ("SAM3Segment … do not import groundingdino") is true but irrelevant; groundingdino was never the blocker. Fix is a Dockerfile change to install sam3, then a rebuild. Tracked as C.12.18.
  2. batch_size: 2 at 896×1152 kills ComfyUI (C.11.04). After 98.8 s the worker answered ComfyUI HTTP unreachable during websocket reconnect — the ComfyUI process died mid-render. That item exists to "confirm the per-job caps and the 4090 memory headroom", and the measured answer is that the headroom is not there for two 896×1152 latents beside a 17.4 GB checkpoint on a 24 GB card. The per-job cap of batch_size ≤ 4 (C.04.07) is therefore too generous at this resolution. Tracked as C.12.19.
  3. rmbg-2.0 fails reproducibly while all three BiRefNet models succeed (C.11.07). Two attempts, 13.8 s and 15.3 s of execution, both Job processing failed with no detail. Not a missing file: the volume holds models/RMBG/RMBG-2.0/model.safetensors (884,878,856 B) with its config.json, birefnet.py and BiRefNet_config.py. Not a poisoned worker either — the retry ran on a fresh worker after a recycle. Needs worker logs to pin. Tracked as C.12.20.

Defect follow-up, 2026-09-12 (offline; no GPU spend)#

C.12.18 — the diagnosis above is wrong in its central claim, corrected here. There is no sam3 package to install. The SAM 3 source is vendored inside the node pack at custom_nodes/ComfyUI-RMBG/models/sam3/ (82 Python modules plus assets/bpe_simple_vocab_16e6.txt.gz), and py/AILab_SAM3Segment.py puts that directory's parent on sys.path before importing it. Verified against the pinned commit 58f1947a by fetching the pack's git tree and walking the module-level import closure of sam3.model_builder (42 modules reached at load time, function-body and try-guarded imports excluded). Exactly one leaf of that closure is absent from the image:

text
sam3.model_builder
  -> sam3.model.sam3_video_inference
    -> sam3.model.sam3_video_base
      -> sam3.train.masks_ops        # import pycocotools.mask as maskUtils

pycocotools is in neither the pack's requirements.txt nor ComfyUI's, so the import raised ModuleNotFoundError, the pack's load_nodes() caught it with a bare print(f"Error loading {file}: {e}"), and the class vanished. The other 13 leaves (einops, ftfy, huggingface_hub, iopath, numpy, psutil, regex, torch, torchvision, tqdm, triton, typing_extensions, PIL) are already installed. Two further checks ruled out the alternatives: the module-level raise RuntimeError("SAM3 assets missing…") on line 27 cannot be firing because the gz is a real gzip in the repo, not a Git-LFS pointer (no .gitattributes at that commit; magic bytes 1f 8b); and triton ships with torch's Linux CUDA wheels.

Landed (unproven until a build runs — the item stays [ ]):

  • pip install pycocotools in the node-pack layer, with the import chain written into the Dockerfile so the reason survives the next reader.
  • A dedicated build layer that performs SAM3Segment's own module-level import (from sam3.model_builder import build_sam3_image_model) under the pack's sys.path. A missing SAM 3 dependency now fails the build with a real traceback instead of shipping as an absent class.
  • The build-time smoke test is fatal again (set -e restored; the full smoke log is dumped to the build output on failure, which is what the set +e was working around — the log is also still baked into the image for entry.sh to print). This is the direct answer to the C.08.10 correction below: a class list is only evidence if the check can fail the build.
  • The smoke test now also fails on ComfyUI-RMBG's own swallow message (Error loading <file>: <err>), which ComfyUI does not count as an import failure, against a named allowlist. The allowlist holds exactly one entry — AILab_SegmentV2.py, for the deliberate groundingdino-py omission. Checked against synthetic logs: a clean log and a groundingdino-only log pass; the AILab_SAM3Segment.py: No module named 'pycocotools' line that this image actually shipped fails.

Build df19db7a (commit 19b645ecce, branch runpod-comfyui-slim, 2026-09-12 00:38:42Z → 00:48:59Z): FAILED, and the failure is the fix working. The build reached the smoke-test layer, which means the two layers before it passed: pip install pycocotools, and the dedicated layer that performs from sam3.model_builder import build_sam3_image_model; from sam3.model.sam3_image_processor import Sam3Processor under the pack's own sys.path. That is the proof of the C.12.18 diagnosis — the import that had been failing now resolves. The smoke then failed on something else entirely, quoted from the console build log:

text
[smoke] node files failed to load and are not on the allowlist:
Error loading /comfyui/custom_nodes/ComfyUI-RMBG/py/AILab_Florence2.py: No module named 'matplotlib'
+ exit 1

SAM3Segment is not in that list. AILab_Florence2.py is a second node file that had been failing silently in every image shipped so far, caught the first time the check was allowed to fail a build. matplotlib is not in the pack's requirements.txt either, Florence 2 captioning is not in the Isis catalog, and no workflow references a Florence class (checked), so it joins the allowlist with its reason rather than pulling ~50 MB of plotting library into a worker image. The prediction that pycocotools would drag matplotlib in with it was wrong — modern pycocotools wheels dropped that dependency.

The RunPod builds API gives only Command '[...docker buildx build...]' returned non-zero exit status 1; the reason is console-only, at Serverless → endpoint → Builds → Build logs, and the log panel is virtualised so it must be read on screen rather than scraped.

Build 05bbde18 (commit 216cdba9ad, 2026-09-12 01:14:55Z → 01:25:52Z): COMPLETED — and completing IS the proof of C.12.18. The chain is:

  1. The smoke test runs test -L on all four volume symlinks, boots ComfyUI, fetches /object_info and fails on any class in REQUIRED_CLASSES that is absent.
  2. SAM3Segment is in REQUIRED_CLASSES and has been all along — that is exactly what made C.08.10's reasoning wrong.
  3. The smoke is now fatal: status is captured, written to smoke.status, and the layer exits non-zero when it is non-zero (the previous build, df19db7a, failed here, so this is demonstrated, not assumed).
  4. The build completed.

So SAM3Segment is registered in the running image's /object_info. This is the difference the item asked for: the same class list that was worthless as evidence yesterday is evidence today, because the check that reads it can now fail the build. sam3-segment and sam3-segment-inpaint have a runtime.

Not observed directly this time: the [smoke] ok — N required classes present, M classes total line. The console's build-log panel rendered blank on three attempts after the earlier successful read of df19db7a's log, and reading it is not worth a worker boot — entry.sh prints the whole smoke log at container start, so the next real job on this release surfaces it in the worker log for free. C.11.08/C.11.09 (a real segmentation) remain unchecked: they need GPU seconds.

C.12.19 — fixed and specced. The batch ceiling now scales with the frame: enforceRenderedGraphCaps refuses a sampling pass whose width × height × batch_size exceeds maxBatchPixels (1024², the largest frame this endpoint has actually rendered — C.11.02/C.11.03), clamped so a lone frame is never refused by the batch rule and maxPixels still governs it alone. The measured killer, 896×1152 × 2 = 2,064,384 latent pixels, is refused before submit with the frame named; 512² × 4 = 1,048,576 still passes. chroma-txt2img.json pins batch_size to max: 1 with the reason, because every aspect preset it offers is ~1 MP and the executor would refuse a 2 at all of them — the form no longer offers a value that can only be rejected. worker-comfyui-submission.spec.ts 14 passed; libs/isis/workflows 426 passed | 1 skipped; apps/isis/generation-api src/workflows/ 13 failed | 123 passed, the documented pre-existing baseline unchanged.

C.12.20 — root cause not determined offline; the next attempt is now instrumented. Ruled out from the pinned sources: it is not timm's removed timm.models.layers shim, because BiRefNetRMBG execs a birefnet.py with the same from timm.models.layers import … lines and those three models pass. The two loaders do differ — AILab_RMBG.py builds the class by exec-ing the remote code and picking the first PreTrainedModel subclass in dir(module), then load_state_dict (strict) on the safetensors, falling back to AutoModelForImageSegmentation.from_pretrained(cache_dir, trust_remote_code=True, local_files_only=True); AILab_BiRefNet.py instantiates module.BiRefNet directly and calls .half(). Which of those steps raises is not decidable from source. What was blocking the answer is that Job processing failed is worker-comfyui's own generic string and the real node error lives in the worker's log, which RunPod deletes with the worker. So scripts/isis/runpod-catalog-run.mjs now, on any non-COMPLETED status, prints output.errors[] and the whole status payload (base64 elided) and fetches GET api.runpod.io/v2/serverless/<ep>/workers/<wid>/logs immediately, writing it beside the run's outputs. Capture never throws — it returns the reason it could not capture, so it can never mask the failure it documents. The next rmbg-2.0 attempt costs the same $0.003 and comes back with the traceback.

Two operational findings worth more than the items#

  • A crashed ComfyUI poisons its worker, and RunPod still reports it healthy. After C.11.04 killed ComfyUI, the next five jobs failed in 0.3–0.7 s each with ComfyUI server (127.0.0.1:8188) not reachable, all on the same worker bh3ek2iz6grdek, while /health showed ready: 1, unhealthy: 0. Without noticing the shared workerId those five would have been recorded as five independent workflow failures. Recycle with workersMax 0 → 1; after that the same jobs passed on a fresh worker.
  • Restoring workersMax is not instant. For several seconds after the restore, /run still answers 409 ENDPOINT_PAUSED (max_workers=0). Submissions in that window are rejected outright and cost nothing.

A correction to this file's own C.08.10 entry#

C.08.10 recorded "0 unguaranteed classes" by checking every catalog node class against the smoke test's REQUIRED_CLASSES. That reasoning was wrong: the Dockerfile runs the smoke test non-fatally (set +e, result baked to smoke.status), so the list states what the image is asserted to provide, not what it does provide. SAM3Segment is on that list and is absent from the running image. A class list is only evidence if the smoke test that checks it is allowed to fail the build.

C.11.13 rest state at session end (2026-09-12)#

Check Required Actual
Workers 0 0 (drained via workersMax → 0)
Pods none 0
Jobs in queue / in progress 0 0 (queue purged after the C.11.12 test)
currentSpendPerHr 0 $0.0490
Network volume — ax0oj250j3, 500 GB, retained

The currentSpendPerHr: 0 requirement cannot be met and should be rewritten. A network volume bills continuously whether or not anything runs: 500 GB at $0.07/GB/month is $0.049/h, and the only way to reach zero is to delete the volume and re-seed 70 GB of models. $0.049/h is the rest state. The meaningful check is that it equals the volume-only figure with no compute on top, which it does.

Day's spend, RunPod's numbers vs the ledger. Balance moved 13.8074 → 13.4977 across the matrix = $0.3097. Of that, $0.2556 is the sum of the per-job delayTime + executionTime figures RunPod returned (table above), and the remaining $0.0541 is ~66 minutes of volume accrual at $0.049/h during the run. The two agree: per-job billing and the balance delta differ only by storage, which the per-job numbers do not and should not include. No unexplained gap.

C.12.20 root cause — KeyError: 'process_res', and it was OUR graph (2026-09-12)#

Two earlier attempts learned nothing because runpod-catalog-run.mjs discarded everything but worker-comfyui's generic Job processing failed. The capture fix worked: the third run returned

text
Workflow execution error: Node Type: RMBG, Node ID: 2,
Message: Error in image processing: Error in batch processing: 'process_res'

The pack is not broken and the model file was never the problem. The catalog's matte_node_class_map sends rmbg-2.0 to node class RMBG and the three BiRefNet options to BiRefNetRMBG, and the node carried a comment asserting the two had the "same input/output signature". They do not:

RMBG (rmbg-2.0) BiRefNetRMBG (the other three)
declares process_res in INPUT_TYPES yes, default 1024 no
reads it params["process_res"], a bare lookup never

In the ComfyUI UI, an unset widget is filled from its default. In an API-format graph only the keys you send exist, and our rendered graph sent none — so RMBG raised KeyError: 'process_res' while the BiRefNet three, which never look at it, were unaffected. That is exactly the 3-pass / 1-fail split C.11.07 recorded, and the asymmetry was in our own template all along.

Fix: process_res is now a declared catalog input (integer, default 1024, 256–2048) emitted on the templated matte node in both affected workflows — matting-remove-background and chroma-txt2img-then-matte. Sending it is inert for BiRefNet: ComfyUI builds a node's call from the class's declared inputs, so an undeclared key is dropped rather than passed, and BiRefNetRMBG.process_image takes **params besides. Verified by rendering both models through the real renderCatalogWorkflow: rmbg-2.0 -> RMBG | process_res = 1024 and birefnet-hr-matting -> BiRefNetRMBG | process_res = 1024. The misleading _comment is replaced with the real contract. libs/isis/workflows 635 passed.

A schema detail worth keeping: the catalog input schema has no step key. The first draft included step: 128 and the migration harness rejected it; the 128-multiple requirement lives in the description instead, and the pack's own normalize_process_res clamps and rounds anyway.

Cost, and an overrun to be straight about. This run cost $0.1032 — delay 94 416 ms (cold start) + exec 12 058 ms on the 96 GB tier at $0.000969/s — against the "under $0.05 for all three" the approval was given for. It was routed to the video endpoint because the image endpoint could not place a worker, and the 96 GB tier is ~3x the image endpoint's rate with a 94-second cold start on top. Job 715d962b, worker r57269ikp5dhys.

The worker-log capture did NOT help and should not be relied on. GET /v2/<ep>/workers/<id>/logs returned nothing but : heartbeat SSE frames. Everything useful came from output.details[] in the /status payload.

The chroma endpoint was wedged — ROOT CAUSE: a 20 GB container disk (2026-09-12)#

The stuck-initializing fault from earlier persisted, and two hypotheses are now dead, which is worth more than the guesses were:

  1. Volume contention: FALSIFIED. The re-run was deliberately done after the loader pod was deleted and the volume was completely quiet. It timed out again after 15 minutes in IN_QUEUE, worker initializing throughout. The 288 GB pull was not the cause.
  2. The worker image: FALSIFIED, and by a clean control. The new video endpoint 2lezocymx2rgot — same repo, same branch, same Dockerfile, same network volume, same datacenter — brought a worker to ready immediately and then to running. The image boots. The fault is the chroma endpoint or its template, not the thing it runs.
chroma 6l1f3tku7z2zoy video 2lezocymx2rgot
worker initializing, indefinitely ready → running
template j1gzx4nccu qcxiqduk01
GPU tiers 24 / 24 / 32 GB 96 / 80 / 141 GB
allowedCudaVersions 12.8, 12.9, 13.0 (unrestricted)
minCudaVersion 12.8 12.0

The leading hypothesis is now the CUDA constraint. The chroma endpoint only accepts hosts reporting CUDA ≥ 12.8, on GPU tiers whose EU-RO-1 supply is already thin (post-volume: 24 GB Unavailable, 24 GB PRO Medium, 32 GB PRO Low). A placement that then fails the CUDA filter would present exactly as a worker that never finishes initializing. PATCH /v1/endpoints/<id> with a widened allowedCudaVersions (12.0–13.0) was accepted; minCudaVersion still reads 12.8, which suggests that field is derived from the image rather than set on the endpoint.

Neither template is inspectable. Both are git-integration templates: GET /v1/templates/<id> returns 404 template not found for each, and neither appears in GraphQL podTemplates. So container disk — the other candidate, since the image grew substantially when the six video packs landed and the working endpoint was given 30 GB — cannot be read or compared except in the console.

Method note for whoever picks this up: the rmbg-2.0 diagnostic was re-routed to the working endpoint rather than waiting on the broken one. Same image, same volume, same models; only the GPU differs, which is irrelevant for a ~1 GB matting model. That yields C.12.20's traceback without depending on the chroma fault. It does not yield a C.11.07 cost row — those are measured on the image endpoint — so C.11.07 stays open regardless.

RESOLVED — container disk, and C.11.07 closed on the rebuilt endpoint#

Root cause: the image endpoint's template allocated a 20 GB container disk. When V.01.01 added the six video node packs and their dependencies the image outgrew it. From Release #10 onward placement succeeded, the container was never created, and the worker sat initializing indefinitely with completely empty logs — which is what made it look like everything else first. Raising the disk on the old endpoint (Release #13) did not recover it; only a rebuild did.

Old 6l1f3tku7z2zoy deleted; new 6jye7vngg4i685, template vfl3iu1dr6, 30 GB disk, same volume, same branch and Dockerfile, 24/24 PRO/32 PRO tiers, workers 0/1, 300 s cap, FlashBoot. Drift check green. Its first worker reached ready, and its first job ran.

C.11.07's fourth model now renders, which needed both fixes — C.12.20's missing process_res AND a worker that could start:

value
endpoint / rate 6jye7vngg4i685, 0.000306 $/s (image tier)
delay / exec 76 410 ms (cold) / 13 987 ms
cost $0.0277
worker crr86xi6bdzmxk
inputs model: rmbg-2.0, process_res: 1024
outputs b081ad54… 1024×1024 RGBA, 1 651 812 B · 96dc6a21… RGB mask, 21 296 B

The alpha was decoded, not assumed: 19.7 % fully opaque, 3.0 % fully transparent, 77.3 % soft edge. A real matte rather than a black one. The large soft-edge fraction is recorded as measured; no claim is made about whether it is good. Files: docs/agents/evidence/isis-chroma-runpod/2026-09-12-rmbg-2.0-{rgba-b081ad54,mask-96dc6a21}.png.

A mistake worth recording. Testing whether NVIDIA RTX PRO 4500 Blackwell is accepted by serverless REST, a PATCH was sent containing only that card. It is not accepted — and RunPod set gpuTypeIds to null rather than rejecting the request, leaving the live endpoint with no GPUs. It was restored immediately, but the test belonged on a throwaway endpoint. The finding stands: that card is reachable only as pool BLACKWELL_32, which REST rejects, and the console's "32 GB PRO" maps to ADA_32_PRO instead — so the best-supplied card in the region is selectable by neither the REST API nor the console form.

Region survey (user decision: do not migrate; keep as a fallback). Measured in the console's create form with exactly one datacenter selected. A first pass was discarded as invalid — "Deselect all" silently does not apply, so regions accumulated and every reading after the first was a union; the method now unchecks each option explicitly and confirms the region label on every read.

Region 24 GB 24 PRO 32 PRO 48 PRO 80 PRO 96 PRO 141 GB
EU-RO-1 Low Low Low — — Low —
US-CA-2 — Low — — Low Low Low
US-IL-1 Low High — Low — — —
US-GA-2 Low — — — Low — Low
US-MO-2 Low — — — — Low —
US-TX-3 — Low — Low — — —
US-NC-2 — — — — — Low —
US-CO-1 — — — — — — Low

EU-RO-1 is the only region with 32 GB PRO, which matters for Chroma1-HD (17.8 GB) + T5-XXL (9.8 GB) in a 24 GB card. US-CA-2 is the only region covering both tiers and is the migration target if no GPU available ever becomes chronic. Labels are point-in-time — EU-RO-1 read differently twice within this session — so one snapshot must not decide a 358 GB move.

An earlier survey attempt was wrong and is withdrawn. GraphQL gpuTypes { lowestPrice(dataCenterId) { stockStatus } } reports the pod market, not serverless: it prices RTX 4090 at $0.34/h and H100 at $2.69/h against serverless $0.69 and $4.79. Do not use it for serverless capacity.

What the console showed, and why the endpoint should be recreated#

Rule 9 says the console is the authority on placement, and it was — the three facts that settled this exist nowhere else.

Release history. The last successful renders (C.11) ran on Release #9, commit 105ad3c. Releases #10 and #11 added the six video packs. No worker has started since. That is the whole timeline of the fault in one view.

Worker detail. o9ikf459jejnd6, EU-RO-1, RTX PRO 4500 — the same card the successful C.11 runs used. Placement and hardware were never the problem.

Logs: completely empty. Not an error, not a traceback — no lines at all. The container is never created, so there is nothing to log. That rules out every in-container explanation, including the pycocotools/SAM3 import work.

Container disk was 20 GB against the working endpoint's 30 GB, and it is invisible to both APIs for a git-integration template. Raised to 30 GB (Release #13, confirmed version: 13). It did not fix it — still initializing 20+ minutes later.

And the image is proven good. git rev-parse origin/runpod-comfyui-slim is a5ac709, which is exactly chroma's Release #11 commit. The video endpoint 2lezocymx2rgot built that same commit today and its worker reached ready in seconds, on the same network volume in the same datacenter.

So: same commit, same volume, same datacenter, same container disk — one endpoint works and one cannot start a container. Four hypotheses are dead (volume contention, the worker image, the CUDA constraint, container disk), and what remains is the endpoint itself, which is C.12.12's recorded fault. The remedy recorded there is a console re-save; two have now been applied (Releases #12 and #13) without effect.

Recommendation: delete and recreate 6l1f3tku7z2zoy the way the video endpoint was made — console → Deploy from a GitHub repository, branch runpod-comfyui-slim, docker/runpod/comfyui-slim/Dockerfile, the 24/32 GB tiers, 30 GB disk, same volume. That took about three minutes for the video endpoint and produced a working worker immediately. It changes the endpoint id, so desired-state.json, the runbook env recipe and RUNPOD_COMFYUI_ENDPOINT_ID all need updating with it.

Two API lessons worth keeping. REST reported chroma's gpuTypeIds as the old four-card list (RTX 4090, A5000, L4, RTX 3090) while the live worker was demonstrably on an RTX PRO 4500 added in Release #5 — REST's GPU list for this endpoint is stale. And container disk is not exposed by REST or GraphQL for a git-integration template at all. When an endpoint misbehaves, the console's Releases, Workers and Logs tabs are not a fallback, they are the only source.

V.02.04 / V.02.05 — the video model pull, done (2026-09-12)#

Pod 689tdcv0qscxgt, CPU-only (cpu3c, 2 vCPU, computeType: CPU) in EU-RO-1, created 12:30:33Z, loader finished 13:27:47Z, pod deleted immediately after.

Figure Value
Wall clock, rented → finished 57.2 min
Pod cost at $0.06/h $0.0572 (the tracker's table guessed ~$3; the estimate before this run was $0.25)
Rows in manifest-result.json 48 = 42 manifest artifacts + 6 companion files
failures 0
Skipped as already correct 16 — the C.02 chroma rows, i.e. the idempotent skip V.02.04 asked to observe
Downloaded this run 32 rows, 288.23 GB in 55.4 min → 86.7 MB/s
Volume after 42/42 rows, 358.30 GB of 500 GB, lint 0 errors 0 warnings, 28.3% headroom

Every one of the 42 artifacts was verified against the sha256 pinned in the manifest before the pull: 42 matched, 0 mismatched. That is the check V.02.05 asks for, and it is meaningful in this direction because V.02.02 filled the hashes from HuggingFace LFS oids before any bytes were fetched — so the pull was verified against a value recorded in advance, not against itself.

A trap worth recording: the loader keys its result rows "<id>:<file>", and two artifacts (rmbg-2.0, birefnet-portrait) carry companion files. Matching on the id alone compares a companion's hash to its parent's and reports six false mismatches. Match on id:file.

At rest afterwards: pods: [], currentSpendPerHr $0.049 — the 500 GB network volume alone ($0.07/GB/month), nothing else running.

V.03.04 — the video endpoint, created in the console (2026-09-12)#

The GPU question is settled (see below), but creating the endpoint hit a structural obstacle that is worth writing down because it is not obvious:

  1. A RunPod template backs exactly one endpoint. Reusing the chroma endpoint's template returns graphql: This template is already bound to an endpoint. D-V2's "one worker image" is satisfiable — one image, two templates — but not "one template".
  2. The chroma endpoint's template j1gzx4nccu cannot be read or cloned. It is a git-integration template: absent from GraphQL podTemplates, and GET /v1/templates/j1gzx4nccu returns 404 template not found. So its image reference is not recoverable through either API.
  3. runpod-provision.mjs template create requires a digest-pinned image (…@sha256:…), and a git-integration build has no digest — that is C.01.09. The guard is correct and should stay; it just means this path cannot create the video template.

So the endpoint has to be created in the console, where a git build can be attached to a second endpoint. That is a UI action, not an API one.

Two corrections to V.03.01's GPU note, both from the live API:

  • REST POST /v1/endpoints does not accept pool ids at all. BLACKWELL_96, AMPERE_80, HOPPER_141 are rejected; the gpuTypeIds enum is concrete card names. GraphQL reports the same endpoint's GPUs as pools (ADA_24,AMPERE_24,BLACKWELL_32 for chroma) while REST reports the cards (RTX 4090, A5000, L4, RTX 3090) — the two APIs are different vocabularies over the same thing, and the pool list is an aggregation, not a constraint.
  • H100 80 GB IS available to serverless, contrary to the inference drawn from "there is no 80 GB Hopper pool". NVIDIA H100 80GB HBM3, NVIDIA H100 PCIe and NVIDIA H100 NVL are all in the accepted enum. The missing HOPPER_80 pool id meant only that no pool aggregates them.

V.03.04 outcome — 2lezocymx2rgot, and what the console revealed#

Created through Deploy from a GitHub repository (GreyChimp/oshun, branch runpod-comfyui-slim, docker/runpod/comfyui-slim/Dockerfile), which is the path the API could not take. RunPod auto-created template qcxiqduk01; the "could not find runpod.serverless.start()" warning is expected and was accepted, because the handler comes from the worker-comfyui base image rather than this repo.

Field Value
endpoint id 2lezocymx2rgot (isis-comfyui-video)
templateId qcxiqduk01
gpuTypeIds RTX PRO 6000 Blackwell Server / Workstation / Max-Q, H100 PCIe / 80GB HBM3 / NVL, H200
workersMin / Max 0 / 1
idleTimeout 10 s
executionTimeoutMs 900 000
flashboot true (Standard)
networkVolumeId ax0oj250j3
scaler QUEUE_DELAY, 4 s
container disk 30 GB

Drift check green on both endpoints. At rest the account is at currentSpendPerHr $0.049 — the 500 GB volume alone.

The finding that matters, and it came from the user: GPU supply labels are global until a network volume is attached. Before attaching, the console showed 96 GB High Supply, 80 GB High Supply, 141 GB Medium Supply. The instant isis-chroma-models was selected the data-center field locked to EU-RO-1 and the same three tiers re-read as:

Tier Global EU-RO-1
96 GB $3.49 High Supply Low Supply
80 GB $4.79 High Supply Unavailable
141 GB $5.93 Medium Supply Unavailable

So the three-deep fallback chain is, in the region where the volume lives, a chain of one. All three stay enabled deliberately — "Unavailable" is a point-in-time reading, not a property — but plan on 96 GB or nothing, and read supply only after the volume is attached. Anyone sizing a RunPod endpoint against the pre-volume labels is reading numbers for a datacenter they will not be deployed in.

Despite the Low Supply label, the endpoint placed a worker immediately: workers: {idle: 1, ready: 1} at rest, which also proves the git build boots.

Two corrections and a defect fixed on the way:

  • The console's 80 GB serverless tier is H100 ($4.79), not the A100 80GB PCIe ($2.72) the pricing page lists — A100 is not offered to serverless here at all. The earlier recommendation priced the middle rung wrong; it does not change the verdict, since 96 GB at $3.49 is both cheaper and larger.
  • GraphQL's gpuIds pool string is lossy in both directions. The live endpoint reports ADA_80_PRO,BLACKWELL_288,BLACKWELL_96,HOPPER_141 while REST reports the seven cards above — with no B300 anywhere. REST is the truth.
  • runpod-endpoint-drift-check.mjs compared desired state against that pool string, so it reported drift on a correctly configured endpoint and, worse, could not have seen a wrong card swapped inside a matching pool. It now reads the concrete card list from REST whenever desired-state is written in card names, warns loudly if REST cannot answer rather than silently falling back to the lossy comparison, and leaves the pool path untouched for the chroma endpoint. 11 tests pass; both endpoints read no drift.

The C.11.07/08/09 re-runs were attempted and abandoned (2026-09-12, $0 spent)#

The user approved a sub-$0.05 session to capture the rmbg-2.0 traceback (C.12.20) and render sam3-segment / sam3-segment-inpaint (C.11.08/09). It did not happen, and nothing was billed. Recorded so the next attempt starts from what was learned rather than repeating it.

Step What happened
/run refused 409 ENDPOINT_PAUSED (max_workers=0) — the endpoint was left at workersMax 0 by an earlier drill.
Raised to 1 REST endpoint update reported workersMax: 1 immediately, but /run kept answering 409 for about 30 seconds afterwards. It is propagation delay, not the C.12.12 wedge; retrying after half a minute worked.
Job queued 68417991-…-e1 accepted, IN_QUEUE.
Worker never became ready One worker initializing, zero in every other state, for 43 minutes. runpod-catalog-run.mjs gave up at its 15-minute cap; the job stayed queued regardless, because that cap abandons the wait, not the job.
Stood down Both queued jobs cancelled (delayTime and executionTime both absent, so no GPU seconds were billed), workersMax back to 0, health confirmed 0 workers / 0 queued.

This is not the wedged endpoint of C.12.12, and the distinction is worth keeping: that fault shows zero workers in every state with jobs queued, because placement itself has stopped. Here placement worked and a worker existed throughout — it simply never finished coming up.

Two candidate explanations, neither confirmed:

  1. The image got much bigger. V.01.01 added six video node packs and their dependencies (cupy, triton, pycocotools, timm, the controlnet_aux annotators). The last measured cold boot, before those, was ~4 minutes (d1e87d06, 14.7 GB).
  2. The network volume was saturated. The V.02.04 loader pod was writing to the same volume at ~3.2 GB/min throughout, and a serverless worker mounts that volume as it starts. This is the cheaper hypothesis to test, and it is why the re-runs were deferred rather than debugged: retry them once the pull is finished and the loader pod is deleted. If a worker comes up normally on a quiet volume, the answer is (2) and the operational rule is simply "do not run jobs against an endpoint whose volume is being seeded".

C.12.04 — what is actually on the volume (2026-09-12, live listing, $0)#

A model inventory built from the manifest alone would have been a fabrication. The manifest is declarative — it says what the loader should put on the volume — and on 2026-09-12 it declared 42 artifacts while the volume held 10. A card reading the manifest and calling it an inventory would have reported 288 GB of models that do not exist.

So presence is observed, and it is a three-state answer: present (seen in a listing), absent (a listing was taken and it was not in it), unknown (no listing could be taken). unknown never degrades to absent — a missing credential is not evidence that a file is missing — and every observed total is null rather than 0 when unobserved, because "0 present" and "we did not look" are different sentences to an operator deciding whether to start a multi-hour pull.

Measured live through the new endpoint (GET /api/v1/jobs/runpod/volume/models, authenticated, against volume ax0oj250j3 in EU-RO-1, S3 LIST only — free):

Figure Value
Objects on the volume 20
Manifest rows present / absent 10 present, 32 absent, 0 unknown of 42
Declared footprint 358.30 GB
Held 70.07 GB
Still to pull (V.02.04) 288.23 GB — the tracker's estimate said 283 GB
Volume used / capacity 70.07 GB of 500 GB, 429.93 GB free — the pull fits
Size mismatches none
Unclaimed objects 10 (companion .py, config.json, __pycache__), ~0.4 MB

Per folder: diffusion_models 3 of 9 present (53.40 GB of 236.94 GB), RMBG 4 of 4, sam3 1 of 2 (the .pt ComfyUI-RMBG reads is there; the .safetensors ComfyUI-SAM3's video nodes want is not), text_encoders 1 of 3, vae 1 of 2, and checkpoints, clip_vision, controlnet_aux, detection, FlashVSR-v1.1, interpolation, latent_upscale_models, loras all 0 — i.e. exactly the C.02 chroma set is there and the entire video set is not.

Two defects the browser pass caught that no unit test would have:

  1. The card printed (0.00 GB of it is companion files and job data no manifest row claims). The real unclaimed total is ~0.4 MB, so the parenthetical fired and said nothing. It now only names the figure above 5 MB, and otherwise reports the object count with "under 5 MB in total". Pinned by a spec.
  2. The six-column folder table was clipped off the card entirely — the card sat in the page's auto-fit minmax(220px, 1fr) grid at 244 px against a 680 px table. Both tables are now in .isis-table-wrap (the app's existing overflow container) and the card spans the grid (gridColumn: 1 / -1), measured after the fix at card 1014 px / table 988 px, fitting, with no sideways page scroll at 400 px width.

Screenshot: docs/agents/evidence/isis-chroma-runpod/2026-09-12-c1204-volume-inventory.png.

A landmine worth the half hour it cost: killing the pnpm dev parent does NOT kill the tsx child. A stale generation-API process from the previous drill kept port 4010 with its own JWT_SECRET, so a freshly minted token came back 401 Invalid token signature on every route — which reads exactly like a mis-minted token. Teardown must check lsof on the port AND pgrep -f generation-api, and kill what is actually listening.

C.11.11 — budget refusal, for real (2026-09-12, $0, nothing dispatched)#

The drill runs against the live dev stack (Postgres + Redis + the generation API on 4010 + the dashboard on 5180), with the endpoint id and RunPod key configured exactly as a real session, and spends nothing: the gate refuses the submit before anything reaches RunPod.

How the exhausted state was created, stated plainly: the API was started with ISIS_RUNPOD_COST_LEDGER_KEY_PREFIX=isis:drill:c1111:cost — a throwaway ledger key — and that key was seeded with usd 0.5, jobs 2, seconds 1600, coldStarts 1, wf:chroma-txt2img:usd 0.5 against a budget of $0.25. That spend was seeded, not earned. The real ledger key (isis:runpod:cost:2026-09-12) was verified absent before the drill, verified absent after it, and the drill key was deleted at the end. Seeding is the precondition; the thing under test is the gate's behaviour in that state, and the alternative — burning $0.50 of real GPU time to reach it — would have measured the same branch for money.

Step Result
GET /jobs/runpod/cost/today totalUsd 0.5, budgetUsd 0.25, remainingUsd 0, usedRatio 2, enabled true
POST /api/v1/jobs (chroma-txt2img, square_1024, consentId set) HTTP 429, code RUNPOD_DAILY_BUDGET_EXCEEDED, reason budget_exhausted, ledger image, estimatedCostUsd 0.011, retryAfter 43393
Response headers retry-after: 43393 (UTC midnight), x-runpod-daily-budget-usd: 0.25, x-runpod-daily-spend-usd: 0.5, x-runpod-daily-remaining-usd: 0
Nothing was created job list total stayed 7, all from 2026-09-11; drill ledger usd still 0.5; no RunPod dispatch in the API log
Dashboard, real chromium studio-submit disabled; reason rendered verbatim: Daily RunPod budget spent: $0.5000 of $0.25 (2026-09-12 UTC). Submits are refused until the UTC day rolls over or ISIS_RUNPOD_DAILY_BUDGET_USD is raised.
RunPod page $0.5000 of $0.25 spent today (2026-09-12 UTC) - $0.0000 remaining · 2 billed jobs · 1600s billed · 1 cold start
Budget restored to $3 submit button enabled, reason line gone, card reads $0.5000 of $3.00 spent today - $2.5000 remaining

Screenshots: docs/agents/evidence/isis-chroma-runpod/2026-09-12-c1111-studio-submit-disabled.png (greyed Submit Job with the reason above it) and docs/agents/evidence/isis-chroma-runpod/2026-09-12-c1111-budget-restored.png.

A finding the drill produced on the way: pointing the dashboard at the live API with a tenant snapshot whose projectId did not match the bearer token's projectId claim produced Ledger unavailable: Project scope mismatch on the budget card — the fail-closed path, reached for a reason that has nothing to do with money. It is the correct behaviour (a scope mismatch must not be allowed to read another project's ledger), but it renders identically to a Redis outage, so an operator seeing that string should check scope before infrastructure. The dashboard's auth snapshot must carry the same tenantId/projectId the minted token claims (tenant_isis_dev / project_isis_dev from scripts/isis/mint-dev-jwt.mjs defaults).

Teardown: both servers killed, lsof -nP -iTCP -sTCP:LISTEN | grep -E ':(3010|3020|4010|5180)' empty, no chromium orphans, drill ledger key deleted, throwaway env script deleted.

V.11.03 — budget refusal on the VIDEO ledger (2026-09-12, $0, nothing dispatched)#

Same shape as C.11.11 and the same honesty about it: the exhausted state was seeded, not earned. The API ran with ISIS_RUNPOD_COST_LEDGER_KEY_PREFIX=isis:drill:v1103:cost, whose video key (isis:drill:v1103:cost:video:2026-09-12) was set to usd 0.5, jobs 2, seconds 375.657, coldStarts 1, wf:wan22-t2v:usd 0.5 against a $0.25 budget — 375.657 s x $0.001331/s = $0.50, so the seed is internally consistent rather than a made-up number. The real key isis:runpod:cost:video:2026-09-12 was verified absent before and after (the only real ledger key on the box is isis:runpod:cost:2026-09-11), and the drill key was deleted at the end.

The point of the drill is the SECOND ledger: that exhausting video refuses video and leaves image alone.

Step Result
GET /jobs/runpod/cost/today (one response, one instant) video totalUsd 0.5, budgetUsd 0.25, remainingUsd 0, usedRatio 2; image totalUsd 0, budgetUsd 3, remainingUsd 3
POST /api/v1/jobs (wan22-t2v, 81 f, 832x480, consentId set) HTTP 429, code RUNPOD_DAILY_BUDGET_EXCEEDED, reason budget_exhausted, ledger video, estimatedCostUsd 0.16 (the catalog figure), retryAfter 24544 (UTC midnight)
Response headers retry-after: 24544, x-runpod-video-daily-budget-usd: 0.25, x-runpod-video-daily-spend-usd: 0.5, x-runpod-video-daily-remaining-usd: 0 — and no x-runpod-daily-*, because only the video ledger was consulted
POST /api/v1/jobs (chroma-txt2img) with the video ledger still exhausted HTTP 201, headers x-runpod-daily-budget-usd: 3, x-runpod-daily-remaining-usd: 3, no 429 — the two budgets are independent in the direction that matters
That image job failed at the consent guard before comfyuiService.submitJob; zero RunPod HTTP calls in the API log, drill ledgers unchanged
Dashboard, real chromium, motion preset wan22-t2v studio-submit disabled; studio-motion-budget-block reads verbatim: The video budget of $0.25 is spent for today. This is separate from the image budget, so image jobs are unaffected. The image block studio-budget-block is absent
Dashboard, no motion preset studio-submit enabled — an image submit is not blocked by a spent video budget
Restored (ISIS_RUNPOD_VIDEO_DAILY_BUDGET_USD=6, drill key deleted) motion block gone; RunPod page reads $0.0000 of $6.00 spent today - $6.0000 remaining, both endpoints configured · routable

Screenshots: docs/agents/evidence/isis-chroma-runpod/2026-09-12-v1103-studio-video-budget-block.png, 2026-09-12-v1103-runpod-two-meters.png, 2026-09-12-v1103-budget-restored.png.

Three defects the drill found, all fixed here. None of them is about budgets; they are what a live path shows that a mocked one does not.

  1. The C.12.17 double-billing guard was being thrown away. createWorkflowError returns a plain object, not an Error subclass, and classifyError only read .code off a value that was instanceof Error. So every error the executor throws was re-derived from its message text and came back UNKNOWN_ERROR, retryable: true. The live witness: a consent-less image submit raises INPUT_VALIDATION_FAILED with retryable: false and was observed running four times (attempts 0-3). The same seam carries retryable: isProviderResultRetryable(result) — the C.12.17 decision that stops a render which has already started billing from being re-submitted — so that fix was live in the code and dead on the wire. classifyError now returns an already-classified error verbatim, message patterns being reserved for foreign errors; details and retryDelayMs survive too, which restores the timeoutPhase the EXECUTION_TIMEOUT handler logs. Re-measured live: one attempt, INPUT_VALIDATION_FAILED, retryable false, zero RunPod calls. error-boundary.spec.ts (10 tests) pins it.

  2. The dashboard's whole video half had never rendered against the live API. /ready spreads its authenticated evidence into the response body, so runpodChroma and checks are top-level; the client typed them under detail and RunPodPage read data?.detail?.runpodChroma, which is always undefined. The video budget meter, the endpoint table and the video kill switch therefore showed their "no video endpoint is configured" empty states against a healthy, configured, routable endpoint. Type flattened, page fixed. After: $0.5000 of $0.25 spent today, Video (motion) — 2le…got · configured · routable, and a working Stop video workers button.

  3. /ready computed checkedAt and never returned it, so no consumer could age the verdict. Returned now. Separately, getReadiness() discarded the 503 body, which for /ready is the answer rather than a failure — a "not ready" snapshot is exactly what an operator needs, and on this dev stack /ready is permanently 503 because MinIO is deliberately not run (runbook §1). It now reads the snapshot off the error (body or details, matching the sibling checkServiceReadiness) and still throws on a 401 or a dead socket. generation.readiness.spec.ts (5 tests) pins it.

Not a defect, for the record: after deselecting a motion preset the Submit button stays disabled. The preset leaves type: text-to-video on the form, which is a cost-warning generation type, so the RunPod cost acknowledgement is required; ticking it enables the button. Verified in chromium.

Also corrected while the preflight was blocking the stack: the runbook billed every image second at the 4090's $0.000306, but the rebuilt endpoint's gpuIds accept an RTX 5090 ($1.58/h = $0.000439/s). A list means "any of these" and the ledger cannot know which card ran, so it must bill the dearest — under-billing is the direction that stops a budget firing. ISIS_COST_RATE_RUNPOD_USD_PER_SECOND is now 0.000439. NVIDIA L4 and NVIDIA GeForce RTX 3090 are in that list with no established serverless flex rate; the preflight names them rather than guessing, and RunPod's API publishes no serverless price to settle it.

V.11.04 — video kill switch, for real (2026-09-12, $0, nothing rendered)#

Engaged from the dashboard, not the API: arm runpod-kill-switch-arm-video, confirm runpod-kill-switch-confirm-video. The card then read Workers stopped at 5:33:21 PM — workersMax is now 0 (was 1). and the image card showed no result at all, which is the target-scoping the control promises.

Step Result
Before video workersMin 0 / workersMax 1, image 0 / 1
After the dashboard kill switch video workersMax 0, image still 1 (REST read)
POST /api/v1/jobs (wan22-t2v) failed, The RunPod endpoint is stopped (workersMax is 0) and is accepting no work. Raise workersMax in the RunPod console to resume. — one attempt, retryable false
RunPod /health for the video endpoint inQueue 0, inProgress 0, every worker count 0
Both ledgers image $0 of $3, video $0 of $6, billedSeconds 0; the only real ledger key on the box is still isis:runpod:cost:2026-09-11
Restore PATCH {"workersMax":1}; a field-by-field diff of the endpoint before and after shows 1 of 19 fields changed and gpuTypeIds intact
Drift check no drift (2 endpoints checked)

Screenshot: docs/agents/evidence/isis-chroma-runpod/2026-09-12-v1104-video-kill-switch.png.

The tracker's expectation was wrong, and the truth is better. V.11.04 said "submit, confirm IN_QUEUE then cancellation by the API timeout". RunPod does not queue against a paused endpoint at all — it refuses /run with 409 ENDPOINT_PAUSED, "Endpoint is paused (max_workers=0). Set max_workers > 0 to accept work." So there was no IN_QUEUE state to observe and nothing to cancel, and the kill switch is stronger than the item assumed: no backlog builds up behind it to stampede when the ceiling goes back up.

Two defects found on the way, both fixed here.

  1. Every motion workflow was refused before dispatch. The node-class preflight read infra/runpod/endpoints/inventory/comfyui-chroma.json unconditionally, so a video graph was validated against the IMAGE worker image. wan22-t2v failed with UNKNOWN_NODE_CLASS naming CreateVideo, EmptyHunyuanLatentVideo, ImageFromBatch, KSamplerAdvanced, ModelSamplingSD3, SaveVideo — classes the video image has had since it was built and which comfyui-video.json lists (53 core + 16 custom). resolveNodeClassInventoryFile now routes on the catalog category, the same rule as the ledger and the endpoint id, with a separate ISIS_RUNPOD_VIDEO_NODE_CLASS_INVENTORY_FILE override so the old single variable cannot capture video jobs. node-class-inventory-routing.spec.ts (7 tests) pins it, including that the video inventory really carries those six classes and the image one really does not. This blocked V.11.01 and V.11.02 too — no clip of any kind could have reached a GPU.

  2. A paused endpoint read as "Unexpected RunPod provider error." 409 fell through the status switch to UNKNOWN. There is now a RUNCOMFY_ENDPOINT_PAUSED code with a message that names the cause and the fix, classified off RunPod's own code/detail before the status switch, and non-retryable — the endpoint stays paused until a human raises it, so a retry is only another 409. runpod-adapter.endpoint-paused.spec.ts (3 tests) pins it, including that an unrelated 409 still reads as UNKNOWN.

A test-fixture landmine worth recording: a mocked fetch response without headers makes the adapter lose the HTTP status while building the error, and the retry predicate then treats a 4xx as a network blip and retries it four times with backoff. The first version of the spec timed out at 15 s for exactly this reason while the live path failed in 588 ms. Mock headers: new Headers().

Also fixed: runpod-endpoint-drift-check.mjs defaulted to --key comfyui-chroma, so from the moment the video endpoint existed a bare "drift check, no drift" was a statement about half the fleet. A bare run now checks every endpoint in the environment and prints how many. Its test file was also still pinning the pre-rebuild chroma endpoint id 6l1f3tku7z2zoy and failing; updated to 6jye7vngg4i685 and extended to cover the video entry.

V.11.01 — the first clip, live (2026-09-12)#

Rendered. wan22-t2v, 33 frames, 832x480, catalog defaults, seed 20260912, submitted through the generation API so the ledger, provenance and output persistence were all exercised.

Isis job id bd345ee6-55c9-4171-8500-a8ad76574e24
RunPod job id e87838c0-da47-45f1-ab31-69a2f205ffa7-e2 (4th attempt; the first three never got a card)
Worker b3vnt2h8obe53d, EU-RO-1
delayTime 528 347 ms (8.8 min queue + cold boot)
executionTime 375 411 ms (6.3 min)
Ledger $1.202902, billedSeconds 903.758, coldStarts 1
Clip wan22-t2v_00001_.mp4, 211 885 bytes, sha256 31e44a9c9d11552851d7075b50004c2c7c547f2c33b42f0d3a128f377160c7a1
Last frame wan22-t2v-lastframe_00001_.png, 358 467 bytes, sha256 57f5cdcef7d6dfd11d4a772073f2d48588d68f6357ea284d2da1c59779e06621
Stamped facts kind video, codec avc1, 33 frames, 16 fps, 832x480, durationMs 2063, hasAudio false
Independent probe ffprobe: h264, 832x480, 16/1, nb_frames 33, duration 2.0625, size 211885 — every stamped fact matches
Provenance bundle prov_bbcfcdf9601fd29a, consent consent_operator_isis_dev_studio, watermark { applied: true, method lsb-blue-v1, images 1 }

Files: docs/agents/evidence/isis-chroma-runpod/v1101/.

It took four attempts and three of them were free. EU-RO-1 had no card: the first attempt was throttled after twenty seconds, the second was handed a ready worker at 18:14:40Z and had it taken away at 18:15:00Z, the third expired in the queue. Each was cancelled by the C.12.17 queue-phase deadline, which reports retryable: true precisely because nothing was billed, so the executor re-submitted without a human. RunPod's billing panel read $0.00 throughout. The fourth attempt got a card at 18:26:26Z — 54 seconds before its own deadline — and ran. Adding the two A100 80 GB variants between attempts 2 and 3 did not cause this; that tier reads Unavailable too (runbook §2f).

Three numbers that are now measured rather than guessed:

  1. The clip cost $1.20, not the $0.16 the catalog sticker implied. The sticker said 120 s for 81 frames; reality is 375 s for 33 frames — about 7.7x the sticker per frame. estimated_seconds_per_megapixel_frame and cost_calibration are V.11.05's to rewrite, and proof_level for wan22-t2v can now go to rendered.

  2. The ledger was over-stating spend by 4x, and the account balance is what caught it. The ledger booked $1.2029; the RunPod balance moved about $0.29. That is not rounding, it is a wrong rule: billedSecondsFor added the whole of delayTime whenever coldStart was set, on the assumption that the delay before a cold start is the worker booting. Here delayTime was 528 s of queue wait for a card — the job made four attempts before one was placed — and queue wait is not worker time and is never billed. $0.29 over 375.4 s of execution is ≈ $0.00077/s, which is the A100 80 GB tier's $2.72/h, so the clip ran on one of the cards added an hour earlier and was charged for its execution and nothing else.

    Over-stating is not the safe direction even though it looks like it: a ledger that invents spend refuses submits the budget could afford — at $1.20 a clip the $6 video budget would have refused V.11.02 partway through on spend that never happened — and it makes the daily figure useless as a record of what was charged. billedSecondsFor now bills execution only; the wait is recorded as unbilledDelaySeconds on the snapshot so a clip that took a quarter of an hour and was charged for six minutes still explains itself.

    What this does not yet separate is the boot, which is inside delayTime and probably is billed. It is small next to a nine-minute queue wait, and V.11.02's warm session is the measurement that isolates it: warm clips have delayTime near zero, so the gap between the session's total charge and the sum of its execution times is the boot. Recorded for V.11.05.

    (Separately, the ledger used the pre-correction rate 0.001331 because the API process predated that change. The rate correction stands for an H200; this job did not run on one.)

  3. The dashboard was calling an estimate a cost. The output file carries costUsd stamped from metadata.estimatedCostUsd — $0.5226 for this clip — and the Outputs badge rendered it as cost, against the ledger's $1.2029 for the same render. Two different quantities, one of them labelled as if it were the other. The badge now reads est. cost, the Jobs line prefixes ~, and the client type says which is which.

V.09.07 — the clip plays in a real browser (2026-09-12, $0, no new generation)#

Chromium, against the live API, on the V.11.01 job. The unit specs assert the markup and jsdom has no media pipeline, so the only thing that can tell a working player from a <video> pointed at a 404 is a browser that decodes the bytes.

Check Result
<video> on the Jobs page present, controls, playsInline
readyState ≥ 2 (HAVE_CURRENT_DATA) — the decoder produced a frame
Decoded dimensions videoWidth 832, videoHeight 480, duration 2.0625 — matching ffprobe and the stamped facts exactly
Playback currentTime advanced 0 → 1.2023 s under play(); it is not a still
Card text seed 20260912 · 2.06s · 33f · 16fps · 832×480 · silent · ~$0.5226 · 375.4s · 528.3s cold start · 2 files

Screenshot: docs/agents/evidence/isis-chroma-runpod/2026-09-12-v0907-clip-card.png.

The card also confirms the estimate-labelling fix is live: the figure renders as ~$0.5226, not as a bare cost.

A finding about the Outputs page, not the clip: /outputs renders no video and its network log shows ERR_CONNECTION_REFUSED against http://localhost:3002/api/v1/outputs and /api/v1/policies. That page is backed by a separate outputs service on :3002 which the dev-stack recipe does not start, so it is empty by configuration rather than broken. The clip card that V.09.07 is about lives on /jobs, which reads the generation API directly. Worth knowing before anyone debugs an empty Outputs page as a persistence failure.

C.09.08 — the first Chroma image in a real browser (2026-09-12, $0)#

Chromium against the live API, on the C.11.01 job 4bae0703-cb80-4a61-be70-3519d057504a.

Check Result
Job row text-to-image, completed, 100%, queued 1.4s / cold start
Thumbnail present, and the browser decoded it: naturalWidth 1024, naturalHeight 1024, from a data:image/png;base64 source
Subject the red bicycle of C.11.01's prompt, so it is that render and not a placeholder

Screenshot: docs/agents/evidence/isis-chroma-runpod/2026-09-12-c0908-chroma-image-card.png. Vite killed afterwards; lsof on 5180 empty.

The item's premise about the Outputs page is wrong, for a structural reason. C.09.08 and V.09.07 both say "the Outputs card", and /outputs will never show either render: it is backed by a separate @isis/output-registry service on :3002, and the generation API never writes to it — there is no call to it anywhere in apps/isis/generation-api/src, and persistJobOutputs writes to its own store. Starting :3002 would not help; it holds a different corpus. The card for anything the RunPod path produced is on /jobs, which reads the generation API directly, and that is where both passes were done.

V.11.02 — the warm-chained matrix, 4 of 6 (2026-09-12)#

One invocation of scripts/isis/motion-live-proof.mjs, all stages submitted up front so they queue behind each other and one worker takes them back to back. The warm chain worked: one cold start for the whole session — stage 1 waited 1 093 417 ms for a card and every stage after it started immediately.

Stage Workflow Status Execution Output
1 wan22-i2v COMPLETED 409.7 s 1-wan22-i2v-0-fdb6408f.mp4 196 548 B, h264 832×480 16 fps 33 f, + last-frame PNG 382 682 B
2a gimm-vfi-interpolate COMPLETED 33.5 s 2a-…-987aca7d.mp4 215 191 B, 65 f at 32 fps — the interpolator really doubled 33→65 and 16→32
3 birefnet-video-matte COMPLETED 27.3 s 3-…-3679e250.webm 85 103 B (vp9, alpha_mode=1) + mask 3-…-2c33ed6a.mp4 46 489 B
4 sam3-track-then-vace COMPLETED 343.2 s 4-…-266b2ab6.mp4 179 583 B, h264 832×480 16 fps 33 f
5 wan22-animate FAILED 100.5 s WebSocket communication error: ComfyUI HTTP unreachable during websocket reconnect
6 ltx23-av FAILED 25.9 s ComfyUI server (127.0.0.1:8188) not reachable after multiple retries.
2b flashvsr-upscale FAILED 0.4 s same, after 391 ms

Cost, execution only (queue wait is unbilled — V.11.01): $1.549 at the H200 rate the ledger is configured for, or $0.711 if the card was the A100 80 GB tier, which V.11.01's billing evidence suggests it is. V.11.01's clip plus this session is therefore roughly $1.00–1.83 against the $1.50–2.60 approved.

Stage 5 killed ComfyUI at the SMALLEST legal shape, and no cap could have stopped it. It ran 33 frames at 832×480, 20 steps, batch 1, on a 33-frame driving clip — the floor of what MOTION_CAPS permits — and every cap passed. There is therefore no tighter number to set. What the caps bound is the job SHAPE; what died was the model FOOTPRINT (Animate's 14B weights beside clip_vision_h and the ONNX pose and face models, resident together), and the table has no expression for that. The cause is not established: the worker log could not be retrieved, because the fetch for it has no timeout and hung the run for fifty minutes — its own fix, now 20 s. "It died" is the whole of what is known, and it is not recorded as an OOM without the log that would say so.

wan22-animate now carries a known_failure note in its catalog entry, and the runner gives it last: true so it is submitted only after every other stage has settled. Nothing queues behind a stage that can take the worker down — which is exactly how FlashVSR lost its only chance to run.

Stages 6 and 2b are collateral, not independent failures. The signature — HTTP unreachable during websocket reconnect — is the same one C.11.04 produced when two 896×1152 latents would not fit 24 GB: the process died rather than raising. wan22-animate loads the Animate 14B model beside clip_vision_h and the ONNX pose/face models, and it went down 100 s in. The two stages behind it in the queue then hit a server that no longer existed, one of them in 391 ms. They never ran and must be re-run once, not counted as proof of anything. Stage 2b in particular has still never exercised the FlashVSR pack.

Stage 3's alpha is proved from the MASK, not the webm, and the distinction matters. The tracker asks for a decoded frame asserting real alpha rather than a black matte. The webm carries alpha_mode=1 at the container level, but decoding it to RGBA yields 399 360 samples all at 255 — one distinct value. That is far more likely to be ffmpeg not decoding VP9's alpha side-channel than an empty matte, so it is recorded as inconclusive as performed rather than as a pass or a failure. The graph's second output settles it: the mask video's first frame has 220 distinct grey levels, min 0 max 255, 85.6 % fully black and 14.4 % soft edge. A black matte would have exactly one level. It is a real matte.

V.11.02 stays [ ] — the item is marked only when all six rows exist, and three are open.

The re-run of the two that never ran (2026-09-12, 20:47Z, $0.172)#

flashvsr-upscale and ltx23-av had failed against a server that no longer existed, so they were owed a FIRST attempt, not a re-run. wan22-animate was deliberately excluded: it had had its attempt, and re-running a known worker-killer buys nothing its known_failure note does not already say.

Stage Status Execution USD Detail
2a gimm-vfi-interpolate COMPLETED (again, to regenerate 2b's input) 22.7 s $0.0374 2a-…-bdba2417.mp4 215 740 B
6 ltx23-av FAILED — and this one was OUR bug 0.3 s $0.0005 Node 8 (LTXVImgToVideoInplace): required_input_missing 'image'
2b flashvsr-upscale FAILED 81.3 s $0.1339 Job processing failed — no detail, worker log fetch timed out

ltx23-av is fixed and is owed a re-run. The graph carried LTXVImgToVideoInplace in every mode with bypass: {{mode == 't2v'}}, on the reasoning that the node knows how to do nothing. ComfyUI validates required inputs BEFORE it honours bypass, so in t2v the whole prompt was rejected in 324 ms. optional_image_input gained passthrough_input, which removes the node and repoints its consumer at the empty latent; t2v renders 19 nodes instead of 20 and LTXVConcatAVLatent.video_latent moves from ["8",0] to ["7",0]. The offline validator V.01.09 describes would have caught this for $0 — that item now has a concrete argument behind it rather than a theoretical one.

flashvsr-upscale remains completely unproven and I cannot say why. It ran 81 seconds and returned a bare Job processing failed, with no output.errors[] and no retrievable worker log (§6c: the log dies with the worker). It is the only pack in the matrix that has never executed successfully.

What has been ruled out, offline and for $0. Both flashvsr-upscale (5 nodes) and wan22-animate (20 nodes) were re-rendered and audited for the class of defect that killed ltx23-av: dangling links, null or empty inputs, and unresolved {{…}} templates. Both are clean on all four. So neither is a graph-shape bug, and the "just render it and look" approach that fixed ltx23-av has nothing left to find here. That narrows both to runtime inside the container — a pack import, a model load, or memory — which is precisely what cannot be seen from outside without either the worker log (a race we lose) or the object-info snapshot V.06.06 needs. "I don't know why" is the accurate answer, and it now has a boundary around it rather than being an absence of effort.

One loose thread, recorded rather than explained: the second gimm-vfi run produced sha256 bdba2417… where the first produced 987aca7d…, from the same input clip and the same seed. Most likely encoder non-determinism, but that is a guess and has not been established.

C.11.08 / C.11.09 / C.11.04 — SAM 3 works, and the cap holds (2026-09-12)#

C.11.08 and C.11.09 were marked BLOCKED and were not. The block was Node 'SAM3Segment' not found — the pack's py/AILab_SAM3Segment.py imports sam3 at module level and the auto-loader swallowed the ImportError. C.12.18 landed that runtime and the rebuilt image's inventory lists SAM3Segment. The blocker note had gone stale; nothing needed fixing, only running.

Both ran on the image endpoint, execution-only cost at $0.000439/s:

Item Workflow Execution USD Output
C.11.08 sam3-segment, prompt the red bicycle 21.2 s $0.0093 RGBA cut-out c1108-0-4a786341.png 1 391 235 B + mask c1108-1-7ab8f633.png 6 260 B
C.11.09 sam3-segment-inpaint, same prompt, a vintage green bicycle… 207.2 s $0.0909 mask c1109-0-7ab8f633.png + result c1109-1-fe8ba740.png 1 192 190 B

The mask covers the object, checked by decoding it rather than by trusting it. Decoded with the repo's own PNG reader: 1024×1024, 3.9 % white / 96.1 % black, 2 distinct values — a hard-edged binary mask — and rendered it is an unmistakable bicycle silhouette: both wheels, frame, saddle, handlebars and rear rack. The 3.9 % coverage is low only because a bicycle is mostly thin tubes and air. The RGBA cut-out's alpha agrees exactly (3.9 % opaque, 96.1 % transparent).

C.11.09's mask is byte-identical to C.11.08's — same sha 7ab8f633, same 6 260 B — which is the determinism you want from the same prompt on the same image. Its result image decodes to 256 distinct levels across the frame (a real photograph, not a flat fill) and shows the red bicycle replaced by a green one with the wall, pavement, sky and lighting preserved. The wall shadow is still the ORIGINAL bicycle's, which is correct: inpaint only repaints the masked region.

C.11.04's cap now refuses its own killer, for $0. The 2026-09-11 run of batch_size 2 at 896×1152 killed ComfyUI outright. C.12.19 made the batch cap scale with resolution. Re-submitting the same combination through the live API now returns batch_size must be <= 1 and never reaches a GPU — the log has no dispatch line for it. The item asked whether the per-job caps and the memory headroom hold; the headroom does not and the cap now enforces that.

D-V2 — the image endpoint's cold start is not measurable while the region is starved#

V.11.01 asks for the image endpoint's cold start after the rebuild, for D-V2. Measured: delayMs 1 314 570, execMs 82 014.

That delay is not a cold start and must not be reported as one. RunPod's delayTime is wall time from submission, and 21.9 minutes of it was waiting for a card in a region with no capacity. C.11.01's pre-rebuild figure — 53 334 ms — was a genuine boot measured when capacity existed. Comparing 1 314 570 against 53 334 would claim a 24× cold-start regression that the numbers do not support. D-V2 stays unanswered, and the reason is recorded rather than papered over: the metric it depends on is not observable while delayTime is dominated by queue wait. A warm-session measurement isolates it (V.11.02's note).

What IS comparable is execution on the identical job — same workflow, checkpoint, seed and size: 150 127 ms before the rebuild, 82 014 ms after. The render got faster, not slower.

The ledger correction is visible here too: the image ledger recorded usd 0.036004, billedSeconds 82.014, unbilledDelaySeconds 1314.57 — the execution billed, the wait reported and not charged.

Region probe — US-CA-2 places a card in under a minute (2026-09-12)#

The question EU-RO-1 has been raising all day: is the starvation regional, or is it our card list? Answered by building the smallest thing that can tell the difference — supply labels only become truthful once a volume pins the datacenter (§2f), so a probe has to own a volume.

Step Result
10 GB network volume mxugcfkswl in US-CA-2 created
Throwaway endpoint bbedw1lqlb7a9u on it, the identical nine cards as the video endpoint created
Worker placement initializing within a minute, then ready/idle — with ZERO jobs queued
EU-RO-1 at the same instant, same nine cards throttled, 2 jobs actually waiting

US-CA-2 handed out a card for nothing; EU-RO-1 would not hand one out for work. That is the comparison that matters, and it says the constraint is the region rather than the selection — the same nine card ids were on both.

What the probe does NOT show. It ran a tiny runpod/base image, so it measures card availability, not boot time; a 288 GB volume and a multi-GB image would still have to be pulled in US-CA-2, and nothing here says how long that takes. It is also one reading at one moment, which is exactly the caveat §2f already records about supply labels.

It cost more than I said it would, and the reason is worth keeping. I quoted "well under $0.05". Creating a serverless endpoint through the REST API defaults workersStandby to 3 and workersMax to 3, and a standby worker starts immediately with no job submitted — so the probe began billing the moment it existed. PATCH {"workersMax":0} did not stop it, because standby is a separate field and the patch that tried to set both returned nulls. Deleting the endpoint was what stopped it. Call it three to four minutes of one worker on an unknown card: $0.17–0.36, not under $0.05.

Both our real endpoints carry workersStandby: 1. They are not billing for it today only because EU-RO-1 cannot place a worker at all — the starvation is suppressing a standing cost rather than the configuration being safe. If supply returns, that setting is worth re-checking before it is discovered on an invoice.

Probe torn down completely: endpoint, volume and template all deleted, verified by listing both — only ax0oj250j3 (the real 500 GB EU-RO-1 volume) remains.

C.11 Live generations#

One row per run. cold = worker had to start; warm = worker already up.

Date Item Workflow Checkpoint Job id RunPod job id GPU State delayTime ms executionTime ms USD Output sha256 Thumbnail
2026-09-11 16:24Z C.11.01 attempt 1 chroma-txt2img chroma1-hd add2adbe-967f (slim image, release v3) RTX 4090 worker never started its container FAILED: RunPod error creating container … exit status 1; Isis timed out at 300 s and cancelled server-side; queue purged — — 0 (nothing billed) — —
2026-09-11 17:09Z C.11.01 attempt 2 chroma-txt2img chroma1-hd 1a90ccfd-9a4f prompt cb66c5bf (stock worker + injected extra_model_paths) RTX PRO 4500 Blackwell (EU-RO-1) RENDERED (Prompt executed in 150.89 s) but Isis refused completion: without provenance consentId — image lost, cost booked 0 154379 0.04724 — —
2026-09-11 17:23Z C.11.01 cold chroma-txt2img chroma1-hd 4bae0703-cb80 prompt 623efed1 RTX PRO 4500 Blackwell (EU-RO-1) completed; provenance bundle prov_6e11f360e7f5f906 (modelVersion 0e0c60ec…, watermarkPolicy unwatermarked) 53334 (cold) 150127 0.08305 0d755be254b495a6… docs/agents/evidence/isis-chroma-runpod/2026-09-11-chroma-txt2img-chroma1-hd-seed42.png
2026-09-11 17:29Z C.11.01 second run, attempt 1 chroma-txt2img chroma1-hd (seed 7) 7d4280f9-e808 prompt 976fcda2 RTX PRO 4500 Blackwell RENDERED (Prompt executed in 148.98 s, RunPod completed=3) but LOST: the API restarted at 17:31:28Z (tsx watch reload from the commit hook) mid-poll; job left running, cancelled by hand; tracker C.12.16 — 148980 (RunPod) ~0.046 billed, not in ledger — —
2026-09-11 17:34Z C.11.01 second run, attempt 2 chroma-txt2img chroma1-hd (seed 7) d1e87d06-a22a first RunPod job cancelled at the 300 s cap (fresh host pulled the 14.7 GB image ~4 min, container stopped 17:39:28Z); retry completed RTX PRO 4500 Blackwell completed; bundle prov_c593445f9478d045; sizeBytes now 1474325 (fix verified) 53750 (cold again: host churn) 148165 0.08195 (+ the cancelled attempt's GPU seconds, not in ledger — C.12.17) 0fdba9a4aa8b59d9… docs/agents/evidence/isis-chroma-runpod/2026-09-11-chroma-txt2img-chroma1-hd-seed7.png
2026-09-11 17:43Z C.11.01 warm chroma-txt2img chroma1-hd (seed 11) 43cca039-348e submitted 20 s after the previous completion RTX PRO 4500 Blackwell completed; bundle prov_3f9980f99f21bab8 9447 (warm) 152871 0.08415 0a41b8eb6dff32da… docs/agents/evidence/isis-chroma-runpod/2026-09-11-chroma-txt2img-chroma1-hd-seed11-warm.png

Session rest-state checks (C.11.13)#

Date Endpoint workers Pods currentSpendPerHr Ledger USD (day) RunPod billing USD (day) Gap explanation

Pre-existing test failures on the branch (not caused by this initiative)#

Bisected 2026-09-11 by reverting every file this initiative touched (executor, comfyui.service, shared/config, ai-providers comfy adapter + types, workflows registry/schema/taxonomy, the chroma catalog folder and the new modules) to commit 480f69fe8b and re-running the executor spec: the same 11 tests fail before and after. They are therefore out of scope here and left as found:

Spec Tests Symptom
apps/isis/generation-api/src/workflows/executor.spec.ts certifies guided-journey / lecture-scene / mandala-animated / meditation-scene; applies voice consistency profiles; executes sequence generation; maps canonical ControlNet major modes result.success false with the deterministic test ComfyUI service (video/audio/control families)
same does not resolve generation types that still have no dedicated catalog workflows; tracks resolve_success/resolve_failure; returns explicit unresolved workflow error; fails closed with production guard the spec assumes inpainting has no catalog workflow, but composition/inpaint-edit (2026-08-02) routes it

The chroma catalog is gated out of the resolver unless the worker-comfyui contract is active, so it does not change any of those outcomes.

Compiled JS beside TypeScript sources (repo hazard, 2026-09-11)#

libs/shared/config/src/{env,features,index,loader,schemas}.js (+ .d.ts) are git-tracked build outputs. Vite/vitest resolves .js before .ts for extension-less relative imports, so an edit to features.ts is invisible at test/runtime until the .js is regenerated. This initiative regenerated features.js/index.js (+ .d.ts) with the library's own tsc after adding getRunPodWorkerKind. 112 such shadow files exist under libs/isis and apps/isis src/ trees; not cleaned up here.

Pre-existing defect fixed on the way (2026-09-11)#

ComfyUIService.cancelJob(isisJobId) forwarded the Isis job id to RunPodAdapter.cancelJob, whose supervision maps are keyed by the RunPod job id, so a cancel never aborted the poll loop and hit /cancel/<isis id> on RunPod. The adapter now records metadata.jobId → RunPod id at submit and resolves either id (runpod-adapter.cancel-mapping.spec.ts); the cancel route additionally aborts the in-flight executor run (jobs-cancel-propagation.route.spec.ts). Without this, a cancelled job kept burning billed GPU seconds until the 300 s cap.

C.09.07 dashboard flow in real chromium (2026-09-11 15:20Z)#

apps/isis/web/e2e/chroma-runpod-dashboard.spec.ts, Playwright 1.60 chromium, one worker, Vite started by Playwright and stopped with it (port 5180 empty afterwards), generation API mocked at the HTTP boundary: Chroma preset applied in the Studio (type + Extra Parameters JSON + steps field), submit payload carried workflowId/checkpoint/aspect/steps 26/cfg 3.8/batch_size 1, the completed job's output thumbnail painted (naturalWidth > 0) with caption seed 42 · chroma1-hd · $0.0107 · 34.2s · cold start, and the RunPod page budget card showed $0.0107 of $3.00. 1 passed (2.9 s). Two defects found by the run and fixed: the Studio's own steps default overrode the preset's catalog value; the budget card was gated behind the operations snapshot.


Video extension evidence (V.01–V.11)#

Same conventions as above. A motion workflow's row must say which proof level it reached: rendered (a real clip came back from a live worker) or validated (golden fixture + static validation against the object-info snapshot + ComfyUI's own /prompt validator in CI, and no live run). Only the eight clips listed in tracker V.11 are rendered; a workflow with no row here is not proved at all.

Every clip row records: job id, endpoint, GPU assigned, delayTime / executionTime in ms, USD at the video rate, and for the artefact itself sha256, duration, fps, resolution, and whether it carries audio.

V.01 Worker image: video node packs#

No rows yet — the video packs are not in the image. Rows here record the build id, the resolved SHA of every pack from /opt/oshun/build-manifest/<pack>.sha, the smoke test's class count, and the image digest both endpoints pin.

V.01.01 — video packs in the image (2026-09-12, pending a build)#

Six packs, each SHA-pinned, cloned by install-video-packs.sh rather than an inline RUN. The reason is specific: Docker expands $1 itself in a shell-form RUN, so a loop using positional parameters would have had them silently replaced with empty strings by an undefined build arg. A COPYed script also gets bash -n before a build is spent on it, which is how the first draft of this was caught.

The script was then run for real against the live repos on this machine, not just syntax-checked. Five of six cloned and every resolved SHA matched its pin (the script exits non-zero when it does not):

text
ComfyUI-VideoHelperSuite.sha    4d907bee61e92c2e65af3bd6383a4e4d356126d1
comfyui_controlnet_aux.sha      59b1fc411ede8623b2997855b8018f0b3b6cf49f
ComfyUI-WanAnimatePreprocess.sha 0e0b6a2a555625acf4d4aefb780e27d06937132f
ComfyUI-GIMM-VFI.sha            4c9a3123762af85e7c796e41737da0b70c75d72d
ComfyUI-FlashVSR_Ultra_Fast.sha 4820b3f02347bddcbbb9a5a85ab7638fe976366e

The sixth (ComfyUI-SAM3) was interrupted twice by the shell timeout, so its clone is unproven locally; its bypass premise is proven from source instead, which is the part that could actually be wrong. nodes/__init__.py at de0ff5d2 assigns both NODE_CLASS_MAPPINGS and NODE_DISPLAY_NAME_MAPPINGS at module level (parsed with ast, not grepped) and does not import comfy_env — so replacing the root __init__.py with a direct re-export is sound, and the build will confirm the clone.

requirements-video.txt is a reconciled list, not a concatenation of the packs' own files, and says why for every omission. The load-bearing one: four packs ask for opencv-python, whose GUI build links libGL that a runtime image does not carry, so the image installs opencv-python-headless once instead. Also corrected — GIMM-VFI's requirement is cupy-cuda12x>=13.3.0; the tracker wrote cupy-cuda12, which is not the PyPI name for the CUDA 12 wheel, so the pin is cupy-cuda12x==13.3.0. The [cuda] extras ComfyUI-SAM3 lists (cc_torch, torch_generic_nms, flash-attn) are deliberately absent: the vendored SAM 3 source imports all three under try/except, and flash-attn compiles from source with no compiler in the base image. ffmpeg is installed as an apt package because SaveVideo, CreateVideo and VHS_VideoCombine shell out to the binary.

REQUIRED_CLASSES (V.01.05) is now 61 classes, no duplicates: the 24 image ones plus 21 core video nodes and 16 pack keys. Every pack key was read from that pack's own NODE_CLASS_MAPPINGS rather than guessed from a class name — VHS_LoadVideo is the class LoadVideoUpload, and they differ.

Not yet proved: that the image builds with all this in it. That is the next build, and with the smoke fatal a missing class fails it.

V.05.02 — video/audio input transport, and a latent stack overflow it exposed (2026-09-12)#

liftInlineMedia mirrors liftInlineImages for video and audio inputs, but sniffs decoded bytes rather than a base64 prefix. It has to: an MP4/MOV is identified by ftyp at byte offset 4, which lands mid-quantum in base64 and has no stable text prefix, so the image sniffer's approach cannot be reused. Covered: mp4, mov (brand qt), webm and mkv (same EBML header, separated by DocType), wav, mp3 (ID3 tag or raw frame sync) and flac. The declared mime is never trusted — a clip labelled video/webm that is really an mp4 is served as .mp4, because worker-comfyui writes the bytes under the name we give and ComfyUI picks its demuxer from that name.

Four refusals, all before a GPU second: unrecognised bytes; audio supplied to a video input (and the reverse); a mime outside the workflow's accept list; and anything over the inline ceiling, which names the V.05.03 volume path rather than quietly posting a 50 MB body.

The bug this turned up is worth more than the feature. The existing RAW_BASE64_RE = /^[A-Za-z0-9+/=\s]{64,}$/ throws RangeError: Maximum call stack size exceeded above about 8 MB of input — measured on this Node: 1 MB fine, 4 MB fine, 8 MB fatal. That was survivable while inline payloads were only stills; a video input makes it certain. And a stack overflow is a much worse failure than a rejected submit: it is not a WorkerComfyUiSubmissionError, so nothing upstream can report it usefully. It is replaced by a bounded head match ({64} on the first 64 characters), which is O(64) at any length and loses nothing — the decisive check is the magic-byte sniff that follows, and no filename is 64 characters of base64 alphabet. The image path shares the fix.

The test fixtures are built from real byte headers rather than pasted base64, and padded to 256 bytes, which is itself a finding: a 32-byte "video" falls under the 64-character floor that separates a payload from a filename. That is a property of an unrealistic fixture, not of any real clip, and the comment says so.

16 specs pass; the src/workflows/ baseline is unchanged at 13 failed / 143 passed — the same 13 pre-existing executor.spec.ts failures.

V.05.01 — the payload-limit probe is INCONCLUSIVE, and here is why not to repeat it#

The endpoint sits at workersMax: 0 (the kill switch from an earlier session, never lifted), so POST /v2/<ep>/run answers:

text
409 {"detail":"Endpoint is paused (max_workers=0)...","code":"ENDPOINT_PAUSED"}

That looked like a free way to measure the limit: a 409 proves no job was created, so no GPU seconds can be spent. Bodies of 10, 20, 30 and 50 MiB all came back 409, which suggested nothing rejects them on size.

Then a 120 MiB body also returned 409 — and that settles it the other way. No service accepts a 120 MiB /run body, so the pause check runs before any size check, and every one of those probes was measuring the pause, not the limit. The right conclusion from the experiment is that the experiment does not work.

Against a nonexistent endpoint id, bodies up to 20 MiB return 404 endpoint not found rather than 413, so the size check is not at the edge either — it is behind endpoint resolution and behind the pause check.

So V.05.01 stays [ ]. Measuring it needs workersMax ≥ 1, and then an accepted oversized body would queue a real job on a real GPU. That is a small spend, not a free one, and it needs the user's go.

The inline threshold is therefore set conservatively rather than measured: ISIS_RUNPOD_VIDEO_INPUT_INLINE_MAX_BYTES defaults to 6 MiB of base64, under RunPod's documented 10 MB for /run, with the volume's S3 path taking everything above it. Being wrong in this direction costs an unnecessary S3 round trip; being wrong the other way costs a rejected job after the upload.

Build 95f5cc15 COMPLETED — the video packs are in the image (2026-09-12 02:24:43Z)#

Commit a5ac709682, the symlink fix. The smoke test's own last line:

text
[smoke] ok — 64 required classes present, 1071 classes total,
        object_info at /opt/oshun/build-manifest/object_info.json

All 64 required classes present, and the smoke is fatal, so that sentence is evidence rather than an assertion. It covers the 24 image classes plus the 21 core video nodes (Wan T2V/I2V/Animate/VACE, the LTX-2.3 audio-video chain, LoadVideo/CreateVideo/SaveVideo/SaveWEBM) and 16 pack keys. SAM3Segment is among them, which re-proves C.12.18 on a second, much larger image.

The pack layer's own output, with every resolved SHA asserted against its pin:

text
[packs] Kosinkadink/ComfyUI-VideoHelperSuite@4d907bee61e92c2e65af3bd6383a4e4d356126d1
[packs] Fannovel16/comfyui_controlnet_aux@59b1fc411ede8623b2997855b8018f0b3b6cf49f
[packs] kijai/ComfyUI-WanAnimatePreprocess@0e0b6a2a555625acf4d4aefb780e27d06937132f
[packs] kijai/ComfyUI-GIMM-VFI@4c9a3123762af85e7c796e41737da0b70c75d72d
[packs] lihaoyun6/ComfyUI-FlashVSR_Ultra_Fast@4820b3f02347bddcbbb9a5a85ab7638fe976366e
[packs] PozzettiAndrea/ComfyUI-SAM3@de0ff5d2c2ea435d29f800abfa568cffdfb94773
[packs] ComfyUI-SAM3: comfy_env bypassed, install.py removed
[packs] ok — 6 packs pinned

So V.01.02's bypass works: replacing the root __init__.py with a direct re-export and deleting install.py is enough, and SAM3VideoSegmentation, SAM3Propagate, SAM3VideoOutput and LoadSAM3Model are all registered. The dependency layer also passed both checks — video deps import ok and gpu-only + optional deps installed ok — confirming that find_spec is the right way to verify cupy where import is not.

Release recorded at 02:24:43Z, image tag registry.runpod.net/greychimp-oshun-runpod-comfyui-slim-docker-runpod-comfyui-slim-dockerfile:a5ac70968. Not yet recorded: the image digest and its compressed size (V.01.06 wants both, plus the object_info.json stored as infra/runpod/endpoints/node-classes/worker-comfyui-5.10.0-video.json). That file exists inside the image; extracting it needs a worker boot, which costs GPU seconds, so V.01.06 stays [ ].

V.10.02 — and a cost check that had been silently broken (2026-09-12)#

Adding the video preflight checks surfaced a pre-existing failure, confirmed by running the suite against the previous revision: 2 of 5 tests were already red, including "a coherent dev environment passes every check".

The cause: GPU_USD_PER_SECOND is keyed by GPU type names (NVIDIA GeForce RTX 4090), but desired-state.json carries GPU pool ids (ADA_24,AMPERE_24,BLACKWELL_32) since the endpoint was re-saved through the console. Every pool id missed the table, so the rate assertion reported "no pricing row" and failed. The check that exists to stop the ledger billing at the wrong rate had not worked since the pool migration.

Two changes, both of which outlive the immediate fix:

  • Pool ids are in the table now, and ADA_24 carries the 4090 flex rate every C.11 measurement was computed with. AMPERE_24, BLACKWELL_32, ADA_32_PRO and AMPERE_80 are listed as UNPRICED rather than guessed — RunPod's GraphQL exposes pod prices (securePrice, communityPrice), not serverless flex rates, so any number there would be invented. The check names them in its detail line instead of rounding the gap away.
  • The rule changed from "matches the FIRST entry within 2%" to "covers the PRICIEST entry". A gpuIds list means "any of these", so billing at the first entry's rate under-bills whenever a pricier one is assigned — and under-billing is precisely the direction that stops a budget firing, because the ledger believes less has been spent than has been.

The video checks are skipped entirely when no video endpoint is configured and all bind once the id is set. That is deliberate: the image stack has to stay usable alone, and a preflight that fails over an optional endpoint teaches people to ignore it. A half-configured video endpoint is the dangerous state, not an unconfigured one. A test asserts the dangerous mistake specifically — setting the video rate to the image rate, which would under-count H100 seconds by ~4.3x. S3 credentials are checked by length and a test asserts the value never appears in the output.

6 of 6 preflight tests pass.

The video-pack builds failed twice — root cause, and how to read a build log (2026-09-12)#

Two failures, ce9ea78c (449 s) and 234449b9 (330 s). The cause is a dangling symlink, and it is worth knowing exactly why.

text
[WARNING] Cannot import /comfyui/custom_nodes/ComfyUI-SAM3 module for custom nodes:
          [Errno 17] File exists: '/comfyui/models/sam3'
FileExistsError: [Errno 17] File exists: '/comfyui/models/sam3'
[INFO] 0.0 seconds (IMPORT FAILED): /comfyui/custom_nodes/ComfyUI-SAM3
[smoke] custom node import failures:

ComfyUI-SAM3's nodes/load_model.py:22 runs os.makedirs(<models_dir>/sam3, exist_ok=True) at import. entry.sh has already created that path as a symlink to the network volume, and in the builder there is no volume, so the link dangles. And here is the part worth remembering:

exist_ok=True does not save you from a dangling symlink. It suppresses the error only when the path is an existing directory. A broken link is os.path.exists() == False — so a guard like if not os.path.exists(p) passes — and mkdir then fails anyway with EEXIST.

Reproduced locally in four lines before changing anything, and re-run after: with a real target the same makedirs(exist_ok=True) succeeds.

The smoke test's own comment asserted the opposite — "the volume is absent, so the links dangle — startup must not care". That was an assumption, it was wrong, and a pack's entire import was the cost. The smoke now stands up a throwaway volume root (/tmp/smoke-volume with models/{RMBG,sam3,FlashVSR-v1.1} and inputs/) and points entry.sh at it, so the links resolve to real empty directories — the runtime shape — instead of an artefact of the builder.

A correction to my own earlier entry here. After the first failure I wrote that import cupy on a GPU-less builder was the likely cause and changed it. That change is still right on its own terms — importing cupy without a CUDA driver does raise — but it was not the failure: the log shows the pip layer completing and the build dying in the smoke test. I had listed four candidates and the real cause was none of them. The lesson is the one C.12.12 already taught and I half-repeated: do not change code against a build you have not read.

How to read a RunPod build log without the console UI — this is what made the diagnosis possible and it is not documented anywhere:

  1. The builds API (GET api.runpod.io/v2/serverless/<ep>/builds) returns only a generic docker buildx build ... exit status 1. There is no /logs route on it (404 on every shape tried).

  2. The console fetches logs with a GraphQL operation. Capture its exact shape by patching window.fetch in the page and expanding a build row:

    graphql
    query getBuildLogs($input: GetBuildLogsInput) {
      getBuildLogs(input: $input) {
        buildId
        timestamp
        message
        level
      }
    }
    

    with variables.input = { buildId, startTime }.

  3. POST https://api.runpod.io/graphql with the normal API key then returns the whole log as JSON — 501 lines here — newest first, so read the tail for the earliest output.

Also worth noting: the console page renders fine while screenshots of it come back blank. document.body.innerText had 11,983 characters when the screenshot was white. If a console screenshot looks empty, read the DOM before concluding the page is broken.

V.01.02 / V.01.05 — node class names and the ComfyUI-SAM3 installer, read from source (2026-09-12)#

All from the pinned SHAs, no guessing. Two of these correct the tracker.

The FlashVSR class name V.01.05 says "was not captured today" is FlashVSRNode, with FlashVSRNodeAdv and FlashVSRInitPipe beside it (nodes.py:543, pack @4820b3f0).

ComfyUI-SAM3 @de0ff5d2 has no prestartup_script.py. The tracker's V.01.02 assumes one; the tree at that SHA is README.md, __init__.py, comfy-env-root.toml, comfy-test.toml, install.py and nodes/. There is no requirements.txt either. The comfy-env coupling is two lines:

python
# install.py
from comfy_env import install
install()

# __init__.py
from comfy_env import register_nodes
NODE_CLASS_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS = register_nodes()

So the bypass is not an env var and not a deleted file: nodes/__init__.py already exports NODE_CLASS_MAPPINGS itself, assembled from four modules, so the image replaces the root __init__.py with a direct from .nodes import NODE_CLASS_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS and installs the dependencies with pip. What those dependencies are is not a guess either — nodes/comfy-env.toml lists them:

toml
[cuda]
packages = ["cc_torch", "torch_generic_nms", "flash-attn"]

[pypi-dependencies]
safetensors, timm, ftfy, regex, iopath, huggingface_hub, einops,
opencv-python, psutil, pycocotools, scikit-image, numpy, pillow, tqdm,
typing_extensions, comfy-aimdo, comfy-kitchen, hf_xet, requests, transformers

Two things worth naming there. First, pycocotools appears in this pack's own dependency list for SAM 3 — an independent confirmation of the C.12.18 diagnosis, from a different author's file, reached without seeing it. Second, the [cuda] three (cc_torch, torch_generic_nms, flash-attn) are the packages the vendored SAM 3 source imports inside try/except blocks, so they are optional accelerators, not requirements — which matters because flash-attn compiles and the runtime base image has no compiler.

comfy-env-root.toml also declares a second node pack as a peer, PozzettiAndrea/ComfyUI-Multiband, which is not in V.01.01's list. Whether any motion graph needs it is a question for V.08.05; it is recorded here so the omission is deliberate rather than overlooked.

Registered node keys, verified from each pack's NODE_CLASS_MAPPINGS (the key is what a graph's class_type must say, and it is not always the class name — VHS_LoadVideo is the class LoadVideoUpload):

Pack Keys
ComfyUI-SAM3 @de0ff5d2 LoadSAM3Model; SAM3Grounding, SAM3Segmentation, SAM3MultipromptSegmentation, SAM3CreateBox, SAM3CreatePoint, SAM3CombineBoxes, SAM3CombinePoints; SAM3VideoSegmentation, SAM3Propagate, SAM3VideoOutput; SAM3PointCollector, SAM3BBoxCollector, SAM3MultiRegionCollector, SAM3InteractiveCollector
ComfyUI-GIMM-VFI @4c9a3123 DownloadAndLoadGIMMVFIModel, GIMMVFI_interpolate
ComfyUI-WanAnimatePreprocess @0e0b6a2a OnnxDetectionModelLoader, PoseAndFaceDetection, DrawViTPose, PoseRetargetPromptHelper, PoseDetectionOneToAllAnimation
ComfyUI-FlashVSR_Ultra_Fast @4820b3f0 FlashVSRNode, FlashVSRNodeAdv, FlashVSRInitPipe
ComfyUI-VideoHelperSuite @4d907bee VHS_LoadVideo (class LoadVideoUpload), VHS_VideoCombine (class VideoCombine), VHS_LoadVideoPath, VHS_LoadVideoFFmpeg, VHS_LoadVideoFFmpegPath

OnnxDetectionModelLoader takes vitpose_model and yolo_model as combo inputs listing whatever is in models/detection — it hard-codes no filename, which is why V.02.03 was free to choose vitpose-l-wholebody.onnx.

V.02 Volume growth (video extension)#

Date Item Result
2026-09-11 15:12Z V.02.01 volume grow User approved 500 GB in conversation. runpod-provision.mjs volume get ax0oj250j3 → size 100; volume update ax0oj250j3 --size 500 --yes → PATCH /networkvolumes/ax0oj250j3 {size:500} → response size 500; re-read volume get → size 500, EU-RO-1, name isis-chroma-models. New storage cost $35.00/month ($0.07/GB), up from $7.00. Irreversible (volumes never shrink).

V.02.02 — video manifest rows (2026-09-12, no pull yet)#

infra/runpod/volumes/isis-chroma-models.manifest.json: 31 rows added, 41 total, 354.85 GB of the 500 GB volume, 29.0 % headroom, lint 0 errors 0 warnings, matching the footprint table's ~353.4 GB projection.

sha256 is filled before the pull rather than after it. For a Hugging Face row the LFS oid is the sha256 of the file content, so recording it now is what lets the loader's download be checked against a value that did not come from the download. Every one was then re-fetched and diffed: a script grouped all 39 HF rows by (repo, revision), called paths-info for each group, and compared both size and the LFS oid — all 39 match. (One repo answered maximum queue size reached on the first pass and was retried on its own.)

Two corrections the reading turned up, both of which would have cost a failed pull or a request-time download:

  • FlashVSR-v1.1, not FlashVSR. ComfyUI-FlashVSR_Ultra_Fast nodes.py:140-141 joins folder_paths.models_dir with the model combo value verbatim, and model_downlod() runs snapshot_download of the entire repo when that directory is absent. The destPaths and the lint's folder list use the -v1.1 name.
  • vitpose-l, not vitpose-h — see V.02.03 below.

Lint changes: ALLOWED_TOP_FOLDERS gained latent_upscale_models, audio_encoders, interpolation, detection, FlashVSR-v1.1, controlnet_aux, each with the pack that reads it written beside it. QUANTIZED_PATTERN gained fpmixed, convrot and a delimited [_-]q4(?![0-9]) (fp4 was already present, but it does not catch fpmixed). The pattern was then run against all 19 real filenames in the new rows and 6 known-quantized ones: no false positive (fp16, fp32 are safe) and no miss. The two checked-in-manifest assertions in runpod-volume-manifest-lint.test.mjs were repinned from 10 rows / ~69 GB to 41 rows / ~355 GB, and the headroom case from 75→100 GB to 400→500 GB; 7 of 7 pass.

V.01.03 — volume folder mapping (2026-09-12)#

extra_model_paths.yaml gained audio_encoders, detection, interpolation and latent_upscale_models. entry.sh gained two symlinks beside the existing RMBG/sam3 pair — models/FlashVSR-v1.1 (the pack bypasses folder_paths entirely) and input/volume → /runpod-volume/inputs, which is what a volume/<jobId>/<name> LoadVideo reference resolves through (D-V4). The Dockerfile sets AUX_ANNOTATOR_CKPTS_PATH=/runpod-volume/models/controlnet_aux and AUX_USE_SYMLINKS=False; custom_hf_download lays files out as <ckpts_dir>/<repo id>/<filename>, which is exactly how the three controlnet_aux manifest rows are addressed, so DWPose and Depth Anything V2 resolve on the volume instead of downloading on a serverless worker.

entry.sh was exercised for real against a throwaway root (OSHUN_ENTRY_PREPARE_ONLY=1, COMFYUI_PATH and OSHUN_VOLUME_ROOT pointed at a temp dir): all four links created, all four dangling because no volume was mounted, exit 0. The smoke test now asserts all four with test -L.

V.02.03 — the two unverified rows, checked against the live HF API (2026-09-12)#

All read-only, anonymous, huggingface.co/api/.../paths-info for exact bytes and the LFS sha256 (so the manifest's sha256 can be filled before the pull rather than after it).

ViTPose wholebody — and a trap. JunkyByte/easy_ViTPose is ungated (repo sha e838052) and carries 28 ONNX files, four of them wholebody:

File Bytes sha256 (LFS oid)
onnx/wholebody/vitpose-l-wholebody.onnx 1,234,579,166 89bdf6692d9224db…
onnx/wholebody/vitpose-b-wholebody.onnx 360,242,862 017c08e161bfa2c7…
onnx/wholebody/vitpose-s-wholebody.onnx 97,427,067 (not fetched)
onnx/wholebody/vitpose-h-wholebody.onnx 412,883 e5dab9693f9fb437…

The -h file is not a model. At 412 KB it is an ONNX graph whose weights live in external-data files; its first bytes are a valid ONNX protobuf (pytorch 1.13.1) with no tensors. Its sha256 is byte-identical to Wan-AI/Wan2.2-Animate-14B's process_checkpoint/pose2d/vitpose_h_wholebody.onnx/end2end.onnx — and in that repo the .onnx is a directory of several hundred external-data tensor files beside the graph, which is why the official copy cannot be a manifest row either (one row per file would mean hundreds). So the manifest row is vitpose-l-wholebody.onnx, the largest usable single file, which also matches the ~1.5 GB the footprint table budgeted for the preprocess pair.

Nothing hard-codes the filename. ComfyUI-WanAnimatePreprocess @0e0b6a2a registers models/detection as a folder_paths folder, adds .onnx to its extensions, and OnnxDetectionModelLoader takes vitpose_model and yolo_model as combo inputs listing whatever is in that folder. So the choice above is ours to make and ours to change, and the volume folder is models/detection/.

Wan-AI/Wan2.2-Animate-14B is ungated (repo sha cb93a22) and its process_checkpoint/det/yolov10m.onnx is 61,659,339 B — that row stands.

controlnet_aux annotator checkpoints. The ckpt-dir env var V.01.03 needs is AUX_ANNOTATOR_CKPTS_PATH (src/custom_controlnet_aux/util.py:41, beside AUX_USE_SYMLINKS and AUX_TEMP_DIR), and custom_hf_download lays files out as <ckpts_dir>/<repo id>/<filename> — so pointing it at /runpod-volume/models/controlnet_aux/ gives these destination paths:

Node Repo (ungated) File Bytes sha256
DWPreprocessor bbox yzd-v/DWPose (sha 1a71441) yolox_l.onnx 216,746,733 7860ae79de6c89a3…
DWPreprocessor pose yzd-v/DWPose dw-ll_ucoco_384.onnx 134,399,116 724f4ff2439ed61a…
DepthAnythingV2Preprocessor depth-anything/Depth-Anything-V2-Large (sha cbbb86a) depth_anything_v2_vitl.pth 1,341,395,338 a7ea19fa0ed99244…

The node defaults are bbox_detector="yolox_l.onnx", pose_estimator="dw-ll_ucoco_384.onnx" (node_wrappers/dwpose.py:54, repo yzd-v/DWPose via DWPOSE_MODEL_NAME) and ckpt_name="depth_anything_v2_vitl.pth" (node_wrappers/depth_anything_v2.py, repo resolved through DEPTH_ANYTHING_V2_MODEL_NAME_DICT). Those three files are what the graphs will actually ask for, so those three are the rows; seeding them is what stops a request-time download on a serverless worker.

Total added by this item: 1.23 + 0.06 + 0.22 + 0.13 + 1.34 ≈ 2.98 GB, about 1.5 GB more than the footprint table's ~1.5 estimate for the Animate preprocess pair, because the table did not count the controlnet_aux annotators. Still far inside the 500 GB volume.

V.03 Video endpoint#

No rows yet. Rows record the endpoint id, the GPU pool ids actually saved (read back through GraphQL, never the REST echo — C.03 landmine), workersMax, executionTimeout, idleTimeout, the image digest, and the drift-check output.

V.04 Video cost control#

No rows yet. Rows record the video ledger key, a refusal observed at the boundary, and the reconciliation of the ledger total against RunPod's own billing view.

V.05 Video transport#

No rows yet. Rows record the measured RunPod request payload limit, the inline/S3 threshold it implies, and one PUT + one read back through the volume's S3 API with byte counts.

V.06 Catalog schema and executor for video#

No rows yet. Rows record the object-info snapshot the static validator runs against (build id and class count) and the CI run of ComfyUI's own /prompt validator over every motion graph against a zero-byte shadow volume.

V.07 Output persistence for video#

No rows yet. Rows record one stored clip: key, content type, byte count, range-request behaviour, and the poster frame's dimensions read from the image header.

V.08 motion workflow catalog#

No rows yet. One row per graph with its proof_level and what proved it.

V.09 Dashboard (video)#

No rows yet. Rows record the chromium run: spec file, what was asserted about playback, and the defects the run found.

V.10 Dev stack (video)#

No rows yet. Rows record the preflight output naming both endpoints, both ledgers and the volume size.

V.11 Live proof matrix#

No rows yet — nothing has been rendered. V.11.01 is the one standalone clip; V.11.02 is a single warm-chained session of seven. Each row: job id, GPU, ms, USD, sha256, duration, fps, resolution, audio.

Found outside this initiative: executor.spec.ts has 11 pre-existing failures#

Measured 2026-09-12 while running the generation-api suite as commit-time evidence for V.05/V.10. Recorded here because this initiative keeps editing apps/isis/generation-api/src/workflows/executor.ts, and a spec file that is already red is not a guard — two regressions of my own hid inside it for several commits before I looked.

The split, measured by checking out and running the file at two points:

commit what it is failures
fc7f3051c3 before the chroma/video work began 11
72ff476aec this branch, before the fix below 13

The two extra were mine and are fixed in the same commit as this note:

  • registers every @isis/workflows catalog entry at startup — expected 66 to be 75. The test compared the resolver against the whole catalog, but V.06.02 (acd8330d9d) put motion into WORKER_COMFYUI_ONLY_CATEGORIES alongside chroma, and those entries are deliberately not registered in the general resolver.
  • validates and resolves all catalog workflows via deterministic override selection — Missing resolver entry for workflow wan22-t2v, the same cause.

Both now filter the catalog through the imported WORKER_COMFYUI_ONLY_CATEGORIES rather than a restated list, and the first additionally asserts the excluded entries really are absent — so the test cannot start passing because the exclusion set quietly grew to cover a registration bug.

The 11 that remain are not from this initiative. Identical assertions at fc7f3051c3. Two of them are diagnosed:

  • tracks resolve_success and resolve_failure counters — asserts resolve({generationType:'inpainting'}) finds nothing. Exactly one workflow routes to inpainting: inpaint-edit, added 2026-07-03 in e8e4b301d2. The test was written 2026-02-21 (b4f667d773). It has been red since the day that workflow landed. The same cause explains does not resolve generation types that still have no dedicated catalog workflows (expected { id: 'inpaint-edit', … } to be undefined).
  • The four certifies <guided-journey|lecture-scene|mandala-animated|meditation-scene> workflow … failures are result.success === false at executor.spec.ts:2899 — a real execution failure of those text-to-video workflows through the deterministic test ComfyUI service, not a stale assertion.

The remaining five (the WorkflowExecutor unresolved workflow runtime guard group, voice consistency profiles, sequence rerender, and the ControlNet mode mapping) are undiagnosed. They belong to whoever owns the catalog-execution contract, not to this tracker; they are written down here so the next person to see "13 failed" on this file knows which number is the baseline and which is new.

V.01.09 — ComfyUI validates every rendered graph at build time (2026-09-13)#

Files: docker/runpod/comfyui-slim/validate-goldens.py (new), docker/runpod/comfyui-slim/smoke-test.sh, docker/runpod/comfyui-slim/Dockerfile, .dockerignore. Log excerpt: docs/agents/evidence/isis-chroma-runpod/build-6bbdae22-smoke-validate.txt.

What Value
Build 6bbdae22-ce4f-4815-951e-21b1eb6c8668 COMPLETED 00:50Z, commit f73b8998d9, image tag …-dockerfile:f73b8998d
Shadow volume [smoke] shadow volume: 42 zero-byte model files under /tmp/smoke-volume/models — one per manifest destPath, created BEFORE boot
Class check 77 required classes present, 1071 classes total (was 64 required on a5ac70968; same 1071 total)
Validator [validate] 18/18 rendered graphs accepted by ComfyUI's validator — 17 motion goldens + chroma, HTTP 200 each, nodes 5–24
Cleanup each accepted prompt: POST /queue {"clear":true} + POST /interrupt; shadow inputs removed; /tmp/smoke-volume removed
Smoke step wall time 7.0 s (#27 DONE 7.0s); the previous good build's was 14.3 s with the same 1071 classes, so ComfyUI does boot in seconds there
Failure mode proven build 055f8d58 FAILED at COPY infra/…/isis-chroma-models.manifest.json — root .dockerignore excludes infra; re-included
Local proof python3 docker/runpod/comfyui-slim/validate-goldens.py --self-test — stub server accepts one graph, rejects one with Bogus; asserts the report, the clear/interrupt calls and that the shadow input was removed
Pre-check on the Mac every model-looking input in all 18 goldens resolves to a manifest destPath (42/42); media refs are ref.png, start.png, clip.mp4, drive.mp4, mask.mp4

What the 18/18 does and does not prove: ComfyUI accepted every graph the catalog renders — every class exists, every required input is present, every literal is in type and range, every combo value (model filename) is one the volume ships, every link's socket types agree. It does not prove the graphs execute; that is what V.11 clips are for. The live endpoints still run a5ac70968; f73b8998d is built and pushed to the registry but not released.

V.01.06 — the object-info snapshot is in the repo (2026-09-13)#

File: infra/runpod/endpoints/node-classes/worker-comfyui-5.10.0-video.json (1071 classes; provenance + classes). Producer: docker/runpod/comfyui-slim/snapshot-emit.py (in the smoke test) and scripts/isis/runpod-build-snapshot.mjs (repo side).

What Value
Build f4fbb9fb-74f2-44b7-a1b7-2217b4531a35 COMPLETED, commit fa81159f3, image …-dockerfile:fa81159f3; 18/18 goldens validated in the same build
Emitted OBJINFO-META classes=1071 bytes=730990 chunks=75 sha256=06a7f453…; 75 lines of 1600 chars, printed last so they sit inside the newest 500 log lines
Decoded all 75 chunks present, sha256 matched, class count matched; wrote …/worker-comfyui-5.10.0-video.json: 1071 classes
Reduction per class: input.required, input.optional (tooltips stripped), output, output_name — the fields the validator reads; 0 tooltips left
Combo lists the shadow volume's names — UNETLoader.unet_name offers 9 files, every one on the manifest (spec asserts it); CLIPLoader.clip_name 3
Why the log US-CO-1 has no S3 and CPU pods cannot hold the image (20 GB cap vs 30); the GPU route sat throttled/initializing in EU-RO-1 for 1 h+ at $0 and was cancelled; a git build's only channel back is its log, and getBuildLogs takes only buildId + startTime (probed: endTime, limit, cursor, page, offset, pageSize all rejected) and returns the newest 500 lines
Refresh any push to runpod-comfyui-slim rebuilds; then node scripts/isis/runpod-build-snapshot.mjs --endpoint 2lezocymx2rgot (refuses a non-COMPLETED build)
Tests scripts/isis/runpod-build-snapshot.test.mjs 5/5 (order-independence, missing chunk, corrupted chunk, no emission, python↔node format); snapshot-emit.py --self-test

Not stored: the raw /object_info with tooltips and display names. It is ~5× larger and carries nothing the validator reads; the reduction is named in the file's provenance.reduced so nobody mistakes it for the whole.

V.06.06 — static validator runs every check, against the real snapshot (2026-09-13)#

Files: libs/isis/workflows/src/graph-validator.ts, graph-validator.spec.ts (+6 cases), graph-validator.snapshot.spec.ts (new), apps/isis/generation-api/src/workflows/executor.ts, worker-comfyui-submission.ts.

Check (tracker wording) Where Evidence
every class_type exists UNKNOWN_CLASS spec
every required input present MISSING_REQUIRED_INPUT spec
literal type and range LITERAL_TYPE_MISMATCH, LITERAL_OUT_OF_RANGE (new) spec: wrong scalar, literal on a socket, 250 outside [0, 100]; "FLOAT,INT" accepts either
link source output type = target input type LINK_TYPE_MISMATCH (was declared, never emitted) spec: LATENT/VAE inputs linked to a MODEL output → two issues
combo values in the list when non-empty BAD_COMBO_VALUE, now BOTH forms: [[…], {}] and ["COMBO", {options}] spec; upload inputs (image_upload etc.) exempt — their list is the builder's input dir, not the job's
model filenames resolve to a manifest destPath combo membership against the shadow-volume lists + MODEL_NOT_ON_VOLUME snapshot spec: every UNETLoader option is on the manifest
LoadVideo/LoadImage references resolve UNRESOLVED_FILE_REFERENCE (pre-submit path only; real upload names) spec
pre-submit path executor reads the snapshot file (fail-loud SERVICE_UNAVAILABLE if missing or shapeless) → submission → validateGraph({ objectInfo }) worker-comfyui-submission + node-class-inventory-routing specs 26/26
vitest over every golden graph-validator.snapshot.spec.ts: 18 goldens (17 motion + chroma), issues: [] AND skipped: [] 43/43 across the two validator specs

Adversarial pass, what it found: two false positives in my own first cut. LTXVEmptyLatentAudio.frame_rate is declared "FLOAT,INT" and the literal check read that as one unknown socket type, failing both ltx23-av goldens that ComfyUI itself had just accepted — fixed by splitting multi-type declarations; and LoadImage.image's option list is ["example.png"] (the builder's input dir), which would have failed every ref.png — fixed by exempting *_upload inputs from membership. Both are the kind of defect only the real snapshot exposes, which is the argument for having it. Adversarial grep on the touched files: zero hits. tsc on libs/isis/workflows: clean. tsc on generation-api: 10 errors, identical with and without these changes (none in the touched files; pre-existing in jobs.ts, job.service.ts, …).

V.07.03 — the OpenAPI half (2026-09-13)#

Files: apps/isis/generation-api/openapi.yaml (baseline), openapi/isis/openapi.yaml (generated), apps/isis/generation-api/src/types.ts, apps/isis/generation-api/src/services/output-metadata-projection.spec.ts.

What Value
Schemas added JobOutputFileFacts (25 properties, additionalProperties: false), JobOutputFile, JobOutput
Operation retyped GET /api/v1/jobs/{id}/output 200 → JobOutput
Semantic diff paths 66 → 66 (none added or removed); schemas +3, −0; operations with changed responses: 1
Generator node scripts/isis/generate_generation_api_openapi.mjs --check → up to date
Drift held by the spec client JobOutputFileFacts = OpenAPI JobOutputFileFacts.properties = API src/types.ts JobOutputFileFacts, all three sorted and compared
Found on the way the API's own JobOutputFileFacts lacked all twelve video and provenance fields (kind, durationMs, fps, frames, hasAudio, codec, alpha, source, encoding, s3FetchError, volumeKey, posterUrl); added
Tests projection spec 7/7; typecheck: correction 2026-09-13 — the tsc --noEmit -p tsconfig.json quoted here as 0 errors checks no files (files: [] plus references); the real app config tsconfig.app.json has 10 pre-existing errors, none in these files

Why the earlier "no target" verdict was wrong: the generator copies any operation the baseline already describes and only invents a JsonObject response for routes the baseline is silent on, so a typed response is a one-operation edit.

V.11.02 — worker logs were never captured, and the reader that fixes it (2026-09-13)#

Files: scripts/isis/runpod-worker-log.mjs (new), scripts/isis/runpod-worker-log.test.mjs (new), scripts/isis/motion-live-proof.mjs, scripts/isis/runpod-catalog-run.mjs, runbook §6c.

What Value
Live probe GET /v2/serverless/2lezocymx2rgot/workers/9lx1gb8ksy3dw6/logs → HTTP/2 200, content-type: text/event-stream; 100 frames in 12 s covering 5 s of worker time (replay is paced, roughly 8 frames/s)
?tail=N honoured: ?tail=3 returned exactly the last three lines (Digest: sha256:af2fba…, Status: Downloaded newer image …:fa81159f3, worker is ready)
Last-Event-ID, ?since= not honoured — nothing replayed within 6 s
Old capture await response.text() on a stream that never closes → 20 s AbortSignal.timeout → "log fetch timed out" on every failure (v1102-summary.json, stages 2b and 6, verbatim)
New reader ?tail=400, incremental read, stop on 2.5 s quiet gap or 20 s cap; truncated reported; 404 → gone, non-2xx → http_error, no frames → empty, socket drop → what was read plus stream ended early
Tests node --test scripts/isis/runpod-worker-log.test.mjs 7/7 against a real node:http SSE server: paced replay split mid-frame stops on the quiet gap and sends ?tail=250; a never-quiet stream is capped at 400 ms; a dropped socket keeps both lines; 404/500/silent; captureWorkerLog writes header plus lines and returns the path
Live proof of the reader node scripts/isis/runpod-worker-log.mjs --endpoint 2lezocymx2rgot --worker 9lx1gb8ksy3dw6 --tail 12 → 12 lines in 5.3 s
Also seen the worker's image digest is in its own log (Digest: sha256:af2fba848bb8… for tag fa81159f3) — a pulled image does have a digest, the build API just never reports one

Both endpoints' templates now show image tag fa81159f3 (GraphQL template.imageName), so the V.01.06 build is what runs; the memory note saying it was unreleased is stale.

D-V4 output leg — never ran, and cannot as configured (2026-09-13)#

What Value
Volume outputs/ node scripts/isis/runpod-volume-s3-put.mjs ls outputs/ → 0 objects; inputs/ → 0 objects; 52 objects total, all under models/ plus .oshun/manifest-result.json
Worker uploader worker-comfyui 5.10.0 handler.py → rp_upload.upload_image(job_id, temp_file_path); runpod~=1.7.12 rp_upload.py:190-193 → bucket = bucket_name if bucket_name else time.strftime("%m-%y"), Key=f"{job_id}/{image_name}{file_extension}"
Consequence against the volume S3 API (one bucket = the volume id) the upload fails; the handler then appends an error and returns no data for that output (handler.py:805-808, no base64 fallback)
Template env unreadable: GraphQL template.env empty, rest.runpod.io/v1/templates/<id> → error; the provision script never injects envFromEnvironment
How V.11 clips arrived inline base64 in images[] (211 885 B for V.11.01) under the measured 10 MiB body cap — the 528 s cold start and every measured cost stand; only the transport claim is corrected

Recorded on C.12.06, V.12.05 and V.12.06; no box changed.

C.12.02 / C.02.09 — Real-ESRGAN on the volume and the optional tail (2026-09-13)#

| What | Value | | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- | | Weight | lllyasviel/Annotators @ 982e7edaec38759d914a963c48c4726685de7d96 RealESRGAN_x4plus.pth, HF LFS size 67 040 989, oid 4fa0d38905f75ac06eb49a7951b426670021be3018265fd191d2125df9d682f1 | | Local check | shasum -a 256 on the download = the LFS oid; 67 040 989 bytes | | Upload (C.02.09) | runpod-volume-s3-put.mjs put … --key models/upscale_models/RealESRGAN_x4plus.pth → single PUT, verified 67040989 bytes by HEAD; ls models/upscale_models/ → 1 object, 2026-09-13T05:00:48Z | | Manifest | row realesrgan-x4plus, upscale_models/RealESRGAN_x4plus.pth, license BSD-3-Clause; runpod-volume-manifest-lint.mjs --strict → 43 artifacts, 358.37 GB of 500 GB, 0 errors, 0 warnings | | Catalog | chroma-txt2img nodes 16–18 and chroma-img2img nodes 13–15: UpscaleModelLoader / ImageUpscaleWithModel / ImageScaleBy(0.5, lanczos), all _condition: {{upscale != 'none'}}, passthrough image; upscale input none | x2; models_required += the file | | Default unchanged | rendering with upscale: none equals chroma-txt2img.square_1024.golden.json byte for byte (script check + spec) | | x2 golden | chroma-txt2img.square_1024.x2.golden.json, 18 nodes, SaveImage.images = ["18", 0] | | Specs | libs/isis/workflows: golden, snapshot (19 goldens), workflows, motion-caps, chain → 5 files / 154 tests; generation-api submission + routing → 26 | | Build | b60db5b1-9ccc-45f8-b2c9-75462df3988e COMPLETED 06:12Z, commit a5118e06871, image tag …:a5118e068; shadow volume 43 files; [validate] 19/19 rendered graphs accepted by ComfyUI's validator incl. chroma-txt2img.square_1024.x2.golden.json nodes=18 http=200; smoke exit 0 — excerpt evidence/isis-chroma-runpod/build-b60db5b1-smoke-validate.txt | | Snapshot refresh | runpod-build-snapshot.mjs --endpoint 2lezocymx2rgot → 1071 classes, 75 chunks, sha256 aa1ac72b…; semantic diff vs the committed file: UpscaleModelLoader.model_name options [] → ["RealESRGAN_x4plus.pth"], VHS_LoadImagePath extension list reordered (set noise); specs 3 files / 56 tests | | Release | both endpoints' templates read …:a5118e068 afterwards (auto-released); balance $10.14, spend $0.049/h (volume only) — no GPU was billed by any of this |

V.12.04 — the four "list, do not pull" families (2026-09-13)#

Anonymous HF API, LFS byte counts. Headroom 141.6 GB. Comfy-Org/Wan_2.2_ComfyUI_Repackaged (Apache-2.0): Fun Control 14B 57.16 GB (5B 10.00), Fun Camera 14B 59.16 GB, Fun Inpaint 14B 57.16 GB (5B 10.00), Fun VACE 14B 69.36 GB, S2V 14B 32.59 GB + wav2vec2 0.63 GB. InfiniteTalk: MeiGen-AI/InfiniteTalk (Apache-2.0) 9.95 / 2.71 / 2.71 GB, Kijai fp16 5.13 / 5.12 GB; needs Wan-AI/Wan2.1-I2V-14B-480P + TencentGameMate/chinese-wav2vec2-base (README) and a wrapper pack — none in the image or on the volume. Recorded in the tracker note; nothing pulled.

V.12.01 — SAM 3.1 support check (2026-09-13)#

ComfyUI-RMBG main = 58f1947a (2026-08-21, pinned SHA); ComfyUI-SAM3 main = de0ff5d2 (2026-05-19, pinned SHA). No path or loader mentions sam3.1 / multiplex. No supported pack; box stays open by its own wording.

V.11.02 — the approved re-run, 3 of 4 (2026-09-13, 07:32–07:58Z, $0.78)#

One invocation of scripts/isis/motion-live-proof.mjs --only 2a,2b,6,5 --out …/v1102-rerun, one worker bkeiqpovpvuuy6 (RTX PRO 6000 Blackwell Server Edition, per GET /v2/serverless/2lezocymx2rgot/workers), image a5118e068. Balance $10.0905 (07:32Z) → $9.2772 once settled (identical at 08:13Z and 08:17Z) = $0.813, of which $0.037 is volume rent; GPU ≈ $0.776 for 739.4 s of execution = $0.00105/s. A reading at 08:00Z, 90 s after the last job, showed $9.4825 — RunPod posts a job's charge minutes after it ends, so an early reading understates by a whole job.

Stage Workflow Status Execution Output
2a gimm-vfi-interpolate COMPLETED 25.4 s 2a-gimm-vfi-interpolate-0-26a1a57a.mp4 214 932 B, h264 832×480 65 f at 32 fps (2a re-run only to feed 2b)
6 ltx23-av COMPLETED 304.9 s 6-ltx23-av-0-93a142c8.mp4 292 869 B, h264 1024×576 25 fps 33 f + AAC audio, 62 frames (ffprobe) — the audio path is real
2b flashvsr-upscale FAILED 71.0 s Job processing failed; worker log captured (worker-bkeiqpovpvuuy6-1789285991353.log, 408 lines) — see below
5 wan22-animate COMPLETED 338.2 s, warm 5-wan22-animate-0-e9fca11a.mp4 139 748 B, h264 832×480 16 fps 33 f — the shape that killed the worker on 2026-09-12 rendered when run alone and last

FlashVSR, diagnosed at last. The log shows Using wan_video_dit from /comfyui/models/FlashVSR-v1.1/diffusion_pytorch_model_streaming_dmd.safetensors (the volume symlink works), [FlashVSR] Processing tile 1/8, then RuntimeError: Failed to find C compiler. Please specify via CC environment variable or set triton.knobs.build.impl. raised from triton/runtime/build.py via sparse_sage/quant_per_block.py — Triton compiles its NVIDIA driver module from C at the first @triton.jit launch, and the runtime base image ships no gcc. Neither the graph, the model files nor memory were ever the problem. Fix: gcc libc6-dev in the image's apt step plus a fatal smoke check (gcc present, Python.h present, a trivial compile succeeds); build a96a0123 (commit 299b763f408).

Cold start, bounded from the worker's own log: create container at 07:46:15Z, and stage 2a (25 s of execution) was complete inside the same minute — container-start-to-first-prompt is under about 20 s with FlashBoot. The 825 720 ms delayTime before it was RunPod placement (the endpoint reported an idle worker at submission time and still took 14 min to create a container after the a5118e068 release), which is not billed.

Also seen in that log: the template env is empty, so COMFY_LOG_LEVEL defaults to DEBUG and an aimdo memory-manager line lands every ~55 ms; a 1 500-line tail (worker-bkeiqpovpvuuy6-full.log) covers only the last 100 s of a job. COMFY_LOG_LEVEL=INFO from desired-state.json was never injected (V.12.05's finding) — it matters for diagnosability now that logs are capturable.

Catalog after this run: ltx23-av and wan22-animate are proof_level: rendered with cost_calibration: measured (15.6647 and 25.6611 s per megapixel-frame); wan22-animate's known_failure note is replaced by the render facts; flashvsr-upscale stays validated with the diagnosis in its proof_note.

V.11.02 — FlashVSR, rendered (2026-09-13, 08:32–08:34Z)#

What Value
Image build 79351a92-09bf-4264-bb57-d37173bef6cc (commit 47a74b7511d) — smoke: C toolchain: gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0; Python.h at /usr/include/python3.12, 19/19 goldens, exit 0; the prior build a96a0123 failed the new check on the missing Python.h (venv on Ubuntu's python3.12, no python3.12-dev)
Run motion-live-proof.mjs --only 2a,2b --out …/v1102-rerun-flashvsr, worker k7lblqd18yoqac
2a gimm-vfi-interpolate COMPLETED, delay 8.5 s, exec 22.7 s, warm — 2a-gimm-vfi-interpolate-0-493ad580.mp4 216 779 B
2b flashvsr-upscale COMPLETED, delay 0.1 s, exec 57.9 s, warm — 2b-flashvsr-upscale-0-c4645d50.mp4 1 117 573 B; ffprobe h264 1664×960, 65 frames, 16 fps (catalog default output_fps)
Rate 0.5579 s per OUTPUT megapixel-frame (65 × 1.5974 MP) — written to the catalog with proof_level: rendered
Snapshot refreshed from build 79351a92 (worker-comfyui-5.10.0-video.json, sha256 81de43ec…); class-level diff vs the committed file: only the VHS_LoadImagePath extension-list order
Cost script estimate $0.107 at its H100 rate; the settled balance is read after the session (RunPod posts charges minutes late)

With this row every one of the eight V.11 workflows has rendered live: T2V, I2V, Animate, SAM 3 track→VACE, BiRefNet matte, GIMM-VFI, FlashVSR, LTX-2.3 t2v+audio.

A.00 Arsenal expansion — volume growth (2026-09-13)#

Date Item Result
2026-09-13 09:55Z A.00.01 volume grow User instructed in conversation ("expand the storage volume as needed to accommodate them all"). runpod-provision.mjs volume get ax0oj250j3 → size 500; volume update ax0oj250j3 --size 1024 --dry-run → PATCH /networkvolumes/ax0oj250j3 {"size":1024}; --yes at 09:55:31Z → response size 1024; volume get at 09:55:33Z → 1024, EU-RO-1, isis-chroma-models; raw REST GET → 1024; GraphQL myself.networkVolumes → 1024. Storage cost $71.20/month tiered (1000 GB × $0.07 plus 24 GB × $0.05), up from $35.00. Irreversible.

Sizing (from the tracker's arsenal footprint table)#

Every survey row present at once with nothing yet retired is 798.36 GB (431.19 GB new on top of the 367.17 GB the lint counted at 09:41Z). The lint demands 20 % headroom: 798.36 ÷ 0.8 = 998 GB, and a 1000 GB volume would pass with 20.16 % — one more row of any size breaks it — so the size is 1024 GB (22.0 % at peak, 33.6 % at the ~680 GB steady state after the LTX-2.3, Animate-1, Depth-Anything-V2 and RMBG-2.0 retirements). The 24 GB above the tier line cost $1.20/month. Lint after the grow (manifest volume.sizeGb 1024): 50 artifacts, 367.17 GB of 1024 GB (64.1% headroom), 0 errors, 0 warnings.

Account readings around the grow#

Time (UTC) Balance currentSpendPerHr Note
09:40:49 $9.1296 $0.049 500 GB volume only; 0 workers, 0 pods
09:55:33 $9.0506 $0.761 right after the grow. Not the volume: /health at 09:57Z showed the chroma endpoint with 1 running worker and 1 job in progress and the video endpoint with 1 throttled worker — another session's live proofs (commits 09:31–09:55Z)
09:57:28 $9.0230 $0.761 unchanged 60 s later; the volume's own share is $0.0989/hour

Runway is the open risk (A.00.02): at $0.0989/hour of storage alone the $9.02 balance lasts about 3.8 days; RunPod's docs say a volume left at $0 "may eventually be terminated and its data cannot be recovered". No arsenal pull starts before the account is topped up.

A.00.03 Arsenal pull into the Hetzner library#

Path per option B: library-pull loader pods stream every A-row missing from the library (Hugging Face and Civitai sources) straight to app-dev-server/isis/models/, mounting the cache volume only for the result file. Rows already present with matching size and recorded sha256 are skipped, so a rerun resumes by object.

Pod Run Window (UTC) Wall Cost USD (0.06/h) Outcome
ed3tappadomybi library-pull, HEAD manifest, par 8 21:51 to 23:01 stopped at 22:30 0.070 none; library 62 to 63 objects (406.8 to 408.6 GB); pod deleted, orphan upload aborted
pa05xxybir0xof library-pull rerun, same manifest 23:01 to 23:17 980 s 0.016 none (swapped); LTX-2.5 transformer lost at 13+ GB, distilled LoRA landed (64 objects, 417.5 GB), Animate-2 upload aborted at 1.2 GB for the swap
732ifrk5muf24t library-pull, Range-resume loader (d78f80df034) 23:17 to 01:55 loader died 01:50 0.158 none; 64 to 81 objects (417.5 to 680.8 GB): LTX-2.5 transformer, Animate-2 base and distilled, SeedVR2, Z-Image, controlnet patches, HiDream-O1, Gemma 4 E4B, Qwen-Image 2512 and its encoder landed; Qwen Edit 2511 upload aborted at 15.7 GB
bhw3iszl7ug0ia library-pull, rerun-on-kill start loop, parallel 4 01:56 to 03:32 5330 s loader 0.096 arsenal-library-pull-result.json: 118 rows, 80 already present, 25 uploaded (50.34 GB, 7 source reopens), 13 failed
920gpxvfzogzez library-pull retry of the 13 failures, open-retry loader (ebbbaf81893) 03:34 to 05:03 3928 s loader 0.089 arsenal-library-pull-retry-result.json: 9 of the 13 already present (their first uploads had landed although the run recorded errors), SCAIL-2 DPO LoRA uploaded, 4 failed
qn2f2onkpve4qw library-pull, part-retry loader (e84471a58ae) 05:05 to 06:06 3299 s loader 0.061 arsenal-library-pull-retry2-result.json: 114 rows, Qwen Edit 2511 (40.86 GB, 609 parts, 1 source reopen), UnGloryHail (17.80 GB) and the Z-Image AIO (20.52 GB) uploaded; 1 failed (CyberRealistic bf16, HTTP 403)

The first pod's last progress was part 101 of the SeedVR2 7B upload at 22:30:27Z (6.71 GB, part 96 never arrived). At 22:58Z it showed 0 % CPU and 0 % memory with the container still up 4010 s, which is the start command's closing sleep after the loader process ended; the multipart upload was still open, so the watchdog's abort never completed. RunPod exposes no pod log over the API key (the REST API has no logs path; the console's log host answers 403 to an API key), and the console's tabs did not respond in the occluded browser window, so the exit code is unknown. The LTX-2.3 cache sync on the same bucket froze over the same minutes (see L.06). The rerun, started 23:01:26Z, was uploading the LTX-2.5 dev transformer within 20 s.

That rerun then lost the transformer too: at 23:07Z its upload stood at 13.02 GB, and by 23:12Z the upload was gone, no object had landed, and the pod had moved to the next row. library-pull read the source as one stream feeding the multipart upload, so a single source-side reset or timeout threw away every byte already sent, most likely also what the first pod's transformer row met. The loader now reopens the source with a Range request at the byte it reached (206 from exactly that offset, 12 reopens, loud failure if the server ignores Range; three new tests). At 23:17Z, after the distilled LoRA landed, the pod was swapped for one running the fixed loader.

That pod carried the transformer through (the Range resume works) and ran for 2 h 36 min, then at about 01:50Z its loader stopped mid-upload the same way the first pod's did: last part 01:49:50Z, 0 % CPU and 0 % memory at 01:54Z with the container still up (no restart), the multipart upload never aborted. An exit through the watchdog aborts uploads and exits 75, which the start loop reruns; an upload left open and a loop that went to sleep mean the process died some other way, most likely killed (memory on the 2-vCPU pod, 41 % in steady state on the previous pod, is the leading suspect but unproven without logs). The start loop now also reruns on any death by signal (exit above 128), still at most five attempts, and the next pod runs at parallel 4 to halve the part buffers held in memory.

It finished at 03:24:54Z with no loader death (memory stayed between 11 % and 30 %). Of the 13 failed rows, 12 failed with "The read operation timed out" raised by opening the source, which the Range-resume code did not retry (it retried reads, not opens): the Qwen Edit 2511 transformer, UnGloryHail, Z-Image AIO, two Z-Image LoRAs, four Wan LoRA halves, SCAIL-2's DPO LoRA and the old SAM 3 safetensors. The thirteenth is the CyberRealistic Z-Image bf16 file, which Civitai answers with 403, the risk A.00.04 recorded. Opens now spend the same retry budget on timeouts and 5xx; other 4xx fail at once.

The retry showed two more things. Nine of the thirteen "failed" rows were already in the library with the manifest size and sha256, so their first uploads had landed and only the client's wait had failed. And Qwen Edit 2511, UnGloryHail and the Z-Image AIO failed again with the same timeout, this time on the upload side: a part had five attempts a few seconds apart, which a Hetzner stall outlasts, so the whole multipart upload was aborted each time (Qwen Edit at about 22 GB). Parts now get twelve attempts, and a Complete that errors is checked by HEAD before anything is aborted. CyberRealistic still answers 403.

Pull closed 2026-09-14 06:00Z. With twelve attempts per part, the three large files that had failed three times landed in one run. The library is 117 objects, 813.38 GB (hetzner-usage: 813.38 GB of the 900 GB alarm line, outputs 0.00 GB). Every manifest row is in it with its manifest size and sha256 except the CyberRealistic Z-Image Turbo v8.0 bf16 file, whose Civitai download answers 403 to the account's key. Six pull pods in all, from 21:51Z to 06:06Z; their pod time at 0.06 USD/hour sums to 0.49 USD (the table's cost column), against the 0.19 USD the plan estimated for a pull with no failures.

Pinned core back on the cache, 2026-09-14. The arsenal rows joined pinned families (the Chroma set, the Wan LoRA pairs, SAM 3.1, SeedVR2 3B), so from the A.00.04 manifest commit until they were synced the API reported those families absent and refused their jobs before any GPU time. Cache sync c2a7313a (pod eu3n20cbc50qnr, 06:07:27Z to 06:32:13Z, 0.0247 USD booked) pulled 33.73 GB with no evictions; afterwards all 11 pinned families read cached and the volume held 304.7 GB of 400.

RMBG-2.0 deleted, 2026-09-14 07:54Z (A.02.08). User instruction: "delete as planned". Deleted by exact listing under the RMBG/RMBG-2.0/ prefix only: library app-dev-server/isis/models/RMBG/RMBG-2.0/ — model.safetensors (884,878,856 B), config.json, birefnet.py, BiRefNet_config.py; cache volume 5bst2pgw7y/models/RMBG/RMBG-2.0/ — the same four plus their .sha256 sidecars. Both prefixes list 0 objects afterwards; RMBG/BiRefNet/ still lists 6 library and 12 cache objects.

A.05.02 Live proof matrix — stills (2026-09-14)#

User approval in conversation 2026-09-14 ("go ahead with the live proof renders") against the tracker's $3–5 estimate. Every render below went through scripts/isis/runpod-catalog-run.mjs (the executor's own renderCatalogWorkflow) straight to the image endpoint 6jye7vngg4i685 on build b5376e6b, one worker (mx03mr7ejhszur), at the configured image rate 0.000439 USD per execution second (queue delay is not billed). Records: docs/agents/evidence/isis-chroma-runpod/a0502/<proof>.json; the UnGloryHail record was reconstructed from RunPod's status endpoint because its console output was not saved. Execution times on the first render of a family include loading its weights from the network volume (UnGloryHail 17.8 GB, Z-Image base, klein 4B, HiDream-O1 16.4 GB, the AIO 20.5 GB), so they are upper bounds for a warm render.

Cache syncs run for these proofs: 711f87c8 (Z-Image base and control, klein 4B, HiDream-O1, Depth Anything 3; 60.09 GB, no evictions, 0.0234 USD) and a4fe57f0 (Z-Image AIO 20.52 GB, evicting the leftover LTX-2.3 cache copy of 47.41 GB, whose library copy stays; 0.0033 USD).

Proof Item Workflow RunPod job Delay s Exec s USD Output (size, sha256)
chroma-ungloryhail-ckpt A.03.01 UnGloryHail checkpoint chroma-txt2img 733d1ab0-535a-48cc-a1f8-4b028babe2bf-e1 272.0 152.8 0.0671 chroma-txt2img-0-54b3febf.png 1282799 B 54b3febf033dc474
chroma-lora-nicegirls A.03.01 NiceGirls LoRA chroma-txt2img e76be00b-b2cb-4a7f-856f-56db28a6da87-e2 119.7 156.8 0.0688 chroma-txt2img-0-5664a65b.png 993197 B 5664a65b9e2501cf (not committed)
chroma-lora-nsfw-realism A.03.01 NSFW Realism LoRA chroma-txt2img aedf71f0-dd4a-4819-adb7-1fd74a09946b-e2 0.1 46.7 0.0205 chroma-txt2img-0-e5cdc1dd.png 1501766 B e5cdc1ddff327146 (not committed)
chroma-seedvr2-3b-tail A.01.08 SeedVR2 3B still tail chroma-txt2img 9de9bb72-81bd-4580-81b4-6db594e0f1a9-e1 0.1 79.2 0.0348 chroma-txt2img-0-f2ee1906.png 5797719 B f2ee190662015048
sam3-segment-sam31 A.01.09 SAM 3.1 core segment sam3-segment 4f43f8b2-423c-43f5-bbaf-cf77afaaa0f5-e2 0.1 18.9 0.0083 sam3-segment-0-4018fa68.png 6910 B 4018fa68a39afd71; sam3-segment-1-b2228ff3.png 1487629 B b2228ff3b30afdb2
zimage-txt2img-base A.02.02 Z-Image base zimage-txt2img 9ba69352-2dfa-4f89-a14a-fcbf61503fd4-e1 170.3 121.6 0.0534 zimage-txt2img-0-7c71c0be.png 1904538 B 7c71c0bec7e44589 (not committed)
zimage-aio-ckpt A.02.02/A.03.02 Z-Image Turbo AIO zimage-txt2img 57bcfbd0-4ba6-48fb-80ec-6d08485ff167-e2 12.4 99.3 0.0436 zimage-txt2img-0-f9f76b32.png 1381606 B f9f76b324ab07154 (not committed)
zimage-control-depth-da3 A.02.02 control, A.02.07 Depth Anything 3 zimage-txt2img-control 742ff7f7-cbb4-4b6d-b4e9-f565ce458a5b-e1 0.1 77.0 0.0338 zimage-txt2img-control-0-5c7c3f4c.png 1425279 B 5c7c3f4cf6634922 (not committed)
zimage-img2img A.02.02 img2img zimage-img2img 398aa014-a328-4a5d-bb7b-0970ac6a458f-e2 0.1 24.2 0.0106 zimage-img2img-0-55474db1.png 1310161 B 55474db1353f02e3 (not committed)
zimage-inpaint A.02.02 inpaint zimage-inpaint b0b870e0-c7a1-4264-99b6-f5e0c2551fb1-e1 0.1 23.8 0.0104 zimage-inpaint-0-96205070.png 1478406 B 962050709a637a6d (not committed)
zimage-lora-photoreal-betternudes-v3 A.03.02 BetterNudes LoRA zimage-txt2img a7b20555-2643-43c0-81b3-bc3d8dec05c0-e1 0.1 9.2 0.0041 zimage-txt2img-0-cbaaed51.png 1515216 B cbaaed51be9a39de (not committed)
zimage-lora-realistic-snapshot-v5 A.03.02 Realistic Snapshot LoRA zimage-txt2img 486d9efc-26cf-4b9e-97bd-6394be3f928b-e2 0.1 9.3 0.0041 zimage-txt2img-0-b4d8a4ed.png 2043347 B b4d8a4ed08360b86 (not committed)
zimage-lora-breast-slider A.03.02 Breast Slider LoRA zimage-txt2img 441c0e6f-6a98-429c-a922-66860acfd2f8-e1 0.1 6.7 0.0029 zimage-txt2img-0-9904cbc8.png 1416530 B 9904cbc8c11b8bad (not committed)
zimage-lora-detail-slider A.03.02 Detail Slider LoRA zimage-txt2img 119d22c0-1cfd-464b-bae2-e868a41effae-e2 0.1 6.6 0.0029 zimage-txt2img-0-d44f0ff4.png 1432440 B d44f0ff46a643213 (not committed)
zimage-lora-detaildaemonz-v1.1 A.03.02 DetailDaemonZ LoRA zimage-txt2img e4888677-b741-4245-9142-27e648773993-e2 0.1 8.7 0.0038 zimage-txt2img-0-9bc7f897.png 1194976 B 9bc7f897676eb629 (not committed)
klein4b-txt2img A.02.04 klein 4B text to image klein4b-txt2img 58eb154b-343e-4d27-949a-7b9e0c6ce69a-e1 0.1 73.1 0.0321 klein4b-txt2img-0-6a1220eb.png 1992456 B 6a1220eb1cd3f2eb (not committed)
klein4b-edit A.02.04 klein 4B edit klein4b-edit 91dbd849-56d9-440b-a846-2982d4355c63-e2 0.1 6.0 0.0026 klein4b-edit-0-57aeeab2.png 1329263 B 57aeeab24079266a
hidream-o1-txt2img A.02.03 HiDream-O1 text to image hidream-o1-txt2img 2cc428f6-498f-4cc2-a762-6208ef24423c-e1 0.1 82.0 0.0360 hidream-o1-txt2img-0-23e66682.png 5030607 B 23e66682995c2f61
hidream-o1-edit A.02.03 HiDream-O1 edit hidream-o1-edit a96fdd37-1455-4c37-9d35-90c9f2c2e443-e1 0.1 42.9 0.0188 hidream-o1-edit-0-ec02a6f7.png 4218304 B ec02a6f7315cde89 (not committed)
hidream-o1-personalize A.02.03 HiDream-O1 personalize hidream-o1-personalize 1de1d17b-9bc3-4c3d-a359-28a58db97ae7-e2 0.1 39.0 0.0171 hidream-o1-personalize-0-c8ccb4a9.png 4388745 B c8ccb4a9b89da4b3 (not committed)

Total: 20 renders, 0.4759 USD of execution; with the two syncs 0.5026 USD. Every job completed on the first submit.

Checks on the outputs, not just the status:

  • Every PNG has the dimensions the workflow declares (1024 square; 2048 square for HiDream-O1 and the SeedVR2 tail).
  • SAM 3.1: the mask is the woman's silhouette; the RGBA cut-out's alpha is 0 at the corner and 255 on the subject, with 80.2 % of pixels transparent.
  • Depth Anything 3 control: the Z-Image output keeps the source still's layout (subject position, foreground leaves, frame bars).
  • klein 4B edit: the dress changed from linen to red and nothing else did.
  • HiDream-O1: the sign reads "HiDream-O1-Image release".
  • LoRAs: the five Z-Image LoRA renders share checkpoint, seed and prompt and produce five different hashes, so each LoRA was applied.
  • SeedVR2 3B tail: the preview showed colour banding in smooth areas; the file does not (3,963 distinct colours in a 300 by 100 pixel out-of-focus patch; the HiDream still has 13,038 in a similar patch), so the banding was the preview's image compression, not the model.

Content finding: the Z-Image control render, whose prompt is the workflow's default and asks for no nudity, came back with a topless subject. The workflow is adult_ok, so this is within policy, but a tenant surface that expects an ordinary prompt to stay SFW cannot rely on this model to do so.

Not committed: outputs with nudity, from the adult LoRAs, or not visually checked are recorded by hash only (marked above); their bytes stay under isis/outputs/<job>/ in the Hetzner bucket (the outputs retention sweep defaults to a dry run, so nothing removes them yet).

A.05.02 video endpoint, first batch (2026-09-14)#

Same harness, video endpoint 2lezocymx2rgot on build b5376e6b, one worker (krxejkbqs9rjxh), at the configured video rate 0.001647 USD per execution second. The first job waited 1031 s in the queue for a card (not billed). Execution on the first render of a family includes loading its weights. Cache sync bfe4bf04 brought Qwen-Image and SeedVR2 7B (74.18 GB, evicting the finished Depth Anything and DWPose, HiDream-O1, klein 4B and Z-Image base families, 48.36 GB; 0.0099 USD).

Proof Item Workflow RunPod job Delay s Exec s USD Output (size, sha256)
sam3-video-track-sam31 A.01.09 video track, prompt the person sam3-video-track ca0e7932-cd3e-49c7-9ee6-c82224c6c54e-e2 1031.5 23.2 0.0382 sam3-video-track-0-5e722314.mp4 9410 B 5e7223147440be72; sam3-video-track-1-cab35f48.mp4 148196 B cab35f48c26683ed
sam3-video-track-sam31-boat A.01.09 video track, prompt the paper boat sam3-video-track 129e7f2e-0d27-40ec-ae1c-afbe8e145ee0-e2 1.1 6.2 0.0103 sam3-video-track-0-ec36dfaa.mp4 22780 B ec36dfaacbc9913e; sam3-video-track-1-d4aacd31.mp4 147495 B d4aacd31d6ed14ca
wan22-i2v-lora-hip-bump-dance A.03.03 hip-bump-dance LoRA pair wan22-i2v b8cb0057-3df0-4063-87b6-6238efc39366-e1 0.1 336.9 0.5548 wan22-i2v-0-f91b1347.mp4 335052 B f91b13476d371c93; wan22-i2v-1-024aa97a.png 527146 B 024aa97a500f5359
qwen-txt2img A.02.05 Qwen-Image text to image qwen-txt2img b94bedff-3461-4741-9aa5-aa9a9f1fdf27-e2 1.1 373.3 0.6148 qwen-txt2img-0-df7db569.png 2269141 B df7db56925093b16
seedvr2-upscale-7b A.01.08 SeedVR2 7B clip upscale seedvr2-upscale 07f38e2a-3635-4566-bd03-816dc30779a5-e2 0.1 129.1 0.2127 seedvr2-upscale-0-745be596.mp4 1113396 B 745be596dc6e7aac

Total: 1.4308 USD of execution plus 0.0099 USD of sync.

Checks on the outputs:

  • SAM 3.1 video: both clips are 33 frames of 832 by 480 at 16 fps, the same as the source clip (a paper boat in a puddle, with no person in it). With the prompt "the person" SAM 3.1 still returned an object: its mask sat on the water below the boat, a false positive at the default 0.5 threshold. With "the paper boat" the mask covers the boat exactly, so the overlay is aligned and the tracker is correct; the false positive is a property of the model at that threshold, recorded rather than hidden.
  • Wan 2.2 i2v with the hip-bump-dance pair: 33 frames of 480 by 832 at 16 fps from the UnGloryHail still as the start image; the frame at 1.2 s shows the same woman mid-dance.
  • Qwen-Image: 1328 square, a coherent full-length street fashion photograph from the workflow's default prompt.
  • SeedVR2 7B: 960 by 1664, 33 frames at 16 fps (2x of the Wan clip); a crop of the face shows resolved hair strands rather than a stretched frame.

A.05.02 video endpoint, second batch (2026-09-14)#

Cache sync 24c91fd4 brought LTX-2.5 and Animate-2 (124.98 GB, evicting the finished Qwen-Image, SeedVR2 7B, Z-Image AIO and Z-Image control families, 108.13 GB; 0.0386 USD). At about 10:29Z the local Postgres and Redis containers exited cleanly while it ran (not stopped by this session); the API's sync driver crashed on the lost database, the containers were brought back with docker compose -f docker/docker-compose.dev.yml up -d, and a new driver took the sync over and finished it with 0 failures. The render harness for LTX-2.5 was killed by the operating system for low memory while polling; the job it had submitted kept its place in RunPod's queue and a small attach poller (GET /status, then download and hash) wrote its record, which says so.

Proof Item Workflow RunPod job Delay s Exec s USD Output (size, sha256)
ltx25-av-t2v-distilled A.01.02 LTX-2.5 t2v, distilled ltx25-av 32e9b0f5-21b1-4cf9-a76e-91e00370e05e-e2 2913.8 400.6 0.6598 ltx25-av-0-9baf52ef.mp4 140901 B 9baf52ef9319e3e7
wan22-animate-2-min A.01.06 Animate-2 minimum shape wan22-animate-2 88646ae9-d2e1-4b43-a430-4eed3ad057c2-e2 1775.1 244.1 0.4021 wan22-animate-2-0-e56d2121.mp4 234059 B e56d2121c0260695

Both waited a long time for a card (48.6 and 29.6 minutes; the video endpoint's worker read throttled in EU-RO-1), which is not billed. Total: 1.0619 USD of execution plus 0.0386 USD of sync.

Checks on the outputs:

  • LTX-2.5: 49 frames of 768 by 448 at 24 fps, 2.04 s, with an AAC stereo 48 kHz track; the frame at 1 s shows a red fox crossing a snowy clearing as prompted, in a stylised rather than photographic look. The audio track is nearly silent (ffmpeg volumedetect: mean -61.8 dB, max -41.6 dB), so the prompted snow sound is not present and audio content is not proven; the catalog proof note says so.
  • Animate-2: 33 frames of 480 by 848 at 16 fps (482 by 854 was requested; the graph rounds down to a multiple of 16); the reference woman's face, dress and greenhouse are preserved. The driving clip showed the same person, so this proves the graph end to end rather than motion transfer to a different person.

Catalog rates: following the V.11 convention, each measured video workflow now carries seconds per megapixel-frame from its executionTime and a headline scaled to its default shape: sam3-video-track 0.4731 s (default 0.0252 USD), seedvr2-upscale 2.4495 s per output megapixel-frame (0.5220 USD), ltx25-av 23.7625 s (4.2673 USD for 121 frames at 1280x704) and wan22-animate-2 18.1742 s (0.9980 USD). The three that loaded weights during the measured run are upper bounds; scaled to its default shape ltx25-av exceeds the video endpoint's 900 s execution cap, which a warm measurement has to settle before that shape is offered.

A.05.02 video endpoint, SCAIL-2 (2026-09-14)#

Cache sync 9a04e6b9 brought SCAIL-2 (35.25 GB, evicting LTX-2.5, 92.19 GB; 0.0384 USD; it took about 36 minutes, the slowest of the day). The job then waited 29.4 minutes for a card.

Proof Item Workflow RunPod job Delay s Exec s USD Output (size, sha256)
scail2-animate-single A.02.06 SCAIL-2, one character, animation mode scail2-animate dc178886-aa79-4db5-b8db-41156ad13ae0-e1 1762.6 417.1 0.6870 scail2-animate-0-be9a0f0c.mp4 320342 B be9a0f0c19412e24

Check on the output: 33 frames of 480 by 832 at 16 fps. Its frames look very close to the driving clip's, because the reference image and the driving clip show the same woman, so a passthrough had to be ruled out: ffmpeg PSNR between the output and the driving clip averages 26.2 dB (minimum 23.6, maximum 38.5), far from the 40 dB and more a re-encoded copy would score (Animate-2's output against the same driver averages 19.0 dB). The graph generates new frames that follow the driver's pose; transfer to a different character is not proven by this run. It ran at 20 steps, half the default, which the catalog cost note says.

Proof spend for the day: 3.6556 USD of execution across 28 renders (20 stills 0.4759, 8 video jobs 3.1797) plus 0.1136 USD for the five proof cache syncs (711f87c8 0.0234, a4fe57f0 0.0033, bfe4bf04 0.0099, 24c91fd4 0.0386, 9a04e6b9 0.0384), 3.7692 USD in all at the ledger rates against the approved 3 to 5 USD. The RunPod balance went from 14.9489 USD (08:13Z) to 12.3962 USD (13:36Z), 2.5526 USD including about 0.20 USD of volume storage over those hours: the ledger rates bill the dearest card each endpoint allows, so they overstate what the cards that ran actually cost, and the balance can settle minutes late.

A.05.02 video endpoint, the approved remainder: lanes A and B (2026-09-14)#

Approved separately after the first 3 to 5 USD ("the rest", about 6.5 to 8 USD). Same harness, endpoint and rate. SCAIL-2, Animate-2 and the Wan 2.2 families were already on the cache, so these jobs needed no sync. The adult and racy LoRA outputs are kept out of git and are recorded here by size and hash only; their records in a0502/ still name the paths the harness wrote.

Proof Item Workflow RunPod job Delay s Exec s USD Output (size, sha256)
scail2-animate-two A.02.06 SCAIL-2, two characters, animation mode scail2-animate 8140f060-a79e-4577-8bb5-b0e131823474-e2 294.4 358.0 0.5896 scail2-animate-0-1506e915.mp4 412329 B 1506e915c58f208d
wan22-animate-2-896 A.01.06 Animate-2 at 512 by 896 wan22-animate-2 37528cd3-9e40-4dd4-b325-ac49fcf34b29-e2 248.4 253.5 0.4174 wan22-animate-2-0-0c92e293.mp4 236586 B 0c92e293027dbb35
wan22-i2v-lora-camera-tiltdown A.03.03 camera tilt-down pair (neutral) wan22-i2v 133e1aa8-26a4-4ca2-b3a1-901321752cbc-e1 65.5 57.9 0.0954 wan22-i2v-0-dcb06490.mp4 323928 B dcb06490bb3c1b7b; wan22-i2v-1-fde272aa.png 468614 B fde272aaf954e6fa
wan22-i2v-lora-camera-tiltup A.03.03 camera tilt-up pair (neutral) wan22-i2v 844ca07e-d4a7-4abf-993b-3815cf7a39e5-e1 60.6 70.8 0.1166 wan22-i2v-0-f50b7dd8.mp4 263688 B f50b7dd8c60a0485; wan22-i2v-1-02a944d0.png 532490 B 02a944d0ff2950a1
wan22-i2v-lora-hip-sway A.03.03 hip-sway pair (racy) wan22-i2v 39d9651b-5202-459d-bd42-12ccad46ba46-e1 413.2 300.5 0.4950 not committed: mp4 255334 B 71d78272154ebdd1; png 542016 B b0fff772dcdbd69e
wan22-i2v-lora-nsfw-posing-nude A.03.03 posing pair (explicit) wan22-i2v 1b4c9f80-53ea-4af3-be5a-8d9f705dd9e2-e1 349.5 68.2 0.1123 not committed: mp4 192342 B c43e7e3a932b2054; png 557527 B 09b16e3371f1c8ce
wan22-i2v-lora-bouncing A.03.03 bouncing pair (explicit) wan22-i2v 9c76252d-8319-434a-a327-3b2126452a74-e2 50.2 69.2 0.1140 not committed: mp4 226285 B 9ea2b98439b09b36; png 527515 B 06a11feed01e4e9a
wan22-i2v-lora-twerking A.03.03 twerking pair (explicit) wan22-i2v ae5f9ca6-4f5f-4f9f-90bd-75594dcd02ab-e2 65.8 59.1 0.0973 not committed: mp4 268534 B 48b21924965cac0c; png 545165 B 3fd9c111b20d7945
wan22-i2v-lora-slow-twerk A.03.03 slow-twerk pair (explicit) wan22-i2v fdae4237-cd6f-4e93-b5db-454c90014812-e2 0.1 58.1 0.0957 not committed: mp4 204766 B 44dee0c2b2bf2407; png 541333 B 0c8e066ebcc59230
wan22-t2v-lora-breast-slider-failed-1 A.03.04 breast-slider pair, first attempt wan22-t2v 04d4b818-05b0-4a5e-8a0b-2972918bf592-e2 0.1 217.1 0.3575 FAILED: ComfyUI HTTP unreachable during websocket reconnect
wan22-t2v-lora-breast-slider A.03.04 breast-slider pair, retry wan22-t2v a3d632ed-34aa-44f3-8249-eb0f9885c1b8-e2 989.9 355.9 0.5862 not committed: mp4 334344 B d8bc4161c9619cfd; png 487396 B c528c9f9f069af9b

Total: 3.0771 USD of execution, 0.3575 of it the failed attempt.

Checks on the outputs:

  • SCAIL-2, two characters: 33 frames of 832 by 480 at 16 fps. The reference image held two different women (the greenhouse woman and a woman in a pink coat on a street); the driving clip was two copies of the greenhouse clip side by side. At frame 16 the output places both reference women, in reference order left to right, in the driver's greenhouse, each in the pose of her driving figure. The right-hand figure is a different person from the driver, so this run proves transfer to a different character, which the single run could not.
  • Animate-2 at 512 by 896: 33 frames at 16 fps, the requested shape exactly (it is already a multiple of 16). The last frame shows the reference woman waving, as the driver does. Its 253.5 s against the 275 s its catalog rate of 18.1742 s per megapixel-frame predicts for 15.14 megapixel-frames keeps that rate an upper bound.
  • All eight Wan clips are 33 frames of 480 by 832 at 16 fps. The two camera pairs move the camera as named: the tilt-down clip ends on a low angle looking up at the woman, the tilt-up clip ends looking down on her from above. Their prompts also describe that camera move, so how much of it the pair adds is not isolated by these clips.
  • The racy and explicit pairs, including the breast slider, ran with LoRA strength 1 on both experts and completed, but at frames 16 and 32 every one of them shows the subject fully clothed and in the start image's framing (hip sway with arms crossed, as prompted). Motion is present (mean absolute frame-to-frame luma difference 3.9 to 7.0 of 255 on the four motion pairs). These runs prove the pairs load and render on the graph; whether a pair changes the render needs a same-seed render without it, below.
  • Same-seed baselines without a pair (approved as part of the Wan LoRA pairs' share): tilt-down (seed 1472, same prompt, job 89468c63-7d55-49ec-a3d9-4573b468aa17-e2, 374.3 s including swapping the Wan i2v experts back in after the Qwen jobs, 0.6165 USD; wan22-i2v-1-c66c83af.mp4 279080 B c66c83aff59b1217, wan22-i2v-0-395a1e80.png 497171 B 395a1e800213bfdf) and posing (seed 1475, same prompt, job 7cca256d-e294-46fe-998a-7147f1d5a8af-e1, 18.8 s warm, 0.0310 USD; wan22-i2v-1-0430d68f.mp4 217013 B 0430d68fc5db5dbe, wan22-i2v-0-c1dabb86.png 547757 B c1dabb86a9f3d4fe). ffmpeg PSNR between each LoRA clip and its baseline averages 14.1 dB for tilt-down and 22.9 dB for posing, so both pairs are applied and change the render. At the last frame the tilt-down clip is a tighter undershot near the chin than the baseline's upward look, the camera move the pair is named for; the posing clip differs in pose from its baseline but both stay clothed. The posing pair's versions list no trained words; bouncing, twerking, slow twerk and hip sway were prompted with their listed phrases and still stayed clothed from a clothed start image, so their named content is not shown by these clips. Spend of this remainder so far: 3.0771 USD for lanes A and B, 1.1179 USD plus the 0.1133 USD sync for Qwen (next section) and 0.6475 USD for the two baselines, 4.9558 USD in all.
  • The first breast-slider attempt ran straight after the slow-twerk job, warm on the same worker (io8c9yys4vg9lp) that still held the i2v experts; ComfyUI stopped answering 217 s in, while it would have been loading the t2v experts. Host memory with four 14B experts touched is the likely cause, but the worker was recycled before its log could be read, so the cause is not established. The retry, on the same worker after a cold start, succeeded. Both records are kept.

A.05.02 video endpoint, Qwen-Image edit and control (2026-09-14)#

Cache sync b80de934 (14:08 to 16:01Z, 0.1133 USD) pulled Qwen-Image, its edit transformer and the two InstantX ControlNets (6 files, 65.47 GB, 0 failures), evicting Animate-2 and SCAIL-2 after their clips (68.04 GB). The video worker then read throttled in EU-RO-1 for about an hour: the first qwen-edit job (fcd5f094-9f22-4843-9798-45d3510df5b6-e2) was cancelled by the harness after 60 minutes in the queue, unbilled, and the union job waited 52.7 minutes.

Proof Item Workflow RunPod job Delay s Exec s USD Output (size, sha256)
qwen-edit-control-union A.02.05 union control, default prompt qwen-edit-control 93050b7b-00d3-4d4c-be50-a54b947cd29d-e2 3159.7 295.5 0.4867 qwen-edit-control-0-ec35c80b.png 1661378 B ec35c80b74477e83
qwen-edit-control-inpaint A.02.05 inpaint control qwen-edit-control 7df56a13-3e4a-4a8e-9a6d-c392f5058022-e1 0.1 77.5 0.1277 qwen-edit-control-0-8343b9b0.png 2357379 B 8343b9b068477888
qwen-edit-one A.02.05 edit, one reference qwen-edit c432e401-648a-4f7f-9d7b-d059af1916cc-e1 0.1 282.5 0.4652 qwen-edit-0-0204f8c6.png 1333336 B 0204f8c6796500b9
qwen-edit-control-union-prompted A.02.05 union control, prompt matching the map qwen-edit-control fc2678f9-b4b9-4f0d-bace-4882d9582692-e2 209.0 23.3 0.0383 qwen-edit-control-0-755eff00.png 1492677 B 755eff0057926107

Total: 1.1179 USD of execution plus 0.1133 USD of sync.

Checks on the outputs:

  • Union, default prompt: the control map was a 1024 square canny edge map of the Qwen-Image street photograph's full-length woman. The output is a 1024 square close-up of a hooded woman, the workflow's default (template) prompt, with no trace of the map's figure: the prompt overrode the control. This is recorded as a finding, not hidden, and the prompt input's description now says to describe what the map shows.
  • Union, prompt describing the map: 1024 square; drawn over the output, the canny edges lie on the woman's outline, coat, handbag and shoes, so the union ControlNet does steer the graph. RunPod flagged the job a cold start, but it ran on the worker that had just run the other Qwen jobs, and its 23.3 s is a warm figure.
  • Inpaint: source the UnGloryHail greenhouse still, mask the SAM 3.1 person silhouette (A.05.02 stills), prompt "a woman in a red leather jacket, natural light". The output is 1536 square (the template scales the source's long side to 1536) with a red leather jacket over a dark top inside the mask and the greenhouse outside it unchanged; the face inside the mask was repainted too.
  • Edit: same source still, one reference, "Change the linen dress to a blue denim jacket over a white t-shirt." The 1024 square output keeps the woman's face, hair, pose and the greenhouse, and replaces the dress with the jacket and t-shirt as asked.

A.05.02 video endpoint, LTX-2.5 audio, Ingredients and pixel x2 (2026-09-14)#

Cache sync cdbb552e (18:14 to 18:54Z, 0.0402 USD, 0 failures) brought LTX-2.5 back (92.19 GB), evicting the three Qwen-Image families after their stills. All three clips ran distilled on one worker (2dohwqc0wosv89); the first waited 7.1 minutes for a card and loaded the weights.

Proof Item Workflow RunPod job Delay s Exec s USD Output (size, sha256)
ltx25-av-t2v-audio A.01.02 t2v with a loud sound prompt, 49 frames ltx25-av 6bb72e9d-04d7-4edf-88d8-958cb92a9a06-e1 426.4 414.4 0.6826 ltx25-av-0-1a1b9424.mp4 340092 B 1a1b9424f79260e0
ltx25-av-ingredients A.01.03 Ingredients, 121 frames, the model card shape ltx25-av 452cf724-4734-4474-b6d0-0d7803970fdd-e2 0.1 134.5 0.2215 ltx25-av-0-a2bdb521.mp4 535041 B a2bdb52171bf3b75
ltx25-av-pixel-x2 A.01.03 pixel x2 tail on a 49-frame t2v ltx25-av 61c74f24-b00a-41a6-8f05-f743a57e6c9f-e2 0.1 187.5 0.3088 ltx25-av-0-dfeb18ce.mp4 682824 B dfeb18ce1416bb0a

Total: 1.2129 USD of execution plus 0.0402 USD of sync.

Checks on the outputs:

  • Audio: prompted "a street drummer hits a snare drum hard and fast … loud crisp drum beats", the clip (49 frames of 768 by 448 at 24 fps, AAC stereo 48 kHz) shows a drummer with a snare on a city street, and its audio is loud (ffmpeg volumedetect mean -17.2 dB, max -0.5 dB); the spectrogram shows evenly spaced broadband bursts, and the 50 ms RMS envelope peaks about every 0.4 s, drum hits rather than noise. LTX-2.5 audio is proven. The earlier fox clip's near-silence (-61.8 dB) and this batch's Ingredients (-52.0 dB) and pixel x2 (-54.0 dB) clips, prompted with birdsong, footsteps, snow and wind, show that soft ambient prompts come out nearly silent.
  • Ingredients: the reference sheet (ltx25-av-reference-sheet.png, composed for this proof from the committed Qwen-Image street still and klein 4B greenhouse still: the woman full length, her face, her yellow handbag and a greenhouse, on black) produced 121 frames of 768 by 448 at 24 fps, 5.04 s: the same woman, braids, rose-gold leather coat and green trousers, walking through a greenhouse with the yellow handbag in the first frame, as prompted. Identity, wardrobe, prop and location all carry from the sheet. Warm, single-stage distilled: 3.23 s per megapixel-frame.
  • Pixel x2: 49 frames of 1536 by 896 at 24 fps, exactly twice the 768 by 448 base, a fox crossing a snowy clearing in a stylised look. Resampling the output down to 768 by 448 and back up scores 37.1 dB PSNR against it, so for this smooth, stylised clip the detail added beyond a plain 2x resample is modest; the tail runs and doubles the frame, and its quality gain is not established by this clip.

Spend of the approved remainder: 3.0771 (lanes A and B) + 1.1179 and 0.1133 sync (Qwen) + 0.6475 (LoRA baselines) + 1.2129 and 0.0402 sync (LTX-2.5) = 6.2089 USD at the ledger rates, against the approved 6.5 to 8 USD. The RunPod balance read 8.9172 USD at about 19:25Z, down 3.4790 USD from 12.3962 USD at 13:36Z, including volume storage (0.039 USD an hour) over those hours; as before, the ledger rates bill the dearest card each endpoint allows.

A.01.09 SAM 3 packs removed from the worker image (2026-09-14)#

Commit 440d66abc98 removed ComfyUI-SAM3 and ComfyUI-RMBG's vendored SAM 3 still node (py/AILab_SAM3Segment.py, models/sam3/) from docker/runpod/comfyui-slim, with pycocotools, the sam3 model folder, its entry symlink and extra-model-paths key, and the two SAM 3 manifest rows and their pinned family. Pushed to runpod-comfyui-slim at 19:54Z; RunPod build a350cf59-3409-4516-b43f-6011395ba314 completed at 20:15Z and both endpoints' templates now name image tag 440d66abc. The build's smoke passed, including its new check that none of SAM3Segment, LoadSAM3Model, SAM3VideoSegmentation, SAM3Propagate and SAM3VideoOutput registers. The object-info snapshot was recaptured from that build (1133 classes, sha256 debec3de373a…): the five classes are absent, SAM3_Detect, SAM3_VideoTrack, SAM3_TrackToMask, RMBG and BiRefNetRMBG are present, and the graph validator specs pass against it.

Live check on the rebuilt image endpoint: sam3-segment with "the woman" on the UnGloryHail still, job 270f413b-e4fc-4dc4-804a-e60fe224c149-e1 (183.7 s queue while the new image was pulled, 18.5 s execution, 0.0081 USD), outputs sam3-segment-0-16fa5a23.png 6906 B 16fa5a2373a5 (mask) and sam3-segment-1-cef297da.png 1487628 B cef297daedb3 (cut-out). The mask is identical to the earlier SAM 3.1 run's (sam3-segment-0-4018fa68.png, intersection over union 1.0).

A.05.02 video endpoint, LTX-2.5 default job and an operator API submission (2026-09-14)#

Approved by the user ("proceed on all") after the remainder closed: one LTX-2.5 clip at the workflow's own default (it could have passed the 900 s cap) and one A.03.04 internal-only Wan pair submitted as the operator tenant through the API rather than the harness.

Proof Item Workflow RunPod job Delay s Exec s USD Output (size, sha256)
ltx25-av-default-shape A.01.02 default job: t2v, 30 steps, 121x1280x704 ltx25-av 70664678-df65-44aa-83d7-4b46fa0a0192-e2 218.6 440.8 0.7259 ltx25-av-0-2c03af8e.mp4 1368785 B 2c03af8eb2b8

Check on the output: 121 frames of 1280 by 704 at 24 fps with AAC stereo 48 kHz; frames 0, 60 and 120 show a drummer at a kit on a club stage under spotlights, sticks moving, as prompted. The audio is loud (mean -11.4 dB, peak 0.0 dB, which may be clipping): a dense broadband wash with a regular low-frequency pulse in the spectrogram. The job took 440.8 s on a cold worker, half the cap; the first measurement's 23.7625 s per megapixel-frame (a 49-frame cold clip, mostly load time) had scaled this job to 2,591 s, so the catalog now carries 4.0423 s per megapixel-frame from this job.

A.02.02 Z-Image VRAM at 1024 and 2048 square (2026-09-15)#

On the same approval. The zimage family (20.88 GB, seven files) was synced into the cache through POST /api/v1/runpod/cache/sync {"families":["zimage"]} (sync 0ab86377), with no eviction. zimage-cyberrealistic was left out because it is still in Civitai early access. Both runs used runpod-catalog-run.mjs on the image endpoint: seed 4242, the base checkpoint, 25 steps, CFG 4. The 2048 run overrode the rendered latent with --graph-set 10.width=2048 --graph-set 10.height=2048, since the catalog offers no 2048 aspect. --capture-log kept each worker log.

Shape RunPod job Delay s Exec s Ledger USD Result
1024x1024 20f079cc-2c5e-404d-858e-bf0168b195a9-e2 102.6 150.8 0.0461 PNG 1 330 497 B ab725798e2cd56cd (the prompted teapot); sampling 25/25 in 1:03 at 2.56 s/it
2048x2048 e8575edb-6aea-47fb-8846-15d205c52e96-e2 1.1 304.8 0.0933 executionTimeout exceeded at step 23 of 25, 13.27 s/it

The worker log names the card and its memory: Device: cuda:0 NVIDIA L4, Total VRAM 22563 MB, with async weight offloading and pinned host memory enabled. At 1024 the text encoder loaded, the Lumina2 diffusion model loaded and sampled, and the VAE decoded with no out-of-memory line (Prompt executed in 145.35 seconds, cold). At 2048 the same warm worker sampled with no out-of-memory line until the image endpoint's 300 s execution cap cancelled it, 2 steps short of 25 and before the VAE decode. Each step took 5.2 times as long as at 1024.

What this measures and what it does not:

  • ComfyUI logs no peak allocation at INFO, and no "loaded partially" line appeared, so there is no peak-GB figure.
  • The finding is fit and time. 1024 square fits a 22.5 GB L4 comfortably. 2048 square did not run out of memory, but cannot finish inside the 300 s cap on that card: about 330 s of sampling plus a 2048 decode. The catalog offers no 2048 aspect, and this is the reason it should not without a larger card or a longer cap.
  • The 2048 run left no image to check.
  • Log lines Error loading … AILab_SegmentV2.py: No module named 'groundingdino' are an unrelated ComfyUI-RMBG import warning; the RMBG nodes the catalog uses load.

A.01.02 multishot shots[] with auto duration, through the API (2026-09-15)#

Approved by the user in this session ("Approve ≈1.0 USD" for a multishot clip and the A.02.02 VRAM stills). The duration head (model_patches/ltx-2.5-duration-head-bf16.safetensors, 3 843 690 B, sha256 2ec71e42…) was pulled into the Hetzner library by a hashed PUT and into the cache by POST /api/v1/runpod/cache/sync (one file, sync df7019d1, 20.5 s of loader pod, 0.000342 USD). The worker image was rebuilt (build 3acf2224, commit 83fafe101) so the snapshot's ModelPatchLoader options include it. Two earlier submissions were refused before any GPU time, and both refusals were right: the executor's "Prompt is required" (fixed so a shots array satisfies it) and the pre-submit validator's "not one of its 2 options" (fixed by the rebuild).

Proof Request RunPod job Delay s Exec s Ledger USD Output (size, sha256)
ltx25-av-multishot t2v, default schedule, 3 shots joined by hard cuts, auto_duration with length 121, 1280x704 442dd359-fd59-48cb-8dc8-adc4568590ac-e1 60.5 500.2 0.8238 isis/outputs/jobs/978264c4…/0.mp4 1 561 493 B 4d6acde37d607b88

The ledger bills the dearest card on the endpoint (0.001647 USD/s). The account balance read 5.85 USD before the session's RunPod work and 5.34 USD after this clip, including about three hours of volume storage, and RunPod settles late.

Check on the output: 121 frames of 1280 by 704 at 24 fps, with audio 5.01 s long, mean -14.3 dB and peak 0.0 dB. ffmpeg scene scores find cuts at 2.21 s and 3.83 s. A contact sheet at frames 10, 45, 60, 85 and 115 shows the three prompted shots in order: the wide club stage under red lights with the crowd, a close-up of the snare as the sticks come down, and the drummer in the black shirt grinning among cymbals. The shots were composed into one paragraph, and the model cut between them.

What it does not show: the predictor's own choice. The clip is exactly the 121-frame cap, so the predicted duration was at or above 5.04 s (the node clamps to max_seconds and floor-snaps to 8k+1). The worker log was captured, but its last 400 lines are all image-pull and container lifecycle lines, so the node's "predicted … frames" line is not in them. The predictor node ran inside the graph: it is the only source of the latent length and audio frame count in this render, and a failing node fails the job. Sizing below the cap is not demonstrated, and a second clip would have gone past the approved spend.

A.03.04 through the API as the operator. The local generation API (restarted at 20:49Z on the current commit) took POST /api/v1/jobs from the dev operator token (tenant tenant_isis_dev) for wan22-i2v with the internal-only nsfw-posing-nude-v2 pair, lightx2v 4-step, 33 frames at 480 by 832, seed 1494, the UnGloryHail still as start_image and presetContent: explicit. The first submission (29607a59-4a98-4bf6-8ce7-350f8a1f39cd) was refused before any GPU time because it carried no consentId, as the provenance ledger requires; the resubmission with the Studio's operator consent id (5aeb8e81-9c61-4ee0-a441-8de5d4ebe953, RunPod job f1cfda18-4c5b-419f-8da3-f8adf4666cd9-e2) completed: 491.2 s in the queue behind the LTX-2.5 job, 347.4 s of execution on a cold worker, 0.4837 USD on the video ledger. The model cache logged hits on wan22-i2v and wan-shared and released both leases; the output carries a provenance bundle (prov_73253336c383c8fc, consent id, prompt hash, model revision) and the watermark on its last-frame still. The clip (33 frames of 480 by 832 at 16 fps, 204358 B, sha256 b9bf91dff6e9a54f) is kept out of git; at frames 16 and 32 the subject is clothed, like the harness clip for the same pair.

A.03.04 diagnosis: why the explicit Wan pairs stayed clothed (2026-09-14)#

Asked for by the user ("i want that") after every explicit-pair clip, harness and API alike, kept the subject clothed. Five jobs on the posing pair nsfw-posing-nude-v2, all at LoRA strength 1, prompt "A woman posing in a greenhouse, natural light." Outputs with nudity and every output of this pair stay out of git; records and hashes only.

Proof What it isolates RunPod job Delay s Exec s USD Output (size, sha256)
wan22-i2v-lora-posing-quality full 20-step schedule, CFG 3.5, no lightx2v, 33 fr f961a792-813a-4218-baba-34a99f2df52c-e1 74.7 508.2 0.8370 mp4 154168 B 76286c167acb9d9e; png 553725 B 86a903784fbbc227
wan22-i2v-lora-posing-81f lightx2v 4-step at 81 frames (the trained length) ab4911c7-fcda-42f8-ab94-8e52328f3347-e2 0.1 69.7 0.1148 mp4 479593 B f6afe43bd00ef7f2; png 545591 B b7fb8bb75fd70c30
chroma-artistic-nude-start an artistic-nude start still (UnGloryHail, seed 1402) 570d69cd-fa61-4bfd-80b9-924d4981dbf3-e1 7.4 140.6 0.0617 png 1558704 B bbd82323fba4a8c6
wan22-i2v-lora-posing-nude-start the pair from that still, 4-step, 33 fr, seed 1495 b5e76dca-5b23-4bdd-91d5-7481e3dcd3d5-e2 267.3 370.1 0.6096 mp4 237147 B 739e850687f20bf0; png 612233 B a1089e9e83321c55
wan22-i2v-baseline-nude-start the same without the pair 08b06a57-ec33-4321-a8f7-26d1fa9e20b0-e2 54.0 381.2 0.6278 mp4 277506 B 5961746db79dd4ab; png 617620 B 8fa20230940d66eb

Total: 2.2509 USD, above the 1 to 2 USD quoted, because the last two jobs each landed on a cold worker and reloaded both experts.

Findings:

  • The pair loads completely. The worker log of the first job (327 lines, streamed while it ran, kept beside the outputs) has no lora key not loaded line, which ComfyUI prints for every LoRA key that matches nothing; the files are in the kohya/musubi lora_unet_blocks_N_* format, 40 blocks, rank 32, which ComfyUI maps onto the Wan model.
  • Neither the schedule nor the length is the cause: on the full 20-step schedule and at 81 frames the subject stays clothed from the clothed start still, as in the 4-step 33-frame clips.
  • The start frame is the cause. Image-to-video holds the first frame's clothing; nothing in these runs removes it, with or without the pair.
  • From the artistic-nude start still, both clips animate the figure naturally (turning from the back view to face the camera, weight shifting, a hand raised), with no visible anatomical artefacts at frames 0, 16 and 32. The clip with the pair and the one without differ by 23.0 dB PSNR and move by about the same amount (mean frame-to-frame luma change 2.00 against 2.56); the pair's posing is not recognisably different from what the prompt alone produces.

Conclusion: the explicit pairs work mechanically and change the render, but no clip shows a result the pair adds over the base model: they cannot turn a clothed start into a nude one, and from a nude start the base model already poses the figure. A.03.04 is reopened on that basis.

A.05.03 Retirements executed (2026-09-14)#

User instruction on 2026-09-14: "proceed on all", in answer to deleting the retired files from the Hetzner library and the cache volume. Each replacement had rendered first (A.01.02/A.01.03 LTX-2.5, A.01.06 Animate-2, A.01.08 SeedVR2 3B, A.01.09 SAM 3.1 on core nodes, A.02.07 Depth Anything 3), and each file had already left the catalog, the manifest and the register in its own commit. Deleted at 20:40 to 20:44Z by exact key (a scratch script listing each manifest destPath and refusing any other key than the file, its .sha256 sidecar or a .partial leftover), after a dry run that listed the same keys and sizes. The first run hit a Hetzner GatewayTimeout after three deletions; the rerun with retries finished, and a final listing of every key returned nothing. The cache database's realesrgan, sam3 and rmbg-2.0 families, which still read cached, were moved to absent with evicted events through the store's own eviction methods.

Retired row Item Library key under isis/models/ Bytes Cache copy deleted
ltx23-22b-dev A.01.04 checkpoints/ltx-2.3-22b-dev.safetensors 46,149,344,974 none
gemma3-12b-it-bf16 A.01.04 text_encoders/gemma_3_12B_it.safetensors 24,379,468,890 a .partial of 8,724,152,320 B from an old pull
ltx23-distilled-lora-1.1 A.01.04 loras/ltx-2.3-22b-distilled-lora-384-1.1.safetensors 7,605,507,256 none
ltx23-spatial-x2 A.01.04 latent_upscale_models/ltx-2.3-spatial-upscaler-x2-1.1.safetensors 995,743,560 none
ltx23-ic-lora-union-control A.01.04 loras/ltx-2.3-22b-ic-lora-union-control-ref0.5.safetensors 654,465,352 none
ltx23-ic-lora-motion-track A.01.04 loras/ltx-2.3-22b-ic-lora-motion-track-control-ref0.5.safetensors 327,309,314 none
ltx23-temporal-x2 A.01.04 latent_upscale_models/ltx-2.3-temporal-upscaler-x2-1.0.safetensors 261,944,000 none
wan22-animate-bf16 A.01.06 diffusion_models/wan2.2_animate_14B_bf16.safetensors 34,549,787,368 none
wan22-animate-relight-lora A.01.06 loras/wan2.2_animate_14B_relight_lora_bf16.safetensors 1,436,673,432 none
sam3 A.01.09 sam3/sam3.pt 3,450,062,241 the file and its sidecar
sam3-safetensors A.01.09 sam3/sam3.safetensors 3,449,860,632 the file and its sidecar
depth-anything-v2-vitl A.02.07 controlnet_aux/depth-anything/Depth-Anything-V2-Large/depth_anything_v2_vitl.pth 1,341,395,338 none
realesrgan-x4plus A.01.08 upscale_models/RealESRGAN_x4plus.pth 67,040,989 the file and its sidecar

Library bytes deleted: 124,668,603,346 (124.67 GB); cache bytes deleted: 15,691,116,377 (15.69 GB, 6.97 GB of retired weights and the 8.72 GB partial). RMBG-2.0 (A.02.08) was deleted earlier the same day (above).

Before and after:

Measure Morning of 2026-09-14 After the retirements
Manifest rows 111 (825.06 GB), 31 families, 11 pinned 98 (700.39 GB), 27 families, 9 pinned
Pinned closure 246.00 GB 239.03 GB
Hetzner library 117 objects, 813.38 GB 100 objects, 687.82 GB
Cache volume (400 GB) varied with the proof syncs 136 objects, 331.22 GB (pinned core + LTX-2.5)

Lint: runpod-volume-manifest-lint.mjs --strict 0 errors, 0 warnings, and the cache invariant holds (239.03 + 92.19 = 331.22 GB of 380 GB allowed). What stays in the library although no catalog workflow requires it: the Animate-2 base transformer and the Gemma 4 E4B prompt-refine encoder (user decisions recorded under A.01.06 and A.02.03), and the YOLOv10m and ViTPose pose models only Animate-1 loaded (1.30 GB, not asked about).

C.12.01 — four Chroma LoRAs, rendered (2026-09-13)#

User picks, all CivitAI with every commercial flag: Lenovo UltraReal (1662740 / 2299345 / file 2190290, trigger l3n0v0), Chroma Realistic Slider (1995853 / 2259125 / 2151331), Chroma Defluxer (2204954 / 2482626 / 2371171), Style > Muk (1944975 / 2715471 / 2601052). Downloaded with CIVITAI_API_KEY, sha256 checked against CivitAI's published hash, uploaded to loras/ over the volume S3 API.

LoRA Job Exec USD Output
lenovo-ultrareal 74b3a8e1-2477-454d-99ab-79e3b3ab7f07-e1 (re-run) 211.3 s $0.0927 chroma-txt2img-0-8d63d12f.png 1024×1024
chroma-realistic-slider 8dadc8a6-562c-4bde-935d-815eedcab3f0-e2 139.6 s $0.0613 …-9ed5aeb4.png 1024×1024
chroma-defluxer e7674db2-863d-488b-ada9-4d6bffa78987-e1 28.4 s $0.0125 …-4f2d611f.png 1024×1024
style-muk ed8c8307-318f-4982-9bf0-3c56daaff364-e1 29.9 s $0.0131 …-fd6106f6.png 1024×1024

Same prompt, chroma1-hd, seed 1213, strength 1.0, worker 1u0p43eprdxnmz. First Lenovo attempt c4e059d5-…-e2 FAILED in validation (value_not_in_list for lora_name, listing the LoRA files present) because its upload had not landed — the multipart completion had failed (InvalidPart: part 1 ETag mismatch) and a single PUT then succeeded. Pairwise mean |Δ| 26.5–44.1 levels, 40–64 % of pixels differing by more than 32. Limits: single-LoRA enum, no stacking; Muk's illustration style not visible on a photographic prompt at 1.0.

LTX weights via a reduced-manifest loader pod (2026-09-13)#

Three rows (distilled LoRA 1.1, union-control, motion-track) pulled by pod create-loader --manifest <reduced>; .oshun/manifest-result.json → finishedAt 09:47:01Z, failures 0, each row status ok with the manifest sha256 (f5d4953f…, a1b888a8…, e279807e…) and exact byte counts (7 605 507 256, 654 465 352, 327 309 314). Pod 8tsjt6bhvoe3e9 deleted; spend back to $0.049/h. Note: that result file now describes only these three rows. The local HTTP downloads of the same files kept dropping (HTTP/2 stream was not closed cleanly); the pod pulled 8.6 GB in about two minutes.

Worker uploader: the start-up hang, found and fixed (2026-09-13)#

Build fd960d7c hit RunPod's 30-minute build limit inside the smoke step with nothing in the log. Reproduced locally (python 3.12, runpod 1.7.13, the module installed through a .pth hook): with BUCKET_ENDPOINT_URL set the interpreter never finished starting; faulthandler showed runpod/serverless/utils/rp_debugger.py:20 → cpuinfo.get_cpu_info() → subprocess.communicate, i.e. the SDK import starts a child python which processed the same hook and recursed. The hook now registers a meta-path finder that patches rp_upload.upload_image when the handler imports it; start-up 1.1 s with and without the env. A real round trip through the patched call against Hetzner (app-dev-server, isis/outputs/_selftest-job/20cbb23d.txt, 46 B) returned identical bytes through the presigned URL and was deleted (GET then 404). The smoke step is now tee'd to the build log and capped at 25 minutes. An earlier attempt with sitecustomize.py failed the install check because Ubuntu's python3.12 ships its own sitecustomize ahead of site-packages (build ea83e467).

L.00 Model library and cache — measurement#

Option B (user decision 2026-09-13). Every pod run records wall clock, pod id, cost and the loader's result file.

Hetzner multipart and the library modes, live from the Mac (2026-09-13, $0)#

Loader modes against app-dev-server under the throwaway prefix isis/models/_livetest/, parts forced small (16 MiB, single-PUT ceiling 20 MiB) so three of four files took the multipart path: library-seed 4/4 ok (Lenovo 112 125 552 B in 7 parts, Slider 28 074 280 B in 2, Muk 74 820 484 B in 5, Defluxer 6 816 088 B single PUT); a second library-seed skipped all four (size and x-amz-meta-sha256 both matched) with no PUT or POST; library-verify re-hashed every byte from the bucket, 4/4 ok; cache-sync into an empty directory pulled all four through .partial, verified and renamed them, 0 failures. All _livetest/ objects deleted (listing of isis/ then empty). Rates of 1–3 MB/s are this home connection and say nothing about the pod path.

L.00.01 transfer probe#

Pod b1k0jr7fdbvror (CPU cpu3c, 2 vCPU, EU-RO-1, ax0oj250j3 mounted, $0.06/h), 13:48Z to 14:42:02Z, wall 3237.3 s, about $0.054, deleted after the read. Result file .oshun/library-probe-result.json, 0 failures, row chroma1-hd (17 800 038 288 B, 266 parts of 64 MiB):

Leg Seconds MiB/s Check
Up, single stream 709.0 23.9 multipart completed
Up, 8 parts 449.0 37.8 multipart completed
Down, single stream 1259.1 13.5 sha256 matches
Down, 8 parts 816.5 20.8 sha256 matches

Full-object checksum: a deliberately wrong x-amz-checksum-sha256 and the right one were both accepted, so Hetzner does not validate full-object sha256 and L.01.05's download-side verification stays in full scope. No probe objects left behind.

L.01 The library on Hetzner#

L.01.03 seed, four launches (2026-09-13)#

  1. Pod 2nz4kgp6emde9p, --parallel 16, 14:43:26Z. The first row's multipart reached 236 of 266 parts by 14:48:04Z at 60.6 MB/s, then nothing for 16 minutes: parts 233 and 236 never landed while 237 and 238 did. The loader's socket timeout was 600 s with 4 retries, so one silent connection could hold the ordered part queue for up to 50 minutes. Container telemetry read 0 % CPU. Deleted 15:05Z (HTTP 204); its open upload aborted (204, ListMultipartUploads then empty).
  2. Pod xws2eg6x6tmkw2, --parallel 8, stall timeout 120 s, 15:06:40Z. Stalls recurred about one part in fifteen (part 44, then 59), each costing the full 120 s: 15.8 MB/s average. Deleted 15:12Z; upload aborted.
  3. Pod 75ml7zmu2t2q1z, --parallel 8, stall timeout 30 s (commit a94f2832105), 15:12:20Z. Ran 35 to 96 MB/s until 16:05:02Z, then stopped reading wan2.2_animate_14B_bf16.safetensors (34.55 GB) from the volume after 504 of 515 parts: 40 minutes at 0 % CPU and 1 % memory, no parts pending, no error. Deleted 16:46Z; upload aborted.
  4. Pod wkd9odg71h3tls, same settings, 16:46:25Z to 17:27:45Z, wall 2478 s. Result file .oshun/library-seed-result.json: 50 rows ok, 0 failures. Library listing: 50 objects, 367.17 GB, every size and every x-amz-meta-sha256 matching the manifest. Deleted 17:28Z.

Pod time across the four launches was about 2.7 hours, about 0.16 USD. The connection stalls are on Hetzner's side of individual connections (other parts on other connections kept landing, and a fresh connection always succeeded). The volume-read hang led to the loader watchdog (111a49e7518).

L.02 Families and the pinned core#

Manifest families (15, 9 pinned) and library/cache blocks committed in 28083a4c432; lint prints cache 400 GB: pinned 212.26 GB + largest unpinned ltx23 80.37 GB = 292.64 GB of 380 GB allowed. requires_families on all 17 workflows and MODEL_NOT_IN_FAMILIES in the static validator committed in 02be697dd61 (sweep over defaults, every enum value and both booleans; library suite 694 passed).

L.03 Cache state, leases and eviction#

Service apps/isis/generation-api/src/services/runpod-model-cache/ committed in 9b31b96481a: 35 specs, the store and race specs against real Postgres (forced lease-versus-eviction race in both orders, and 25 concurrent rounds with exactly one winner each).

L.04 Sync executor#

Service and routes committed in 0fe7d06ba70 (L.04.02, L.04.03). Every loader pod of 2026-09-13, CPU cpu3c in EU-RO-1 at 0.06 USD per hour (cost is pod time at that rate, or the API's booked cache_sync line where it drove the pod). Result files are copied to docs/agents/evidence/isis-chroma-runpod/l-results/.

Pod Mode, run Window (UTC) Wall Cost USD Result file, outcome
b1k0jr7fdbvror library-probe (L.00.01) 13:48 to 14:42 3237 s 0.054 library-probe-result.json, 0 failures
2nz4kgp6emde9p library-seed, parallel 16 14:43 to 15:05 hung 0.022 none; deleted, upload aborted
xws2eg6x6tmkw2 library-seed, 120 s stall 15:06 to 15:12 stalls 0.006 none; deleted, upload aborted
75ml7zmu2t2q1z library-seed, 30 s stall 15:12 to 16:46 volume read hung 0.094 none; deleted, upload aborted
wkd9odg71h3tls library-seed 16:46 to 17:28 2478 s 0.042 library-seed-result.json, 50 ok
ktbpl2bdda9vgf library-verify, 16 non-core rows 17:31 to 18:08 2245 s 0.037 library-verify-result.json, 14 ok, 2 timeouts
9t25i5c9aoe0j6 library-verify retry, 2 rows 18:09 to 18:41 1080 s run 0.032 library-verify-retry-result.json, Gemma ok, VACE timeout
uzcxhv22fkl5na cache-sync 6e9cc338, pinned core 17:37 to 18:40 3631 s 0.063 booked cache-sync-core-result.json, 31 of 34; reconcile hit a 524
1q4axik9vvhuv6 cache-sync 7620b385, core remainder 18:46 to 19:01 739 s 0.015 booked cache-sync-core-followup-result.json, 6 of 6
gr5anr0ft8vc5r cache-sync eb61d6df, VACE miss 19:10 to 19:35 1369 s 0.024 booked cache-sync-vace-result.json, 4 of 4
tcjz2pz3tav1je cache-sync 015b7753, rmbg-2.0 20:03 to 20:04 about 25 s 0.0004 booked succeeded; file overwritten by the next sync before it was copied
904j331z943wbd cache-sync 7b013797, eviction proof 20:05 to 20:46 eviction done, pull stalled 0.046 booked none; pod deleted, sync recorded failed by takeover
z3uhi38hc9hx76 cache-sync 4d053b49, LTX-2.3 resume 20:52 to 22:55 Gemma pull froze at 22:13 0.123 booked none; 3 of 7 files landed, pod deleted, sync failed by takeover

L.04 defect found in use: a live sync taken over (2026-09-14)#

Cache sync b80de934 (the Qwen-Image edit families, 106.33 GB) spent minutes in its applying phase listing a 361 GB volume. The driver refreshed its heartbeat only in its result-polling loop, so the heartbeat went stale; the cache status route calls resumeStaleSync(), and the next status read at 16:00:27Z claimed the sync and launched a second driver in the same process. The second driver tried to complete an eviction the first had finished ("cannot complete eviction of wan22-animate2 ... it is not evicting") and recorded the sync failed with 0.112043 USD booked; the first then finished it as succeeded and booked 0.113316 USD, so the pod's cost was booked twice. The families were cached correctly.

Fix: a running driver refreshes its heartbeat on a timer in every phase (touchSyncHeartbeat), and resumeStaleSync() never takes over a sync this process is still driving. Specs: "never takes over a sync its own process is still driving, so it is finalized once" and "keeps a running driver heart-beating in every phase", both shown to fail with the fix removed. The sync spec had also been failing unnoticed since L.04 moved the plan to PLAN_GZ_B64 (its fake pod still decoded PLAN_B64, and the spec only runs with ISIS_TEST_DATABASE_URL set); the harness now decodes the compressed plan. ISIS_TEST_DATABASE_URL=postgresql://oshun:oshun_dev@127.0.0.1:5432/oshun_dev npx vitest run src/services/runpod-model-cache → 60 passed.

L.05 Dashboard#

Cache panel committed in 52d7fd1fbb9. Chromium apps/isis/web/e2e/runpod-model-cache-panel.spec.ts 2/2 with the API mocked at the network boundary (re-run by the reviewing session, not only the building agent); view logic 7 tests, client 3, API/client/OpenAPI projection 11, RunPodPage.spec.tsx 31. No listener left on 3010, 3020, 4010 or 5180.

L.06 Migration of the volumes#

L.06.01 cache volume#

Balance first (GraphQL myself, 15:26Z): clientBalance $17.94, currentSpendPerHr $0.132, spendLimit $80. volume create --dry-run showed {name: isis-model-cache, size: 400, dataCenterId: EU-RO-1}; the real call at 15:26:33Z returned id 5bst2pgw7y. Storage is now $71.20 + $28.00 a month until L.06.05.

L.06.02 to L.06.05 (2026-09-13)#

  • Core sync. Sync 6e9cc338, pod uzcxhv22fkl5na, 17:37 to 18:37:53Z: 31 of 34 files, 3 lost to ranged reads that stalled five times on one range, then the API's reconcile failed on a 1000-key volume listing (Cloudflare 524). Follow-up sync 7620b385, pod 1q4axik9vvhuv6, wall 739 s: the 3 files plus the 3 BiRefNet companions, 0 failures. Booked cost 0.063 and 0.015 USD. The API then reads the 9 pinned families cached, 212.26 GB used, 0 unmanaged.
  • Repoint. Console edit on both endpoints, 18:12Z; GraphQL and the drift checker read networkVolumeId 5bst2pgw7y.
  • Deletion. ax0oj250j3 deleted 18:13:21Z on the user's explicit go; currentSpendPerHr 0.222 before, 0.162 after with the same two CPU pods, and 0.039 once the pods ended (the 400 GB volume alone).
  • Verification (L.01.05). First pass over the 16 non-core rows: 14 ok, VACE and Gemma 3 hit read timeouts while three transfers shared Hetzner. Retry pod 9t25i5c9aoe0j6: Gemma 3 ok (24.38 GB at 25.8 MiB/s), VACE timed out again; VACE is verified by its cache sync below.

L.06.04 proofs on the cache volume#

Proof Job Result
Model-free probe RunPod 224dcfc6, worker uv0s0976i8isy9 delay 121.1 s, execution 8.9 s, output s3_url on Hetzner
Chroma still (first) Isis f792b235 rendered, but unwatermarked: the still came back as a signed URL and three mappings dropped requiresSigning
Chroma still (after 6e3691ad7b9) Isis 9f0b2633 1024x1024, execution 177.1 s, 0.098 USD, watermark.applied, payload decoded with 512 of 512 copies intact
Wan 2.2 I2V, 4-step, 33 frames Isis b964e556 832x480, h264, 16 fps, 33 frames, execution 323.5 s, 0.450 USD; last-frame still watermarked

Frames of the clip and the watermarked still are in docs/agents/evidence/isis-chroma-runpod/l0604/. The VACE miss, sync and render cycle, and the forced eviction, follow.

VACE miss. Submitting wan21-vace-edit at 19:10:25Z answered HTTP 409 models_not_cached: families controlnet-aux (1.69 GB) and wan21-vace (46.30 GB, of which 36.37 GB to pull because umt5 is shared with the cached wan-shared), an estimated wait, and the exact sync body {"workflows": ["wan21-vace-edit"]}. That sync (eb61d6df, pod gr5anr0ft8vc5r) started at 19:10:37Z.

VACE sync and render. Sync eb61d6df finished 19:35:03Z, state succeeded, 0.024 USD, wall 1368.7 s: wan2.1_vace_14B_fp16.safetensors (34.68 GB at 24.9 MiB/s) and the three controlnet_aux files, all hashed ok. The resubmit first failed with HTTP 500: the route accepted type: video-to-video, but the live jobs_type_check constraint lists 18 types while the code's ISIS_GENERATION_TYPES has more, so the insert was refused. That is a pre-existing defect outside this section (recorded, not fixed here). Submitted as text-to-video (what the dashboard's motion presets send), job 7bcfc7ea ran 19:35:57 to 20:02:42Z: 832x480, h264, 16 fps, 33 frames, execution 344.8 s, 0.480 USD, queue delay 1254.7 s waiting for a card. Frames 0, 16 and 32 follow the source clip's boat and grate with the restyled palette (l0604/wan21-vace-frames-0-16-32.png). Miss to render: 52 min, of which 24.4 sync, about 21 GPU queue and 5.7 render.

Forced eviction, set-up. The whole library (367 GB) fits the real cache's 380 GB usable, so no real request can force an eviction. The API was restarted with ISIS_RUNPOD_VOLUME_MANIFEST_FILE pointing at a proof-only copy of the manifest with cache.sizeGb 320 (usable 304 GB; the lint invariant still holds, pinned 212.26 plus ltx23 80.37 = 292.64). Before, with the real manifest: used 249.52 GB; unpinned cached families wan21-vace (last used 20:02:42Z, the VACE render), controlnet-aux (same) and rmbg-2.0 (synced 20:04:09Z by a 30 s, 0.0004 USD sync done for this proof). Dry run of {"workflows": ["ltx23-av"]} (needs ltx23 and controlnet-aux): evict wan21-vace, 34.68 GB in one file (umt5 stays, wan-shared is cached and pinned), pull ltx23 80.37 GB. Check: 249.52 - 34.68 + 80.37 = 295.21 GB fits 304; without the eviction 329.89 does not. rmbg-2.0, touched later, is kept. The real sync 7b013797 (pod 904j331z943wbd) started 20:05:13Z.

Forced eviction, result. The loader evicts before it pulls, and the pull was slow (Hetzner bursts and stalls; a watchdog rerun would also have restarted the 46 GB checkpoint from zero, fixed in e0c6d74442e), so after the eviction had landed the pod was deleted at 20:46Z and the API restarted with a 60 s sync timeout: it took the sync over at 20:51:15Z and recorded it failed ("no result file 60 s after the pod was created"), 0.046 USD booked. Restarted on the real 400 GB manifest, the API reads: wan21-vace absent; controlnet-aux and rmbg-2.0 cached; used 228.80 GB of which 13.96 GB is the checkpoint's .partial, so managed bytes 214.84 GB = 249.52 - 34.68, exactly what the plan said it would free. A fresh ltx23-av sync (4d053b49, pod z3uhi38hc9hx76, 20:52:17Z, no evictions at 400 GB) resumes the partial.

That sync landed the 46.15 GB checkpoint and both latent upscalers by 21:55Z (the resume worked: the checkpoint finished from its partial), then stopped on the Gemma 3 encoder. Its .partial sat at 8,724,152,320 bytes, exactly 130 whole 64 MiB parts, from 22:13:52Z: the shape of a watchdog rerun cutting the file back to whole parts and then fetching nothing. Forty minutes later (four watchdog periods, and past the start loop's five attempts) it had not moved, so at 22:54Z the API was restarted with a 60 s sync timeout, took the sync over, recorded it failed ("no result file 60 s after the pod was created", 0.123 USD booked) and deleted the pod; the API then restarted on its normal settings. The likely cause is contention: the arsenal library pull (ed3tappadomybi, from 21:51Z) was uploading to the same Hetzner bucket at about 26 MB/s the whole time, and Hetzner stalls connections under contention (L.00). The LTX-2.3 sync is to be relaunched after that pull finishes; the partial and the three landed files stay on the volume for it to resume from.

L.07 Operation#

L.07.02 outputs retention and the usage alarm#

Commit dfbdea14099. Live, 2026-09-13:

  • 15:02Z, CLI from the Mac: hetzner-usage listed one bucket (app-dev-server, 0 objects); hetzner-sweep --days 7 dry run, 0 outputs.
  • 15:20:55Z, the API guard's own runtime loader (loadHetznerStorageOps) with defaults: dry-run sweep of isis/outputs/ (0 deleted, 0 kept); usage 17 800 038 288 B in 1 object, all under isis/models/, not over 900 GB.
  • Same live reading with the line set to 1 byte: the first check raised storage.hetzner.usage_high (payload date 2026-09-13, usedRatio as measured), the second check the same day logged the error line but sent no second webhook.

Found on the way: budget-alarm webhooks (C.04.06) read url from store records that call it endpointUrl; fixed for both alarm paths with a regression spec.

H.00 The hosted OpenRouter lane — policy and terms (2026-09-15)#

The lane exists because the owner asked for the Meshy treatment to be extended: "openrouter has a number of image and video models can we do the same as we are doing with meshy … and only use runpod stuff when we want / need to?".

The gate that had to be opened first. The Eve SOTA provider/subprocessor review (docs/audits/eve-sota-provider-subprocessor-review/) marks every hosted media route blocked-unattested, OpenRouter image and video among them. The owner chose "Attest Meshy + OpenRouter media" on 2026-09-15; the record is owner-attestations/2026-09-15-isis-hosted-media.json (operator only, operator-owned non-personal SFW inputs, vendor posture accepted as published). Regenerate and verify with node tools/eve-everywhere/generate-provider-subprocessor-review.mjs and its verify- twin after touching any bound source file; the node --test suite carries 58 semantic mutations.

What is offered, and what is not. libs/isis/workflows/src/hosted-media/openrouter-model-register.ts holds one row per model with the vendor terms that were read and the date they were read.

  • Offered: the FLUX.2 family — klein 4B, flex, pro — for txt2img and img-edit, under Black Forest Labs' terms of 2026-08-01.
  • Excluded, each with the reason on the row: OpenAI (terms answered 403), Alibaba Wan (Model Studio terms answered 404, and Article 4.48 is not served), Krea 2 and MiniMax H3 (already excluded for their weights), Google's image and video models (terms read, but the owner's proof approval excludes frontier vendors, so offering them needs a separate owner decision).
  • No video model is offered at all. OPENROUTER_VIDEO_BLOCKER states it: no non-frontier vendor's video terms could be read. A video task is refused naming that blocker rather than half-served.

Tenant exposure (H.00.02). Asked and answered the same day: "Operator only for now". OPENROUTER_LANE_TENANT_EXPOSURE carries the decision, its date and its source, and a non-operator sees an empty catalog and is refused at submit.

H.00.01 closed 2026-09-16 — the terms are fingerprinted and the two verdicts are fields#

The register was built on 2026-09-15 and left open for three things. The first (the catalog offering more than the register admits) was gone by 2026-09-16. The other two are done here.

Territory and training are fields now, not prose. OpenRouterVendorTerms gained unitedKingdom and trainsOnInputs, each a verdict plus the clause it rests on, and openRouterRegisterProblems() refuses to offer a model whose UK verdict is anything but served or not_excluded. The verdicts, read from the documents:

  • Black Forest Labs — UK not_excluded (12.1 Export Laws is the only territory clause: embargoed countries and restricted-parties lists, no served-territory list). Training opt_out_held_by_openrouter: 1.2(e) licenses Input, Output and Tasks to "train and improve our technologies", opt-out by writing to legal@blackforestlabs.ai — and that opt-out belongs to Black Forest Labs' customer, which is OpenRouter.
  • Google — UK conditional, which is not offerable: "You may use only Paid Services when making API Clients available to users in the European Economic Area, Switzerland, or the United Kingdom", and Isis cannot see which tier OpenRouter buys. Training depends_on_tier for the same reason. This is now a second, independent reason the Google rows are excluded — until today the exclusion rested only on the owner's frontier-vendor decision.

Two clauses the 2026-09-15 reading had missed, found by re-reading the BFL document in full and added to its findings:

  • 1.3(n) forbids using Output "to train, distill or fine-tune any other AI models". A hosted still may not become training data — which touches E.01.07 (the body-physics LoRA) directly.
  • 1.3(p) forbids removing or obscuring "any AI content marking or labelling or transparency metadata". The lane's watermark and C2PA stages are a vendor requirement, not only Isis policy.

The fingerprint, and why it is over text and not bytes. Measured 2026-09-16 by fetching each document twice within a minute:

document bytes raw sha256 across two fetches text sha256
bfl.ai/legal/terms-of-service 321 843 identical e98d98402c3ad246…
ai.google.dev/gemini-api/terms 111 409 different (3 149 bytes, all inside per-request CSP nonce="…") 7395f597b16af2ed… identical

So a raw-body hash would report drift on a document nobody touched. The drift signal is textSha256 under the named normalization isis-terms-text-v1 (terms-document.ts: drop <script>, <style>, <noscript>, comments and tags, decode the entities a legal page uses, collapse whitespace; refuse an extraction under 2 000 characters rather than hash a consent wall). rawSha256 is still recorded with a measured rawStable flag — it pins the exact bytes that were read without pretending they are comparable.

The check. tsx libs/isis/workflows/scripts/openrouter-terms-drift.ts:

  • --check (offline, and now a dependsOn of the lib's lint target via hosted-terms:check) runs the register's rules and re-hashes the checked-in normalized copies under docs/agents/evidence/isis-chroma-runpod/h0001/, proving the recorded hash is the hash of the recorded reading. Verified to fail: tampering one clause in the BFL evidence copy gave evidence copy hashes 788c189206e180d4, row records e98d98402c3ad246 — one of them is stale and exit 1; restoring it gave exit 0.
  • --check-remote re-fetches and compares. Run live 2026-09-16: Black Forest Labs: unchanged since 2026-09-15 and Google (Gemini API terms): cosmetic only — body 111409 -> 111409 bytes, sha256 8c70ed744366c53f -> 4f82168dfe6225ef, same text — a third distinct raw hash, same prose, correctly not an alarm.
  • --write prints the digest literals and refuses to edit the register: a new hash without a fresh reading of the clauses would be a machine silently re-approving terms a person never read.

What it cannot do, written down because the approximation is real: it cannot tell a reworded clause from a reformatted page, and a vendor that renders its terms client-side would extract near-nothing. Both end in the same place — a person re-reads the document.

Specs: 12 in terms-document.spec.ts (normalization, entity ordering, rawStable measured rather than assumed, and the unchanged/cosmetic/changed/ incomparable verdicts), 14 new in openrouter-model-register.spec.ts (11 → 25), seven of them negative cases: a conditional UK verdict refused, an excluded one refused, a verdict with no clause behind it, a bad digest, a document too short to be an agreement, an evidence path outside the tree, and two rows quoting one URL with different hashes. 59 pass in src/hosted-media/.

H.01 One OpenRouter media client (2026-09-15, moved 2026-09-16)#

libs/isis/ai-providers/src/providers/openrouter-media/ is the only place in the repository that knows OpenRouter's media wire format: image create (buffered and SSE), video create / get / wait / content download, both catalogues with per-endpoint pricing, one typed error per documented failure, and the webhook signature check (raw body, five-minute replay window in both directions, constant-time compare, @noble/hashes because node:crypto is banned in client-transpiled libs). The key is injected through @oshun/config; a grep for process.env over the production files is empty. 38 client specs, 47 support specs.

openrouter-media-surface.snapshot.json (fetched 2026-09-15) is the checked-in catalogue. scripts/openrouter-media-surface.ts --check compares the client and the register against it offline; --check-remote re-fetches and names added, removed and repriced models. The negative cases are real: a named model or parameter missing from the snapshot fails, a parameter the client and the catalogue describe differently fails in both directions, and a route the client calls that the snapshot does not list fails.

The second copy is gone (H.01.03). Oshun's BFF carried its own OpenRouter video provider — its own poll loop, its own resolution and aspect-ratio tables, its own HTTP error strings. On 2026-09-16 it moved onto this client and became an adapter (364 → 271 lines). The move was possible because the boundary rule already lets scope:oshun depend on scope:isis and the BFF already declared @isis/ai-providers. The route gains three properties it did not have: the bearer key reaches a content URL only when that URL is on the API host, failures arrive typed, and a transient poll failure no longer abandons a job that is running and billing. Its admission is unchanged — it stays behind isEveExtendedProviderAdmitted('video-openrouter'), and sharing a transport grants it nothing. Specs: 16 provider, 3 hero-gate, 12 executor, 20 provider-env, 85 openrouter-media; tsc --noEmit clean on both projects.

Using the client here taught it two things, both now in the shared code: an optional poll backoff (pollBackoffFactor / maxPollIntervalMs, defaulting to the documented fixed cadence) so a five-second clip is seen quickly without polling a long job hard for its whole deadline, and describeJobError, so a failed job's reason reaches the operator instead of [object Object].

H.06.01 — the OpenRouter lane, live (2026-09-16)#

On the owner's approval of ~0.6 USD (asked and answered 2026-09-15: "Approve both (~1.1 USD)", cheap models only) and under the 2026-09-15 hosted-media attestation. Every job went through POST /api/v1/jobs on the local generation API (port 4111, inline runner) as the operator tenant tenant_isis_dev with a minted dev JWT, the Valkey ledger and the local outputs store. The daily budget was pinned to ISIS_OPENROUTER_DAILY_BUDGET_USD=0.6 so the approval was enforced by the API, not only by intent.

Spend: 0.043 USD of the 0.6 approved, read live from the submit headers after the runs: x-openrouter-daily-spend-usd: 0.043, x-openrouter-daily-remaining-usd: 0.557, key isis:openrouter:cost:2026-09-16.

Group Job Model Actual Estimate Wall s Output
txt2img 05bfced8 flux.2-klein-4b 0.014 0.01468 6.9 PNG 1024×1024, 1 485 838 B, sha256 3d6bdf6cd20694dd…, bundle prov_6533ce5e6b9314b2
img-edit 1d8db4d0 flux.2-klein-4b 0.015 0.01468 16.0 PNG 1024×1024, 1 411 737 B, sha256 22ff1732433075a6…, bundle prov_c83ff3282c7414ff, one material input

The edit was given the first image as its only reference (operator-owned, the lane's own earlier output) and honoured it: same table, framing, window light, notebook and lamp base; the shade is deep green enamel and a potted fern stands behind the notebook, which is what the prompt asked for and nothing else.

The two video groups the item asks for could not run and were not faked. The register offers no video model (H.00.01), and the lane says so live: POST /api/v1/jobs with openrouter:t2v:alibaba/wan-3.0 answered 422 hosted_lane_video_unavailable, naming the blocker — Alibaba's terms 404, ByteDance and Kuaishou unread, BFL video at 0.17 USD/s, Google excluded by the owner's frontier rule.

Also proved live at no cost: a racy content rating on a hosted request is refused 422 before anything is created, and the budget headers ride on the refusal.

Two defects the live run found, both fixed#

  1. The first job was created, billed and then failed to complete. Job d5e98732 spent 0.014 USD, stored its PNG, and then persistGenerationProvenance refused the bundle: "Cannot complete job … without provenance modelVersion". The lane had set modelVersion to the model's canonicalSlug, which OpenRouter publishes only for video models; an image row carries no version at all. The lane now uses the model id — flux.2-klein-4b names the family and the size — and records modelVersionSource beside it so nobody reads it as a vendor-published version. That 0.014 USD bought a stored image and a red job; it is counted in the 0.043 above.
  2. The lane's own generation id was being overwritten. The provenance bundle writes its own generationId, which is the JOB id, over whatever the lane put there — so the minted OpenRouter id survived only in the dispatch record, while generationIdSource still claimed to describe it. It is now recorded as laneGenerationId / laneGenerationIdSource, which the bundle does not touch: img-1d8db4d0-…-0 is on the second job's output.

One smaller thing worth writing down: OpenRouter answered created: 0 on both image calls. The minted id's uniqueness never rested on that field — one job creates at most one image, so the job id carries it — but the comment that described it as "the answer's created second" was corrected to say so.

H.02–H.05 — the lane through the API and the Studio (2026-09-16)#

The API. A job naming openrouter:<task>:<model> runs on the hosted lane with no GPU lease. The policy is judged at submit, on retry (which re-submits), at every chain step and again in the runner before any HTTP call; the daily budget is checked at submit and again before the runner creates. The catalog carries openRouterLanes and /workflows/hosted-lanes/openrouter serves the listing, both empty for a non-operator.

Refusals that cost nothing and say why: hosted_lane_operator_only (403), hosted_lane_video_unavailable (422, naming the blocker), hosted_lane_runpod_only (422, carrying runpodOnlyReasons() and the self-hosted workflows that serve the task), hosted_lane_parameter_unsupported (the catalog snapshot is the allowlist), hosted_lane_provenance_consent_required, and OPENROUTER_DAILY_BUDGET_EXCEEDED (429, or 503 when the ledger is unreachable).

The Studio. HostedLanePanel offers only the lanes the catalog served this caller, builds its form from the snapshot's parameter descriptors, shows the estimate from the API's own estimate route (never computed in the browser) and the measured cost afterwards. 3 chromium Playwright specs cover the picker, a refusal and the cost line.

One rule worth stating plainly. Every offered model's register row says the vendor's acceptable-use policy binds its outputs, so a hosted output may not feed an adult_ok workflow — and wan22-i2v is declared adult_ok. The sfw_only motion workflow ltx25-av is the one that may take a hosted still.

H.04.01 — a RunPod output now carries its own rating (2026-09-16, later)#

The chain rule shipped earlier today depended on the caller telling it the rating of material the caller did not make (inputs.hostedLane.inputContentRatings). That is the half this closes.

The judgement. judgeSubmitContentRating sits in @isis/workflows beside evaluateContentPolicy, because it answers the same question from the same declaration. It gives three answers and unknown is a real one:

the request the judgement why
sfw_only workflow, nothing non-neutral selected neutral the submit gate refuses a racy or explicit option or preset on such a workflow, so reaching the recording IS the evidence
anything selecting a weight or preset rated racy / explicit that rating, most restrictive first the declaration says so, and it names what carried it
adult_ok workflow that nothing rated unknown the prompt is unconstrained and no declaration rated it; recording neutral here is the exact guess the chain rule exists to prevent

What it is a judgement about is written into the function's own doc comment: the declared inputs of one request, not the pixels that come back. Nothing here looks at an image. That is the same standard the hosted lane already holds its own outputs to, and the vendor applies its own moderation to what it receives.

Where it is recorded and read. judgeSubmitContent runs in JobService.submit immediately after assertSubmitContentPolicy — after the gate on purpose, since the gate is what makes neutral an observation — and stamps metadata.contentJudgement. The advancer reads it off the producing job and it outranks the caller's declaration: a declaration is a claim about someone else's material, a judgement is what that job's own gate concluded, so a caller cannot declare its way past a rating Isis recorded. A rating this build does not know reads back as no rating at all.

What this does and does not unblock. An ltx25-av or klein4b output now crosses to a hosted step on its own record, with nothing declared. A chroma-txt2img still is still refused without a declaration — chroma is adult_ok and nothing rated it — and that is the correct answer, not a remaining gap.

Specs: 6 in content-policy.spec.ts (113 in that file), 6 in catalog-content-policy.spec.ts, 5 new in job-chain-advancer.spec.ts (21 there). The five advancer cases are the ones that matter: a recorded neutral advances undeclared; a recorded explicit refuses even when the chain declares neutral; a recorded unknown refuses and the message names the producing job and its reason; a rating from a later build is not read as permission; and a hosted output still cannot feed adult_ok whatever the judgement says. One grammar fix in passing: the refusal read "A explicit input cannot be sent…".

H.04.01 stays open on its one remaining blocker, which is external: RunPod → hosted in the video direction cannot exist while no video vendor's terms can be read (OPENROUTER_VIDEO_BLOCKER).

H.06.02 — the mixed-lane chain, attempted live and blocked (2026-09-16)#

The chain half works: POST /jobs/chain accepted a hosted txt2img step followed by ltx25-av, the still was created and stored, the chain advanced, and the RunPod step received it. Two blockers stop the second half, and both are written down rather than worked around.

1. The volume input channel does not resolve on the live workers. The worker's own start-up log shows oshun-entry: /comfyui/input/volume -> /runpod-volume/inputs, both endpoints report networkVolumeId: 5bst2pgw7y, and the RunPod S3 API writes objects to that volume and reads them back. ComfyUI nevertheless answered an invalid-image-file error for every shape tried:

Reference Object on the volume Endpoint Answer
volume/jobs/<jobId>/0.png inputs/jobs/<jobId>/0.png video Invalid image file
volume/h0602probe/still.png inputs/h0602probe/still.png image Invalid image file
volume/h0602-flat.png inputs/h0602-flat.png image Invalid image file

The cause is not isolated. The next step is a shell on a worker to look at /runpod-volume/inputs while an object is known to be there. Until then, no chain step can read a previous step's bytes in place, which also puts V.05.03's large-media path in doubt.

2. Budget. The cheapest ltx25-av shape the catalog accepts estimates 0.7259 USD — above the ~0.5 USD approved for RunPod this round — so the video budget gate refused before dispatch. That is the gate working.

Three defects the attempt found, all fixed: workerPathForOutput was minting volume/<jobId>/<name> for objects under outputs/, which no symlink reaches; an output the store WRITES to the volume carried no volume key at all; and one transient S3 answer lost an already-billed generation, so a paid-for write is now retried three times before it is called lost. A fourth, mine from the same evening: the new cross-lane chain rule read only metadata.lane, so a Meshy output — which records provider — looked like unrated self-hosted material and stopped every Meshy chain.

E.01 Benchmark suite (2026-09-16)#

E.01.01 — the suite, and the spec that checks it against the catalog. libs/isis/workflows/src/benchmarks/benchmark-suite.ts holds 14 entries across the eight lanes E.01.01 names, 12 of them runnable today. Prompts, seeds and shapes are checked-in literals; nothing is generated at run time, because a benchmark whose inputs drift is not a benchmark.

lane entries workflows runnable
photoreal-portrait 2 chroma-txt2img, zimage-txt2img 2
nude-yoga 2 chroma-txt2img, zimage-txt2img 2
figure-study 1 chroma-txt2img 1
dance-and-movement 3 wan22-t2v, wan22-i2v, scail2-animate 2
sets-and-environments 2 qwen-txt2img, chroma-txt2img 2
costume-and-wardrobe 1 qwen-edit 1
anime-and-stylised 2 hidream-o1-txt2img ×2 1
typography 1 qwen-txt2img 1

What the spec caught. It validates every entry's workflow id, every pinned input name and every enum value against the real catalog JSON on disk. On the first run it failed 12 of 93 assertions, all of them real: invented aspect values (portrait, landscape, square) where the catalog names portrait_896x1152, landscape_1152x896, square_1024; a Z-Image entry using Chroma's aspect vocabulary; a Qwen typography entry pinned to a HiDream aspect (portrait_3x4_1728x2304, which Qwen does not have); and resolution: '480p' where Wan names 480p_landscape_832x480. Every one of those is either a submit refusal or a silently different shape — the failure mode the item exists to prevent, found for free instead of on a billed GPU minute.

Policy is in the data, not in the runner. The three adult-lane entries (nude-yoga ×2, figure-study ×1) are apiReference: false and name adult_ok workflows, and the spec asserts both — so an adult prompt cannot reach E.01.04's external SFW reference through an edit to the runner. Nine entries are cleared for that reference, all neutral.

References, and the two entries that cannot run. wan22-i2v and qwen-edit take their start frame from an earlier entry in the same suite, so the material is Isis's own and carries the operator self-consent C.12.14 settled as correct for a typed prompt. Two entries are carried as blocked rather than dropped or faked:

  • dance-and-movement/scail2-animate-driven needs a driving clip Oshun owns or has cleared (E.03.08). Stock footage would put uncleared likeness through the lane and make the score meaningless.
  • anime-and-stylised/netayume holds the prompt and seed for a dedicated anime base that is not in the catalog (E.06.03), so the eventual comparison is against the same input.

Dropping either would have made its lane read as covered in a results file while measuring less than it claims. A further spec refuses a lane whose entries are all blocked, so this cannot hollow out a lane silently.

93 specs, all green. No spend: this item creates nothing and runs nothing.

E.01.02 — the rubric and the aggregate (2026-09-16). Built and specified; the score store and the dashboard view are the remaining half, so the item stays open.

benchmark-rubric.ts makes four decisions that are the difference between a rubric and a mood:

  • Every point is anchored. Six criteria, each with what a 1 and a 5 look like, so "3" means the same thing in March as in January.
  • Criteria are per lane. BENCHMARK_LANE_CRITERIA gives each lane only what it can answer. Typography is scored on prompt adherence alone; asking it about soft-tissue motion would produce a number that means nothing and then get averaged into a routing decision.
  • Act-bleed is a defect count, not a quality. Sexual pose, sexualised framing, unrequested second person — higher is worse, the opposite direction from every other column, so it is typed separately rather than left as a trap for whoever averages them. E.01.06 needs it reported per body-LoRA.
  • An unscored output stays unscored. summariseBenchmarkLane reports scored and unscored beside every mean and returns null when nobody has looked — 0 would read as "scored, and terrible".

validateBenchmarkScore rejects rather than coerces, and each refusal is a number that would otherwise enter E.01.05 without a person giving it: no rater, a criterion the lane does not use, a value off the 1–5 scale, a score made under an older rubricVersion, an outputHash that is not a sha256.

Cost per accepted output is priced against everything the lane spent, not just the keepers, because the rejected renders are part of what the accepted one cost. Blinding is sample-<8 hex> from the output bytes: stable across sessions, and carrying no workflow, model or checkpoint — the rater sees the lane, because they need it to know which criteria apply, and nothing else.

24 specs; src/benchmarks/ is 117.

E.04 Plates across content policies (2026-09-16)#

E.04.04 — the rule is now called, at both places it had to be. The library half landed 2026-09-15 (libs/isis/workflows/src/output-chain-policy.ts, which reads a producing workflow's models_required against the volume manifest and treats an output as bound when any of those licences carries an acceptable-use clause). It had no callers. It has two now.

At submit. JobService.submit calls assertSubmitOutputChainPolicy immediately after the content-policy gate and before the model-cache lease, so a refusal leaves no lease, no job row and no outbox entry. The producing workflow is read, not guessed: an input naming an Isis output carries the producing job in its own URL (/api/v1/outputs/<jobId>/<file>, the shape output-store.service.ts mints), so the job is looked up and its workflow read off it using the executor's own override order (parameters first, then metadata).

An input that is not an Isis output — an upload, an external URL, a data URI — has no producing workflow here, and the rule does not apply to it. That is deliberate and the specs pin it: E.04.04 is about what Isis made and whose licence Isis therefore knows, not about every byte that reaches a graph.

On each chain step. The advancer answers it exactly where a RunPod output feeds a RunPod step — which was previously an early return null, because no lane boundary is crossed there. A licence boundary can be, and now is checked, before the budget, so a step that must not run is never weighed against money.

Fail-closed in three places, each on purpose:

  • a manifest that cannot be read yields no artifacts, which makes every model unknown, which the library already treats as bound — so an unreadable manifest closes the gate rather than opening it;
  • a producing job that cannot be found is output_provenance_missing, not "allowed";
  • a producing workflow the catalog does not have (a rename) is the same.

One real behaviour change, stated plainly. An adult_ok step whose plate has no readable producing workflow is now refused. Three existing advancer fixtures tripped on it because they omitted a workflowId that a job which actually rendered always carries — reference resolution, ledger choice and the budget stop, none of them about content policy. The shared completedJob fixture now supplies one unless a test sets it itself, and the provenance- missing case is its own named test rather than an accident of an under-specified fixture.

Specs, both ways. 17 in output-chain-policy.service.spec.ts against the real catalog and the real manifest (ltx25-av into wan22-i2v refused with the licence quoted; Qwen-Image, klein 4B and HiDream-O1 allowed; an sfw_only consumer never even looks a job up; a non-Isis input never looked up; job-id extraction and its negatives) and 4 new in job-chain-advancer.spec.ts, which is now 25.

Unrelated red noted, not fixed. catalog-content-policy.projection.spec.ts has 2 failures on this branch: the server's workflow-catalog interface carries modelFamilyPolicy and the client projection does not. git log -S puts that field in 74e9a68b437 (sam 3d gated family and hunyuan exclusions, T.01.03), another session's commit that arrived here through a merge of origin/main. Nothing in E.04.04 touches that route, that interface or the client. It is that session's to close. The rest of the service suite is 589 passed, 35 skipped.