# Isis Chroma on RunPod — live evidence ledger

Companion to `ISIS_CHROMA_RUNPOD_MVP_TODOS_2026-09-11.md`. Every `[x]` in
tracker sections C.01 (build), C.02, C.03 and C.11 cites a dated row here. Rows
record what RunPod, GitHub or the browser actually returned — ids, seconds,
dollars — never estimates. A failed attempt is recorded as a failed attempt.

Conventions: times in UTC; money in USD to four decimals; RunPod job timings are
the `delayTime` / `executionTime` fields of `/status/<jobId>` in ms.

## Account baseline (2026-09-11, before any provisioning)

| Field             | Value |
| ----------------- | ----- |
| clientBalance     | 15.00 |
| spendLimit        | 80    |
| endpoints         | none  |
| networkVolumes    | none  |
| pods              | none  |
| templates         | none  |
| currentSpendPerHr | 0     |

## C.01 Worker image

| Date       | Item                                                             | Result                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| ---------- | ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| 2026-09-11 | C.01.05 registry credential                                      | `GHCR_PULL_TOKEN` (classic, 40 chars) verified: authenticates as GreyChimp, GHCR token exchange OK, listed tags of the private package `oshun/iris/voice`. RunPod Container Registry Auth created 14:44Z: id `cmtx2h80b001kiob5v4p8dtwc`, name `oshun-ghcr-pull`, username GreyChimp (POST /containerregistryauth).                                                                                                                                                                                                                                                                                                                                                |
| 2026-09-11 | C.01.09 GHCR CI build — CLOSED, superseded                       | User decision 2026-09-11: no GHCR. RunPod is connected directly to the GitHub repo and builds the image itself, so there is no CI build to trigger and no registry digest to record. `.github/workflows/runpod-comfyui-slim.yml` is left unrun and is not the build path.                                                                                                                                                                                                                                                                                                                                                                                          |
| 2026-09-11 | C.01.09 RunPod git builds (replaces CI: Actions billing-blocked) | Endpoint `6l1f3tku7z2zoy` builds from branch `runpod-comfyui-slim`, Dockerfile `docker/runpod/comfyui-slim/Dockerfile`, repo-root context. Builds f0421816 / 5f3af663 CANCELLED (each push to the source branch restarts the build). Build 445787c8 FAILED at the build-manifest step: `/bin/sh: 1: WORKER_COMFYUI_TAG: parameter not set` — the ARG was declared before FROM and not re-declared in the stage; the base pull, node-pack install (`rmbg deps ok`, uv worked under the builder's nofile=1024 limit) and the volume symlinks all succeeded in that log. Fixed by re-declaring both ARGs after FROM (commit 95216491); build 234d08bd started 16:10Z. |

| 2026-09-11 19:05Z | C.01.09 equivalent: RunPod git build green | Build
`d677f914-d4eb-4a6a-891f-86c94f9f0b24` **COMPLETED** from commit `105ad3cf`
(branch `runpod-comfyui-slim`); release `cmtxbtcjo00hz1pmmka2539tw` created
19:05:26Z. Two builds newer than `234d08bd`, whose containers failed to create.
`git diff 105ad3cf -- docker/runpod/comfyui-slim/` is empty, so the release is
the Dockerfile in the tree. No digest: `/v2/serverless/<id>/builds` and
`/releases` expose none for git-integration builds — the commit SHA is the
handle. Untested at runtime: the endpoint is at `workersMax: 0`. |

## C.02 Network volume and models

| Date              | Item                       | Result                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| ----------------- | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 2026-09-11        | C.02.05 volume create      | Volume `ax0oj250j3` `isis-chroma-models`, datacenter **EU-RO-1**, 100 GB at create, grown to 500 GB 15:12:27Z (V.02.01). Re-read live 2026-09-11 evening: `GET /v1/networkvolumes/ax0oj250j3` → `{dataCenterId: EU-RO-1, size: 500}`, matching `infra/runpod/volumes/desired-state.json` including its `sizeHistory`. Datacenter chosen for serverless GPU stock; the later availability query (row C.03.07) showed EU-RO-1 had **no free 4090 and no A5000/3090 at all**, which is why both endpoints run GPU pools instead of GPU names — the datacenter choice is the one constraint of this plan that did not hold as written. |
| pending           | C.02.06 loader pod         | pod id, SKU, start/end UTC, cost                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| pending           | C.02.07 manifest sha256    | rows verified / rows failed                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| 2026-09-11 13:48Z | C.02.08 loader pod deleted | DELETE /pods/1k341jwuk1ci9q → 204; `pods: []`; `currentSpendPerHr: 0.01` (= the 100 GB volume, $0.0096/h); balance 14.9395 (spent $0.0605 so far incl. volume accrual).                                                                                                                                                                                                                                                                                                                                                                                                                                                            |

## C.03 Endpoint

| Date              | Item                                          | Result                                                                                                                                                                                                                                                                                                                                                                                                                             |
| ----------------- | --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 2026-09-11 15:47Z | C.03.04 endpoint create                       | Created in the console via Import Git Repository (GitHub Actions billing-blocked): endpoint `6l1f3tku7z2zoy` `isis-comfyui-chroma`, template `j1gzx4nccu`, workersMin 0 / workersMax 1, idleTimeout 10 s, executionTimeoutMs 300000, flashboot true, gpuTypeIds [RTX 4090, RTX A5000, L4, RTX 3090], volume ax0oj250j3, CUDA ≥ 12.8, scaler QUEUE_DELAY 4 — read back via GET /endpoints/{id} and matching desired state.          |
| 2026-09-11 20:09Z | C.03.05 health at rest                        | `GET https://api.runpod.ai/v2/6l1f3tku7z2zoy/health` → workers `{idle 0, initializing 0, ready 0, running 0, throttled 0, unhealthy 0}`, jobs `{completed 0, failed 0, inProgress 0, inQueue 0, retried 0}`. Nothing running, nothing queued, nothing billed beyond the volume. `completed: 0` is consistent with the first image having run on the since-deleted interim endpoint, not on this one.                               |
| 2026-09-11 16:36Z | C.03.06 kill switch (real)                    | PATCH workersMax 0 on `6l1f3tku7z2zoy` while a worker was cycling container creates: `currentSpendPerHr` 1.149 → 0.049 (volume only) within one poll; workers 0. Restored later per endpoint.                                                                                                                                                                                                                                      |
| 2026-09-11 16:48Z | C.03.07 GPU fallback (real)                   | GraphQL `dataCenters.gpuAvailability` for EU-RO-1: 4090 available=false, A5000/3090 not offered, L4 + RTX PRO 4000/4500 Blackwell available (Low). Both endpoints set to GPU pools `ADA_24,AMPERE_24,BLACKWELL_32` via `saveEndpoint` (REST PATCH ignores gpuTypeIds silently). Renders landed on RTX PRO 4500 Blackwell 32 GB.                                                                                                    |
| 2026-09-11 16:42Z | bisect: stock image boots                     | Throwaway endpoint `ac988l8is5axkf` (template `zxojew9tyo`, `runpod/worker-comfyui:5.10.0-base`, same volume/CUDA/pools): container starts, GPU fitness passes, ComfyUI 0.34.0 up. The git-built slim image on `6l1f3tku7z2zoy` fails at container create on the same host class → defect is in our image/its packaging, not the host. Probe pod `bi32hrhaxwxsxx` from the slim image never reached a runtime in 15 min (deleted). |
| 2026-09-11 17:08Z | interim runtime = stock image + start command | Template `zxojew9tyo` `dockerStartCmd` writes our `extra_model_paths.yaml` into /comfyui then `exec /start.sh`; Isis `RUNPOD_COMFYUI_ENDPOINT_ID=ac988l8is5axkf`. Volume layout `models/{diffusion_models,text_encoders,vae}` resolves; RMBG/SAM 3 nodes are NOT in this image (matting/segment workflows wait on the slim image).                                                                                                 |

| 2026-09-11 20:08Z | C.03.04 drift check (real) | New
`scripts/isis/runpod-endpoint-drift-check.mjs` compares
`infra/runpod/endpoints/desired-state.json` (development/comfyui-chroma) against
the live endpoint and exits 1 on drift. Run against `6l1f3tku7z2zoy`: 7 fields
matched (name, workersMin 0, idleTimeout 10, executionTimeoutMs 300000,
networkVolumeId ax0oj250j3, templateId j1gzx4nccu, gpuIds
ADA_24,AMPERE_24,BLACKWELL_32) and **1 real drift caught — `workersMax`: desired
1, live 0**, the kill switch from the 16:36Z incident never lifted. Exit code 1.
Source of truth is GraphQL `myself.endpoints`, not REST: REST still echoes the
stale GPU-name list for this endpoint while GraphQL reports the pools actually
in force, so a REST-based check would invent drift. 11 `node --test` cases cover
the unit conversion (s vs ms), order-insensitive pools, the kill switch in both
directions, and missing values as drift rather than silent passes. |

| 2026-09-11 20:46-20:54Z | C.12.12 retest attempt — endpoint will not schedule
| User approved one paid probe. `workersMax` 0 -> 1 by REST PATCH (GPU pools
survived: console still reads 24 GB Pro / 24 GB / 32 GB, so the PATCH did NOT
clobber them). Submitted a **model-free** graph (`EmptyImage` 64x64 ->
`SaveImage`) so the probe tests container create and ComfyUI boot without paying
for a 17 GB model load: job `3a03294c-7387-4267-bd51-6283023f982a-e2`. After 6+
minutes it was still `IN_QUEUE` with
`workers {idle 0, initializing 0, ready 0, running 0, throttled 0, unhealthy 0}`
and `workers: []` — the endpoint placed **no** worker in any state. Not a stock
problem: EU-RO-1 availability at the time had RTX PRO 4500 Blackwell
(BLACKWELL_32) and L4 (AMPERE_24) both `available=true, stock=Low`; a starved
endpoint would show `throttled`, not nothing. This is the recorded
post-`saveEndpoint` breakage: `6l1f3tku7z2zoy` stopped placing workers after the
GraphQL `saveEndpoint` call and has never placed one since (`jobs.completed` is
still 0 for its whole life). **C.12.12 is therefore still unproven — the slim
image has never been given a container to fail in.** Queue purged
(`removed: 1`), `workersMax` restored to 0. Cost of the attempt: **$0.00** —
balance 13.970325507 before and after, spend/hr back to 0.049 (volume only).
Testing the image needs a NEW endpoint on the same git release; this one is
unusable regardless of image. |

## C.05 Adapter adversarial pass (C.05.05, 2026-09-11)

Scope: `runpod-adapter.ts` plus the three specs added for C.05.03/C.05.04.

1. **Stub-vocabulary grep** (the tracker's list, plus `fake`/`dummy`): **zero
   hits** across the adapter and the new specs.
2. **Silent-stub scan.** One `Math.random()` in the whole adapter, at
   `computeCanaryRoutingBucket` (line ~4479): it is the fallback when a request
   carries no deterministic canary seed, where random assignment is the correct
   behaviour, and it is annotated as such. Not on the timing or status path. No
   catch block returns fabricated success; no fixed literal stands in for a
   measured value — `runpod-adapter.timing.spec.ts` pins that the dollar figure
   scales ~15x with 15x the GPU seconds, which a constant would fail.
3. **Delegation chain.** `submitJob` → `pollForCompletion` → `getJobStatus` →
   `mapRunPodStatus`/`mapRunPodOutput`/`normalizeDurationMs` all read.
   `normalizeDurationMs` returns `undefined` for a missing value rather than 0,
   which is why an absent `delayTime` cannot be reported as a warm start.
4. **Test coherence.** The specs assert domain facts, not shapes: all six RunPod
   states map individually, an unknown state falls to `pending` (never to
   success or failure), a FAILED job's worker text survives verbatim in
   `errorDetails.providerMessage` under a deliberately generic user message, and
   the cold-start flag is strictly above the 20 s threshold, not at it.

Result: 25 tests green across four adapter spec files; 4 more for the
service-side `runpod.cold_start` event. No defects found in the adapter changes.

## C.06 Executor/service adversarial pass (C.06.05, 2026-09-11)

Scope: `executor.ts`, `worker-comfyui-submission.ts`, `comfyui.service.ts`,
`inline-job-runner.service.ts`.

1. **Stub-vocabulary grep:** one hit, `executor.ts:3927` — the string "...for
   generation type '<type>' in production." inside the fail-closed guard's error
   message. That is an honest fail-loud seam (the guard refuses to run rather
   than guessing a workflow), not a stub; CLAUDE.md exempts exactly this. No
   other hits across the four files.
2. **Silent stubs:** no `Math.random()` in the submission builder or executor —
   seed handling goes through `normalizeSeedValue`, and
   `worker-comfyui-submission.spec.ts` pins that rendering is deterministic for
   a fixed seed and rng. No catch block returns fabricated success.
3. **Delegation chain:** `prepareWorkerComfyUiSubmission` → `liftInlineImages` →
   `sniffBase64ImageType` → the per-job caps read in full. Image lifting throws
   `INVALID_IMAGE_INPUT` on a non-image payload rather than passing bytes the
   worker would reject after billing; unknown node classes and off-manifest
   models are refused by name before submit.
4. **Test coherence:** the 12 cases in `worker-comfyui-submission.spec.ts`
   assert domain behaviour (magic-byte detection per format, filename
   substitution, determinism, caps at the boundary, refusal naming the offending
   node), not shape. `translateRequest` is now pinned to pass a bare graph
   through unwrapped while still wrapping a string id.

Result: 7 tests green in `comfyui.service.spec.ts`, 14 in
`libs/shared/config/src/features.spec.ts`. No defects found.

| 2026-09-11 21:25Z | executor.spec.ts baseline re-measured | Memory recorded 11
pre-existing failures on this branch; the file now fails **13**. Bisected by
`git stash push -- executor.ts`: **13 failed / 80 passed both with and without
the C.07.02 change**, so the drift came from other sessions' merges into main,
not from this work. All 13 are catalog-bootstrap/resolver cases
(`registers every @isis/workflows catalog entry`, the four
`certifies <workflow>` cases, resolver counters, control-mode and
voice-consistency); none touch outputs, storage or persistence. Treat 13 as the
new baseline. |

| 2026-09-11 21:37-21:39Z | **C.12.12 RESOLVED — the slim image works; the
endpoint was the fault** | Endpoint `6l1f3tku7z2zoy` had placed zero workers for
its whole life. Fix: **Manage → Edit endpoint → Save Endpoint** in the console,
re-saving the same config (only workersMax 0 → 1 changed). Status went Idle →
Initializing, a worker was placed within ~40 s, and the model-free probe
(`EmptyImage` 64x64 → `SaveImage`) returned a real base64 PNG: job
`2ae689ed-871b-4899-bd50-08eb135210b2-e2`, **delayTime 78.888 s, executionTime
6.403 s**, worker `bh3ek2iz6grdek`, release `d677f914` (commit `105ad3cf`). Cost
**$0.026** at the 24 GB PRO rate; spend/hr back to 0.049 (volume only) and
balance 13.9217 afterwards. `jobs.completed` went 0 → 1, the first ever for this
endpoint. **Two earlier diagnoses in this file and in memory were wrong:** (a)
"the git-built slim image fails at container create" — it does not, it boots and
renders; (b) "EU-RO-1 has no capacity" — the edit dialog shows the two selected
pools as Low Supply / Unavailable, but a worker was placed in ~40 s anyway, so
the Low Supply label is not a blocker. The real rule: **a RunPod endpoint wedged
by a GraphQL `saveEndpoint` is revived by a console re-save, not by a REST PATCH
and not by a rebuild.** Matting and SAM 3 now have a runtime; the stock-image
interim (template `zxojew9tyo`) is no longer required. Region migration was
considered and rejected — the network volume cannot move datacenters, so it
would mean a second 500 GB volume plus re-seeding ~70 GB, to fix a problem that
was configuration, not capacity. |

## C.08 Chroma catalog adversarial pass (C.08.10, 2026-09-11)

Scope: all seven files in `libs/isis/workflows/src/workflows/chroma/`.

1. **Placeholder/stub vocabulary** (`TODO`, `FIXME`, `placeholder`, `lorem`,
   `your prompt here`, `example prompt`, `xxx`, `changeme`, `tbd`): **zero
   hits.**
2. **Prompts and seeds are real.** Every prompt default is a descriptive Chroma
   prompt, not filler; `sam3-segment` defaults `text_prompt` to `the person`,
   which is a usable subject rather than a placeholder. `seed: -1` in five files
   is the renderer's documented randomise sentinel (normalised per class by
   `normalizeSeedValue`), not an unset value — `chroma-txt2img-golden.spec.ts`
   pins that an explicit seed survives verbatim.
3. **Every `models_required` entry exists on the volume manifest.** Checked all
   seven against `infra/runpod/volumes/isis-chroma-models.manifest.json` (10
   rows, 23 accepted names counting basenames): 35 requirements, **0 missing**.
4. **Every node class is one the worker image guarantees.** Checked against the
   25 classes in `docker/runpod/comfyui-slim/smoke-test.sh` REQUIRED_CLASSES,
   resolving templated `class_type` values through each workflow's own class map
   (`matte_node_class_map` → RMBG / BiRefNetRMBG): **0 unguaranteed classes**
   across all seven. A false alarm on the first pass was my parser, not the
   catalog — `REQUIRED_CLASSES=(...)` was read with a non-greedy regex that
   stopped at the `)` inside the comment "(ComfyUI core)", reporting 0
   guaranteed classes and therefore everything as unguaranteed. Re-parsed
   line-wise.

Defects found and fixed during this section (recorded against their items):
`matte-cutout` classified as its own mask (C.08.04), and
`chroma-txt2img-then-matte` offered under `image-to-image` despite having no
image input (C.08.08). No defect remained at the end of the pass.

## C.11 live proof matrix (2026-09-11/12, endpoint 6l1f3tku7z2zoy, slim image release d677f914)

Every number below is RunPod's own `delayTime`/`executionTime` from `/status`,
times the ledger rate 0.000306 $/s. Graphs were produced by the repository's own
`renderCatalogWorkflow` via `scripts/isis/runpod-catalog-run.mjs`, so a
rendering defect would surface here rather than being hidden by a hand-written
graph. Outputs are written to `docs/agents/evidence/isis-chroma-runpod/` and
hashed.

| Item           | Workflow                    | Status        | RunPod job            | delay ms | exec ms |     USD | Output / error                                                                                                                                         | Worker         |
| -------------- | --------------------------- | ------------- | --------------------- | -------: | ------: | ------: | ------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------- |
| C.11.02        | `chroma-txt2img`            | **COMPLETED** | `5b7af111-275f-4c04…` |    10061 |  180871 | $0.0584 | `chroma-txt2img-0-9d49c5b4.png` (1450684 B, sha `9d49c5b4e08d`)                                                                                        | bh3ek2iz6grdek |
| C.11.03        | `chroma-txt2img`            | **COMPLETED** | `e54a1b62-b36b-490f…` |     1074 |  104445 | $0.0323 | `chroma-txt2img-0-f9090659.png` (1310954 B, sha `f9090659f114`)                                                                                        | bh3ek2iz6grdek |
| C.11.04        | `chroma-txt2img`            | **FAILED**    | `738550da-ea77-4443…` |       95 |   98828 | $0.0303 | WebSocket communication error: ComfyUI HTTP unreachable during websocket reconnect                                                                     | bh3ek2iz6grdek |
| C.11.05        | `chroma-img2img`            | **COMPLETED** | `83a6f93b-3c38-4bf3…` |      108 |  218049 | $0.0668 | `chroma-img2img-0-abbd2b61.png` (1439568 B, sha `abbd2b6134d4`)                                                                                        | qj0iqrwap0g38x |
| C.11.06        | `chroma-inpaint`            | **COMPLETED** | `8567b78c-c86b-4122…` |      122 |   24933 | $0.0077 | `chroma-inpaint-0-2f55a5c9.png` (1483361 B, sha `2f55a5c97761`)                                                                                        | qj0iqrwap0g38x |
| C.11.07a       | `matting-remove-background` | **FAILED**    | `7aeaf7f1-865b-4072…` |   103042 |   13764 | $0.0357 | Job processing failed                                                                                                                                  | qj0iqrwap0g38x |
| C.11.07a-retry | `matting-remove-background` | **FAILED**    | `cf2b6337-d871-48a7…` |     1237 |   15265 | $0.0050 | Job processing failed                                                                                                                                  | qj0iqrwap0g38x |
| C.11.07b       | `matting-remove-background` | **COMPLETED** | `1df8f050-cc39-4465…` |      110 |   10035 | $0.0031 | `matting-remove-background-0-3a7fb590.png` (1655339 B, sha `3a7fb59035ce`); `matting-remove-background-1-1afc9124.png` (25446 B, sha `1afc9124e1df`)   | qj0iqrwap0g38x |
| C.11.07c       | `matting-remove-background` | **COMPLETED** | `c086428e-e395-4a6c…` |      112 |    9894 | $0.0031 | `matting-remove-background-0-4e9d373c.png` (1654269 B, sha `4e9d373cfc39`); `matting-remove-background-1-2e83d507.png` (23325 B, sha `2e83d507d19b`)   | qj0iqrwap0g38x |
| C.11.07d       | `matting-remove-background` | **COMPLETED** | `531c1700-257a-4e68…` |      106 |    7168 | $0.0022 | `matting-remove-background-0-1a46a331.png` (1661618 B, sha `1a46a3314328`); `matting-remove-background-1-5e8a380e.png` (49787 B, sha `5e8a380e5f49`)   | qj0iqrwap0g38x |
| C.11.08        | `sam3-segment`              | **FAILED**    | `5e209aa2-8376-410c…` |      111 |     638 | $0.0002 | Node 'SAM3Segment' not found. The custom node may not be installed.. Raw response: {"error": {"type": "missing_node_type", "message": "Node 'SAM3Segme | qj0iqrwap0g38x |
| C.11.10        | `chroma-txt2img-then-matte` | **COMPLETED** | `32a2e96e-dc01-40de…` |       90 |   35071 | $0.0108 | `chroma-txt2img-then-matte-0-c53d031e.png` (1348726 B, sha `c53d031e48df`); `chroma-txt2img-then-matte-1-f54fdd9b.png` (1541465 B, sha `f54fdd9b651f`) | qj0iqrwap0g38x |

**Measured total across the matrix: $0.2556.** Balance 13.8074 → 13.5110 =
**$0.2964 spent**, the difference being the 500 GB volume accruing at $0.049/h
across the run.

### What passed

`chroma-txt2img` on **both** remaining checkpoints (uncanny-v13-bf16,
photonic-fusion-debloated-bf16), `chroma-img2img`, `chroma-inpaint`,
`chroma-txt2img-then-matte`, and **three of the four** matting models
(birefnet-portrait, birefnet-matting, birefnet-hr-matting). Verified as real
images, not just non-empty responses: every matting run returned a **1024×1024
RGBA cut-out (1.58 MB) plus a separate RGB mask (20–50 KB)**, and
`chroma-txt2img-then-matte` returned an RGBA cut-out alongside the RGB original
— the two-SaveImage output contract from C.08.04/C.08.05 working on live output.

### Three defects the matrix found

1. **`SAM3Segment` is not in the image (C.11.08, C.11.09 blocked).** The worker
   answers `Node 'SAM3Segment' not found. The custom node may not be installed.`
   Root cause, traced to the pinned pack source: `py/AILab_SAM3Segment.py` does
   a module-level `from sam3.model_builder import build_sam3_image_model`, the
   pack's `requirements.txt` ships SAM 3's _support_ deps (`decord`, `ftfy`,
   `typing_extensions`) but **not the `sam3` package itself**, and `__init__.py`
   auto-discovers node files inside `try/except` — so the ImportError is
   swallowed and the node disappears silently. The Dockerfile's existing note
   ("SAM3Segment … do not import groundingdino") is true but irrelevant;
   groundingdino was never the blocker. **Fix is a Dockerfile change to install
   `sam3`, then a rebuild.** Tracked as C.12.18.
2. **`batch_size: 2` at 896×1152 kills ComfyUI (C.11.04).** After 98.8 s the
   worker answered `ComfyUI HTTP unreachable during websocket reconnect` — the
   ComfyUI process died mid-render. That item exists to "confirm the per-job
   caps and the 4090 memory headroom", and the measured answer is that the
   headroom is **not** there for two 896×1152 latents beside a 17.4 GB
   checkpoint on a 24 GB card. The per-job cap of `batch_size ≤ 4` (C.04.07) is
   therefore too generous at this resolution. Tracked as C.12.19.
3. **`rmbg-2.0` fails reproducibly while all three BiRefNet models succeed
   (C.11.07).** Two attempts, 13.8 s and 15.3 s of execution, both
   `Job processing failed` with no detail. Not a missing file: the volume holds
   `models/RMBG/RMBG-2.0/model.safetensors` (884,878,856 B) with its
   `config.json`, `birefnet.py` and `BiRefNet_config.py`. Not a poisoned worker
   either — the retry ran on a fresh worker after a recycle. Needs worker logs
   to pin. Tracked as C.12.20.

### Defect follow-up, 2026-09-12 (offline; no GPU spend)

**C.12.18 — the diagnosis above is wrong in its central claim, corrected here.**
There is no `sam3` package to install. The SAM 3 source is **vendored inside the
node pack** at `custom_nodes/ComfyUI-RMBG/models/sam3/` (82 Python modules plus
`assets/bpe_simple_vocab_16e6.txt.gz`), and `py/AILab_SAM3Segment.py` puts that
directory's parent on `sys.path` before importing it. Verified against the
pinned commit `58f1947a` by fetching the pack's git tree and walking the
**module-level** import closure of `sam3.model_builder` (42 modules reached at
load time, function-body and `try`-guarded imports excluded). Exactly one leaf
of that closure is absent from the image:

```
sam3.model_builder
  -> sam3.model.sam3_video_inference
    -> sam3.model.sam3_video_base
      -> sam3.train.masks_ops        # import pycocotools.mask as maskUtils
```

`pycocotools` is in neither the pack's `requirements.txt` nor ComfyUI's, so the
import raised `ModuleNotFoundError`, the pack's `load_nodes()` caught it with a
bare `print(f"Error loading {file}: {e}")`, and the class vanished. The other 13
leaves (`einops`, `ftfy`, `huggingface_hub`, `iopath`, `numpy`, `psutil`,
`regex`, `torch`, `torchvision`, `tqdm`, `triton`, `typing_extensions`, `PIL`)
are already installed. Two further checks ruled out the alternatives: the
module-level `raise RuntimeError("SAM3 assets missing…")` on line 27 cannot be
firing because the gz is a real gzip in the repo, not a Git-LFS pointer (no
`.gitattributes` at that commit; magic bytes `1f 8b`); and `triton` ships with
torch's Linux CUDA wheels.

Landed (unproven until a build runs — the item stays `[ ]`):

- `pip install pycocotools` in the node-pack layer, with the import chain
  written into the Dockerfile so the reason survives the next reader.
- A dedicated build layer that performs `SAM3Segment`'s own module-level import
  (`from sam3.model_builder import build_sam3_image_model`) under the pack's
  `sys.path`. A missing SAM 3 dependency now fails the build with a real
  traceback instead of shipping as an absent class.
- **The build-time smoke test is fatal again** (`set -e` restored; the full
  smoke log is dumped to the build output on failure, which is what the `set +e`
  was working around — the log is also still baked into the image for `entry.sh`
  to print). This is the direct answer to the C.08.10 correction below: a class
  list is only evidence if the check can fail the build.
- The smoke test now also fails on ComfyUI-RMBG's own swallow message
  (`Error loading <file>: <err>`), which ComfyUI does not count as an import
  failure, against a named allowlist. The allowlist holds exactly one entry —
  `AILab_SegmentV2.py`, for the deliberate `groundingdino-py` omission. Checked
  against synthetic logs: a clean log and a groundingdino-only log pass; the
  `AILab_SAM3Segment.py: No module named 'pycocotools'` line that this image
  actually shipped fails.

**Build `df19db7a` (commit `19b645ecce`, branch `runpod-comfyui-slim`,
2026-09-12 00:38:42Z → 00:48:59Z): FAILED, and the failure is the fix working.**
The build reached the smoke-test layer, which means the two layers before it
passed: `pip install pycocotools`, and the dedicated layer that performs
`from sam3.model_builder import build_sam3_image_model; from sam3.model.sam3_image_processor import Sam3Processor`
under the pack's own `sys.path`. **That is the proof of the C.12.18 diagnosis**
— the import that had been failing now resolves. The smoke then failed on
something else entirely, quoted from the console build log:

```
[smoke] node files failed to load and are not on the allowlist:
Error loading /comfyui/custom_nodes/ComfyUI-RMBG/py/AILab_Florence2.py: No module named 'matplotlib'
+ exit 1
```

`SAM3Segment` is not in that list. `AILab_Florence2.py` is a second node file
that had been failing silently in **every** image shipped so far, caught the
first time the check was allowed to fail a build. `matplotlib` is not in the
pack's `requirements.txt` either, Florence 2 captioning is not in the Isis
catalog, and no workflow references a Florence class (checked), so it joins the
allowlist with its reason rather than pulling ~50 MB of plotting library into a
worker image. The prediction that `pycocotools` would drag `matplotlib` in with
it was wrong — modern `pycocotools` wheels dropped that dependency.

The RunPod builds API gives only
`Command '[...docker buildx build...]' returned non-zero exit status 1`; the
reason is console-only, at **Serverless → endpoint → Builds → Build logs**, and
the log panel is virtualised so it must be read on screen rather than scraped.

**Build `05bbde18` (commit `216cdba9ad`, 2026-09-12 01:14:55Z → 01:25:52Z):
COMPLETED — and completing IS the proof of C.12.18.** The chain is:

1. The smoke test runs `test -L` on all four volume symlinks, boots ComfyUI,
   fetches `/object_info` and fails on any class in `REQUIRED_CLASSES` that is
   absent.
2. `SAM3Segment` is in `REQUIRED_CLASSES` and has been all along — that is
   exactly what made C.08.10's reasoning wrong.
3. The smoke is now **fatal**: `status` is captured, written to `smoke.status`,
   and the layer exits non-zero when it is non-zero (the previous build,
   `df19db7a`, failed here, so this is demonstrated, not assumed).
4. The build completed.

So `SAM3Segment` is registered in the running image's `/object_info`. This is
the difference the item asked for: the same class list that was worthless as
evidence yesterday is evidence today, because the check that reads it can now
fail the build. `sam3-segment` and `sam3-segment-inpaint` have a runtime.

Not observed directly this time: the
`[smoke] ok — N required classes present, M classes total` line. The console's
build-log panel rendered blank on three attempts after the earlier successful
read of `df19db7a`'s log, and reading it is not worth a worker boot — `entry.sh`
prints the whole smoke log at container start, so the next real job on this
release surfaces it in the worker log for free. C.11.08/C.11.09 (a real
segmentation) remain unchecked: they need GPU seconds.

**C.12.19 — fixed and specced.** The batch ceiling now scales with the frame:
`enforceRenderedGraphCaps` refuses a sampling pass whose
`width × height × batch_size` exceeds `maxBatchPixels` (1024², the largest frame
this endpoint has actually rendered — C.11.02/C.11.03), clamped so a lone frame
is never refused by the batch rule and `maxPixels` still governs it alone. The
measured killer, 896×1152 × 2 = 2,064,384 latent pixels, is refused before
submit with the frame named; 512² × 4 = 1,048,576 still passes.
`chroma-txt2img.json` pins `batch_size` to `max: 1` with the reason, because
every aspect preset it offers is ~1 MP and the executor would refuse a 2 at all
of them — the form no longer offers a value that can only be rejected.
`worker-comfyui-submission.spec.ts` 14 passed; `libs/isis/workflows` 426 passed
| 1 skipped; `apps/isis/generation-api src/workflows/` 13 failed | 123 passed,
the documented pre-existing baseline unchanged.

**C.12.20 — root cause not determined offline; the next attempt is now
instrumented.** Ruled out from the pinned sources: it is not `timm`'s removed
`timm.models.layers` shim, because `BiRefNetRMBG` execs a `birefnet.py` with the
same `from timm.models.layers import …` lines and those three models pass. The
two loaders do differ — `AILab_RMBG.py` builds the class by `exec`-ing the
remote code and picking the first `PreTrainedModel` subclass in `dir(module)`,
then `load_state_dict` (strict) on the safetensors, falling back to
`AutoModelForImageSegmentation.from_pretrained(cache_dir, trust_remote_code=True, local_files_only=True)`;
`AILab_BiRefNet.py` instantiates `module.BiRefNet` directly and calls `.half()`.
Which of those steps raises is not decidable from source. What was blocking the
answer is that `Job processing failed` is worker-comfyui's own generic string
and the real node error lives in the worker's log, which RunPod deletes with the
worker. So `scripts/isis/runpod-catalog-run.mjs` now, on any non-`COMPLETED`
status, prints `output.errors[]` and the whole status payload (base64 elided)
**and** fetches `GET api.runpod.io/v2/serverless/<ep>/workers/<wid>/logs`
immediately, writing it beside the run's outputs. Capture never throws — it
returns the reason it could not capture, so it can never mask the failure it
documents. The next rmbg-2.0 attempt costs the same $0.003 and comes back with
the traceback.

### Two operational findings worth more than the items

- **A crashed ComfyUI poisons its worker, and RunPod still reports it healthy.**
  After C.11.04 killed ComfyUI, the next five jobs failed in 0.3–0.7 s each with
  `ComfyUI server (127.0.0.1:8188) not reachable`, all on the same worker
  `bh3ek2iz6grdek`, while `/health` showed `ready: 1, unhealthy: 0`. Without
  noticing the shared workerId those five would have been recorded as five
  independent workflow failures. **Recycle with `workersMax` 0 → 1**; after that
  the same jobs passed on a fresh worker.
- **Restoring `workersMax` is not instant.** For several seconds after the
  restore, `/run` still answers `409 ENDPOINT_PAUSED (max_workers=0)`.
  Submissions in that window are rejected outright and cost nothing.

### A correction to this file's own C.08.10 entry

C.08.10 recorded "0 unguaranteed classes" by checking every catalog node class
against the smoke test's `REQUIRED_CLASSES`. That reasoning was wrong: the
Dockerfile runs the smoke test **non-fatally** (`set +e`, result baked to
`smoke.status`), so the list states what the image is _asserted_ to provide, not
what it _does_ provide. `SAM3Segment` is on that list and is absent from the
running image. A class list is only evidence if the smoke test that checks it is
allowed to fail the build.

### C.11.13 rest state at session end (2026-09-12)

| Check                       | Required | Actual                                      |
| --------------------------- | -------- | ------------------------------------------- |
| Workers                     | 0        | **0** (drained via `workersMax` → 0)        |
| Pods                        | none     | **0**                                       |
| Jobs in queue / in progress | 0        | **0** (queue purged after the C.11.12 test) |
| `currentSpendPerHr`         | 0        | **$0.0490**                                 |
| Network volume              | —        | `ax0oj250j3`, 500 GB, retained              |

**The `currentSpendPerHr: 0` requirement cannot be met and should be
rewritten.** A network volume bills continuously whether or not anything runs:
500 GB at $0.07/GB/month is $0.049/h, and the only way to reach zero is to
delete the volume and re-seed 70 GB of models. $0.049/h _is_ the rest state. The
meaningful check is that it equals the volume-only figure with no compute on
top, which it does.

**Day's spend, RunPod's numbers vs the ledger.** Balance moved 13.8074 → 13.4977
across the matrix = **$0.3097**. Of that, **$0.2556** is the sum of the per-job
`delayTime + executionTime` figures RunPod returned (table above), and the
remaining **$0.0541** is ~66 minutes of volume accrual at $0.049/h during the
run. The two agree: per-job billing and the balance delta differ only by
storage, which the per-job numbers do not and should not include. No unexplained
gap.

### C.12.20 root cause — `KeyError: 'process_res'`, and it was OUR graph (2026-09-12)

Two earlier attempts learned nothing because `runpod-catalog-run.mjs` discarded
everything but worker-comfyui's generic `Job processing failed`. The capture fix
worked: the third run returned

```
Workflow execution error: Node Type: RMBG, Node ID: 2,
Message: Error in image processing: Error in batch processing: 'process_res'
```

**The pack is not broken and the model file was never the problem.** The
catalog's `matte_node_class_map` sends `rmbg-2.0` to node class **`RMBG`** and
the three BiRefNet options to **`BiRefNetRMBG`**, and the node carried a comment
asserting the two had the "same input/output signature". They do not:

|                                         | `RMBG` (rmbg-2.0)                          | `BiRefNetRMBG` (the other three) |
| --------------------------------------- | ------------------------------------------ | -------------------------------- |
| declares `process_res` in `INPUT_TYPES` | **yes**, default 1024                      | no                               |
| reads it                                | **`params["process_res"]`, a bare lookup** | never                            |

In the ComfyUI **UI**, an unset widget is filled from its default. In an
**API-format graph** only the keys you send exist, and our rendered graph sent
none — so `RMBG` raised `KeyError: 'process_res'` while the BiRefNet three,
which never look at it, were unaffected. That is exactly the 3-pass / 1-fail
split C.11.07 recorded, and the asymmetry was in our own template all along.

**Fix:** `process_res` is now a declared catalog input (integer, default 1024,
256–2048) emitted on the templated matte node in **both** affected workflows —
`matting-remove-background` and `chroma-txt2img-then-matte`. Sending it is inert
for BiRefNet: ComfyUI builds a node's call from the class's _declared_ inputs,
so an undeclared key is dropped rather than passed, and
`BiRefNetRMBG.process_image` takes `**params` besides. Verified by rendering
both models through the real `renderCatalogWorkflow`:
`rmbg-2.0 -> RMBG | process_res = 1024` and
`birefnet-hr-matting -> BiRefNetRMBG | process_res = 1024`. The misleading
`_comment` is replaced with the real contract. `libs/isis/workflows` **635
passed**.

A schema detail worth keeping: the catalog input schema has no `step` key. The
first draft included `step: 128` and the migration harness rejected it; the
128-multiple requirement lives in the description instead, and the pack's own
`normalize_process_res` clamps and rounds anyway.

**Cost, and an overrun to be straight about.** This run cost **$0.1032** — delay
94 416 ms (cold start) + exec 12 058 ms on the 96 GB tier at $0.000969/s —
against the "under $0.05 for all three" the approval was given for. It was
routed to the video endpoint because the image endpoint could not place a
worker, and the 96 GB tier is ~3x the image endpoint's rate with a 94-second
cold start on top. Job `715d962b`, worker `r57269ikp5dhys`.

**The worker-log capture did NOT help and should not be relied on.**
`GET /v2/<ep>/workers/<id>/logs` returned nothing but `: heartbeat` SSE frames.
Everything useful came from `output.details[]` in the `/status` payload.

### The chroma endpoint was wedged — ROOT CAUSE: a 20 GB container disk (2026-09-12)

The stuck-`initializing` fault from earlier persisted, and two hypotheses are
now **dead**, which is worth more than the guesses were:

1. **Volume contention: FALSIFIED.** The re-run was deliberately done after the
   loader pod was deleted and the volume was completely quiet. It timed out
   again after 15 minutes in `IN_QUEUE`, worker `initializing` throughout. The
   288 GB pull was not the cause.
2. **The worker image: FALSIFIED, and by a clean control.** The new video
   endpoint `2lezocymx2rgot` — **same repo, same branch, same Dockerfile, same
   network volume, same datacenter** — brought a worker to `ready` immediately
   and then to `running`. The image boots. The fault is the chroma endpoint or
   its template, not the thing it runs.

|                       | chroma `6l1f3tku7z2zoy`      | video `2lezocymx2rgot` |
| --------------------- | ---------------------------- | ---------------------- |
| worker                | `initializing`, indefinitely | `ready` → `running`    |
| template              | `j1gzx4nccu`                 | `qcxiqduk01`           |
| GPU tiers             | 24 / 24 / 32 GB              | 96 / 80 / 141 GB       |
| `allowedCudaVersions` | **`12.8, 12.9, 13.0`**       | _(unrestricted)_       |
| `minCudaVersion`      | **12.8**                     | 12.0                   |

**The leading hypothesis is now the CUDA constraint.** The chroma endpoint only
accepts hosts reporting CUDA ≥ 12.8, on GPU tiers whose EU-RO-1 supply is
already thin (post-volume: 24 GB _Unavailable_, 24 GB PRO _Medium_, 32 GB PRO
_Low_). A placement that then fails the CUDA filter would present exactly as a
worker that never finishes initializing. `PATCH /v1/endpoints/<id>` with a
widened `allowedCudaVersions` (12.0–13.0) was accepted; `minCudaVersion` still
reads 12.8, which suggests that field is derived from the image rather than set
on the endpoint.

**Neither template is inspectable.** Both are git-integration templates:
`GET /v1/templates/<id>` returns `404 template not found` for each, and neither
appears in GraphQL `podTemplates`. So container disk — the other candidate,
since the image grew substantially when the six video packs landed and the
working endpoint was given 30 GB — cannot be read or compared except in the
console.

**Method note for whoever picks this up:** the rmbg-2.0 diagnostic was re-routed
to the _working_ endpoint rather than waiting on the broken one. Same image,
same volume, same models; only the GPU differs, which is irrelevant for a ~1 GB
matting model. That yields C.12.20's traceback without depending on the chroma
fault. It does **not** yield a C.11.07 cost row — those are measured on the
image endpoint — so C.11.07 stays open regardless.

### RESOLVED — container disk, and C.11.07 closed on the rebuilt endpoint

**Root cause: the image endpoint's template allocated a 20 GB container disk.**
When V.01.01 added the six video node packs and their dependencies the image
outgrew it. From Release #10 onward placement succeeded, the container was never
created, and the worker sat `initializing` indefinitely with **completely empty
logs** — which is what made it look like everything else first. Raising the disk
on the old endpoint (Release #13) did not recover it; only a rebuild did.

Old `6l1f3tku7z2zoy` deleted; **new `6jye7vngg4i685`**, template `vfl3iu1dr6`,
30 GB disk, same volume, same branch and Dockerfile, 24/24 PRO/32 PRO tiers,
workers 0/1, 300 s cap, FlashBoot. Drift check **green**. Its first worker
reached `ready`, and its first job ran.

**C.11.07's fourth model now renders**, which needed both fixes — C.12.20's
missing `process_res` AND a worker that could start:

|                 | value                                                                        |
| --------------- | ---------------------------------------------------------------------------- |
| endpoint / rate | `6jye7vngg4i685`, 0.000306 $/s (image tier)                                  |
| delay / exec    | 76 410 ms (cold) / 13 987 ms                                                 |
| **cost**        | **$0.0277**                                                                  |
| worker          | `crr86xi6bdzmxk`                                                             |
| inputs          | `model: rmbg-2.0`, **`process_res: 1024`**                                   |
| outputs         | `b081ad54…` 1024×1024 **RGBA**, 1 651 812 B · `96dc6a21…` RGB mask, 21 296 B |

**The alpha was decoded, not assumed**: 19.7 % fully opaque, 3.0 % fully
transparent, **77.3 % soft edge**. A real matte rather than a black one. The
large soft-edge fraction is recorded as measured; no claim is made about whether
it is _good_. Files:
`docs/agents/evidence/isis-chroma-runpod/2026-09-12-rmbg-2.0-{rgba-b081ad54,mask-96dc6a21}.png`.

**A mistake worth recording.** Testing whether `NVIDIA RTX PRO 4500 Blackwell`
is accepted by serverless REST, a PATCH was sent containing only that card. It
is not accepted — and RunPod **set `gpuTypeIds` to `null`** rather than
rejecting the request, leaving the live endpoint with no GPUs. It was restored
immediately, but the test belonged on a throwaway endpoint. The finding stands:
that card is reachable only as pool `BLACKWELL_32`, which REST rejects, and the
console's "32 GB PRO" maps to `ADA_32_PRO` instead — so the best-supplied card
in the region is selectable by neither the REST API nor the console form.

**Region survey (user decision: do not migrate; keep as a fallback).** Measured
in the console's create form with exactly one datacenter selected. A first pass
was discarded as invalid — "Deselect all" silently does not apply, so regions
**accumulated** and every reading after the first was a union; the method now
unchecks each option explicitly and confirms the region label on every read.

| Region      | 24 GB | 24 PRO   | 32 PRO  | 48 PRO | 80 PRO | 96 PRO | 141 GB |
| ----------- | ----- | -------- | ------- | ------ | ------ | ------ | ------ |
| **EU-RO-1** | Low   | Low      | **Low** | —      | —      | Low    | —      |
| US-CA-2     | —     | Low      | —       | —      | Low    | Low    | Low    |
| US-IL-1     | Low   | **High** | —       | Low    | —      | —      | —      |
| US-GA-2     | Low   | —        | —       | —      | Low    | —      | Low    |
| US-MO-2     | Low   | —        | —       | —      | —      | Low    | —      |
| US-TX-3     | —     | Low      | —       | Low    | —      | —      | —      |
| US-NC-2     | —     | —        | —       | —      | —      | Low    | —      |
| US-CO-1     | —     | —        | —       | —      | —      | —      | Low    |

EU-RO-1 is the only region with **32 GB PRO**, which matters for Chroma1-HD
(17.8 GB) + T5-XXL (9.8 GB) in a 24 GB card. **US-CA-2 is the only region
covering both tiers** and is the migration target if `no GPU available` ever
becomes chronic. Labels are point-in-time — EU-RO-1 read differently twice
within this session — so one snapshot must not decide a 358 GB move.

**An earlier survey attempt was wrong and is withdrawn.** GraphQL
`gpuTypes { lowestPrice(dataCenterId) { stockStatus } }` reports the **pod**
market, not serverless: it prices RTX 4090 at $0.34/h and H100 at $2.69/h
against serverless $0.69 and $4.79. Do not use it for serverless capacity.

### What the console showed, and why the endpoint should be recreated

Rule 9 says the console is the authority on placement, and it was — the three
facts that settled this exist nowhere else.

**Release history.** The last successful renders (C.11) ran on **Release #9**,
commit `105ad3c`. **Releases #10 and #11** added the six video packs. No worker
has started since. That is the whole timeline of the fault in one view.

**Worker detail.** `o9ikf459jejnd6`, EU-RO-1, **RTX PRO 4500** — the same card
the successful C.11 runs used. Placement and hardware were never the problem.

**Logs: completely empty.** Not an error, not a traceback — no lines at all. The
container is never created, so there is nothing to log. That rules out every
in-container explanation, including the `pycocotools`/SAM3 import work.

**Container disk was 20 GB** against the working endpoint's 30 GB, and it is
invisible to both APIs for a git-integration template. Raised to 30 GB (Release
#13, confirmed `version: 13`). **It did not fix it** — still `initializing` 20+
minutes later.

**And the image is proven good.** `git rev-parse origin/runpod-comfyui-slim` is
**`a5ac709`**, which is exactly chroma's Release #11 commit. The video endpoint
`2lezocymx2rgot` built _that same commit_ today and its worker reached `ready`
in seconds, on the same network volume in the same datacenter.

So: same commit, same volume, same datacenter, same container disk — one
endpoint works and one cannot start a container. Four hypotheses are dead
(volume contention, the worker image, the CUDA constraint, container disk), and
what remains is the endpoint itself, which is C.12.12's recorded fault. The
remedy recorded there is a console re-save; two have now been applied (Releases
#12 and #13) without effect.

**Recommendation: delete and recreate `6l1f3tku7z2zoy` the way the video
endpoint was made** — console → Deploy from a GitHub repository, branch
`runpod-comfyui-slim`, `docker/runpod/comfyui-slim/Dockerfile`, the 24/32 GB
tiers, 30 GB disk, same volume. That took about three minutes for the video
endpoint and produced a working worker immediately. It changes the endpoint id,
so `desired-state.json`, the runbook env recipe and `RUNPOD_COMFYUI_ENDPOINT_ID`
all need updating with it.

**Two API lessons worth keeping.** REST reported chroma's `gpuTypeIds` as the
_old_ four-card list (`RTX 4090, A5000, L4, RTX 3090`) while the live worker was
demonstrably on an RTX PRO 4500 added in Release #5 — REST's GPU list for this
endpoint is stale. And container disk is not exposed by REST or GraphQL for a
git-integration template at all. When an endpoint misbehaves, the console's
Releases, Workers and Logs tabs are not a fallback, they are the only source.

### V.02.04 / V.02.05 — the video model pull, done (2026-09-12)

Pod `689tdcv0qscxgt`, CPU-only (`cpu3c`, 2 vCPU, `computeType: CPU`) in EU-RO-1,
created 12:30:33Z, loader finished **13:27:47Z**, pod deleted immediately after.

| Figure                         | Value                                                                                 |
| ------------------------------ | ------------------------------------------------------------------------------------- |
| Wall clock, rented → finished  | **57.2 min**                                                                          |
| Pod cost at $0.06/h            | **$0.0572** (the tracker's table guessed ~$3; the estimate before this run was $0.25) |
| Rows in `manifest-result.json` | 48 = 42 manifest artifacts + 6 companion files                                        |
| `failures`                     | **0**                                                                                 |
| Skipped as already correct     | **16** — the C.02 chroma rows, i.e. the idempotent skip V.02.04 asked to observe      |
| Downloaded this run            | **32 rows, 288.23 GB in 55.4 min → 86.7 MB/s**                                        |
| Volume after                   | **42/42 rows, 358.30 GB of 500 GB**, lint 0 errors 0 warnings, 28.3% headroom         |

**Every one of the 42 artifacts was verified against the sha256 pinned in the
manifest before the pull: 42 matched, 0 mismatched.** That is the check V.02.05
asks for, and it is meaningful in this direction because V.02.02 filled the
hashes from HuggingFace LFS oids _before_ any bytes were fetched — so the pull
was verified against a value recorded in advance, not against itself.

A trap worth recording: the loader keys its result rows `"<id>:<file>"`, and two
artifacts (`rmbg-2.0`, `birefnet-portrait`) carry companion files. Matching on
the id alone compares a companion's hash to its parent's and reports six false
mismatches. Match on `id:file`.

At rest afterwards: `pods: []`, `currentSpendPerHr` **$0.049** — the 500 GB
network volume alone ($0.07/GB/month), nothing else running.

### V.03.04 — the video endpoint, created in the console (2026-09-12)

The GPU question is settled (see below), but creating the endpoint hit a
structural obstacle that is worth writing down because it is not obvious:

1. **A RunPod template backs exactly one endpoint.** Reusing the chroma
   endpoint's template returns
   `graphql: This template is already bound to an endpoint.` D-V2's "one worker
   image" is satisfiable — one image, two templates — but not "one template".
2. **The chroma endpoint's template `j1gzx4nccu` cannot be read or cloned.** It
   is a git-integration template: absent from GraphQL `podTemplates`, and
   `GET /v1/templates/j1gzx4nccu` returns `404 template not found`. So its image
   reference is not recoverable through either API.
3. **`runpod-provision.mjs template create` requires a digest-pinned image**
   (`…@sha256:…`), and a git-integration build has no digest — that is C.01.09.
   The guard is correct and should stay; it just means this path cannot create
   the video template.

So the endpoint has to be created in the console, where a git build can be
attached to a second endpoint. That is a UI action, not an API one.

**Two corrections to V.03.01's GPU note, both from the live API:**

- **REST `POST /v1/endpoints` does not accept pool ids at all.** `BLACKWELL_96`,
  `AMPERE_80`, `HOPPER_141` are rejected; the `gpuTypeIds` enum is concrete card
  names. GraphQL reports the same endpoint's GPUs _as_ pools
  (`ADA_24,AMPERE_24,BLACKWELL_32` for chroma) while REST reports the cards
  (`RTX 4090, A5000, L4, RTX 3090`) — the two APIs are different vocabularies
  over the same thing, and the pool list is an aggregation, not a constraint.
- **H100 80 GB IS available to serverless**, contrary to the inference drawn
  from "there is no 80 GB Hopper pool". `NVIDIA H100 80GB HBM3`,
  `NVIDIA H100 PCIe` and `NVIDIA H100 NVL` are all in the accepted enum. The
  missing `HOPPER_80` pool id meant only that no _pool_ aggregates them.

### V.03.04 outcome — `2lezocymx2rgot`, and what the console revealed

Created through **Deploy from a GitHub repository** (`GreyChimp/oshun`, branch
`runpod-comfyui-slim`, `docker/runpod/comfyui-slim/Dockerfile`), which is the
path the API could not take. RunPod auto-created template `qcxiqduk01`; the
"could not find `runpod.serverless.start()`" warning is expected and was
accepted, because the handler comes from the `worker-comfyui` base image rather
than this repo.

| Field              | Value                                                                                              |
| ------------------ | -------------------------------------------------------------------------------------------------- |
| endpoint id        | **`2lezocymx2rgot`** (`isis-comfyui-video`)                                                        |
| templateId         | `qcxiqduk01`                                                                                       |
| gpuTypeIds         | RTX PRO 6000 Blackwell **Server / Workstation / Max-Q**, H100 **PCIe / 80GB HBM3 / NVL**, **H200** |
| workersMin / Max   | 0 / 1                                                                                              |
| idleTimeout        | 10 s                                                                                               |
| executionTimeoutMs | 900 000                                                                                            |
| flashboot          | true (Standard)                                                                                    |
| networkVolumeId    | `ax0oj250j3`                                                                                       |
| scaler             | QUEUE_DELAY, 4 s                                                                                   |
| container disk     | 30 GB                                                                                              |

Drift check **green on both endpoints**. At rest the account is at
`currentSpendPerHr` **$0.049** — the 500 GB volume alone.

**The finding that matters, and it came from the user: GPU supply labels are
global until a network volume is attached.** Before attaching, the console
showed 96 GB _High Supply_, 80 GB _High Supply_, 141 GB _Medium Supply_. The
instant `isis-chroma-models` was selected the data-center field locked to
**EU-RO-1** and the same three tiers re-read as:

| Tier         | Global        | EU-RO-1         |
| ------------ | ------------- | --------------- |
| 96 GB $3.49  | High Supply   | **Low Supply**  |
| 80 GB $4.79  | High Supply   | **Unavailable** |
| 141 GB $5.93 | Medium Supply | **Unavailable** |

So the three-deep fallback chain is, in the region where the volume lives, a
chain of one. All three stay enabled deliberately — "Unavailable" is a
point-in-time reading, not a property — but **plan on 96 GB or nothing**, and
read supply only after the volume is attached. Anyone sizing a RunPod endpoint
against the pre-volume labels is reading numbers for a datacenter they will not
be deployed in.

Despite the Low Supply label, the endpoint placed a worker immediately:
`workers: {idle: 1, ready: 1}` at rest, which also proves the git build boots.

**Two corrections and a defect fixed on the way:**

- The console's **80 GB serverless tier is H100** ($4.79), not the A100 80GB
  PCIe ($2.72) the pricing page lists — A100 is not offered to serverless here
  at all. The earlier recommendation priced the middle rung wrong; it does not
  change the verdict, since 96 GB at $3.49 is both cheaper and larger.
- GraphQL's `gpuIds` pool string is **lossy in both directions**. The live
  endpoint reports `ADA_80_PRO,BLACKWELL_288,BLACKWELL_96,HOPPER_141` while REST
  reports the seven cards above — with **no B300 anywhere**. REST is the truth.
- `runpod-endpoint-drift-check.mjs` compared desired state against that pool
  string, so it reported drift on a correctly configured endpoint and, worse,
  **could not have seen a wrong card swapped inside a matching pool**. It now
  reads the concrete card list from REST whenever desired-state is written in
  card names, warns loudly if REST cannot answer rather than silently falling
  back to the lossy comparison, and leaves the pool path untouched for the
  chroma endpoint. 11 tests pass; both endpoints read `no drift`.

### The C.11.07/08/09 re-runs were attempted and abandoned (2026-09-12, $0 spent)

The user approved a sub-$0.05 session to capture the `rmbg-2.0` traceback
(C.12.20) and render `sam3-segment` / `sam3-segment-inpaint` (C.11.08/09). It
did not happen, and nothing was billed. Recorded so the next attempt starts from
what was learned rather than repeating it.

| Step                      | What happened                                                                                                                                                                                                        |
| ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `/run` refused            | `409 ENDPOINT_PAUSED (max_workers=0)` — the endpoint was left at `workersMax 0` by an earlier drill.                                                                                                                 |
| Raised to 1               | REST `endpoint update` reported `workersMax: 1` immediately, but `/run` kept answering 409 for about **30 seconds** afterwards. It is propagation delay, not the C.12.12 wedge; retrying after half a minute worked. |
| Job queued                | `68417991-…-e1` accepted, `IN_QUEUE`.                                                                                                                                                                                |
| Worker never became ready | One worker `initializing`, zero in every other state, for **43 minutes**. `runpod-catalog-run.mjs` gave up at its 15-minute cap; the job stayed queued regardless, because that cap abandons the wait, not the job.  |
| Stood down                | Both queued jobs cancelled (`delayTime` and `executionTime` both absent, so **no GPU seconds were billed**), `workersMax` back to 0, health confirmed 0 workers / 0 queued.                                          |

**This is not the wedged endpoint of C.12.12**, and the distinction is worth
keeping: that fault shows **zero** workers in _every_ state with jobs queued,
because placement itself has stopped. Here placement worked and a worker existed
throughout — it simply never finished coming up.

Two candidate explanations, neither confirmed:

1. **The image got much bigger.** V.01.01 added six video node packs and their
   dependencies (cupy, triton, pycocotools, timm, the controlnet_aux
   annotators). The last measured cold boot, before those, was ~4 minutes
   (`d1e87d06`, 14.7 GB).
2. **The network volume was saturated.** The V.02.04 loader pod was writing to
   the same volume at ~3.2 GB/min throughout, and a serverless worker mounts
   that volume as it starts. This is the cheaper hypothesis to test, and it is
   why the re-runs were deferred rather than debugged: **retry them once the
   pull is finished and the loader pod is deleted.** If a worker comes up
   normally on a quiet volume, the answer is (2) and the operational rule is
   simply "do not run jobs against an endpoint whose volume is being seeded".

### C.12.04 — what is actually on the volume (2026-09-12, live listing, $0)

A model inventory built from the manifest alone would have been a fabrication.
The manifest is **declarative** — it says what the loader should put on the
volume — and on 2026-09-12 it declared 42 artifacts while the volume held 10. A
card reading the manifest and calling it an inventory would have reported 288 GB
of models that do not exist.

So presence is observed, and it is a three-state answer: `present` (seen in a
listing), `absent` (a listing was taken and it was not in it), `unknown` (no
listing could be taken). `unknown` never degrades to `absent` — a missing
credential is not evidence that a file is missing — and every observed total is
`null` rather than `0` when unobserved, because "0 present" and "we did not
look" are different sentences to an operator deciding whether to start a
multi-hour pull.

**Measured live through the new endpoint**
(`GET /api/v1/jobs/runpod/volume/models`, authenticated, against volume
`ax0oj250j3` in EU-RO-1, S3 LIST only — free):

| Figure                         | Value                                                       |
| ------------------------------ | ----------------------------------------------------------- |
| Objects on the volume          | 20                                                          |
| Manifest rows present / absent | **10 present, 32 absent, 0 unknown** of 42                  |
| Declared footprint             | 358.30 GB                                                   |
| Held                           | 70.07 GB                                                    |
| **Still to pull (V.02.04)**    | **288.23 GB** — the tracker's estimate said 283 GB          |
| Volume used / capacity         | 70.07 GB of 500 GB, **429.93 GB free** — the pull fits      |
| Size mismatches                | none                                                        |
| Unclaimed objects              | 10 (companion `.py`, `config.json`, `__pycache__`), ~0.4 MB |

Per folder: `diffusion_models` 3 of 9 present (53.40 GB of 236.94 GB), `RMBG` 4
of 4, `sam3` 1 of 2 (the `.pt` ComfyUI-RMBG reads is there; the `.safetensors`
ComfyUI-SAM3's video nodes want is not), `text_encoders` 1 of 3, `vae` 1 of 2,
and `checkpoints`, `clip_vision`, `controlnet_aux`, `detection`,
`FlashVSR-v1.1`, `interpolation`, `latent_upscale_models`, `loras` all 0 — i.e.
exactly the C.02 chroma set is there and the entire video set is not.

**Two defects the browser pass caught that no unit test would have:**

1. The card printed
   `(0.00 GB of it is companion files and job data no manifest row claims)`. The
   real unclaimed total is ~0.4 MB, so the parenthetical fired and said nothing.
   It now only names the figure above 5 MB, and otherwise reports the object
   count with "under 5 MB in total". Pinned by a spec.
2. The six-column folder table was **clipped off the card entirely** — the card
   sat in the page's `auto-fit minmax(220px, 1fr)` grid at 244 px against a 680
   px table. Both tables are now in `.isis-table-wrap` (the app's existing
   overflow container) and the card spans the grid (`gridColumn: 1 / -1`),
   measured after the fix at card 1014 px / table 988 px, fitting, with no
   sideways page scroll at 400 px width.

Screenshot:
`docs/agents/evidence/isis-chroma-runpod/2026-09-12-c1204-volume-inventory.png`.

**A landmine worth the half hour it cost:** killing the `pnpm dev` parent does
NOT kill the tsx child. A stale generation-API process from the previous drill
kept port 4010 with its own `JWT_SECRET`, so a freshly minted token came back
`401 Invalid token signature` on every route — which reads exactly like a
mis-minted token. Teardown must check `lsof` on the port AND
`pgrep -f generation-api`, and kill what is actually listening.

### C.11.11 — budget refusal, for real (2026-09-12, $0, nothing dispatched)

The drill runs against the live dev stack (Postgres + Redis + the generation API
on 4010 + the dashboard on 5180), with the endpoint id and RunPod key configured
exactly as a real session, and **spends nothing**: the gate refuses the submit
before anything reaches RunPod.

**How the exhausted state was created, stated plainly:** the API was started
with `ISIS_RUNPOD_COST_LEDGER_KEY_PREFIX=isis:drill:c1111:cost` — a throwaway
ledger key — and that key was seeded with
`usd 0.5, jobs 2, seconds 1600, coldStarts 1, wf:chroma-txt2img:usd 0.5` against
a budget of `$0.25`. **That spend was seeded, not earned.** The real ledger key
(`isis:runpod:cost:2026-09-12`) was verified absent before the drill, verified
absent after it, and the drill key was deleted at the end. Seeding is the
precondition; the thing under test is the gate's behaviour in that state, and
the alternative — burning $0.50 of real GPU time to reach it — would have
measured the same branch for money.

| Step                                                             | Result                                                                                                                                                                                                              |
| ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GET /jobs/runpod/cost/today`                                    | `totalUsd 0.5`, `budgetUsd 0.25`, `remainingUsd 0`, `usedRatio 2`, `enabled true`                                                                                                                                   |
| `POST /api/v1/jobs` (chroma-txt2img, square_1024, consentId set) | **HTTP 429**, `code RUNPOD_DAILY_BUDGET_EXCEEDED`, `reason budget_exhausted`, `ledger image`, `estimatedCostUsd 0.011`, `retryAfter 43393`                                                                          |
| Response headers                                                 | `retry-after: 43393` (UTC midnight), `x-runpod-daily-budget-usd: 0.25`, `x-runpod-daily-spend-usd: 0.5`, `x-runpod-daily-remaining-usd: 0`                                                                          |
| Nothing was created                                              | job list total stayed **7**, all from 2026-09-11; drill ledger `usd` still `0.5`; no RunPod dispatch in the API log                                                                                                 |
| Dashboard, real chromium                                         | `studio-submit` **disabled**; reason rendered verbatim: `Daily RunPod budget spent: $0.5000 of $0.25 (2026-09-12 UTC). Submits are refused until the UTC day rolls over or ISIS_RUNPOD_DAILY_BUDGET_USD is raised.` |
| RunPod page                                                      | `$0.5000 of $0.25 spent today (2026-09-12 UTC) - $0.0000 remaining · 2 billed jobs · 1600s billed · 1 cold start`                                                                                                   |
| Budget restored to `$3`                                          | submit button **enabled**, reason line gone, card reads `$0.5000 of $3.00 spent today - $2.5000 remaining`                                                                                                          |

Screenshots:
`docs/agents/evidence/isis-chroma-runpod/2026-09-12-c1111-studio-submit-disabled.png`
(greyed Submit Job with the reason above it) and
`docs/agents/evidence/isis-chroma-runpod/2026-09-12-c1111-budget-restored.png`.

**A finding the drill produced on the way:** pointing the dashboard at the live
API with a tenant snapshot whose `projectId` did not match the bearer token's
`projectId` claim produced `Ledger unavailable: Project scope mismatch` on the
budget card — the fail-closed path, reached for a reason that has nothing to do
with money. It is the correct behaviour (a scope mismatch must not be allowed to
read another project's ledger), but it renders identically to a Redis outage, so
an operator seeing that string should check scope before infrastructure. The
dashboard's auth snapshot must carry the same `tenantId`/`projectId` the minted
token claims (`tenant_isis_dev` / `project_isis_dev` from
`scripts/isis/mint-dev-jwt.mjs` defaults).

Teardown: both servers killed,
`lsof -nP -iTCP -sTCP:LISTEN | grep -E ':(3010|3020|4010|5180)'` empty, no
chromium orphans, drill ledger key deleted, throwaway env script deleted.

### V.11.03 — budget refusal on the VIDEO ledger (2026-09-12, $0, nothing dispatched)

Same shape as C.11.11 and the same honesty about it: the exhausted state was
**seeded, not earned**. The API ran with
`ISIS_RUNPOD_COST_LEDGER_KEY_PREFIX=isis:drill:v1103:cost`, whose video key
(`isis:drill:v1103:cost:video:2026-09-12`) was set to
`usd 0.5, jobs 2, seconds 375.657, coldStarts 1, wf:wan22-t2v:usd 0.5` against a
$0.25 budget — 375.657 s x $0.001331/s = $0.50, so the seed is internally
consistent rather than a made-up number. The real key
`isis:runpod:cost:video:2026-09-12` was verified **absent before and after**
(the only real ledger key on the box is `isis:runpod:cost:2026-09-11`), and the
drill key was deleted at the end.

The point of the drill is the SECOND ledger: that exhausting video refuses video
and leaves image alone.

| Step                                                                         | Result                                                                                                                                                                                                                                               |
| ---------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GET /jobs/runpod/cost/today` (one response, one instant)                    | video `totalUsd 0.5`, `budgetUsd 0.25`, `remainingUsd 0`, `usedRatio 2`; image `totalUsd 0`, `budgetUsd 3`, `remainingUsd 3`                                                                                                                         |
| `POST /api/v1/jobs` (`wan22-t2v`, 81 f, 832x480, consentId set)              | **HTTP 429**, `code RUNPOD_DAILY_BUDGET_EXCEEDED`, `reason budget_exhausted`, **`ledger video`**, `estimatedCostUsd 0.16` (the catalog figure), `retryAfter 24544` (UTC midnight)                                                                    |
| Response headers                                                             | `retry-after: 24544`, `x-runpod-video-daily-budget-usd: 0.25`, `x-runpod-video-daily-spend-usd: 0.5`, `x-runpod-video-daily-remaining-usd: 0` — and **no** `x-runpod-daily-*`, because only the video ledger was consulted                           |
| `POST /api/v1/jobs` (`chroma-txt2img`) with the video ledger still exhausted | **HTTP 201**, headers `x-runpod-daily-budget-usd: 3`, `x-runpod-daily-remaining-usd: 3`, no 429 — the two budgets are independent in the direction that matters                                                                                      |
| That image job                                                               | failed at the consent guard before `comfyuiService.submitJob`; **zero** RunPod HTTP calls in the API log, drill ledgers unchanged                                                                                                                    |
| Dashboard, real chromium, motion preset `wan22-t2v`                          | `studio-submit` **disabled**; `studio-motion-budget-block` reads verbatim: `The video budget of $0.25 is spent for today. This is separate from the image budget, so image jobs are unaffected.` The image block `studio-budget-block` is **absent** |
| Dashboard, no motion preset                                                  | `studio-submit` **enabled** — an image submit is not blocked by a spent video budget                                                                                                                                                                 |
| Restored (`ISIS_RUNPOD_VIDEO_DAILY_BUDGET_USD=6`, drill key deleted)         | motion block gone; RunPod page reads `$0.0000 of $6.00 spent today - $6.0000 remaining`, both endpoints `configured · routable`                                                                                                                      |

Screenshots:
`docs/agents/evidence/isis-chroma-runpod/2026-09-12-v1103-studio-video-budget-block.png`,
`2026-09-12-v1103-runpod-two-meters.png`,
`2026-09-12-v1103-budget-restored.png`.

**Three defects the drill found, all fixed here.** None of them is about
budgets; they are what a live path shows that a mocked one does not.

1. **The C.12.17 double-billing guard was being thrown away.**
   `createWorkflowError` returns a plain object, not an `Error` subclass, and
   `classifyError` only read `.code` off a value that was `instanceof Error`. So
   every error the executor _throws_ was re-derived from its message text and
   came back `UNKNOWN_ERROR`, `retryable: true`. The live witness: a
   consent-less image submit raises `INPUT_VALIDATION_FAILED` with
   `retryable: false` and was observed running **four times** (attempts 0-3).
   The same seam carries `retryable: isProviderResultRetryable(result)` — the
   C.12.17 decision that stops a render which has already started billing from
   being re-submitted — so that fix was live in the code and dead on the wire.
   `classifyError` now returns an already-classified error verbatim, message
   patterns being reserved for foreign errors; `details` and `retryDelayMs`
   survive too, which restores the `timeoutPhase` the `EXECUTION_TIMEOUT`
   handler logs. Re-measured live: **one** attempt, `INPUT_VALIDATION_FAILED`,
   `retryable false`, zero RunPod calls. `error-boundary.spec.ts` (10 tests)
   pins it.

2. **The dashboard's whole video half had never rendered against the live API.**
   `/ready` **spreads** its authenticated evidence into the response body, so
   `runpodChroma` and `checks` are top-level; the client typed them under
   `detail` and `RunPodPage` read `data?.detail?.runpodChroma`, which is always
   `undefined`. The video budget meter, the endpoint table and the **video kill
   switch** therefore showed their "no video endpoint is configured" empty
   states against a healthy, configured, routable endpoint. Type flattened, page
   fixed. After: `$0.5000 of $0.25 spent today`,
   `Video (motion) — 2le…got · configured · routable`, and a working
   `Stop video workers` button.

3. **`/ready` computed `checkedAt` and never returned it,** so no consumer could
   age the verdict. Returned now. Separately, `getReadiness()` discarded the
   **503** body, which for `/ready` is the answer rather than a failure — a "not
   ready" snapshot is exactly what an operator needs, and on this dev stack
   `/ready` is permanently 503 because MinIO is deliberately not run (runbook
   §1). It now reads the snapshot off the error (`body` or `details`, matching
   the sibling `checkServiceReadiness`) and still throws on a 401 or a dead
   socket. `generation.readiness.spec.ts` (5 tests) pins it.

**Not a defect, for the record:** after deselecting a motion preset the Submit
button stays disabled. The preset leaves `type: text-to-video` on the form,
which is a cost-warning generation type, so the RunPod cost acknowledgement is
required; ticking it enables the button. Verified in chromium.

**Also corrected while the preflight was blocking the stack:** the runbook
billed every image second at the 4090's $0.000306, but the rebuilt endpoint's
`gpuIds` accept an **RTX 5090** ($1.58/h = $0.000439/s). A list means "any of
these" and the ledger cannot know which card ran, so it must bill the dearest —
under-billing is the direction that stops a budget firing.
`ISIS_COST_RATE_RUNPOD_USD_PER_SECOND` is now `0.000439`. `NVIDIA L4` and
`NVIDIA GeForce RTX 3090` are in that list with no established serverless flex
rate; the preflight names them rather than guessing, and RunPod's API publishes
no serverless price to settle it.

### V.11.04 — video kill switch, for real (2026-09-12, $0, nothing rendered)

Engaged from the **dashboard**, not the API: arm `runpod-kill-switch-arm-video`,
confirm `runpod-kill-switch-confirm-video`. The card then read
`Workers stopped at 5:33:21 PM — workersMax is now 0 (was 1).` and the image
card showed no result at all, which is the target-scoping the control promises.

| Step                                    | Result                                                                                                                                                                      |
| --------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Before                                  | video `workersMin 0 / workersMax 1`, image `0 / 1`                                                                                                                          |
| After the dashboard kill switch         | video **`workersMax 0`**, image still `1` (REST read)                                                                                                                       |
| `POST /api/v1/jobs` (`wan22-t2v`)       | **failed**, `The RunPod endpoint is stopped (workersMax is 0) and is accepting no work. Raise workersMax in the RunPod console to resume.` — one attempt, `retryable false` |
| RunPod `/health` for the video endpoint | `inQueue 0`, `inProgress 0`, every worker count 0                                                                                                                           |
| Both ledgers                            | image `$0 of $3`, video `$0 of $6`, `billedSeconds 0`; the only real ledger key on the box is still `isis:runpod:cost:2026-09-11`                                           |
| Restore                                 | `PATCH {"workersMax":1}`; a field-by-field diff of the endpoint before and after shows **1 of 19 fields changed** and `gpuTypeIds` intact                                   |
| Drift check                             | `no drift (2 endpoints checked)`                                                                                                                                            |

Screenshot:
`docs/agents/evidence/isis-chroma-runpod/2026-09-12-v1104-video-kill-switch.png`.

**The tracker's expectation was wrong, and the truth is better.** V.11.04 said
"submit, confirm `IN_QUEUE` then cancellation by the API timeout". RunPod does
not queue against a paused endpoint at all — it refuses `/run` with **409
`ENDPOINT_PAUSED`**,
`"Endpoint is paused (max_workers=0). Set max_workers > 0 to accept work."` So
there was no `IN_QUEUE` state to observe and nothing to cancel, and the kill
switch is stronger than the item assumed: no backlog builds up behind it to
stampede when the ceiling goes back up.

**Two defects found on the way, both fixed here.**

1. **Every `motion` workflow was refused before dispatch.** The node-class
   preflight read `infra/runpod/endpoints/inventory/comfyui-chroma.json`
   unconditionally, so a video graph was validated against the IMAGE worker
   image. `wan22-t2v` failed with `UNKNOWN_NODE_CLASS` naming
   `CreateVideo, EmptyHunyuanLatentVideo, ImageFromBatch, KSamplerAdvanced, ModelSamplingSD3, SaveVideo`
   — classes the video image has had since it was built and which
   `comfyui-video.json` lists (53 core + 16 custom).
   `resolveNodeClassInventoryFile` now routes on the catalog category, the same
   rule as the ledger and the endpoint id, with a separate
   `ISIS_RUNPOD_VIDEO_NODE_CLASS_INVENTORY_FILE` override so the old single
   variable cannot capture video jobs. `node-class-inventory-routing.spec.ts` (7
   tests) pins it, including that the video inventory really carries those six
   classes and the image one really does not. **This blocked V.11.01 and V.11.02
   too** — no clip of any kind could have reached a GPU.

2. **A paused endpoint read as "Unexpected RunPod provider error."** 409 fell
   through the status switch to `UNKNOWN`. There is now a
   `RUNCOMFY_ENDPOINT_PAUSED` code with a message that names the cause and the
   fix, classified off RunPod's own `code`/`detail` before the status switch,
   and non-retryable — the endpoint stays paused until a human raises it, so a
   retry is only another 409. `runpod-adapter.endpoint-paused.spec.ts` (3 tests)
   pins it, including that an unrelated 409 still reads as `UNKNOWN`.

**A test-fixture landmine worth recording:** a mocked `fetch` response without
`headers` makes the adapter lose the HTTP status while building the error, and
the retry predicate then treats a 4xx as a network blip and retries it four
times with backoff. The first version of the spec timed out at 15 s for exactly
this reason while the live path failed in 588 ms. Mock `headers: new Headers()`.

**Also fixed:** `runpod-endpoint-drift-check.mjs` defaulted to
`--key comfyui-chroma`, so from the moment the video endpoint existed a bare
"drift check, no drift" was a statement about half the fleet. A bare run now
checks every endpoint in the environment and prints how many. Its test file was
also still pinning the pre-rebuild chroma endpoint id `6l1f3tku7z2zoy` and
failing; updated to `6jye7vngg4i685` and extended to cover the video entry.

### V.11.01 — the first clip, live (2026-09-12)

**Rendered.** `wan22-t2v`, 33 frames, 832x480, catalog defaults, seed 20260912,
submitted through the generation API so the ledger, provenance and output
persistence were all exercised.

|                   |                                                                                                                                         |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| Isis job id       | `bd345ee6-55c9-4171-8500-a8ad76574e24`                                                                                                  |
| RunPod job id     | `e87838c0-da47-45f1-ab31-69a2f205ffa7-e2` (4th attempt; the first three never got a card)                                               |
| Worker            | `b3vnt2h8obe53d`, EU-RO-1                                                                                                               |
| `delayTime`       | **528 347 ms** (8.8 min queue + cold boot)                                                                                              |
| `executionTime`   | **375 411 ms** (6.3 min)                                                                                                                |
| Ledger            | **$1.202902**, `billedSeconds 903.758`, `coldStarts 1`                                                                                  |
| Clip              | `wan22-t2v_00001_.mp4`, 211 885 bytes, sha256 `31e44a9c9d11552851d7075b50004c2c7c547f2c33b42f0d3a128f377160c7a1`                        |
| Last frame        | `wan22-t2v-lastframe_00001_.png`, 358 467 bytes, sha256 `57f5cdcef7d6dfd11d4a772073f2d48588d68f6357ea284d2da1c59779e06621`              |
| Stamped facts     | `kind video`, `codec avc1`, `33 frames`, `16 fps`, `832x480`, `durationMs 2063`, `hasAudio false`                                       |
| Independent probe | `ffprobe`: `h264`, `832x480`, `16/1`, `nb_frames 33`, `duration 2.0625`, `size 211885` — every stamped fact matches                     |
| Provenance        | bundle `prov_bbcfcdf9601fd29a`, consent `consent_operator_isis_dev_studio`, `watermark { applied: true, method lsb-blue-v1, images 1 }` |

Files: `docs/agents/evidence/isis-chroma-runpod/v1101/`.

**It took four attempts and three of them were free.** EU-RO-1 had no card: the
first attempt was `throttled` after twenty seconds, the second was handed a
`ready` worker at 18:14:40Z and had it taken away at 18:15:00Z, the third
expired in the queue. Each was cancelled by the C.12.17 **queue-phase**
deadline, which reports `retryable: true` precisely because nothing was billed,
so the executor re-submitted without a human. RunPod's billing panel read
`$0.00` throughout. The fourth attempt got a card at 18:26:26Z — 54 seconds
before its own deadline — and ran. Adding the two A100 80 GB variants between
attempts 2 and 3 did not cause this; that tier reads Unavailable too (runbook
§2f).

**Three numbers that are now measured rather than guessed:**

1. **The clip cost $1.20, not the $0.16 the catalog sticker implied.** The
   sticker said 120 s for 81 frames; reality is 375 s for 33 frames — about 7.7x
   the sticker per frame. `estimated_seconds_per_megapixel_frame` and
   `cost_calibration` are V.11.05's to rewrite, and `proof_level` for
   `wan22-t2v` can now go to `rendered`.
2. **The ledger was over-stating spend by 4x, and the account balance is what
   caught it.** The ledger booked **$1.2029**; the RunPod balance moved about
   **$0.29**. That is not rounding, it is a wrong rule: `billedSecondsFor` added
   the whole of `delayTime` whenever `coldStart` was set, on the assumption that
   the delay before a cold start is the worker booting. Here `delayTime` was 528
   s of **queue wait for a card** — the job made four attempts before one was
   placed — and queue wait is not worker time and is never billed. $0.29 over
   375.4 s of execution is ≈ $0.00077/s, which is the **A100 80 GB tier's
   $2.72/h**, so the clip ran on one of the cards added an hour earlier and was
   charged for its execution and nothing else.

   Over-stating is not the safe direction even though it looks like it: a ledger
   that invents spend refuses submits the budget could afford — at $1.20 a clip
   the $6 video budget would have refused V.11.02 partway through on spend that
   never happened — and it makes the daily figure useless as a record of what
   was charged. `billedSecondsFor` now bills execution only; the wait is
   recorded as `unbilledDelaySeconds` on the snapshot so a clip that took a
   quarter of an hour and was charged for six minutes still explains itself.

   **What this does not yet separate is the boot**, which is inside `delayTime`
   and probably is billed. It is small next to a nine-minute queue wait, and
   V.11.02's warm session is the measurement that isolates it: warm clips have
   `delayTime` near zero, so the gap between the session's total charge and the
   sum of its execution times is the boot. Recorded for V.11.05.

   (Separately, the ledger used the pre-correction rate 0.001331 because the API
   process predated that change. The rate correction stands for an H200; this
   job did not run on one.)

3. **The dashboard was calling an estimate a cost.** The output file carries
   `costUsd` stamped from `metadata.estimatedCostUsd` — $0.5226 for this clip —
   and the Outputs badge rendered it as `cost`, against the ledger's $1.2029 for
   the same render. Two different quantities, one of them labelled as if it were
   the other. The badge now reads `est. cost`, the Jobs line prefixes `~`, and
   the client type says which is which.

### V.09.07 — the clip plays in a real browser (2026-09-12, $0, no new generation)

Chromium, against the live API, on the V.11.01 job. The unit specs assert the
markup and jsdom has no media pipeline, so the only thing that can tell a
working player from a `<video>` pointed at a 404 is a browser that decodes the
bytes.

| Check                      | Result                                                                                                     |
| -------------------------- | ---------------------------------------------------------------------------------------------------------- |
| `<video>` on the Jobs page | present, `controls`, `playsInline`                                                                         |
| `readyState`               | **≥ 2 (HAVE_CURRENT_DATA)** — the decoder produced a frame                                                 |
| Decoded dimensions         | `videoWidth 832`, `videoHeight 480`, `duration 2.0625` — matching `ffprobe` and the stamped facts exactly  |
| Playback                   | `currentTime` advanced **0 → 1.2023 s** under `play()`; it is not a still                                  |
| Card text                  | `seed 20260912 · 2.06s · 33f · 16fps · 832×480 · silent · ~$0.5226 · 375.4s · 528.3s cold start · 2 files` |

Screenshot:
`docs/agents/evidence/isis-chroma-runpod/2026-09-12-v0907-clip-card.png`.

The card also confirms the estimate-labelling fix is live: the figure renders as
`~$0.5226`, not as a bare cost.

**A finding about the Outputs page, not the clip:** `/outputs` renders no video
and its network log shows `ERR_CONNECTION_REFUSED` against
`http://localhost:3002/api/v1/outputs` and `/api/v1/policies`. That page is
backed by a **separate outputs service on :3002** which the dev-stack recipe
does not start, so it is empty by configuration rather than broken. The clip
card that V.09.07 is about lives on **`/jobs`**, which reads the generation API
directly. Worth knowing before anyone debugs an empty Outputs page as a
persistence failure.

### C.09.08 — the first Chroma image in a real browser (2026-09-12, $0)

Chromium against the live API, on the C.11.01 job
`4bae0703-cb80-4a61-be70-3519d057504a`.

| Check     | Result                                                                                                                    |
| --------- | ------------------------------------------------------------------------------------------------------------------------- |
| Job row   | `text-to-image`, **completed**, 100%, `queued 1.4s` / `cold start`                                                        |
| Thumbnail | present, and the browser **decoded it**: `naturalWidth 1024`, `naturalHeight 1024`, from a `data:image/png;base64` source |
| Subject   | the red bicycle of C.11.01's prompt, so it is that render and not a placeholder                                           |

Screenshot:
`docs/agents/evidence/isis-chroma-runpod/2026-09-12-c0908-chroma-image-card.png`.
Vite killed afterwards; `lsof` on 5180 empty.

**The item's premise about the Outputs page is wrong, for a structural reason.**
C.09.08 and V.09.07 both say "the Outputs card", and `/outputs` will never show
either render: it is backed by a **separate `@isis/output-registry` service on
:3002**, and the generation API never writes to it — there is no call to it
anywhere in `apps/isis/generation-api/src`, and `persistJobOutputs` writes to
its own store. Starting :3002 would not help; it holds a different corpus. The
card for anything the RunPod path produced is on **`/jobs`**, which reads the
generation API directly, and that is where both passes were done.

### V.11.02 — the warm-chained matrix, 4 of 6 (2026-09-12)

One invocation of `scripts/isis/motion-live-proof.mjs`, all stages submitted up
front so they queue behind each other and one worker takes them back to back.
**The warm chain worked**: one cold start for the whole session — stage 1 waited
1 093 417 ms for a card and every stage after it started immediately.

| Stage | Workflow               | Status        | Execution | Output                                                                                              |
| ----- | ---------------------- | ------------- | --------- | --------------------------------------------------------------------------------------------------- |
| 1     | `wan22-i2v`            | **COMPLETED** | 409.7 s   | `1-wan22-i2v-0-fdb6408f.mp4` 196 548 B, h264 832×480 16 fps **33 f**, + last-frame PNG 382 682 B    |
| 2a    | `gimm-vfi-interpolate` | **COMPLETED** | 33.5 s    | `2a-…-987aca7d.mp4` 215 191 B, **65 f at 32 fps** — the interpolator really doubled 33→65 and 16→32 |
| 3     | `birefnet-video-matte` | **COMPLETED** | 27.3 s    | `3-…-3679e250.webm` 85 103 B (vp9, `alpha_mode=1`) + mask `3-…-2c33ed6a.mp4` 46 489 B               |
| 4     | `sam3-track-then-vace` | **COMPLETED** | 343.2 s   | `4-…-266b2ab6.mp4` 179 583 B, h264 832×480 16 fps 33 f                                              |
| 5     | `wan22-animate`        | **FAILED**    | 100.5 s   | `WebSocket communication error: ComfyUI HTTP unreachable during websocket reconnect`                |
| 6     | `ltx23-av`             | **FAILED**    | 25.9 s    | `ComfyUI server (127.0.0.1:8188) not reachable after multiple retries.`                             |
| 2b    | `flashvsr-upscale`     | **FAILED**    | 0.4 s     | same, after 391 ms                                                                                  |

**Cost, execution only** (queue wait is unbilled — V.11.01): **$1.549** at the
H200 rate the ledger is configured for, or **$0.711** if the card was the A100
80 GB tier, which V.11.01's billing evidence suggests it is. V.11.01's clip plus
this session is therefore roughly **$1.00–1.83** against the $1.50–2.60
approved.

**Stage 5 killed ComfyUI at the SMALLEST legal shape, and no cap could have
stopped it.** It ran 33 frames at 832×480, 20 steps, batch 1, on a 33-frame
driving clip — the floor of what `MOTION_CAPS` permits — and every cap passed.
There is therefore no tighter number to set. What the caps bound is the job
SHAPE; what died was the model FOOTPRINT (Animate's 14B weights beside
`clip_vision_h` and the ONNX pose and face models, resident together), and the
table has no expression for that. **The cause is not established**: the worker
log could not be retrieved, because the fetch for it has no timeout and hung the
run for fifty minutes — its own fix, now 20 s. "It died" is the whole of what is
known, and it is not recorded as an OOM without the log that would say so.

`wan22-animate` now carries a `known_failure` note in its catalog entry, and the
runner gives it `last: true` so it is submitted only after every other stage has
settled. Nothing queues behind a stage that can take the worker down — which is
exactly how FlashVSR lost its only chance to run.

**Stages 6 and 2b are collateral, not independent failures.** The signature —
`HTTP unreachable during websocket reconnect` — is the same one C.11.04 produced
when two 896×1152 latents would not fit 24 GB: the process died rather than
raising. `wan22-animate` loads the Animate 14B model beside `clip_vision_h` and
the ONNX pose/face models, and it went down 100 s in. The two stages behind it
in the queue then hit a server that no longer existed, one of them in 391 ms.
**They never ran and must be re-run once, not counted as proof of anything.**
Stage 2b in particular has still never exercised the FlashVSR pack.

**Stage 3's alpha is proved from the MASK, not the webm, and the distinction
matters.** The tracker asks for a decoded frame asserting real alpha rather than
a black matte. The webm carries `alpha_mode=1` at the container level, but
decoding it to RGBA yields 399 360 samples all at 255 — one distinct value. That
is far more likely to be **ffmpeg not decoding VP9's alpha side-channel** than
an empty matte, so it is recorded as **inconclusive as performed** rather than
as a pass or a failure. The graph's second output settles it: the mask video's
first frame has **220 distinct grey levels, min 0 max 255, 85.6 % fully black
and 14.4 % soft edge**. A black matte would have exactly one level. It is a real
matte.

**V.11.02 stays `[ ]`** — the item is marked only when all six rows exist, and
three are open.

#### The re-run of the two that never ran (2026-09-12, 20:47Z, $0.172)

`flashvsr-upscale` and `ltx23-av` had failed against a server that no longer
existed, so they were owed a FIRST attempt, not a re-run. `wan22-animate` was
deliberately excluded: it had had its attempt, and re-running a known
worker-killer buys nothing its `known_failure` note does not already say.

| Stage                     | Status                                          | Execution | USD     | Detail                                                           |
| ------------------------- | ----------------------------------------------- | --------- | ------- | ---------------------------------------------------------------- |
| 2a `gimm-vfi-interpolate` | **COMPLETED** (again, to regenerate 2b's input) | 22.7 s    | $0.0374 | `2a-…-bdba2417.mp4` 215 740 B                                    |
| 6 `ltx23-av`              | **FAILED — and this one was OUR bug**           | 0.3 s     | $0.0005 | `Node 8 (LTXVImgToVideoInplace): required_input_missing 'image'` |
| 2b `flashvsr-upscale`     | **FAILED**                                      | 81.3 s    | $0.1339 | `Job processing failed` — no detail, worker log fetch timed out  |

**`ltx23-av` is fixed and is owed a re-run.** The graph carried
`LTXVImgToVideoInplace` in every mode with `bypass: {{mode == 't2v'}}`, on the
reasoning that the node knows how to do nothing. **ComfyUI validates required
inputs BEFORE it honours bypass**, so in `t2v` the whole prompt was rejected in
324 ms. `optional_image_input` gained `passthrough_input`, which removes the
node and repoints its consumer at the empty latent; t2v renders 19 nodes instead
of 20 and `LTXVConcatAVLatent.video_latent` moves from `["8",0]` to `["7",0]`.
**The offline validator V.01.09 describes would have caught this for $0** — that
item now has a concrete argument behind it rather than a theoretical one.

**`flashvsr-upscale` remains completely unproven** and I cannot say why. It ran
81 seconds and returned a bare `Job processing failed`, with no
`output.errors[]` and no retrievable worker log (§6c: the log dies with the
worker). It is the only pack in the matrix that has never executed successfully.

**What has been ruled out, offline and for $0.** Both `flashvsr-upscale` (5
nodes) and `wan22-animate` (20 nodes) were re-rendered and audited for the class
of defect that killed `ltx23-av`: dangling links, null or empty inputs, and
unresolved `{{…}}` templates. **Both are clean on all four.** So neither is a
graph-shape bug, and the "just render it and look" approach that fixed
`ltx23-av` has nothing left to find here. That narrows both to runtime inside
the container — a pack import, a model load, or memory — which is precisely what
cannot be seen from outside without either the worker log (a race we lose) or
the object-info snapshot V.06.06 needs. **"I don't know why" is the accurate
answer, and it now has a boundary around it rather than being an absence of
effort.**

**One loose thread, recorded rather than explained:** the second `gimm-vfi` run
produced sha256 `bdba2417…` where the first produced `987aca7d…`, from the same
input clip and the same seed. Most likely encoder non-determinism, but that is a
guess and has not been established.

### C.11.08 / C.11.09 / C.11.04 — SAM 3 works, and the cap holds (2026-09-12)

**C.11.08 and C.11.09 were marked BLOCKED and were not.** The block was
`Node 'SAM3Segment' not found` — the pack's `py/AILab_SAM3Segment.py` imports
`sam3` at module level and the auto-loader swallowed the ImportError. C.12.18
landed that runtime and the rebuilt image's inventory lists `SAM3Segment`. The
blocker note had gone stale; nothing needed fixing, only running.

Both ran on the image endpoint, execution-only cost at $0.000439/s:

| Item    | Workflow                                                        | Execution | USD         | Output                                                                                |
| ------- | --------------------------------------------------------------- | --------- | ----------- | ------------------------------------------------------------------------------------- |
| C.11.08 | `sam3-segment`, prompt `the red bicycle`                        | 21.2 s    | **$0.0093** | RGBA cut-out `c1108-0-4a786341.png` 1 391 235 B + mask `c1108-1-7ab8f633.png` 6 260 B |
| C.11.09 | `sam3-segment-inpaint`, same prompt, `a vintage green bicycle…` | 207.2 s   | **$0.0909** | mask `c1109-0-7ab8f633.png` + result `c1109-1-fe8ba740.png` 1 192 190 B               |

**The mask covers the object, checked by decoding it rather than by trusting
it.** Decoded with the repo's own PNG reader: 1024×1024, **3.9 % white / 96.1 %
black, 2 distinct values** — a hard-edged binary mask — and rendered it is an
unmistakable bicycle silhouette: both wheels, frame, saddle, handlebars and rear
rack. The 3.9 % coverage is low only because a bicycle is mostly thin tubes and
air. The RGBA cut-out's alpha agrees exactly (3.9 % opaque, 96.1 % transparent).

**C.11.09's mask is byte-identical to C.11.08's** — same sha `7ab8f633`, same 6
260 B — which is the determinism you want from the same prompt on the same
image. Its result image decodes to 256 distinct levels across the frame (a real
photograph, not a flat fill) and shows the red bicycle replaced by a green one
with the wall, pavement, sky and lighting preserved. The wall shadow is still
the ORIGINAL bicycle's, which is correct: inpaint only repaints the masked
region.

**C.11.04's cap now refuses its own killer, for $0.** The 2026-09-11 run of
`batch_size 2` at 896×1152 killed ComfyUI outright. C.12.19 made the batch cap
scale with resolution. Re-submitting **the same combination** through the live
API now returns `batch_size must be <= 1` and **never reaches a GPU** — the log
has no dispatch line for it. The item asked whether the per-job caps and the
memory headroom hold; the headroom does not and the cap now enforces that.

### D-V2 — the image endpoint's cold start is not measurable while the region is starved

V.11.01 asks for the image endpoint's cold start after the rebuild, for D-V2.
Measured: `delayMs 1 314 570`, `execMs 82 014`.

**That delay is not a cold start and must not be reported as one.** RunPod's
`delayTime` is wall time from submission, and 21.9 minutes of it was waiting for
a card in a region with no capacity. C.11.01's pre-rebuild figure — 53 334 ms —
was a genuine boot measured when capacity existed. Comparing 1 314 570 against
53 334 would claim a 24× cold-start regression that the numbers do not support.
**D-V2 stays unanswered**, and the reason is recorded rather than papered over:
the metric it depends on is not observable while `delayTime` is dominated by
queue wait. A warm-session measurement isolates it (V.11.02's note).

What IS comparable is execution on the identical job — same workflow,
checkpoint, seed and size: **150 127 ms before the rebuild, 82 014 ms after**.
The render got faster, not slower.

The ledger correction is visible here too: the image ledger recorded
`usd 0.036004`, `billedSeconds 82.014`, `unbilledDelaySeconds 1314.57` — the
execution billed, the wait reported and not charged.

### Region probe — US-CA-2 places a card in under a minute (2026-09-12)

The question EU-RO-1 has been raising all day: is the starvation regional, or is
it our card list? Answered by building the smallest thing that can tell the
difference — supply labels only become truthful once a volume pins the
datacenter (§2f), so a probe has to own a volume.

| Step                                                                                          | Result                                                                          |
| --------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| 10 GB network volume `mxugcfkswl` in **US-CA-2**                                              | created                                                                         |
| Throwaway endpoint `bbedw1lqlb7a9u` on it, **the identical nine cards** as the video endpoint | created                                                                         |
| Worker placement                                                                              | **`initializing` within a minute, then `ready`/`idle` — with ZERO jobs queued** |
| EU-RO-1 at the same instant, same nine cards                                                  | **`throttled`, 2 jobs actually waiting**                                        |

**US-CA-2 handed out a card for nothing; EU-RO-1 would not hand one out for
work.** That is the comparison that matters, and it says the constraint is the
region rather than the selection — the same nine card ids were on both.

**What the probe does NOT show.** It ran a tiny `runpod/base` image, so it
measures **card availability, not boot time**; a 288 GB volume and a multi-GB
image would still have to be pulled in US-CA-2, and nothing here says how long
that takes. It is also one reading at one moment, which is exactly the caveat
§2f already records about supply labels.

**It cost more than I said it would, and the reason is worth keeping.** I quoted
"well under $0.05". Creating a serverless endpoint through the REST API
**defaults `workersStandby` to 3 and `workersMax` to 3**, and a standby worker
starts immediately **with no job submitted** — so the probe began billing the
moment it existed. `PATCH {"workersMax":0}` did not stop it, because standby is
a separate field and the patch that tried to set both returned nulls. Deleting
the endpoint was what stopped it. Call it three to four minutes of one worker on
an unknown card: **$0.17–0.36**, not under $0.05.

Both our real endpoints carry `workersStandby: 1`. They are not billing for it
today only because EU-RO-1 cannot place a worker at all — the starvation is
suppressing a standing cost rather than the configuration being safe. **If
supply returns, that setting is worth re-checking before it is discovered on an
invoice.**

Probe torn down completely: endpoint, volume and template all deleted, verified
by listing both — only `ax0oj250j3` (the real 500 GB EU-RO-1 volume) remains.

## C.11 Live generations

One row per run. `cold` = worker had to start; `warm` = worker already up.

| Date              | Item                          | Workflow       | Checkpoint           | Job id        | RunPod job id                                                                                                                          | GPU                                         | State                                                                                                                                                                                                        | delayTime ms                   | executionTime ms | USD                                                                      | Output sha256     | Thumbnail                                                                                    |
| ----------------- | ----------------------------- | -------------- | -------------------- | ------------- | -------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------ | ---------------- | ------------------------------------------------------------------------ | ----------------- | -------------------------------------------------------------------------------------------- |
| 2026-09-11 16:24Z | C.11.01 attempt 1             | chroma-txt2img | chroma1-hd           | add2adbe-967f | (slim image, release v3)                                                                                                               | RTX 4090 worker never started its container | FAILED: RunPod `error creating container … exit status 1`; Isis timed out at 300 s and cancelled server-side; queue purged                                                                                   | —                              | —                | 0 (nothing billed)                                                       | —                 | —                                                                                            |
| 2026-09-11 17:09Z | C.11.01 attempt 2             | chroma-txt2img | chroma1-hd           | 1a90ccfd-9a4f | prompt cb66c5bf (stock worker + injected extra_model_paths)                                                                            | RTX PRO 4500 Blackwell (EU-RO-1)            | RENDERED (Prompt executed in 150.89 s) but Isis refused completion: `without provenance consentId` — image lost, cost booked                                                                                 | 0                              | 154379           | 0.04724                                                                  | —                 | —                                                                                            |
| 2026-09-11 17:23Z | C.11.01 cold                  | chroma-txt2img | chroma1-hd           | 4bae0703-cb80 | prompt 623efed1                                                                                                                        | RTX PRO 4500 Blackwell (EU-RO-1)            | completed; provenance bundle prov_6e11f360e7f5f906 (modelVersion 0e0c60ec…, watermarkPolicy unwatermarked)                                                                                                   | 53334 (cold)                   | 150127           | 0.08305                                                                  | 0d755be254b495a6… | docs/agents/evidence/isis-chroma-runpod/2026-09-11-chroma-txt2img-chroma1-hd-seed42.png      |
| 2026-09-11 17:29Z | C.11.01 second run, attempt 1 | chroma-txt2img | chroma1-hd (seed 7)  | 7d4280f9-e808 | prompt 976fcda2                                                                                                                        | RTX PRO 4500 Blackwell                      | RENDERED (Prompt executed in 148.98 s, RunPod completed=3) but LOST: the API restarted at 17:31:28Z (tsx watch reload from the commit hook) mid-poll; job left `running`, cancelled by hand; tracker C.12.16 | —                              | 148980 (RunPod)  | ~0.046 billed, not in ledger                                             | —                 | —                                                                                            |
| 2026-09-11 17:34Z | C.11.01 second run, attempt 2 | chroma-txt2img | chroma1-hd (seed 7)  | d1e87d06-a22a | first RunPod job cancelled at the 300 s cap (fresh host pulled the 14.7 GB image ~4 min, container stopped 17:39:28Z); retry completed | RTX PRO 4500 Blackwell                      | completed; bundle prov_c593445f9478d045; sizeBytes now 1474325 (fix verified)                                                                                                                                | 53750 (cold again: host churn) | 148165           | 0.08195 (+ the cancelled attempt's GPU seconds, not in ledger — C.12.17) | 0fdba9a4aa8b59d9… | docs/agents/evidence/isis-chroma-runpod/2026-09-11-chroma-txt2img-chroma1-hd-seed7.png       |
| 2026-09-11 17:43Z | C.11.01 warm                  | chroma-txt2img | chroma1-hd (seed 11) | 43cca039-348e | submitted 20 s after the previous completion                                                                                           | RTX PRO 4500 Blackwell                      | completed; bundle prov_3f9980f99f21bab8                                                                                                                                                                      | 9447 (warm)                    | 152871           | 0.08415                                                                  | 0a41b8eb6dff32da… | docs/agents/evidence/isis-chroma-runpod/2026-09-11-chroma-txt2img-chroma1-hd-seed11-warm.png |

## Session rest-state checks (C.11.13)

| Date | Endpoint workers | Pods | currentSpendPerHr | Ledger USD (day) | RunPod billing USD (day) | Gap explanation |
| ---- | ---------------- | ---- | ----------------- | ---------------- | ------------------------ | --------------- |

## Pre-existing test failures on the branch (not caused by this initiative)

Bisected 2026-09-11 by reverting every file this initiative touched (executor,
comfyui.service, shared/config, ai-providers comfy adapter + types, workflows
registry/schema/taxonomy, the chroma catalog folder and the new modules) to
commit `480f69fe8b` and re-running the executor spec: the same 11 tests fail
before and after. They are therefore out of scope here and left as found:

| Spec                                                      | Tests                                                                                                                                                                                                    | Symptom                                                                                                      |
| --------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| `apps/isis/generation-api/src/workflows/executor.spec.ts` | certifies guided-journey / lecture-scene / mandala-animated / meditation-scene; applies voice consistency profiles; executes sequence generation; maps canonical ControlNet major modes                  | `result.success` false with the deterministic test ComfyUI service (video/audio/control families)            |
| same                                                      | does not resolve generation types that still have no dedicated catalog workflows; tracks resolve_success/resolve_failure; returns explicit unresolved workflow error; fails closed with production guard | the spec assumes `inpainting` has no catalog workflow, but `composition/inpaint-edit` (2026-08-02) routes it |

The chroma catalog is gated out of the resolver unless the worker-comfyui
contract is active, so it does not change any of those outcomes.

## Compiled JS beside TypeScript sources (repo hazard, 2026-09-11)

`libs/shared/config/src/{env,features,index,loader,schemas}.js` (+ `.d.ts`) are
git-tracked build outputs. Vite/vitest resolves `.js` before `.ts` for
extension-less relative imports, so an edit to `features.ts` is invisible at
test/runtime until the `.js` is regenerated. This initiative regenerated
`features.js`/`index.js` (+ `.d.ts`) with the library's own `tsc` after adding
`getRunPodWorkerKind`. 112 such shadow files exist under `libs/isis` and
`apps/isis` `src/` trees; not cleaned up here.

## Pre-existing defect fixed on the way (2026-09-11)

`ComfyUIService.cancelJob(isisJobId)` forwarded the Isis job id to
`RunPodAdapter.cancelJob`, whose supervision maps are keyed by the RunPod job
id, so a cancel never aborted the poll loop and hit `/cancel/<isis id>` on
RunPod. The adapter now records `metadata.jobId → RunPod id` at submit and
resolves either id (`runpod-adapter.cancel-mapping.spec.ts`); the cancel route
additionally aborts the in-flight executor run
(`jobs-cancel-propagation.route.spec.ts`). Without this, a cancelled job kept
burning billed GPU seconds until the 300 s cap.

## C.09.07 dashboard flow in real chromium (2026-09-11 15:20Z)

`apps/isis/web/e2e/chroma-runpod-dashboard.spec.ts`, Playwright 1.60 chromium,
one worker, Vite started by Playwright and stopped with it (port 5180 empty
afterwards), generation API mocked at the HTTP boundary: Chroma preset applied
in the Studio (type + Extra Parameters JSON + steps field), submit payload
carried `workflowId/checkpoint/aspect/steps 26/cfg 3.8/batch_size 1`, the
completed job's output thumbnail painted (`naturalWidth > 0`) with caption
`seed 42 · chroma1-hd · $0.0107 · 34.2s · cold start`, and the RunPod page
budget card showed `$0.0107 of $3.00`. 1 passed (2.9 s). Two defects found by
the run and fixed: the Studio's own `steps` default overrode the preset's
catalog value; the budget card was gated behind the operations snapshot.

---

# Video extension evidence (V.01–V.11)

Same conventions as above. A `motion` workflow's row must say which proof level
it reached: `rendered` (a real clip came back from a live worker) or `validated`
(golden fixture + static validation against the object-info snapshot + ComfyUI's
own `/prompt` validator in CI, and no live run). Only the eight clips listed in
tracker V.11 are rendered; a workflow with no row here is not proved at all.

Every clip row records: job id, endpoint, GPU assigned, `delayTime` /
`executionTime` in ms, USD at the video rate, and for the artefact itself
sha256, duration, fps, resolution, and whether it carries audio.

## V.01 Worker image: video node packs

_No rows yet — the video packs are not in the image._ Rows here record the build
id, the resolved SHA of every pack from `/opt/oshun/build-manifest/<pack>.sha`,
the smoke test's class count, and the image digest both endpoints pin.

### V.01.01 — video packs in the image (2026-09-12, pending a build)

Six packs, each SHA-pinned, cloned by `install-video-packs.sh` rather than an
inline `RUN`. The reason is specific: **Docker expands `$1` itself in a
shell-form RUN**, so a loop using positional parameters would have had them
silently replaced with empty strings by an undefined build arg. A COPYed script
also gets `bash -n` before a build is spent on it, which is how the first draft
of this was caught.

The script was then run for real against the live repos on this machine, not
just syntax-checked. Five of six cloned and **every resolved SHA matched its
pin** (the script exits non-zero when it does not):

```
ComfyUI-VideoHelperSuite.sha    4d907bee61e92c2e65af3bd6383a4e4d356126d1
comfyui_controlnet_aux.sha      59b1fc411ede8623b2997855b8018f0b3b6cf49f
ComfyUI-WanAnimatePreprocess.sha 0e0b6a2a555625acf4d4aefb780e27d06937132f
ComfyUI-GIMM-VFI.sha            4c9a3123762af85e7c796e41737da0b70c75d72d
ComfyUI-FlashVSR_Ultra_Fast.sha 4820b3f02347bddcbbb9a5a85ab7638fe976366e
```

The sixth (ComfyUI-SAM3) was interrupted twice by the shell timeout, so its
clone is unproven locally; its **bypass premise is proven from source instead**,
which is the part that could actually be wrong. `nodes/__init__.py` at
`de0ff5d2` assigns both `NODE_CLASS_MAPPINGS` and `NODE_DISPLAY_NAME_MAPPINGS`
at module level (parsed with `ast`, not grepped) and does **not** import
`comfy_env` — so replacing the root `__init__.py` with a direct re-export is
sound, and the build will confirm the clone.

`requirements-video.txt` is a reconciled list, not a concatenation of the packs'
own files, and says why for every omission. The load-bearing one: **four packs
ask for `opencv-python`**, whose GUI build links libGL that a runtime image does
not carry, so the image installs `opencv-python-headless` once instead. Also
corrected — GIMM-VFI's requirement is `cupy-cuda12x>=13.3.0`; the tracker wrote
`cupy-cuda12`, which is not the PyPI name for the CUDA 12 wheel, so the pin is
`cupy-cuda12x==13.3.0`. The `[cuda]` extras ComfyUI-SAM3 lists (`cc_torch`,
`torch_generic_nms`, `flash-attn`) are deliberately absent: the vendored SAM 3
source imports all three under `try`/`except`, and `flash-attn` compiles from
source with no compiler in the base image. `ffmpeg` is installed as an apt
package because `SaveVideo`, `CreateVideo` and `VHS_VideoCombine` shell out to
the binary.

`REQUIRED_CLASSES` (V.01.05) is now **61 classes, no duplicates**: the 24 image
ones plus 21 core video nodes and 16 pack keys. Every pack key was read from
that pack's own `NODE_CLASS_MAPPINGS` rather than guessed from a class name —
`VHS_LoadVideo` is the class `LoadVideoUpload`, and they differ.

Not yet proved: that the image builds with all this in it. That is the next
build, and with the smoke fatal a missing class fails it.

### V.05.02 — video/audio input transport, and a latent stack overflow it exposed (2026-09-12)

`liftInlineMedia` mirrors `liftInlineImages` for `video` and `audio` inputs, but
sniffs **decoded bytes** rather than a base64 prefix. It has to: an MP4/MOV is
identified by `ftyp` at **byte offset 4**, which lands mid-quantum in base64 and
has no stable text prefix, so the image sniffer's approach cannot be reused.
Covered: mp4, mov (brand `qt`), webm and mkv (same EBML header, separated by
DocType), wav, mp3 (ID3 tag or raw frame sync) and flac. The declared mime is
never trusted — a clip labelled `video/webm` that is really an mp4 is served as
`.mp4`, because worker-comfyui writes the bytes under the name we give and
ComfyUI picks its demuxer from that name.

Four refusals, all before a GPU second: unrecognised bytes; audio supplied to a
video input (and the reverse); a mime outside the workflow's `accept` list; and
anything over the inline ceiling, which names the V.05.03 volume path rather
than quietly posting a 50 MB body.

**The bug this turned up is worth more than the feature.** The existing
`RAW_BASE64_RE = /^[A-Za-z0-9+/=\s]{64,}$/` **throws
`RangeError: Maximum call stack size exceeded` above about 8 MB of input** —
measured on this Node: 1 MB fine, 4 MB fine, 8 MB fatal. That was survivable
while inline payloads were only stills; a video input makes it certain. And a
stack overflow is a much worse failure than a rejected submit: it is not a
`WorkerComfyUiSubmissionError`, so nothing upstream can report it usefully. It
is replaced by a bounded head match (`{64}` on the first 64 characters), which
is O(64) at any length and loses nothing — the decisive check is the magic-byte
sniff that follows, and no filename is 64 characters of base64 alphabet. The
image path shares the fix.

The test fixtures are built from real byte headers rather than pasted base64,
and padded to 256 bytes, which is itself a finding: a 32-byte "video" falls
under the 64-character floor that separates a payload from a filename. That is a
property of an unrealistic fixture, not of any real clip, and the comment says
so.

16 specs pass; the `src/workflows/` baseline is unchanged at 13 failed / 143
passed — the same 13 pre-existing `executor.spec.ts` failures.

### V.05.01 — the payload-limit probe is INCONCLUSIVE, and here is why not to repeat it

The endpoint sits at `workersMax: 0` (the kill switch from an earlier session,
never lifted), so `POST /v2/<ep>/run` answers:

```
409 {"detail":"Endpoint is paused (max_workers=0)...","code":"ENDPOINT_PAUSED"}
```

That looked like a free way to measure the limit: a 409 proves no job was
created, so no GPU seconds can be spent. Bodies of 10, 20, 30 and 50 MiB all
came back 409, which suggested nothing rejects them on size.

**Then a 120 MiB body also returned 409** — and that settles it the other way.
No service accepts a 120 MiB `/run` body, so the **pause check runs before any
size check**, and every one of those probes was measuring the pause, not the
limit. The right conclusion from the experiment is that the experiment does not
work.

Against a nonexistent endpoint id, bodies up to 20 MiB return
`404 endpoint not found` rather than 413, so the size check is not at the edge
either — it is behind endpoint resolution _and_ behind the pause check.

**So V.05.01 stays `[ ]`.** Measuring it needs `workersMax` ≥ 1, and then an
accepted oversized body would queue a real job on a real GPU. That is a small
spend, not a free one, and it needs the user's go.

The inline threshold is therefore set conservatively rather than measured:
`ISIS_RUNPOD_VIDEO_INPUT_INLINE_MAX_BYTES` defaults to **6 MiB of base64**,
under RunPod's documented 10 MB for `/run`, with the volume's S3 path taking
everything above it. Being wrong in this direction costs an unnecessary S3 round
trip; being wrong the other way costs a rejected job after the upload.

### Build `95f5cc15` COMPLETED — the video packs are in the image (2026-09-12 02:24:43Z)

Commit `a5ac709682`, the symlink fix. The smoke test's own last line:

```
[smoke] ok — 64 required classes present, 1071 classes total,
        object_info at /opt/oshun/build-manifest/object_info.json
```

**All 64 required classes present**, and the smoke is fatal, so that sentence is
evidence rather than an assertion. It covers the 24 image classes plus the 21
core video nodes (Wan T2V/I2V/Animate/VACE, the LTX-2.3 audio-video chain,
LoadVideo/CreateVideo/SaveVideo/SaveWEBM) and 16 pack keys. `SAM3Segment` is
among them, which re-proves C.12.18 on a second, much larger image.

The pack layer's own output, with every resolved SHA asserted against its pin:

```
[packs] Kosinkadink/ComfyUI-VideoHelperSuite@4d907bee61e92c2e65af3bd6383a4e4d356126d1
[packs] Fannovel16/comfyui_controlnet_aux@59b1fc411ede8623b2997855b8018f0b3b6cf49f
[packs] kijai/ComfyUI-WanAnimatePreprocess@0e0b6a2a555625acf4d4aefb780e27d06937132f
[packs] kijai/ComfyUI-GIMM-VFI@4c9a3123762af85e7c796e41737da0b70c75d72d
[packs] lihaoyun6/ComfyUI-FlashVSR_Ultra_Fast@4820b3f02347bddcbbb9a5a85ab7638fe976366e
[packs] PozzettiAndrea/ComfyUI-SAM3@de0ff5d2c2ea435d29f800abfa568cffdfb94773
[packs] ComfyUI-SAM3: comfy_env bypassed, install.py removed
[packs] ok — 6 packs pinned
```

So **V.01.02's bypass works**: replacing the root `__init__.py` with a direct
re-export and deleting `install.py` is enough, and `SAM3VideoSegmentation`,
`SAM3Propagate`, `SAM3VideoOutput` and `LoadSAM3Model` are all registered. The
dependency layer also passed both checks — `video deps import ok` and
`gpu-only + optional deps installed ok` — confirming that `find_spec` is the
right way to verify `cupy` where `import` is not.

Release recorded at 02:24:43Z, image tag
`registry.runpod.net/greychimp-oshun-runpod-comfyui-slim-docker-runpod-comfyui-slim-dockerfile:a5ac70968`.
**Not yet recorded: the image digest and its compressed size** (V.01.06 wants
both, plus the `object_info.json` stored as
`infra/runpod/endpoints/node-classes/worker-comfyui-5.10.0-video.json`). That
file exists _inside_ the image; extracting it needs a worker boot, which costs
GPU seconds, so V.01.06 stays `[ ]`.

### V.10.02 — and a cost check that had been silently broken (2026-09-12)

Adding the video preflight checks surfaced a pre-existing failure, confirmed by
running the suite against the previous revision: **2 of 5 tests were already
red**, including "a coherent dev environment passes every check".

The cause: `GPU_USD_PER_SECOND` is keyed by GPU **type** names
(`NVIDIA GeForce RTX 4090`), but `desired-state.json` carries GPU **pool** ids
(`ADA_24,AMPERE_24,BLACKWELL_32`) since the endpoint was re-saved through the
console. Every pool id missed the table, so the rate assertion reported "no
pricing row" and failed. **The check that exists to stop the ledger billing at
the wrong rate had not worked since the pool migration.**

Two changes, both of which outlive the immediate fix:

- Pool ids are in the table now, and `ADA_24` carries the 4090 flex rate every
  C.11 measurement was computed with. `AMPERE_24`, `BLACKWELL_32`, `ADA_32_PRO`
  and `AMPERE_80` are listed as **UNPRICED rather than guessed** — RunPod's
  GraphQL exposes pod prices (`securePrice`, `communityPrice`), not serverless
  flex rates, so any number there would be invented. The check names them in its
  detail line instead of rounding the gap away.
- The rule changed from "matches the FIRST entry within 2%" to "**covers the
  PRICIEST entry**". A `gpuIds` list means "any of these", so billing at the
  first entry's rate under-bills whenever a pricier one is assigned — and
  under-billing is precisely the direction that stops a budget firing, because
  the ledger believes less has been spent than has been.

The video checks are **skipped entirely when no video endpoint is configured**
and all bind once the id is set. That is deliberate: the image stack has to stay
usable alone, and a preflight that fails over an optional endpoint teaches
people to ignore it. A half-configured video endpoint is the dangerous state,
not an unconfigured one. A test asserts the dangerous mistake specifically —
setting the video rate to the image rate, which would under-count H100 seconds
by ~4.3x. S3 credentials are checked by length and a test asserts the value
never appears in the output.

6 of 6 preflight tests pass.

### The video-pack builds failed twice — root cause, and how to read a build log (2026-09-12)

Two failures, `ce9ea78c` (449 s) and `234449b9` (330 s). **The cause is a
dangling symlink, and it is worth knowing exactly why.**

```
[WARNING] Cannot import /comfyui/custom_nodes/ComfyUI-SAM3 module for custom nodes:
          [Errno 17] File exists: '/comfyui/models/sam3'
FileExistsError: [Errno 17] File exists: '/comfyui/models/sam3'
[INFO] 0.0 seconds (IMPORT FAILED): /comfyui/custom_nodes/ComfyUI-SAM3
[smoke] custom node import failures:
```

ComfyUI-SAM3's `nodes/load_model.py:22` runs
`os.makedirs(<models_dir>/sam3, exist_ok=True)` at import. `entry.sh` has
already created that path as a symlink to the network volume, and in the
**builder there is no volume**, so the link dangles. And here is the part worth
remembering:

> **`exist_ok=True` does not save you from a dangling symlink.** It suppresses
> the error only when the path is an existing _directory_. A broken link is
> `os.path.exists() == False` — so a guard like `if not os.path.exists(p)`
> passes — and `mkdir` then fails anyway with `EEXIST`.

Reproduced locally in four lines before changing anything, and re-run after:
with a real target the same `makedirs(exist_ok=True)` succeeds.

The smoke test's own comment asserted the opposite — "the volume is absent, so
the links dangle — startup must not care". That was an assumption, it was wrong,
and a pack's entire import was the cost. The smoke now stands up a throwaway
volume root (`/tmp/smoke-volume` with `models/{RMBG,sam3,FlashVSR-v1.1}` and
`inputs/`) and points `entry.sh` at it, so the links resolve to real empty
directories — the runtime shape — instead of an artefact of the builder.

**A correction to my own earlier entry here.** After the first failure I wrote
that `import cupy` on a GPU-less builder was the likely cause and changed it.
That change is still right on its own terms — importing cupy without a CUDA
driver does raise — but it was **not** the failure: the log shows the pip layer
completing and the build dying in the smoke test. I had listed four candidates
and the real cause was none of them. The lesson is the one C.12.12 already
taught and I half-repeated: do not change code against a build you have not
read.

**How to read a RunPod build log without the console UI** — this is what made
the diagnosis possible and it is not documented anywhere:

1. The builds API (`GET api.runpod.io/v2/serverless/<ep>/builds`) returns only a
   generic `docker buildx build ... exit status 1`. There is no `/logs` route on
   it (404 on every shape tried).
2. The console fetches logs with a GraphQL operation. Capture its exact shape by
   patching `window.fetch` in the page and expanding a build row:

   ```graphql
   query getBuildLogs($input: GetBuildLogsInput) {
     getBuildLogs(input: $input) {
       buildId
       timestamp
       message
       level
     }
   }
   ```

   with `variables.input = { buildId, startTime }`.

3. `POST https://api.runpod.io/graphql` with the normal API key then returns the
   whole log as JSON — 501 lines here — **newest first**, so read the tail for
   the earliest output.

Also worth noting: the console page renders fine while **screenshots of it come
back blank**. `document.body.innerText` had 11,983 characters when the
screenshot was white. If a console screenshot looks empty, read the DOM before
concluding the page is broken.

### V.01.02 / V.01.05 — node class names and the ComfyUI-SAM3 installer, read from source (2026-09-12)

All from the pinned SHAs, no guessing. Two of these correct the tracker.

**The FlashVSR class name V.01.05 says "was not captured today"** is
`FlashVSRNode`, with `FlashVSRNodeAdv` and `FlashVSRInitPipe` beside it
(`nodes.py:543`, pack `@4820b3f0`).

**ComfyUI-SAM3 `@de0ff5d2` has no `prestartup_script.py`.** The tracker's
V.01.02 assumes one; the tree at that SHA is `README.md`, `__init__.py`,
`comfy-env-root.toml`, `comfy-test.toml`, `install.py` and `nodes/`. There is no
`requirements.txt` either. The comfy-env coupling is two lines:

```python
# install.py
from comfy_env import install
install()

# __init__.py
from comfy_env import register_nodes
NODE_CLASS_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS = register_nodes()
```

So the bypass is not an env var and not a deleted file: **`nodes/__init__.py`
already exports `NODE_CLASS_MAPPINGS` itself**, assembled from four modules, so
the image replaces the root `__init__.py` with a direct
`from .nodes import NODE_CLASS_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS` and
installs the dependencies with pip. What those dependencies are is not a guess
either — `nodes/comfy-env.toml` lists them:

```toml
[cuda]
packages = ["cc_torch", "torch_generic_nms", "flash-attn"]

[pypi-dependencies]
safetensors, timm, ftfy, regex, iopath, huggingface_hub, einops,
opencv-python, psutil, pycocotools, scikit-image, numpy, pillow, tqdm,
typing_extensions, comfy-aimdo, comfy-kitchen, hf_xet, requests, transformers
```

Two things worth naming there. First, **`pycocotools` appears in this pack's own
dependency list for SAM 3** — an independent confirmation of the C.12.18
diagnosis, from a different author's file, reached without seeing it. Second,
the `[cuda]` three (`cc_torch`, `torch_generic_nms`, `flash-attn`) are the
packages the vendored SAM 3 source imports inside `try`/`except` blocks, so they
are optional accelerators, not requirements — which matters because `flash-attn`
compiles and the runtime base image has no compiler.

`comfy-env-root.toml` also declares a second node pack as a peer,
`PozzettiAndrea/ComfyUI-Multiband`, which is **not** in V.01.01's list. Whether
any `motion` graph needs it is a question for V.08.05; it is recorded here so
the omission is deliberate rather than overlooked.

**Registered node keys, verified from each pack's `NODE_CLASS_MAPPINGS`** (the
key is what a graph's `class_type` must say, and it is not always the class name
— `VHS_LoadVideo` is the class `LoadVideoUpload`):

| Pack                                     | Keys                                                                                                                                                                                                                                                                                                                             |
| ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| ComfyUI-SAM3 `@de0ff5d2`                 | `LoadSAM3Model`; `SAM3Grounding`, `SAM3Segmentation`, `SAM3MultipromptSegmentation`, `SAM3CreateBox`, `SAM3CreatePoint`, `SAM3CombineBoxes`, `SAM3CombinePoints`; `SAM3VideoSegmentation`, `SAM3Propagate`, `SAM3VideoOutput`; `SAM3PointCollector`, `SAM3BBoxCollector`, `SAM3MultiRegionCollector`, `SAM3InteractiveCollector` |
| ComfyUI-GIMM-VFI `@4c9a3123`             | `DownloadAndLoadGIMMVFIModel`, `GIMMVFI_interpolate`                                                                                                                                                                                                                                                                             |
| ComfyUI-WanAnimatePreprocess `@0e0b6a2a` | `OnnxDetectionModelLoader`, `PoseAndFaceDetection`, `DrawViTPose`, `PoseRetargetPromptHelper`, `PoseDetectionOneToAllAnimation`                                                                                                                                                                                                  |
| ComfyUI-FlashVSR_Ultra_Fast `@4820b3f0`  | `FlashVSRNode`, `FlashVSRNodeAdv`, `FlashVSRInitPipe`                                                                                                                                                                                                                                                                            |
| ComfyUI-VideoHelperSuite `@4d907bee`     | `VHS_LoadVideo` (class `LoadVideoUpload`), `VHS_VideoCombine` (class `VideoCombine`), `VHS_LoadVideoPath`, `VHS_LoadVideoFFmpeg`, `VHS_LoadVideoFFmpegPath`                                                                                                                                                                      |

`OnnxDetectionModelLoader` takes `vitpose_model` and `yolo_model` as **combo
inputs listing whatever is in `models/detection`** — it hard-codes no filename,
which is why V.02.03 was free to choose `vitpose-l-wholebody.onnx`.

## V.02 Volume growth (video extension)

| Date              | Item                | Result                                                                                                                                                                                                                                                                                                                                                                           |
| ----------------- | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 2026-09-11 15:12Z | V.02.01 volume grow | User approved 500 GB in conversation. `runpod-provision.mjs volume get ax0oj250j3` → size 100; `volume update ax0oj250j3 --size 500 --yes` → PATCH /networkvolumes/ax0oj250j3 `{size:500}` → response size 500; re-read `volume get` → size 500, EU-RO-1, name isis-chroma-models. New storage cost $35.00/month ($0.07/GB), up from $7.00. Irreversible (volumes never shrink). |

### V.02.02 — video manifest rows (2026-09-12, no pull yet)

`infra/runpod/volumes/isis-chroma-models.manifest.json`: **31 rows added, 41
total, 354.85 GB of the 500 GB volume, 29.0 % headroom, lint 0 errors 0
warnings**, matching the footprint table's ~353.4 GB projection.

`sha256` is filled **before** the pull rather than after it. For a Hugging Face
row the LFS oid _is_ the sha256 of the file content, so recording it now is what
lets the loader's download be checked against a value that did not come from the
download. Every one was then re-fetched and diffed: a script grouped all 39 HF
rows by `(repo, revision)`, called `paths-info` for each group, and compared
both `size` and the LFS oid — **all 39 match**. (One repo answered
`maximum queue size reached` on the first pass and was retried on its own.)

Two corrections the reading turned up, both of which would have cost a failed
pull or a request-time download:

- **`FlashVSR-v1.1`, not `FlashVSR`.** `ComfyUI-FlashVSR_Ultra_Fast`
  `nodes.py:140-141` joins `folder_paths.models_dir` with the model combo value
  verbatim, and `model_downlod()` runs `snapshot_download` of the entire repo
  when that directory is absent. The destPaths and the lint's folder list use
  the `-v1.1` name.
- **`vitpose-l`, not `vitpose-h`** — see V.02.03 below.

Lint changes: `ALLOWED_TOP_FOLDERS` gained `latent_upscale_models`,
`audio_encoders`, `interpolation`, `detection`, `FlashVSR-v1.1`,
`controlnet_aux`, each with the pack that reads it written beside it.
`QUANTIZED_PATTERN` gained `fpmixed`, `convrot` and a delimited
`[_-]q4(?![0-9])` (`fp4` was already present, but it does not catch `fpmixed`).
The pattern was then run against all 19 real filenames in the new rows and 6
known-quantized ones: no false positive (`fp16`, `fp32` are safe) and no miss.
The two checked-in-manifest assertions in `runpod-volume-manifest-lint.test.mjs`
were repinned from 10 rows / ~69 GB to 41 rows / ~355 GB, and the headroom case
from 75→100 GB to 400→500 GB; 7 of 7 pass.

### V.01.03 — volume folder mapping (2026-09-12)

`extra_model_paths.yaml` gained `audio_encoders`, `detection`, `interpolation`
and `latent_upscale_models`. `entry.sh` gained two symlinks beside the existing
RMBG/sam3 pair — `models/FlashVSR-v1.1` (the pack bypasses `folder_paths`
entirely) and `input/volume → /runpod-volume/inputs`, which is what a
`volume/<jobId>/<name>` `LoadVideo` reference resolves through (D-V4). The
Dockerfile sets `AUX_ANNOTATOR_CKPTS_PATH=/runpod-volume/models/controlnet_aux`
and `AUX_USE_SYMLINKS=False`; `custom_hf_download` lays files out as
`<ckpts_dir>/<repo id>/<filename>`, which is exactly how the three
controlnet_aux manifest rows are addressed, so DWPose and Depth Anything V2
resolve on the volume instead of downloading on a serverless worker.

`entry.sh` was exercised for real against a throwaway root
(`OSHUN_ENTRY_PREPARE_ONLY=1`, `COMFYUI_PATH` and `OSHUN_VOLUME_ROOT` pointed at
a temp dir): all four links created, all four dangling because no volume was
mounted, exit 0. The smoke test now asserts all four with `test -L`.

### V.02.03 — the two unverified rows, checked against the live HF API (2026-09-12)

All read-only, anonymous, `huggingface.co/api/.../paths-info` for exact bytes
and the LFS sha256 (so the manifest's `sha256` can be filled before the pull
rather than after it).

**ViTPose wholebody — and a trap.** `JunkyByte/easy_ViTPose` is **ungated**
(repo sha `e838052`) and carries 28 ONNX files, four of them wholebody:

| File                                      | Bytes         | sha256 (LFS oid)    |
| ----------------------------------------- | ------------- | ------------------- |
| `onnx/wholebody/vitpose-l-wholebody.onnx` | 1,234,579,166 | `89bdf6692d9224db…` |
| `onnx/wholebody/vitpose-b-wholebody.onnx` | 360,242,862   | `017c08e161bfa2c7…` |
| `onnx/wholebody/vitpose-s-wholebody.onnx` | 97,427,067    | (not fetched)       |
| `onnx/wholebody/vitpose-h-wholebody.onnx` | 412,883       | `e5dab9693f9fb437…` |

**The `-h` file is not a model.** At 412 KB it is an ONNX _graph_ whose weights
live in external-data files; its first bytes are a valid ONNX protobuf
(`pytorch 1.13.1`) with no tensors. Its sha256 is byte-identical to
`Wan-AI/Wan2.2-Animate-14B`'s
`process_checkpoint/pose2d/vitpose_h_wholebody.onnx/end2end.onnx` — and in that
repo the `.onnx` is a **directory of several hundred external-data tensor
files** beside the graph, which is why the official copy cannot be a manifest
row either (one row per file would mean hundreds). So the manifest row is
**`vitpose-l-wholebody.onnx`**, the largest usable single file, which also
matches the ~1.5 GB the footprint table budgeted for the preprocess pair.

**Nothing hard-codes the filename.** `ComfyUI-WanAnimatePreprocess` `@0e0b6a2a`
registers `models/detection` as a `folder_paths` folder, adds `.onnx` to its
extensions, and `OnnxDetectionModelLoader` takes `vitpose_model` and
`yolo_model` as **combo inputs listing whatever is in that folder**. So the
choice above is ours to make and ours to change, and the volume folder is
`models/detection/`.

`Wan-AI/Wan2.2-Animate-14B` is **ungated** (repo sha `cb93a22`) and its
`process_checkpoint/det/yolov10m.onnx` is **61,659,339 B** — that row stands.

**controlnet_aux annotator checkpoints.** The ckpt-dir env var V.01.03 needs is
**`AUX_ANNOTATOR_CKPTS_PATH`** (`src/custom_controlnet_aux/util.py:41`, beside
`AUX_USE_SYMLINKS` and `AUX_TEMP_DIR`), and `custom_hf_download` lays files out
as **`<ckpts_dir>/<repo id>/<filename>`** — so pointing it at
`/runpod-volume/models/controlnet_aux/` gives these destination paths:

| Node                          | Repo (ungated)                                           | File                         | Bytes         | sha256              |
| ----------------------------- | -------------------------------------------------------- | ---------------------------- | ------------- | ------------------- |
| `DWPreprocessor` bbox         | `yzd-v/DWPose` (sha `1a71441`)                           | `yolox_l.onnx`               | 216,746,733   | `7860ae79de6c89a3…` |
| `DWPreprocessor` pose         | `yzd-v/DWPose`                                           | `dw-ll_ucoco_384.onnx`       | 134,399,116   | `724f4ff2439ed61a…` |
| `DepthAnythingV2Preprocessor` | `depth-anything/Depth-Anything-V2-Large` (sha `cbbb86a`) | `depth_anything_v2_vitl.pth` | 1,341,395,338 | `a7ea19fa0ed99244…` |

The node defaults are `bbox_detector="yolox_l.onnx"`,
`pose_estimator="dw-ll_ucoco_384.onnx"` (`node_wrappers/dwpose.py:54`, repo
`yzd-v/DWPose` via `DWPOSE_MODEL_NAME`) and
`ckpt_name="depth_anything_v2_vitl.pth"` (`node_wrappers/depth_anything_v2.py`,
repo resolved through `DEPTH_ANYTHING_V2_MODEL_NAME_DICT`). Those three files
are what the graphs will actually ask for, so those three are the rows; seeding
them is what stops a request-time download on a serverless worker.

Total added by this item: 1.23 + 0.06 + 0.22 + 0.13 + 1.34 ≈ **2.98 GB**, about
1.5 GB more than the footprint table's `~1.5` estimate for the Animate
preprocess pair, because the table did not count the controlnet_aux annotators.
Still far inside the 500 GB volume.

## V.03 Video endpoint

_No rows yet._ Rows record the endpoint id, the GPU pool ids actually saved
(read back through GraphQL, never the REST echo — C.03 landmine), workersMax,
executionTimeout, idleTimeout, the image digest, and the drift-check output.

## V.04 Video cost control

_No rows yet._ Rows record the video ledger key, a refusal observed at the
boundary, and the reconciliation of the ledger total against RunPod's own
billing view.

## V.05 Video transport

_No rows yet._ Rows record the measured RunPod request payload limit, the
inline/S3 threshold it implies, and one PUT + one read back through the volume's
S3 API with byte counts.

## V.06 Catalog schema and executor for video

_No rows yet._ Rows record the object-info snapshot the static validator runs
against (build id and class count) and the CI run of ComfyUI's own `/prompt`
validator over every `motion` graph against a zero-byte shadow volume.

## V.07 Output persistence for video

_No rows yet._ Rows record one stored clip: key, content type, byte count,
range-request behaviour, and the poster frame's dimensions read from the image
header.

## V.08 `motion` workflow catalog

_No rows yet._ One row per graph with its `proof_level` and what proved it.

## V.09 Dashboard (video)

_No rows yet._ Rows record the chromium run: spec file, what was asserted about
playback, and the defects the run found.

## V.10 Dev stack (video)

_No rows yet._ Rows record the preflight output naming both endpoints, both
ledgers and the volume size.

## V.11 Live proof matrix

_No rows yet — nothing has been rendered._ V.11.01 is the one standalone clip;
V.11.02 is a single warm-chained session of seven. Each row: job id, GPU, ms,
USD, sha256, duration, fps, resolution, audio.

## Found outside this initiative: `executor.spec.ts` has 11 pre-existing failures

Measured 2026-09-12 while running the generation-api suite as commit-time
evidence for V.05/V.10. Recorded here because this initiative keeps editing
`apps/isis/generation-api/src/workflows/executor.ts`, and a spec file that is
already red is not a guard — two regressions of my own hid inside it for several
commits before I looked.

**The split, measured by checking out and running the file at two points:**

| commit       | what it is                         | failures |
| ------------ | ---------------------------------- | -------- |
| `fc7f3051c3` | before the chroma/video work began | **11**   |
| `72ff476aec` | this branch, before the fix below  | **13**   |

The two extra were mine and are fixed in the same commit as this note:

- `registers every @isis/workflows catalog entry at startup` —
  `expected 66 to be 75`. The test compared the resolver against the **whole**
  catalog, but V.06.02 (`acd8330d9d`) put `motion` into
  `WORKER_COMFYUI_ONLY_CATEGORIES` alongside `chroma`, and those entries are
  deliberately not registered in the general resolver.
- `validates and resolves all catalog workflows via deterministic override selection`
  — `Missing resolver entry for workflow wan22-t2v`, the same cause.

Both now filter the catalog through the **imported**
`WORKER_COMFYUI_ONLY_CATEGORIES` rather than a restated list, and the first
additionally asserts the excluded entries really are absent — so the test cannot
start passing because the exclusion set quietly grew to cover a registration
bug.

**The 11 that remain are not from this initiative.** Identical assertions at
`fc7f3051c3`. Two of them are diagnosed:

- `tracks resolve_success and resolve_failure counters` — asserts
  `resolve({generationType:'inpainting'})` finds nothing. Exactly one workflow
  routes to `inpainting`: `inpaint-edit`, added **2026-07-03** in `e8e4b301d2`.
  The test was written **2026-02-21** (`b4f667d773`). It has been red since the
  day that workflow landed. The same cause explains
  `does not resolve generation types that still have no dedicated catalog workflows`
  (`expected { id: 'inpaint-edit', … } to be undefined`).
- The four
  `certifies <guided-journey|lecture-scene|mandala-animated|meditation-scene> workflow …`
  failures are `result.success === false` at `executor.spec.ts:2899` — a real
  execution failure of those `text-to-video` workflows through the deterministic
  test ComfyUI service, not a stale assertion.

The remaining five (the `WorkflowExecutor unresolved workflow runtime guard`
group, voice consistency profiles, sequence rerender, and the ControlNet mode
mapping) are undiagnosed. They belong to whoever owns the catalog-execution
contract, not to this tracker; they are written down here so the next person to
see "13 failed" on this file knows which number is the baseline and which is
new.

## V.01.09 — ComfyUI validates every rendered graph at build time (2026-09-13)

Files: `docker/runpod/comfyui-slim/validate-goldens.py` (new),
`docker/runpod/comfyui-slim/smoke-test.sh`,
`docker/runpod/comfyui-slim/Dockerfile`, `.dockerignore`. Log excerpt:
`docs/agents/evidence/isis-chroma-runpod/build-6bbdae22-smoke-validate.txt`.

| What                 | Value                                                                                                                                                                                                               |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Build                | `6bbdae22-ce4f-4815-951e-21b1eb6c8668` COMPLETED 00:50Z, commit `f73b8998d9`, image tag `…-dockerfile:f73b8998d`                                                                                                    |
| Shadow volume        | `[smoke] shadow volume: 42 zero-byte model files under /tmp/smoke-volume/models` — one per manifest destPath, created BEFORE boot                                                                                   |
| Class check          | `77 required classes present, 1071 classes total` (was 64 required on `a5ac70968`; same 1071 total)                                                                                                                 |
| Validator            | `[validate] 18/18 rendered graphs accepted by ComfyUI's validator` — 17 motion goldens + chroma, HTTP 200 each, nodes 5–24                                                                                          |
| Cleanup              | each accepted prompt: `POST /queue {"clear":true}` + `POST /interrupt`; shadow inputs removed; `/tmp/smoke-volume` removed                                                                                          |
| Smoke step wall time | 7.0 s (`#27 DONE 7.0s`); the previous good build's was 14.3 s with the same 1071 classes, so ComfyUI does boot in seconds there                                                                                     |
| Failure mode proven  | build `055f8d58` FAILED at `COPY infra/…/isis-chroma-models.manifest.json` — root `.dockerignore` excludes `infra`; re-included                                                                                     |
| Local proof          | `python3 docker/runpod/comfyui-slim/validate-goldens.py --self-test` — stub server accepts one graph, rejects one with `Bogus`; asserts the report, the clear/interrupt calls and that the shadow input was removed |
| Pre-check on the Mac | every model-looking input in all 18 goldens resolves to a manifest destPath (42/42); media refs are `ref.png`, `start.png`, `clip.mp4`, `drive.mp4`, `mask.mp4`                                                     |

What the 18/18 does and does not prove: ComfyUI accepted every graph the catalog
renders — every class exists, every required input is present, every literal is
in type and range, every combo value (model filename) is one the volume ships,
every link's socket types agree. It does not prove the graphs execute; that is
what V.11 clips are for. The live endpoints still run `a5ac70968`; `f73b8998d`
is built and pushed to the registry but not released.

## V.01.06 — the object-info snapshot is in the repo (2026-09-13)

File: `infra/runpod/endpoints/node-classes/worker-comfyui-5.10.0-video.json`
(1071 classes; `provenance` + `classes`). Producer:
`docker/runpod/comfyui-slim/snapshot-emit.py` (in the smoke test) and
`scripts/isis/runpod-build-snapshot.mjs` (repo side).

| What        | Value                                                                                                                                                                                                                                                                                                                                                                                  |
| ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Build       | `f4fbb9fb-74f2-44b7-a1b7-2217b4531a35` COMPLETED, commit `fa81159f3`, image `…-dockerfile:fa81159f3`; 18/18 goldens validated in the same build                                                                                                                                                                                                                                        |
| Emitted     | `OBJINFO-META classes=1071 bytes=730990 chunks=75 sha256=06a7f453…`; 75 lines of 1600 chars, printed last so they sit inside the newest 500 log lines                                                                                                                                                                                                                                  |
| Decoded     | all 75 chunks present, sha256 matched, class count matched; `wrote …/worker-comfyui-5.10.0-video.json: 1071 classes`                                                                                                                                                                                                                                                                   |
| Reduction   | per class: `input.required`, `input.optional` (tooltips stripped), `output`, `output_name` — the fields the validator reads; 0 tooltips left                                                                                                                                                                                                                                           |
| Combo lists | the shadow volume's names — `UNETLoader.unet_name` offers 9 files, every one on the manifest (spec asserts it); `CLIPLoader.clip_name` 3                                                                                                                                                                                                                                               |
| Why the log | US-CO-1 has no S3 and CPU pods cannot hold the image (20 GB cap vs 30); the GPU route sat `throttled`/`initializing` in EU-RO-1 for 1 h+ at $0 and was cancelled; a git build's only channel back is its log, and `getBuildLogs` takes only `buildId` + `startTime` (probed: `endTime`, `limit`, `cursor`, `page`, `offset`, `pageSize` all rejected) and returns the newest 500 lines |
| Refresh     | any push to `runpod-comfyui-slim` rebuilds; then `node scripts/isis/runpod-build-snapshot.mjs --endpoint 2lezocymx2rgot` (refuses a non-COMPLETED build)                                                                                                                                                                                                                               |
| Tests       | `scripts/isis/runpod-build-snapshot.test.mjs` 5/5 (order-independence, missing chunk, corrupted chunk, no emission, python↔node format); `snapshot-emit.py --self-test`                                                                                                                                                                                                                |

Not stored: the raw /object_info with tooltips and display names. It is ~5×
larger and carries nothing the validator reads; the reduction is named in the
file's `provenance.reduced` so nobody mistakes it for the whole.

## V.06.06 — static validator runs every check, against the real snapshot (2026-09-13)

Files: `libs/isis/workflows/src/graph-validator.ts`, `graph-validator.spec.ts`
(+6 cases), `graph-validator.snapshot.spec.ts` (new),
`apps/isis/generation-api/src/workflows/executor.ts`,
`worker-comfyui-submission.ts`.

| Check (tracker wording)                        | Where                                                                                                                                     | Evidence                                                                                                |
| ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| every `class_type` exists                      | `UNKNOWN_CLASS`                                                                                                                           | spec                                                                                                    |
| every required input present                   | `MISSING_REQUIRED_INPUT`                                                                                                                  | spec                                                                                                    |
| literal type and range                         | `LITERAL_TYPE_MISMATCH`, `LITERAL_OUT_OF_RANGE` (new)                                                                                     | spec: wrong scalar, literal on a socket, 250 outside `[0, 100]`; `"FLOAT,INT"` accepts either           |
| link source output type = target input type    | `LINK_TYPE_MISMATCH` (was declared, never emitted)                                                                                        | spec: LATENT/VAE inputs linked to a MODEL output → two issues                                           |
| combo values in the list when non-empty        | `BAD_COMBO_VALUE`, now BOTH forms: `[[…], {}]` and `["COMBO", {options}]`                                                                 | spec; upload inputs (`image_upload` etc.) exempt — their list is the builder's input dir, not the job's |
| model filenames resolve to a manifest destPath | combo membership against the shadow-volume lists + `MODEL_NOT_ON_VOLUME`                                                                  | snapshot spec: every `UNETLoader` option is on the manifest                                             |
| LoadVideo/LoadImage references resolve         | `UNRESOLVED_FILE_REFERENCE` (pre-submit path only; real upload names)                                                                     | spec                                                                                                    |
| pre-submit path                                | executor reads the snapshot file (fail-loud `SERVICE_UNAVAILABLE` if missing or shapeless) → submission → `validateGraph({ objectInfo })` | `worker-comfyui-submission` + `node-class-inventory-routing` specs 26/26                                |
| vitest over every golden                       | `graph-validator.snapshot.spec.ts`: 18 goldens (17 motion + chroma), `issues: []` AND `skipped: []`                                       | 43/43 across the two validator specs                                                                    |

Adversarial pass, what it found: **two false positives in my own first cut**.
`LTXVEmptyLatentAudio.frame_rate` is declared `"FLOAT,INT"` and the literal
check read that as one unknown socket type, failing both `ltx23-av` goldens that
ComfyUI itself had just accepted — fixed by splitting multi-type declarations;
and `LoadImage.image`'s option list is `["example.png"]` (the builder's input
dir), which would have failed every `ref.png` — fixed by exempting `*_upload`
inputs from membership. Both are the kind of defect only the real snapshot
exposes, which is the argument for having it. Adversarial grep on the touched
files: zero hits. `tsc` on `libs/isis/workflows`: clean. `tsc` on
`generation-api`: 10 errors, identical with and without these changes (none in
the touched files; pre-existing in `jobs.ts`, `job.service.ts`, …).

## V.07.03 — the OpenAPI half (2026-09-13)

Files: `apps/isis/generation-api/openapi.yaml` (baseline),
`openapi/isis/openapi.yaml` (generated),
`apps/isis/generation-api/src/types.ts`,
`apps/isis/generation-api/src/services/output-metadata-projection.spec.ts`.

| What                   | Value                                                                                                                                                                                                                                                          |
| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Schemas added          | `JobOutputFileFacts` (25 properties, `additionalProperties: false`), `JobOutputFile`, `JobOutput`                                                                                                                                                              |
| Operation retyped      | `GET /api/v1/jobs/{id}/output` 200 → `JobOutput`                                                                                                                                                                                                               |
| Semantic diff          | paths 66 → 66 (none added or removed); schemas +3, −0; operations with changed responses: 1                                                                                                                                                                    |
| Generator              | `node scripts/isis/generate_generation_api_openapi.mjs --check` → up to date                                                                                                                                                                                   |
| Drift held by the spec | client `JobOutputFileFacts` = OpenAPI `JobOutputFileFacts.properties` = API `src/types.ts` `JobOutputFileFacts`, all three sorted and compared                                                                                                                 |
| Found on the way       | the API's own `JobOutputFileFacts` lacked all twelve video and provenance fields (`kind`, `durationMs`, `fps`, `frames`, `hasAudio`, `codec`, `alpha`, `source`, `encoding`, `s3FetchError`, `volumeKey`, `posterUrl`); added                                  |
| Tests                  | projection spec 7/7; typecheck: **correction 2026-09-13** — the `tsc --noEmit -p tsconfig.json` quoted here as 0 errors checks no files (`files: []` plus references); the real app config `tsconfig.app.json` has 10 pre-existing errors, none in these files |

Why the earlier "no target" verdict was wrong: the generator copies any
operation the baseline already describes and only invents a `JsonObject`
response for routes the baseline is silent on, so a typed response is a
one-operation edit.

## V.11.02 — worker logs were never captured, and the reader that fixes it (2026-09-13)

Files: `scripts/isis/runpod-worker-log.mjs` (new),
`scripts/isis/runpod-worker-log.test.mjs` (new),
`scripts/isis/motion-live-proof.mjs`, `scripts/isis/runpod-catalog-run.mjs`,
runbook §6c.

| What                       | Value                                                                                                                                                                                                                                                                                                                                     |
| -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Live probe                 | `GET /v2/serverless/2lezocymx2rgot/workers/9lx1gb8ksy3dw6/logs` → `HTTP/2 200`, `content-type: text/event-stream`; 100 frames in 12 s covering 5 s of worker time (replay is paced, roughly 8 frames/s)                                                                                                                                   |
| `?tail=N`                  | honoured: `?tail=3` returned exactly the last three lines (`Digest: sha256:af2fba…`, `Status: Downloaded newer image …:fa81159f3`, `worker is ready`)                                                                                                                                                                                     |
| `Last-Event-ID`, `?since=` | not honoured — nothing replayed within 6 s                                                                                                                                                                                                                                                                                                |
| Old capture                | `await response.text()` on a stream that never closes → 20 s `AbortSignal.timeout` → "log fetch timed out" on every failure (v1102-summary.json, stages 2b and 6, verbatim)                                                                                                                                                               |
| New reader                 | `?tail=400`, incremental read, stop on 2.5 s quiet gap or 20 s cap; `truncated` reported; 404 → `gone`, non-2xx → `http_error`, no frames → `empty`, socket drop → what was read plus `stream ended early`                                                                                                                                |
| Tests                      | `node --test scripts/isis/runpod-worker-log.test.mjs` 7/7 against a real `node:http` SSE server: paced replay split mid-frame stops on the quiet gap and sends `?tail=250`; a never-quiet stream is capped at 400 ms; a dropped socket keeps both lines; 404/500/silent; `captureWorkerLog` writes header plus lines and returns the path |
| Live proof of the reader   | `node scripts/isis/runpod-worker-log.mjs --endpoint 2lezocymx2rgot --worker 9lx1gb8ksy3dw6 --tail 12` → 12 lines in 5.3 s                                                                                                                                                                                                                 |
| Also seen                  | the worker's image digest is in its own log (`Digest: sha256:af2fba848bb8…` for tag `fa81159f3`) — a pulled image does have a digest, the build API just never reports one                                                                                                                                                                |

Both endpoints' templates now show image tag `fa81159f3` (GraphQL
`template.imageName`), so the V.01.06 build is what runs; the memory note saying
it was unreleased is stale.

## D-V4 output leg — never ran, and cannot as configured (2026-09-13)

| What                   | Value                                                                                                                                                                                                                                                |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Volume `outputs/`      | `node scripts/isis/runpod-volume-s3-put.mjs ls outputs/` → 0 objects; `inputs/` → 0 objects; 52 objects total, all under `models/` plus `.oshun/manifest-result.json`                                                                                |
| Worker uploader        | `worker-comfyui` 5.10.0 `handler.py` → `rp_upload.upload_image(job_id, temp_file_path)`; `runpod~=1.7.12` `rp_upload.py:190-193` → `bucket = bucket_name if bucket_name else time.strftime("%m-%y")`, `Key=f"{job_id}/{image_name}{file_extension}"` |
| Consequence            | against the volume S3 API (one bucket = the volume id) the upload fails; the handler then appends an error and returns no data for that output (`handler.py:805-808`, no base64 fallback)                                                            |
| Template env           | unreadable: GraphQL `template.env` empty, `rest.runpod.io/v1/templates/<id>` → error; the provision script never injects `envFromEnvironment`                                                                                                        |
| How V.11 clips arrived | inline `base64` in `images[]` (211 885 B for V.11.01) under the measured 10 MiB body cap — the 528 s cold start and every measured cost stand; only the transport claim is corrected                                                                 |

Recorded on C.12.06, V.12.05 and V.12.06; no box changed.

## C.12.02 / C.02.09 — Real-ESRGAN on the volume and the optional tail (2026-09-13)

| What              | Value                                                                                                                                                                                                                                                                                                                                                            |
| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- |
| Weight            | `lllyasviel/Annotators` @ `982e7edaec38759d914a963c48c4726685de7d96` `RealESRGAN_x4plus.pth`, HF LFS size 67 040 989, oid `4fa0d38905f75ac06eb49a7951b426670021be3018265fd191d2125df9d682f1`                                                                                                                                                                     |
| Local check       | `shasum -a 256` on the download = the LFS oid; 67 040 989 bytes                                                                                                                                                                                                                                                                                                  |
| Upload (C.02.09)  | `runpod-volume-s3-put.mjs put … --key models/upscale_models/RealESRGAN_x4plus.pth` → single PUT, `verified 67040989 bytes` by HEAD; `ls models/upscale_models/` → 1 object, `2026-09-13T05:00:48Z`                                                                                                                                                               |
| Manifest          | row `realesrgan-x4plus`, `upscale_models/RealESRGAN_x4plus.pth`, license BSD-3-Clause; `runpod-volume-manifest-lint.mjs --strict` → 43 artifacts, 358.37 GB of 500 GB, 0 errors, 0 warnings                                                                                                                                                                      |
| Catalog           | `chroma-txt2img` nodes 16–18 and `chroma-img2img` nodes 13–15: `UpscaleModelLoader` / `ImageUpscaleWithModel` / `ImageScaleBy(0.5, lanczos)`, all `_condition: {{upscale != 'none'}}`, passthrough `image`; `upscale` input `none                                                                                                                                | x2`; `models_required` += the file |
| Default unchanged | rendering with `upscale: none` equals `chroma-txt2img.square_1024.golden.json` byte for byte (script check + spec)                                                                                                                                                                                                                                               |
| x2 golden         | `chroma-txt2img.square_1024.x2.golden.json`, 18 nodes, `SaveImage.images = ["18", 0]`                                                                                                                                                                                                                                                                            |
| Specs             | `libs/isis/workflows`: golden, snapshot (19 goldens), workflows, motion-caps, chain → 5 files / 154 tests; `generation-api` submission + routing → 26                                                                                                                                                                                                            |
| Build             | `b60db5b1-9ccc-45f8-b2c9-75462df3988e` COMPLETED 06:12Z, commit `a5118e06871`, image tag `…:a5118e068`; shadow volume 43 files; `[validate] 19/19 rendered graphs accepted by ComfyUI's validator` incl. `chroma-txt2img.square_1024.x2.golden.json nodes=18 http=200`; `smoke exit 0` — excerpt `evidence/isis-chroma-runpod/build-b60db5b1-smoke-validate.txt` |
| Snapshot refresh  | `runpod-build-snapshot.mjs --endpoint 2lezocymx2rgot` → 1071 classes, 75 chunks, sha256 `aa1ac72b…`; semantic diff vs the committed file: `UpscaleModelLoader.model_name` options `[]` → `["RealESRGAN_x4plus.pth"]`, `VHS_LoadImagePath` extension list reordered (set noise); specs 3 files / 56 tests                                                         |
| Release           | both endpoints' templates read `…:a5118e068` afterwards (auto-released); balance $10.14, spend $0.049/h (volume only) — no GPU was billed by any of this                                                                                                                                                                                                         |

## V.12.04 — the four "list, do not pull" families (2026-09-13)

Anonymous HF API, LFS byte counts. Headroom 141.6 GB.
Comfy-Org/Wan_2.2_ComfyUI_Repackaged (Apache-2.0): Fun Control 14B 57.16 GB (5B
10.00), Fun Camera 14B 59.16 GB, Fun Inpaint 14B 57.16 GB (5B 10.00), Fun VACE
14B 69.36 GB, S2V 14B 32.59 GB + wav2vec2 0.63 GB. InfiniteTalk:
MeiGen-AI/InfiniteTalk (Apache-2.0) 9.95 / 2.71 / 2.71 GB, Kijai fp16 5.13 /
5.12 GB; needs `Wan-AI/Wan2.1-I2V-14B-480P` +
`TencentGameMate/chinese-wav2vec2-base` (README) and a wrapper pack — none in
the image or on the volume. Recorded in the tracker note; nothing pulled.

## V.12.01 — SAM 3.1 support check (2026-09-13)

ComfyUI-RMBG `main` = `58f1947a` (2026-08-21, pinned SHA); ComfyUI-SAM3 `main` =
`de0ff5d2` (2026-05-19, pinned SHA). No path or loader mentions `sam3.1` /
`multiplex`. No supported pack; box stays open by its own wording.

### V.11.02 — the approved re-run, 3 of 4 (2026-09-13, 07:32–07:58Z, $0.78)

One invocation of
`scripts/isis/motion-live-proof.mjs --only 2a,2b,6,5 --out …/v1102-rerun`, one
worker `bkeiqpovpvuuy6` (**RTX PRO 6000 Blackwell Server Edition**, per
`GET /v2/serverless/2lezocymx2rgot/workers`), image `a5118e068`. Balance
$10.0905 (07:32Z) → $9.2772 once settled (identical at 08:13Z and 08:17Z) =
**$0.813**, of which $0.037 is volume rent; GPU ≈ **$0.776 for 739.4 s of
execution = $0.00105/s**. A reading at 08:00Z, 90 s after the last job, showed
$9.4825 — RunPod posts a job's charge minutes after it ends, so an early reading
understates by a whole job.

| Stage | Workflow               | Status        | Execution     | Output                                                                                                                                                 |
| ----- | ---------------------- | ------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| 2a    | `gimm-vfi-interpolate` | **COMPLETED** | 25.4 s        | `2a-gimm-vfi-interpolate-0-26a1a57a.mp4` 214 932 B, h264 832×480 **65 f at 32 fps** (2a re-run only to feed 2b)                                        |
| 6     | `ltx23-av`             | **COMPLETED** | 304.9 s       | `6-ltx23-av-0-93a142c8.mp4` 292 869 B, h264 1024×576 25 fps 33 f **+ AAC audio, 62 frames** (ffprobe) — the audio path is real                         |
| 2b    | `flashvsr-upscale`     | **FAILED**    | 71.0 s        | `Job processing failed`; **worker log captured** (`worker-bkeiqpovpvuuy6-1789285991353.log`, 408 lines) — see below                                    |
| 5     | `wan22-animate`        | **COMPLETED** | 338.2 s, warm | `5-wan22-animate-0-e9fca11a.mp4` 139 748 B, h264 832×480 16 fps 33 f — the shape that killed the worker on 2026-09-12 rendered when run alone and last |

**FlashVSR, diagnosed at last.** The log shows
`Using wan_video_dit from /comfyui/models/FlashVSR-v1.1/diffusion_pytorch_model_streaming_dmd.safetensors`
(the volume symlink works), `[FlashVSR] Processing tile 1/8`, then
`RuntimeError: Failed to find C compiler. Please specify via CC environment variable or set triton.knobs.build.impl.`
raised from `triton/runtime/build.py` via `sparse_sage/quant_per_block.py` —
Triton compiles its NVIDIA driver module from C at the first `@triton.jit`
launch, and the runtime base image ships no gcc. Neither the graph, the model
files nor memory were ever the problem. Fix: `gcc libc6-dev` in the image's apt
step plus a fatal smoke check (gcc present, `Python.h` present, a trivial
compile succeeds); build `a96a0123` (commit `299b763f408`).

**Cold start, bounded from the worker's own log:** `create container` at
07:46:15Z, and stage 2a (25 s of execution) was complete inside the same minute
— container-start-to-first-prompt is under about 20 s with FlashBoot. The 825
720 ms `delayTime` before it was RunPod placement (the endpoint reported an idle
worker at submission time and still took 14 min to create a container after the
`a5118e068` release), which is not billed.

**Also seen in that log:** the template env is empty, so `COMFY_LOG_LEVEL`
defaults to DEBUG and an `aimdo` memory-manager line lands every ~55 ms; a 1
500-line tail (`worker-bkeiqpovpvuuy6-full.log`) covers only the last 100 s of a
job. `COMFY_LOG_LEVEL=INFO` from `desired-state.json` was never injected
(V.12.05's finding) — it matters for diagnosability now that logs are
capturable.

Catalog after this run: `ltx23-av` and `wan22-animate` are
`proof_level: rendered` with `cost_calibration: measured` (15.6647 and 25.6611 s
per megapixel-frame); `wan22-animate`'s `known_failure` note is replaced by the
render facts; `flashvsr-upscale` stays `validated` with the diagnosis in its
`proof_note`.

### V.11.02 — FlashVSR, rendered (2026-09-13, 08:32–08:34Z)

| What     | Value                                                                                                                                                                                                                                                                                                                             |
| -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Image    | build `79351a92-09bf-4264-bb57-d37173bef6cc` (commit `47a74b7511d`) — smoke: `C toolchain: gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0; Python.h at /usr/include/python3.12`, 19/19 goldens, exit 0; the prior build `a96a0123` failed the new check on the missing `Python.h` (venv on Ubuntu's python3.12, no `python3.12-dev`) |
| Run      | `motion-live-proof.mjs --only 2a,2b --out …/v1102-rerun-flashvsr`, worker `k7lblqd18yoqac`                                                                                                                                                                                                                                        |
| 2a       | `gimm-vfi-interpolate` COMPLETED, delay 8.5 s, exec 22.7 s, warm — `2a-gimm-vfi-interpolate-0-493ad580.mp4` 216 779 B                                                                                                                                                                                                             |
| 2b       | `flashvsr-upscale` **COMPLETED**, delay 0.1 s, exec **57.9 s**, warm — `2b-flashvsr-upscale-0-c4645d50.mp4` 1 117 573 B; ffprobe h264 **1664×960**, 65 frames, 16 fps (catalog default `output_fps`)                                                                                                                              |
| Rate     | 0.5579 s per OUTPUT megapixel-frame (65 × 1.5974 MP) — written to the catalog with `proof_level: rendered`                                                                                                                                                                                                                        |
| Snapshot | refreshed from build 79351a92 (`worker-comfyui-5.10.0-video.json`, sha256 `81de43ec…`); class-level diff vs the committed file: only the `VHS_LoadImagePath` extension-list order                                                                                                                                                 |
| Cost     | script estimate $0.107 at its H100 rate; the settled balance is read after the session (RunPod posts charges minutes late)                                                                                                                                                                                                        |

With this row every one of the eight V.11 workflows has rendered live: T2V, I2V,
Animate, SAM 3 track→VACE, BiRefNet matte, GIMM-VFI, FlashVSR, LTX-2.3
t2v+audio.

## A.00 Arsenal expansion — volume growth (2026-09-13)

| Date              | Item                | Result                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| ----------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| 2026-09-13 09:55Z | A.00.01 volume grow | User instructed in conversation ("expand the storage volume as needed to accommodate them all"). `runpod-provision.mjs volume get ax0oj250j3` → size 500; `volume update ax0oj250j3 --size 1024 --dry-run` → `PATCH /networkvolumes/ax0oj250j3 {"size":1024}`; `--yes` at 09:55:31Z → response size 1024; `volume get` at 09:55:33Z → 1024, EU-RO-1, isis-chroma-models; raw REST GET → 1024; GraphQL `myself.networkVolumes` → 1024. Storage cost $71.20/month tiered (1000 GB × $0.07 plus 24 GB × $0.05), up from $35.00. Irreversible. |

### Sizing (from the tracker's arsenal footprint table)

Every survey row present at once with nothing yet retired is **798.36 GB**
(431.19 GB new on top of the 367.17 GB the lint counted at 09:41Z). The lint
demands 20 % headroom: 798.36 ÷ 0.8 = 998 GB, and a 1000 GB volume would pass
with 20.16 % — one more row of any size breaks it — so the size is **1024 GB**
(22.0 % at peak, 33.6 % at the ~680 GB steady state after the LTX-2.3,
Animate-1, Depth-Anything-V2 and RMBG-2.0 retirements). The 24 GB above the tier
line cost $1.20/month. Lint after the grow (manifest `volume.sizeGb` 1024):
`50 artifacts, 367.17 GB of 1024 GB (64.1% headroom), 0 errors, 0 warnings`.

### Account readings around the grow

| Time (UTC) | Balance | currentSpendPerHr | Note                                                                                                                                                                                                                                   |
| ---------- | ------- | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 09:40:49   | $9.1296 | $0.049            | 500 GB volume only; 0 workers, 0 pods                                                                                                                                                                                                  |
| 09:55:33   | $9.0506 | $0.761            | right after the grow. Not the volume: `/health` at 09:57Z showed the chroma endpoint with 1 running worker and 1 job in progress and the video endpoint with 1 throttled worker — another session's live proofs (commits 09:31–09:55Z) |
| 09:57:28   | $9.0230 | $0.761            | unchanged 60 s later; the volume's own share is $0.0989/hour                                                                                                                                                                           |

**Runway is the open risk (A.00.02):** at $0.0989/hour of storage alone the
$9.02 balance lasts about 3.8 days; RunPod's docs say a volume left at $0 "may
eventually be terminated and its data cannot be recovered". No arsenal pull
starts before the account is topped up.

### A.00.03 Arsenal pull into the Hetzner library

Path per option B: `library-pull` loader pods stream every A-row missing from
the library (Hugging Face and Civitai sources) straight to
`app-dev-server/isis/models/`, mounting the cache volume only for the result
file. Rows already present with matching size and recorded sha256 are skipped,
so a rerun resumes by object.

| Pod              | Run                                                                      | Window (UTC)   | Wall              | Cost USD (0.06/h) | Outcome                                                                                                                                                                                                                                        |
| ---------------- | ------------------------------------------------------------------------ | -------------- | ----------------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ed3tappadomybi` | library-pull, HEAD manifest, par 8                                       | 21:51 to 23:01 | stopped at 22:30  | 0.070             | none; library 62 to 63 objects (406.8 to 408.6 GB); pod deleted, orphan upload aborted                                                                                                                                                         |
| `pa05xxybir0xof` | library-pull rerun, same manifest                                        | 23:01 to 23:17 | 980 s             | 0.016             | none (swapped); LTX-2.5 transformer lost at 13+ GB, distilled LoRA landed (64 objects, 417.5 GB), Animate-2 upload aborted at 1.2 GB for the swap                                                                                              |
| `732ifrk5muf24t` | library-pull, Range-resume loader (`d78f80df034`)                        | 23:17 to 01:55 | loader died 01:50 | 0.158             | none; 64 to 81 objects (417.5 to 680.8 GB): LTX-2.5 transformer, Animate-2 base and distilled, SeedVR2, Z-Image, controlnet patches, HiDream-O1, Gemma 4 E4B, Qwen-Image 2512 and its encoder landed; Qwen Edit 2511 upload aborted at 15.7 GB |
| `bhw3iszl7ug0ia` | library-pull, rerun-on-kill start loop, parallel 4                       | 01:56 to 03:32 | 5330 s loader     | 0.096             | `arsenal-library-pull-result.json`: 118 rows, 80 already present, 25 uploaded (50.34 GB, 7 source reopens), 13 failed                                                                                                                          |
| `920gpxvfzogzez` | library-pull retry of the 13 failures, open-retry loader (`ebbbaf81893`) | 03:34 to 05:03 | 3928 s loader     | 0.089             | `arsenal-library-pull-retry-result.json`: 9 of the 13 already present (their first uploads had landed although the run recorded errors), SCAIL-2 DPO LoRA uploaded, 4 failed                                                                   |
| `qn2f2onkpve4qw` | library-pull, part-retry loader (`e84471a58ae`)                          | 05:05 to 06:06 | 3299 s loader     | 0.061             | `arsenal-library-pull-retry2-result.json`: 114 rows, Qwen Edit 2511 (40.86 GB, 609 parts, 1 source reopen), UnGloryHail (17.80 GB) and the Z-Image AIO (20.52 GB) uploaded; 1 failed (CyberRealistic bf16, HTTP 403)                           |

The first pod's last progress was part 101 of the SeedVR2 7B upload at 22:30:27Z
(6.71 GB, part 96 never arrived). At 22:58Z it showed 0 % CPU and 0 % memory
with the container still up 4010 s, which is the start command's closing `sleep`
after the loader process ended; the multipart upload was still open, so the
watchdog's abort never completed. RunPod exposes no pod log over the API key
(the REST API has no logs path; the console's log host answers 403 to an API
key), and the console's tabs did not respond in the occluded browser window, so
the exit code is unknown. The LTX-2.3 cache sync on the same bucket froze over
the same minutes (see L.06). The rerun, started 23:01:26Z, was uploading the
LTX-2.5 dev transformer within 20 s.

That rerun then lost the transformer too: at 23:07Z its upload stood at 13.02
GB, and by 23:12Z the upload was gone, no object had landed, and the pod had
moved to the next row. `library-pull` read the source as one stream feeding the
multipart upload, so a single source-side reset or timeout threw away every byte
already sent, most likely also what the first pod's transformer row met. The
loader now reopens the source with a Range request at the byte it reached (206
from exactly that offset, 12 reopens, loud failure if the server ignores Range;
three new tests). At 23:17Z, after the distilled LoRA landed, the pod was
swapped for one running the fixed loader.

That pod carried the transformer through (the Range resume works) and ran for 2
h 36 min, then at about 01:50Z its loader stopped mid-upload the same way the
first pod's did: last part 01:49:50Z, 0 % CPU and 0 % memory at 01:54Z with the
container still up (no restart), the multipart upload never aborted. An exit
through the watchdog aborts uploads and exits 75, which the start loop reruns;
an upload left open and a loop that went to sleep mean the process died some
other way, most likely killed (memory on the 2-vCPU pod, 41 % in steady state on
the previous pod, is the leading suspect but unproven without logs). The start
loop now also reruns on any death by signal (exit above 128), still at most five
attempts, and the next pod runs at parallel 4 to halve the part buffers held in
memory.

It finished at 03:24:54Z with no loader death (memory stayed between 11 % and 30
%). Of the 13 failed rows, 12 failed with "The read operation timed out" raised
by opening the source, which the Range-resume code did not retry (it retried
reads, not opens): the Qwen Edit 2511 transformer, UnGloryHail, Z-Image AIO, two
Z-Image LoRAs, four Wan LoRA halves, SCAIL-2's DPO LoRA and the old SAM 3
safetensors. The thirteenth is the CyberRealistic Z-Image bf16 file, which
Civitai answers with 403, the risk A.00.04 recorded. Opens now spend the same
retry budget on timeouts and 5xx; other 4xx fail at once.

The retry showed two more things. Nine of the thirteen "failed" rows were
already in the library with the manifest size and sha256, so their first uploads
had landed and only the client's wait had failed. And Qwen Edit 2511,
UnGloryHail and the Z-Image AIO failed again with the same timeout, this time on
the upload side: a part had five attempts a few seconds apart, which a Hetzner
stall outlasts, so the whole multipart upload was aborted each time (Qwen Edit
at about 22 GB). Parts now get twelve attempts, and a Complete that errors is
checked by HEAD before anything is aborted. CyberRealistic still answers 403.

**Pull closed 2026-09-14 06:00Z.** With twelve attempts per part, the three
large files that had failed three times landed in one run. The library is 117
objects, 813.38 GB (`hetzner-usage`: 813.38 GB of the 900 GB alarm line, outputs
0.00 GB). Every manifest row is in it with its manifest size and sha256 except
the CyberRealistic Z-Image Turbo v8.0 bf16 file, whose Civitai download answers
403 to the account's key. Six pull pods in all, from 21:51Z to 06:06Z; their pod
time at 0.06 USD/hour sums to 0.49 USD (the table's cost column), against the
0.19 USD the plan estimated for a pull with no failures.

**Pinned core back on the cache, 2026-09-14.** The arsenal rows joined pinned
families (the Chroma set, the Wan LoRA pairs, SAM 3.1, SeedVR2 3B), so from the
A.00.04 manifest commit until they were synced the API reported those families
absent and refused their jobs before any GPU time. Cache sync `c2a7313a` (pod
`eu3n20cbc50qnr`, 06:07:27Z to 06:32:13Z, 0.0247 USD booked) pulled 33.73 GB
with no evictions; afterwards all 11 pinned families read cached and the volume
held 304.7 GB of 400.

**RMBG-2.0 deleted, 2026-09-14 07:54Z (A.02.08).** User instruction: "delete as
planned". Deleted by exact listing under the `RMBG/RMBG-2.0/` prefix only:
library `app-dev-server/isis/models/RMBG/RMBG-2.0/` — `model.safetensors`
(884,878,856 B), `config.json`, `birefnet.py`, `BiRefNet_config.py`; cache
volume `5bst2pgw7y/models/RMBG/RMBG-2.0/` — the same four plus their `.sha256`
sidecars. Both prefixes list 0 objects afterwards; `RMBG/BiRefNet/` still lists
6 library and 12 cache objects.

## A.05.02 Live proof matrix — stills (2026-09-14)

User approval in conversation 2026-09-14 ("go ahead with the live proof
renders") against the tracker's $3–5 estimate. Every render below went through
`scripts/isis/runpod-catalog-run.mjs` (the executor's own
`renderCatalogWorkflow`) straight to the image endpoint `6jye7vngg4i685` on
build `b5376e6b`, one worker (`mx03mr7ejhszur`), at the configured image rate
0.000439 USD per execution second (queue delay is not billed). Records:
`docs/agents/evidence/isis-chroma-runpod/a0502/<proof>.json`; the UnGloryHail
record was reconstructed from RunPod's status endpoint because its console
output was not saved. Execution times on the first render of a family include
loading its weights from the network volume (UnGloryHail 17.8 GB, Z-Image base,
klein 4B, HiDream-O1 16.4 GB, the AIO 20.5 GB), so they are upper bounds for a
warm render.

Cache syncs run for these proofs: `711f87c8` (Z-Image base and control, klein
4B, HiDream-O1, Depth Anything 3; 60.09 GB, no evictions, 0.0234 USD) and
`a4fe57f0` (Z-Image AIO 20.52 GB, evicting the leftover LTX-2.3 cache copy of
47.41 GB, whose library copy stays; 0.0033 USD).

| Proof                                  | Item                                      | Workflow                 | RunPod job                                | Delay s | Exec s | USD    | Output (size, sha256)                                                                                               |
| -------------------------------------- | ----------------------------------------- | ------------------------ | ----------------------------------------- | ------- | ------ | ------ | ------------------------------------------------------------------------------------------------------------------- |
| `chroma-ungloryhail-ckpt`              | A.03.01 UnGloryHail checkpoint            | `chroma-txt2img`         | `733d1ab0-535a-48cc-a1f8-4b028babe2bf-e1` | 272.0   | 152.8  | 0.0671 | `chroma-txt2img-0-54b3febf.png` 1282799 B `54b3febf033dc474`                                                        |
| `chroma-lora-nicegirls`                | A.03.01 NiceGirls LoRA                    | `chroma-txt2img`         | `e76be00b-b2cb-4a7f-856f-56db28a6da87-e2` | 119.7   | 156.8  | 0.0688 | `chroma-txt2img-0-5664a65b.png` 993197 B `5664a65b9e2501cf` (not committed)                                         |
| `chroma-lora-nsfw-realism`             | A.03.01 NSFW Realism LoRA                 | `chroma-txt2img`         | `aedf71f0-dd4a-4819-adb7-1fd74a09946b-e2` | 0.1     | 46.7   | 0.0205 | `chroma-txt2img-0-e5cdc1dd.png` 1501766 B `e5cdc1ddff327146` (not committed)                                        |
| `chroma-seedvr2-3b-tail`               | A.01.08 SeedVR2 3B still tail             | `chroma-txt2img`         | `9de9bb72-81bd-4580-81b4-6db594e0f1a9-e1` | 0.1     | 79.2   | 0.0348 | `chroma-txt2img-0-f2ee1906.png` 5797719 B `f2ee190662015048`                                                        |
| `sam3-segment-sam31`                   | A.01.09 SAM 3.1 core segment              | `sam3-segment`           | `4f43f8b2-423c-43f5-bbaf-cf77afaaa0f5-e2` | 0.1     | 18.9   | 0.0083 | `sam3-segment-0-4018fa68.png` 6910 B `4018fa68a39afd71`; `sam3-segment-1-b2228ff3.png` 1487629 B `b2228ff3b30afdb2` |
| `zimage-txt2img-base`                  | A.02.02 Z-Image base                      | `zimage-txt2img`         | `9ba69352-2dfa-4f89-a14a-fcbf61503fd4-e1` | 170.3   | 121.6  | 0.0534 | `zimage-txt2img-0-7c71c0be.png` 1904538 B `7c71c0bec7e44589` (not committed)                                        |
| `zimage-aio-ckpt`                      | A.02.02/A.03.02 Z-Image Turbo AIO         | `zimage-txt2img`         | `57bcfbd0-4ba6-48fb-80ec-6d08485ff167-e2` | 12.4    | 99.3   | 0.0436 | `zimage-txt2img-0-f9f76b32.png` 1381606 B `f9f76b324ab07154` (not committed)                                        |
| `zimage-control-depth-da3`             | A.02.02 control, A.02.07 Depth Anything 3 | `zimage-txt2img-control` | `742ff7f7-cbb4-4b6d-b4e9-f565ce458a5b-e1` | 0.1     | 77.0   | 0.0338 | `zimage-txt2img-control-0-5c7c3f4c.png` 1425279 B `5c7c3f4cf6634922` (not committed)                                |
| `zimage-img2img`                       | A.02.02 img2img                           | `zimage-img2img`         | `398aa014-a328-4a5d-bb7b-0970ac6a458f-e2` | 0.1     | 24.2   | 0.0106 | `zimage-img2img-0-55474db1.png` 1310161 B `55474db1353f02e3` (not committed)                                        |
| `zimage-inpaint`                       | A.02.02 inpaint                           | `zimage-inpaint`         | `b0b870e0-c7a1-4264-99b6-f5e0c2551fb1-e1` | 0.1     | 23.8   | 0.0104 | `zimage-inpaint-0-96205070.png` 1478406 B `962050709a637a6d` (not committed)                                        |
| `zimage-lora-photoreal-betternudes-v3` | A.03.02 BetterNudes LoRA                  | `zimage-txt2img`         | `a7b20555-2643-43c0-81b3-bc3d8dec05c0-e1` | 0.1     | 9.2    | 0.0041 | `zimage-txt2img-0-cbaaed51.png` 1515216 B `cbaaed51be9a39de` (not committed)                                        |
| `zimage-lora-realistic-snapshot-v5`    | A.03.02 Realistic Snapshot LoRA           | `zimage-txt2img`         | `486d9efc-26cf-4b9e-97bd-6394be3f928b-e2` | 0.1     | 9.3    | 0.0041 | `zimage-txt2img-0-b4d8a4ed.png` 2043347 B `b4d8a4ed08360b86` (not committed)                                        |
| `zimage-lora-breast-slider`            | A.03.02 Breast Slider LoRA                | `zimage-txt2img`         | `441c0e6f-6a98-429c-a922-66860acfd2f8-e1` | 0.1     | 6.7    | 0.0029 | `zimage-txt2img-0-9904cbc8.png` 1416530 B `9904cbc8c11b8bad` (not committed)                                        |
| `zimage-lora-detail-slider`            | A.03.02 Detail Slider LoRA                | `zimage-txt2img`         | `119d22c0-1cfd-464b-bae2-e868a41effae-e2` | 0.1     | 6.6    | 0.0029 | `zimage-txt2img-0-d44f0ff4.png` 1432440 B `d44f0ff46a643213` (not committed)                                        |
| `zimage-lora-detaildaemonz-v1.1`       | A.03.02 DetailDaemonZ LoRA                | `zimage-txt2img`         | `e4888677-b741-4245-9142-27e648773993-e2` | 0.1     | 8.7    | 0.0038 | `zimage-txt2img-0-9bc7f897.png` 1194976 B `9bc7f897676eb629` (not committed)                                        |
| `klein4b-txt2img`                      | A.02.04 klein 4B text to image            | `klein4b-txt2img`        | `58eb154b-343e-4d27-949a-7b9e0c6ce69a-e1` | 0.1     | 73.1   | 0.0321 | `klein4b-txt2img-0-6a1220eb.png` 1992456 B `6a1220eb1cd3f2eb` (not committed)                                       |
| `klein4b-edit`                         | A.02.04 klein 4B edit                     | `klein4b-edit`           | `91dbd849-56d9-440b-a846-2982d4355c63-e2` | 0.1     | 6.0    | 0.0026 | `klein4b-edit-0-57aeeab2.png` 1329263 B `57aeeab24079266a`                                                          |
| `hidream-o1-txt2img`                   | A.02.03 HiDream-O1 text to image          | `hidream-o1-txt2img`     | `2cc428f6-498f-4cc2-a762-6208ef24423c-e1` | 0.1     | 82.0   | 0.0360 | `hidream-o1-txt2img-0-23e66682.png` 5030607 B `23e66682995c2f61`                                                    |
| `hidream-o1-edit`                      | A.02.03 HiDream-O1 edit                   | `hidream-o1-edit`        | `a96fdd37-1455-4c37-9d35-90c9f2c2e443-e1` | 0.1     | 42.9   | 0.0188 | `hidream-o1-edit-0-ec02a6f7.png` 4218304 B `ec02a6f7315cde89` (not committed)                                       |
| `hidream-o1-personalize`               | A.02.03 HiDream-O1 personalize            | `hidream-o1-personalize` | `1de1d17b-9bc3-4c3d-a359-28a58db97ae7-e2` | 0.1     | 39.0   | 0.0171 | `hidream-o1-personalize-0-c8ccb4a9.png` 4388745 B `c8ccb4a9b89da4b3` (not committed)                                |

**Total:** 20 renders, 0.4759 USD of execution; with the two syncs 0.5026 USD.
Every job completed on the first submit.

**Checks on the outputs, not just the status:**

- Every PNG has the dimensions the workflow declares (1024 square; 2048 square
  for HiDream-O1 and the SeedVR2 tail).
- SAM 3.1: the mask is the woman's silhouette; the RGBA cut-out's alpha is 0 at
  the corner and 255 on the subject, with 80.2 % of pixels transparent.
- Depth Anything 3 control: the Z-Image output keeps the source still's layout
  (subject position, foreground leaves, frame bars).
- klein 4B edit: the dress changed from linen to red and nothing else did.
- HiDream-O1: the sign reads "HiDream-O1-Image release".
- LoRAs: the five Z-Image LoRA renders share checkpoint, seed and prompt and
  produce five different hashes, so each LoRA was applied.
- SeedVR2 3B tail: the preview showed colour banding in smooth areas; the file
  does not (3,963 distinct colours in a 300 by 100 pixel out-of-focus patch; the
  HiDream still has 13,038 in a similar patch), so the banding was the preview's
  image compression, not the model.

**Content finding:** the Z-Image control render, whose prompt is the workflow's
default and asks for no nudity, came back with a topless subject. The workflow
is `adult_ok`, so this is within policy, but a tenant surface that expects an
ordinary prompt to stay SFW cannot rely on this model to do so.

**Not committed:** outputs with nudity, from the adult LoRAs, or not visually
checked are recorded by hash only (marked above); their bytes stay under
`isis/outputs/<job>/` in the Hetzner bucket (the outputs retention sweep
defaults to a dry run, so nothing removes them yet).

### A.05.02 video endpoint, first batch (2026-09-14)

Same harness, video endpoint `2lezocymx2rgot` on build `b5376e6b`, one worker
(`krxejkbqs9rjxh`), at the configured video rate 0.001647 USD per execution
second. The first job waited 1031 s in the queue for a card (not billed).
Execution on the first render of a family includes loading its weights. Cache
sync `bfe4bf04` brought Qwen-Image and SeedVR2 7B (74.18 GB, evicting the
finished Depth Anything and DWPose, HiDream-O1, klein 4B and Z-Image base
families, 48.36 GB; 0.0099 USD).

| Proof                           | Item                                       | Workflow           | RunPod job                                | Delay s | Exec s | USD    | Output (size, sha256)                                                                                                       |
| ------------------------------- | ------------------------------------------ | ------------------ | ----------------------------------------- | ------- | ------ | ------ | --------------------------------------------------------------------------------------------------------------------------- |
| `sam3-video-track-sam31`        | A.01.09 video track, prompt the person     | `sam3-video-track` | `ca0e7932-cd3e-49c7-9ee6-c82224c6c54e-e2` | 1031.5  | 23.2   | 0.0382 | `sam3-video-track-0-5e722314.mp4` 9410 B `5e7223147440be72`; `sam3-video-track-1-cab35f48.mp4` 148196 B `cab35f48c26683ed`  |
| `sam3-video-track-sam31-boat`   | A.01.09 video track, prompt the paper boat | `sam3-video-track` | `129e7f2e-0d27-40ec-ae1c-afbe8e145ee0-e2` | 1.1     | 6.2    | 0.0103 | `sam3-video-track-0-ec36dfaa.mp4` 22780 B `ec36dfaacbc9913e`; `sam3-video-track-1-d4aacd31.mp4` 147495 B `d4aacd31d6ed14ca` |
| `wan22-i2v-lora-hip-bump-dance` | A.03.03 hip-bump-dance LoRA pair           | `wan22-i2v`        | `b8cb0057-3df0-4063-87b6-6238efc39366-e1` | 0.1     | 336.9  | 0.5548 | `wan22-i2v-0-f91b1347.mp4` 335052 B `f91b13476d371c93`; `wan22-i2v-1-024aa97a.png` 527146 B `024aa97a500f5359`              |
| `qwen-txt2img`                  | A.02.05 Qwen-Image text to image           | `qwen-txt2img`     | `b94bedff-3461-4741-9aa5-aa9a9f1fdf27-e2` | 1.1     | 373.3  | 0.6148 | `qwen-txt2img-0-df7db569.png` 2269141 B `df7db56925093b16`                                                                  |
| `seedvr2-upscale-7b`            | A.01.08 SeedVR2 7B clip upscale            | `seedvr2-upscale`  | `07f38e2a-3635-4566-bd03-816dc30779a5-e2` | 0.1     | 129.1  | 0.2127 | `seedvr2-upscale-0-745be596.mp4` 1113396 B `745be596dc6e7aac`                                                               |

**Total:** 1.4308 USD of execution plus 0.0099 USD of sync.

**Checks on the outputs:**

- SAM 3.1 video: both clips are 33 frames of 832 by 480 at 16 fps, the same as
  the source clip (a paper boat in a puddle, with no person in it). With the
  prompt "the person" SAM 3.1 still returned an object: its mask sat on the
  water below the boat, a false positive at the default 0.5 threshold. With "the
  paper boat" the mask covers the boat exactly, so the overlay is aligned and
  the tracker is correct; the false positive is a property of the model at that
  threshold, recorded rather than hidden.
- Wan 2.2 i2v with the hip-bump-dance pair: 33 frames of 480 by 832 at 16 fps
  from the UnGloryHail still as the start image; the frame at 1.2 s shows the
  same woman mid-dance.
- Qwen-Image: 1328 square, a coherent full-length street fashion photograph from
  the workflow's default prompt.
- SeedVR2 7B: 960 by 1664, 33 frames at 16 fps (2x of the Wan clip); a crop of
  the face shows resolved hair strands rather than a stretched frame.

### A.05.02 video endpoint, second batch (2026-09-14)

Cache sync `24c91fd4` brought LTX-2.5 and Animate-2 (124.98 GB, evicting the
finished Qwen-Image, SeedVR2 7B, Z-Image AIO and Z-Image control families,
108.13 GB; 0.0386 USD). At about 10:29Z the local Postgres and Redis containers
exited cleanly while it ran (not stopped by this session); the API's sync driver
crashed on the lost database, the containers were brought back with
`docker compose -f docker/docker-compose.dev.yml up -d`, and a new driver took
the sync over and finished it with 0 failures. The render harness for LTX-2.5
was killed by the operating system for low memory while polling; the job it had
submitted kept its place in RunPod's queue and a small attach poller
(`GET /status`, then download and hash) wrote its record, which says so.

| Proof                    | Item                            | Workflow          | RunPod job                                | Delay s | Exec s | USD    | Output (size, sha256)                                        |
| ------------------------ | ------------------------------- | ----------------- | ----------------------------------------- | ------- | ------ | ------ | ------------------------------------------------------------ |
| `ltx25-av-t2v-distilled` | A.01.02 LTX-2.5 t2v, distilled  | `ltx25-av`        | `32e9b0f5-21b1-4cf9-a76e-91e00370e05e-e2` | 2913.8  | 400.6  | 0.6598 | `ltx25-av-0-9baf52ef.mp4` 140901 B `9baf52ef9319e3e7`        |
| `wan22-animate-2-min`    | A.01.06 Animate-2 minimum shape | `wan22-animate-2` | `88646ae9-d2e1-4b43-a430-4eed3ad057c2-e2` | 1775.1  | 244.1  | 0.4021 | `wan22-animate-2-0-e56d2121.mp4` 234059 B `e56d2121c0260695` |

Both waited a long time for a card (48.6 and 29.6 minutes; the video endpoint's
worker read `throttled` in EU-RO-1), which is not billed. **Total:** 1.0619 USD
of execution plus 0.0386 USD of sync.

**Checks on the outputs:**

- LTX-2.5: 49 frames of 768 by 448 at 24 fps, 2.04 s, with an AAC stereo 48 kHz
  track; the frame at 1 s shows a red fox crossing a snowy clearing as prompted,
  in a stylised rather than photographic look. The audio track is nearly silent
  (ffmpeg volumedetect: mean -61.8 dB, max -41.6 dB), so the prompted snow sound
  is not present and audio content is not proven; the catalog proof note says
  so.
- Animate-2: 33 frames of 480 by 848 at 16 fps (482 by 854 was requested; the
  graph rounds down to a multiple of 16); the reference woman's face, dress and
  greenhouse are preserved. The driving clip showed the same person, so this
  proves the graph end to end rather than motion transfer to a different person.

**Catalog rates:** following the V.11 convention, each measured video workflow
now carries seconds per megapixel-frame from its executionTime and a headline
scaled to its default shape: sam3-video-track 0.4731 s (default 0.0252 USD),
seedvr2-upscale 2.4495 s per output megapixel-frame (0.5220 USD), ltx25-av
23.7625 s (4.2673 USD for 121 frames at 1280x704) and wan22-animate-2 18.1742 s
(0.9980 USD). The three that loaded weights during the measured run are upper
bounds; scaled to its default shape ltx25-av exceeds the video endpoint's 900 s
execution cap, which a warm measurement has to settle before that shape is
offered.

### A.05.02 video endpoint, SCAIL-2 (2026-09-14)

Cache sync `9a04e6b9` brought SCAIL-2 (35.25 GB, evicting LTX-2.5, 92.19 GB;
0.0384 USD; it took about 36 minutes, the slowest of the day). The job then
waited 29.4 minutes for a card.

| Proof                   | Item                                           | Workflow         | RunPod job                                | Delay s | Exec s | USD    | Output (size, sha256)                                       |
| ----------------------- | ---------------------------------------------- | ---------------- | ----------------------------------------- | ------- | ------ | ------ | ----------------------------------------------------------- |
| `scail2-animate-single` | A.02.06 SCAIL-2, one character, animation mode | `scail2-animate` | `dc178886-aa79-4db5-b8db-41156ad13ae0-e1` | 1762.6  | 417.1  | 0.6870 | `scail2-animate-0-be9a0f0c.mp4` 320342 B `be9a0f0c19412e24` |

**Check on the output:** 33 frames of 480 by 832 at 16 fps. Its frames look very
close to the driving clip's, because the reference image and the driving clip
show the same woman, so a passthrough had to be ruled out: ffmpeg PSNR between
the output and the driving clip averages 26.2 dB (minimum 23.6, maximum 38.5),
far from the 40 dB and more a re-encoded copy would score (Animate-2's output
against the same driver averages 19.0 dB). The graph generates new frames that
follow the driver's pose; transfer to a different character is not proven by
this run. It ran at 20 steps, half the default, which the catalog cost note
says.

**Proof spend for the day:** 3.6556 USD of execution across 28 renders (20
stills 0.4759, 8 video jobs 3.1797) plus 0.1136 USD for the five proof cache
syncs (711f87c8 0.0234, a4fe57f0 0.0033, bfe4bf04 0.0099, 24c91fd4 0.0386,
9a04e6b9 0.0384), 3.7692 USD in all at the ledger rates against the approved 3
to 5 USD. The RunPod balance went from 14.9489 USD (08:13Z) to 12.3962 USD
(13:36Z), 2.5526 USD including about 0.20 USD of volume storage over those
hours: the ledger rates bill the dearest card each endpoint allows, so they
overstate what the cards that ran actually cost, and the balance can settle
minutes late.

### A.05.02 video endpoint, the approved remainder: lanes A and B (2026-09-14)

Approved separately after the first 3 to 5 USD ("the rest", about 6.5 to 8 USD).
Same harness, endpoint and rate. SCAIL-2, Animate-2 and the Wan 2.2 families
were already on the cache, so these jobs needed no sync. The adult and racy LoRA
outputs are kept out of git and are recorded here by size and hash only; their
records in `a0502/` still name the paths the harness wrote.

| Proof                                   | Item                                            | Workflow          | RunPod job                                | Delay s | Exec s | USD    | Output (size, sha256)                                                                                          |
| --------------------------------------- | ----------------------------------------------- | ----------------- | ----------------------------------------- | ------- | ------ | ------ | -------------------------------------------------------------------------------------------------------------- |
| `scail2-animate-two`                    | A.02.06 SCAIL-2, two characters, animation mode | `scail2-animate`  | `8140f060-a79e-4577-8bb5-b0e131823474-e2` | 294.4   | 358.0  | 0.5896 | `scail2-animate-0-1506e915.mp4` 412329 B `1506e915c58f208d`                                                    |
| `wan22-animate-2-896`                   | A.01.06 Animate-2 at 512 by 896                 | `wan22-animate-2` | `37528cd3-9e40-4dd4-b325-ac49fcf34b29-e2` | 248.4   | 253.5  | 0.4174 | `wan22-animate-2-0-0c92e293.mp4` 236586 B `0c92e293027dbb35`                                                   |
| `wan22-i2v-lora-camera-tiltdown`        | A.03.03 camera tilt-down pair (neutral)         | `wan22-i2v`       | `133e1aa8-26a4-4ca2-b3a1-901321752cbc-e1` | 65.5    | 57.9   | 0.0954 | `wan22-i2v-0-dcb06490.mp4` 323928 B `dcb06490bb3c1b7b`; `wan22-i2v-1-fde272aa.png` 468614 B `fde272aaf954e6fa` |
| `wan22-i2v-lora-camera-tiltup`          | A.03.03 camera tilt-up pair (neutral)           | `wan22-i2v`       | `844ca07e-d4a7-4abf-993b-3815cf7a39e5-e1` | 60.6    | 70.8   | 0.1166 | `wan22-i2v-0-f50b7dd8.mp4` 263688 B `f50b7dd8c60a0485`; `wan22-i2v-1-02a944d0.png` 532490 B `02a944d0ff2950a1` |
| `wan22-i2v-lora-hip-sway`               | A.03.03 hip-sway pair (racy)                    | `wan22-i2v`       | `39d9651b-5202-459d-bd42-12ccad46ba46-e1` | 413.2   | 300.5  | 0.4950 | not committed: mp4 255334 B `71d78272154ebdd1`; png 542016 B `b0fff772dcdbd69e`                                |
| `wan22-i2v-lora-nsfw-posing-nude`       | A.03.03 posing pair (explicit)                  | `wan22-i2v`       | `1b4c9f80-53ea-4af3-be5a-8d9f705dd9e2-e1` | 349.5   | 68.2   | 0.1123 | not committed: mp4 192342 B `c43e7e3a932b2054`; png 557527 B `09b16e3371f1c8ce`                                |
| `wan22-i2v-lora-bouncing`               | A.03.03 bouncing pair (explicit)                | `wan22-i2v`       | `9c76252d-8319-434a-a327-3b2126452a74-e2` | 50.2    | 69.2   | 0.1140 | not committed: mp4 226285 B `9ea2b98439b09b36`; png 527515 B `06a11feed01e4e9a`                                |
| `wan22-i2v-lora-twerking`               | A.03.03 twerking pair (explicit)                | `wan22-i2v`       | `ae5f9ca6-4f5f-4f9f-90bd-75594dcd02ab-e2` | 65.8    | 59.1   | 0.0973 | not committed: mp4 268534 B `48b21924965cac0c`; png 545165 B `3fd9c111b20d7945`                                |
| `wan22-i2v-lora-slow-twerk`             | A.03.03 slow-twerk pair (explicit)              | `wan22-i2v`       | `fdae4237-cd6f-4e93-b5db-454c90014812-e2` | 0.1     | 58.1   | 0.0957 | not committed: mp4 204766 B `44dee0c2b2bf2407`; png 541333 B `0c8e066ebcc59230`                                |
| `wan22-t2v-lora-breast-slider-failed-1` | A.03.04 breast-slider pair, first attempt       | `wan22-t2v`       | `04d4b818-05b0-4a5e-8a0b-2972918bf592-e2` | 0.1     | 217.1  | 0.3575 | FAILED: ComfyUI HTTP unreachable during websocket reconnect                                                    |
| `wan22-t2v-lora-breast-slider`          | A.03.04 breast-slider pair, retry               | `wan22-t2v`       | `a3d632ed-34aa-44f3-8249-eb0f9885c1b8-e2` | 989.9   | 355.9  | 0.5862 | not committed: mp4 334344 B `d8bc4161c9619cfd`; png 487396 B `c528c9f9f069af9b`                                |

**Total:** 3.0771 USD of execution, 0.3575 of it the failed attempt.

**Checks on the outputs:**

- SCAIL-2, two characters: 33 frames of 832 by 480 at 16 fps. The reference
  image held two different women (the greenhouse woman and a woman in a pink
  coat on a street); the driving clip was two copies of the greenhouse clip side
  by side. At frame 16 the output places both reference women, in reference
  order left to right, in the driver's greenhouse, each in the pose of her
  driving figure. The right-hand figure is a different person from the driver,
  so this run proves transfer to a different character, which the single run
  could not.
- Animate-2 at 512 by 896: 33 frames at 16 fps, the requested shape exactly (it
  is already a multiple of 16). The last frame shows the reference woman waving,
  as the driver does. Its 253.5 s against the 275 s its catalog rate of 18.1742
  s per megapixel-frame predicts for 15.14 megapixel-frames keeps that rate an
  upper bound.
- All eight Wan clips are 33 frames of 480 by 832 at 16 fps. The two camera
  pairs move the camera as named: the tilt-down clip ends on a low angle looking
  up at the woman, the tilt-up clip ends looking down on her from above. Their
  prompts also describe that camera move, so how much of it the pair adds is not
  isolated by these clips.
- The racy and explicit pairs, including the breast slider, ran with LoRA
  strength 1 on both experts and completed, but at frames 16 and 32 every one of
  them shows the subject fully clothed and in the start image's framing (hip
  sway with arms crossed, as prompted). Motion is present (mean absolute
  frame-to-frame luma difference 3.9 to 7.0 of 255 on the four motion pairs).
  These runs prove the pairs load and render on the graph; whether a pair
  changes the render needs a same-seed render without it, below.
- Same-seed baselines without a pair (approved as part of the Wan LoRA pairs'
  share): tilt-down (seed 1472, same prompt, job
  `89468c63-7d55-49ec-a3d9-4573b468aa17-e2`, 374.3 s including swapping the Wan
  i2v experts back in after the Qwen jobs, 0.6165 USD;
  `wan22-i2v-1-c66c83af.mp4` 279080 B `c66c83aff59b1217`,
  `wan22-i2v-0-395a1e80.png` 497171 B `395a1e800213bfdf`) and posing (seed 1475,
  same prompt, job `7cca256d-e294-46fe-998a-7147f1d5a8af-e1`, 18.8 s warm,
  0.0310 USD; `wan22-i2v-1-0430d68f.mp4` 217013 B `0430d68fc5db5dbe`,
  `wan22-i2v-0-c1dabb86.png` 547757 B `c1dabb86a9f3d4fe`). ffmpeg PSNR between
  each LoRA clip and its baseline averages 14.1 dB for tilt-down and 22.9 dB for
  posing, so both pairs are applied and change the render. At the last frame the
  tilt-down clip is a tighter undershot near the chin than the baseline's upward
  look, the camera move the pair is named for; the posing clip differs in pose
  from its baseline but both stay clothed. The posing pair's versions list no
  trained words; bouncing, twerking, slow twerk and hip sway were prompted with
  their listed phrases and still stayed clothed from a clothed start image, so
  their named content is not shown by these clips. **Spend of this remainder so
  far:** 3.0771 USD for lanes A and B, 1.1179 USD plus the 0.1133 USD sync for
  Qwen (next section) and 0.6475 USD for the two baselines, 4.9558 USD in all.
- The first breast-slider attempt ran straight after the slow-twerk job, warm on
  the same worker (`io8c9yys4vg9lp`) that still held the i2v experts; ComfyUI
  stopped answering 217 s in, while it would have been loading the t2v experts.
  Host memory with four 14B experts touched is the likely cause, but the worker
  was recycled before its log could be read, so the cause is not established.
  The retry, on the same worker after a cold start, succeeded. Both records are
  kept.

### A.05.02 video endpoint, Qwen-Image edit and control (2026-09-14)

Cache sync `b80de934` (14:08 to 16:01Z, 0.1133 USD) pulled Qwen-Image, its edit
transformer and the two InstantX ControlNets (6 files, 65.47 GB, 0 failures),
evicting Animate-2 and SCAIL-2 after their clips (68.04 GB). The video worker
then read `throttled` in EU-RO-1 for about an hour: the first qwen-edit job
(`fcd5f094-9f22-4843-9798-45d3510df5b6-e2`) was cancelled by the harness after
60 minutes in the queue, unbilled, and the union job waited 52.7 minutes.

| Proof                              | Item                                           | Workflow            | RunPod job                                | Delay s | Exec s | USD    | Output (size, sha256)                                           |
| ---------------------------------- | ---------------------------------------------- | ------------------- | ----------------------------------------- | ------- | ------ | ------ | --------------------------------------------------------------- |
| `qwen-edit-control-union`          | A.02.05 union control, default prompt          | `qwen-edit-control` | `93050b7b-00d3-4d4c-be50-a54b947cd29d-e2` | 3159.7  | 295.5  | 0.4867 | `qwen-edit-control-0-ec35c80b.png` 1661378 B `ec35c80b74477e83` |
| `qwen-edit-control-inpaint`        | A.02.05 inpaint control                        | `qwen-edit-control` | `7df56a13-3e4a-4a8e-9a6d-c392f5058022-e1` | 0.1     | 77.5   | 0.1277 | `qwen-edit-control-0-8343b9b0.png` 2357379 B `8343b9b068477888` |
| `qwen-edit-one`                    | A.02.05 edit, one reference                    | `qwen-edit`         | `c432e401-648a-4f7f-9d7b-d059af1916cc-e1` | 0.1     | 282.5  | 0.4652 | `qwen-edit-0-0204f8c6.png` 1333336 B `0204f8c6796500b9`         |
| `qwen-edit-control-union-prompted` | A.02.05 union control, prompt matching the map | `qwen-edit-control` | `fc2678f9-b4b9-4f0d-bace-4882d9582692-e2` | 209.0   | 23.3   | 0.0383 | `qwen-edit-control-0-755eff00.png` 1492677 B `755eff0057926107` |

**Total:** 1.1179 USD of execution plus 0.1133 USD of sync.

**Checks on the outputs:**

- Union, default prompt: the control map was a 1024 square canny edge map of the
  Qwen-Image street photograph's full-length woman. The output is a 1024 square
  close-up of a hooded woman, the workflow's default (template) prompt, with no
  trace of the map's figure: the prompt overrode the control. This is recorded
  as a finding, not hidden, and the prompt input's description now says to
  describe what the map shows.
- Union, prompt describing the map: 1024 square; drawn over the output, the
  canny edges lie on the woman's outline, coat, handbag and shoes, so the union
  ControlNet does steer the graph. RunPod flagged the job a cold start, but it
  ran on the worker that had just run the other Qwen jobs, and its 23.3 s is a
  warm figure.
- Inpaint: source the UnGloryHail greenhouse still, mask the SAM 3.1 person
  silhouette (A.05.02 stills), prompt "a woman in a red leather jacket, natural
  light". The output is 1536 square (the template scales the source's long side
  to 1536) with a red leather jacket over a dark top inside the mask and the
  greenhouse outside it unchanged; the face inside the mask was repainted too.
- Edit: same source still, one reference, "Change the linen dress to a blue
  denim jacket over a white t-shirt." The 1024 square output keeps the woman's
  face, hair, pose and the greenhouse, and replaces the dress with the jacket
  and t-shirt as asked.

### A.05.02 video endpoint, LTX-2.5 audio, Ingredients and pixel x2 (2026-09-14)

Cache sync `cdbb552e` (18:14 to 18:54Z, 0.0402 USD, 0 failures) brought LTX-2.5
back (92.19 GB), evicting the three Qwen-Image families after their stills. All
three clips ran distilled on one worker (`2dohwqc0wosv89`); the first waited 7.1
minutes for a card and loaded the weights.

| Proof                  | Item                                                  | Workflow   | RunPod job                                | Delay s | Exec s | USD    | Output (size, sha256)                                 |
| ---------------------- | ----------------------------------------------------- | ---------- | ----------------------------------------- | ------- | ------ | ------ | ----------------------------------------------------- |
| `ltx25-av-t2v-audio`   | A.01.02 t2v with a loud sound prompt, 49 frames       | `ltx25-av` | `6bb72e9d-04d7-4edf-88d8-958cb92a9a06-e1` | 426.4   | 414.4  | 0.6826 | `ltx25-av-0-1a1b9424.mp4` 340092 B `1a1b9424f79260e0` |
| `ltx25-av-ingredients` | A.01.03 Ingredients, 121 frames, the model card shape | `ltx25-av` | `452cf724-4734-4474-b6d0-0d7803970fdd-e2` | 0.1     | 134.5  | 0.2215 | `ltx25-av-0-a2bdb521.mp4` 535041 B `a2bdb52171bf3b75` |
| `ltx25-av-pixel-x2`    | A.01.03 pixel x2 tail on a 49-frame t2v               | `ltx25-av` | `61c74f24-b00a-41a6-8f05-f743a57e6c9f-e2` | 0.1     | 187.5  | 0.3088 | `ltx25-av-0-dfeb18ce.mp4` 682824 B `dfeb18ce1416bb0a` |

**Total:** 1.2129 USD of execution plus 0.0402 USD of sync.

**Checks on the outputs:**

- Audio: prompted "a street drummer hits a snare drum hard and fast … loud crisp
  drum beats", the clip (49 frames of 768 by 448 at 24 fps, AAC stereo 48 kHz)
  shows a drummer with a snare on a city street, and its audio is loud (ffmpeg
  volumedetect mean -17.2 dB, max -0.5 dB); the spectrogram shows evenly spaced
  broadband bursts, and the 50 ms RMS envelope peaks about every 0.4 s, drum
  hits rather than noise. LTX-2.5 audio is proven. The earlier fox clip's
  near-silence (-61.8 dB) and this batch's Ingredients (-52.0 dB) and pixel x2
  (-54.0 dB) clips, prompted with birdsong, footsteps, snow and wind, show that
  soft ambient prompts come out nearly silent.
- Ingredients: the reference sheet (`ltx25-av-reference-sheet.png`, composed for
  this proof from the committed Qwen-Image street still and klein 4B greenhouse
  still: the woman full length, her face, her yellow handbag and a greenhouse,
  on black) produced 121 frames of 768 by 448 at 24 fps, 5.04 s: the same woman,
  braids, rose-gold leather coat and green trousers, walking through a
  greenhouse with the yellow handbag in the first frame, as prompted. Identity,
  wardrobe, prop and location all carry from the sheet. Warm, single-stage
  distilled: 3.23 s per megapixel-frame.
- Pixel x2: 49 frames of 1536 by 896 at 24 fps, exactly twice the 768 by 448
  base, a fox crossing a snowy clearing in a stylised look. Resampling the
  output down to 768 by 448 and back up scores 37.1 dB PSNR against it, so for
  this smooth, stylised clip the detail added beyond a plain 2x resample is
  modest; the tail runs and doubles the frame, and its quality gain is not
  established by this clip.

**Spend of the approved remainder:** 3.0771 (lanes A and B) + 1.1179 and 0.1133
sync (Qwen) + 0.6475 (LoRA baselines) + 1.2129 and 0.0402 sync (LTX-2.5) =
6.2089 USD at the ledger rates, against the approved 6.5 to 8 USD. The RunPod
balance read 8.9172 USD at about 19:25Z, down 3.4790 USD from 12.3962 USD at
13:36Z, including volume storage (0.039 USD an hour) over those hours; as
before, the ledger rates bill the dearest card each endpoint allows.

### A.01.09 SAM 3 packs removed from the worker image (2026-09-14)

Commit `440d66abc98` removed ComfyUI-SAM3 and ComfyUI-RMBG's vendored SAM 3
still node (`py/AILab_SAM3Segment.py`, `models/sam3/`) from
`docker/runpod/comfyui-slim`, with pycocotools, the `sam3` model folder, its
entry symlink and extra-model-paths key, and the two SAM 3 manifest rows and
their pinned family. Pushed to `runpod-comfyui-slim` at 19:54Z; RunPod build
`a350cf59-3409-4516-b43f-6011395ba314` completed at 20:15Z and both endpoints'
templates now name image tag `440d66abc`. The build's smoke passed, including
its new check that none of `SAM3Segment`, `LoadSAM3Model`,
`SAM3VideoSegmentation`, `SAM3Propagate` and `SAM3VideoOutput` registers. The
object-info snapshot was recaptured from that build (1133 classes, sha256
`debec3de373a…`): the five classes are absent, `SAM3_Detect`, `SAM3_VideoTrack`,
`SAM3_TrackToMask`, `RMBG` and `BiRefNetRMBG` are present, and the graph
validator specs pass against it.

Live check on the rebuilt image endpoint: `sam3-segment` with "the woman" on the
UnGloryHail still, job `270f413b-e4fc-4dc4-804a-e60fe224c149-e1` (183.7 s queue
while the new image was pulled, 18.5 s execution, 0.0081 USD), outputs
`sam3-segment-0-16fa5a23.png` 6906 B `16fa5a2373a5` (mask) and
`sam3-segment-1-cef297da.png` 1487628 B `cef297daedb3` (cut-out). The mask is
identical to the earlier SAM 3.1 run's (`sam3-segment-0-4018fa68.png`,
intersection over union 1.0).

### A.05.02 video endpoint, LTX-2.5 default job and an operator API submission (2026-09-14)

Approved by the user ("proceed on all") after the remainder closed: one LTX-2.5
clip at the workflow's own default (it could have passed the 900 s cap) and one
A.03.04 internal-only Wan pair submitted as the operator tenant through the API
rather than the harness.

| Proof                    | Item                                             | Workflow   | RunPod job                                | Delay s | Exec s | USD    | Output (size, sha256)                              |
| ------------------------ | ------------------------------------------------ | ---------- | ----------------------------------------- | ------- | ------ | ------ | -------------------------------------------------- |
| `ltx25-av-default-shape` | A.01.02 default job: t2v, 30 steps, 121x1280x704 | `ltx25-av` | `70664678-df65-44aa-83d7-4b46fa0a0192-e2` | 218.6   | 440.8  | 0.7259 | `ltx25-av-0-2c03af8e.mp4` 1368785 B `2c03af8eb2b8` |

**Check on the output:** 121 frames of 1280 by 704 at 24 fps with AAC stereo 48
kHz; frames 0, 60 and 120 show a drummer at a kit on a club stage under
spotlights, sticks moving, as prompted. The audio is loud (mean -11.4 dB, peak
0.0 dB, which may be clipping): a dense broadband wash with a regular
low-frequency pulse in the spectrogram. The job took 440.8 s on a cold worker,
half the cap; the first measurement's 23.7625 s per megapixel-frame (a 49-frame
cold clip, mostly load time) had scaled this job to 2,591 s, so the catalog now
carries 4.0423 s per megapixel-frame from this job.

### A.02.02 Z-Image VRAM at 1024 and 2048 square (2026-09-15)

On the same approval. The `zimage` family (20.88 GB, seven files) was synced
into the cache through `POST /api/v1/runpod/cache/sync {"families":["zimage"]}`
(sync `0ab86377`), with no eviction. `zimage-cyberrealistic` was left out
because it is still in Civitai early access. Both runs used
`runpod-catalog-run.mjs` on the image endpoint: seed 4242, the base checkpoint,
25 steps, CFG 4. The 2048 run overrode the rendered latent with
`--graph-set 10.width=2048 --graph-set 10.height=2048`, since the catalog offers
no 2048 aspect. `--capture-log` kept each worker log.

| Shape     | RunPod job                                | Delay s | Exec s | Ledger USD | Result                                                                                        |
| --------- | ----------------------------------------- | ------- | ------ | ---------- | --------------------------------------------------------------------------------------------- |
| 1024x1024 | `20f079cc-2c5e-404d-858e-bf0168b195a9-e2` | 102.6   | 150.8  | 0.0461     | PNG 1 330 497 B `ab725798e2cd56cd` (the prompted teapot); sampling 25/25 in 1:03 at 2.56 s/it |
| 2048x2048 | `e8575edb-6aea-47fb-8846-15d205c52e96-e2` | 1.1     | 304.8  | 0.0933     | `executionTimeout exceeded` at step 23 of 25, 13.27 s/it                                      |

The worker log names the card and its memory: `Device: cuda:0 NVIDIA L4`,
`Total VRAM 22563 MB`, with async weight offloading and pinned host memory
enabled. At 1024 the text encoder loaded, the Lumina2 diffusion model loaded and
sampled, and the VAE decoded with no out-of-memory line
(`Prompt executed in 145.35 seconds`, cold). At 2048 the same warm worker
sampled with no out-of-memory line until the image endpoint's 300 s execution
cap cancelled it, 2 steps short of 25 and before the VAE decode. Each step took
5.2 times as long as at 1024.

What this measures and what it does not:

- ComfyUI logs no peak allocation at INFO, and no "loaded partially" line
  appeared, so there is no peak-GB figure.
- The finding is fit and time. 1024 square fits a 22.5 GB L4 comfortably. 2048
  square did not run out of memory, but cannot finish inside the 300 s cap on
  that card: about 330 s of sampling plus a 2048 decode. The catalog offers no
  2048 aspect, and this is the reason it should not without a larger card or a
  longer cap.
- The 2048 run left no image to check.
- Log lines
  `Error loading … AILab_SegmentV2.py: No module named 'groundingdino'` are an
  unrelated ComfyUI-RMBG import warning; the RMBG nodes the catalog uses load.

### A.01.02 multishot shots[] with auto duration, through the API (2026-09-15)

Approved by the user in this session ("Approve ≈1.0 USD" for a multishot clip
and the A.02.02 VRAM stills). The duration head
(`model_patches/ltx-2.5-duration-head-bf16.safetensors`, 3 843 690 B, sha256
`2ec71e42…`) was pulled into the Hetzner library by a hashed PUT and into the
cache by `POST /api/v1/runpod/cache/sync` (one file, sync `df7019d1`, 20.5 s of
loader pod, 0.000342 USD). The worker image was rebuilt (build `3acf2224`,
commit 83fafe101) so the snapshot's `ModelPatchLoader` options include it. Two
earlier submissions were refused before any GPU time, and both refusals were
right: the executor's "Prompt is required" (fixed so a `shots` array satisfies
it) and the pre-submit validator's "not one of its 2 options" (fixed by the
rebuild).

| Proof                | Request                                                                                         | RunPod job                                | Delay s | Exec s | Ledger USD | Output (size, sha256)                                              |
| -------------------- | ----------------------------------------------------------------------------------------------- | ----------------------------------------- | ------- | ------ | ---------- | ------------------------------------------------------------------ |
| `ltx25-av-multishot` | t2v, default schedule, 3 shots joined by hard cuts, `auto_duration` with `length` 121, 1280x704 | `442dd359-fd59-48cb-8dc8-adc4568590ac-e1` | 60.5    | 500.2  | 0.8238     | `isis/outputs/jobs/978264c4…/0.mp4` 1 561 493 B `4d6acde37d607b88` |

The ledger bills the dearest card on the endpoint (0.001647 USD/s). The account
balance read 5.85 USD before the session's RunPod work and 5.34 USD after this
clip, including about three hours of volume storage, and RunPod settles late.

**Check on the output:** 121 frames of 1280 by 704 at 24 fps, with audio 5.01 s
long, mean -14.3 dB and peak 0.0 dB. ffmpeg scene scores find cuts at 2.21 s and
3.83 s. A contact sheet at frames 10, 45, 60, 85 and 115 shows the three
prompted shots in order: the wide club stage under red lights with the crowd, a
close-up of the snare as the sticks come down, and the drummer in the black
shirt grinning among cymbals. The shots were composed into one paragraph, and
the model cut between them.

**What it does not show:** the predictor's own choice. The clip is exactly the
121-frame cap, so the predicted duration was at or above 5.04 s (the node clamps
to `max_seconds` and floor-snaps to 8k+1). The worker log was captured, but its
last 400 lines are all image-pull and container lifecycle lines, so the node's
"predicted … frames" line is not in them. The predictor node ran inside the
graph: it is the only source of the latent length and audio frame count in this
render, and a failing node fails the job. Sizing below the cap is not
demonstrated, and a second clip would have gone past the approved spend.

**A.03.04 through the API as the operator.** The local generation API (restarted
at 20:49Z on the current commit) took `POST /api/v1/jobs` from the dev operator
token (tenant `tenant_isis_dev`) for `wan22-i2v` with the internal-only
`nsfw-posing-nude-v2` pair, lightx2v 4-step, 33 frames at 480 by 832, seed 1494,
the UnGloryHail still as `start_image` and `presetContent: explicit`. The first
submission (`29607a59-4a98-4bf6-8ce7-350f8a1f39cd`) was refused before any GPU
time because it carried no `consentId`, as the provenance ledger requires; the
resubmission with the Studio's operator consent id
(`5aeb8e81-9c61-4ee0-a441-8de5d4ebe953`, RunPod job
`f1cfda18-4c5b-419f-8da3-f8adf4666cd9-e2`) completed: 491.2 s in the queue
behind the LTX-2.5 job, 347.4 s of execution on a cold worker, 0.4837 USD on the
video ledger. The model cache logged hits on wan22-i2v and wan-shared and
released both leases; the output carries a provenance bundle
(`prov_73253336c383c8fc`, consent id, prompt hash, model revision) and the
watermark on its last-frame still. The clip (33 frames of 480 by 832 at 16 fps,
204358 B, sha256 `b9bf91dff6e9a54f`) is kept out of git; at frames 16 and 32 the
subject is clothed, like the harness clip for the same pair.

### A.03.04 diagnosis: why the explicit Wan pairs stayed clothed (2026-09-14)

Asked for by the user ("i want that") after every explicit-pair clip, harness
and API alike, kept the subject clothed. Five jobs on the posing pair
`nsfw-posing-nude-v2`, all at LoRA strength 1, prompt "A woman posing in a
greenhouse, natural light." Outputs with nudity and every output of this pair
stay out of git; records and hashes only.

| Proof                              | What it isolates                                      | RunPod job                                | Delay s | Exec s | USD    | Output (size, sha256)                                            |
| ---------------------------------- | ----------------------------------------------------- | ----------------------------------------- | ------- | ------ | ------ | ---------------------------------------------------------------- |
| `wan22-i2v-lora-posing-quality`    | full 20-step schedule, CFG 3.5, no lightx2v, 33 fr    | `f961a792-813a-4218-baba-34a99f2df52c-e1` | 74.7    | 508.2  | 0.8370 | mp4 154168 B `76286c167acb9d9e`; png 553725 B `86a903784fbbc227` |
| `wan22-i2v-lora-posing-81f`        | lightx2v 4-step at 81 frames (the trained length)     | `ab4911c7-fcda-42f8-ab94-8e52328f3347-e2` | 0.1     | 69.7   | 0.1148 | mp4 479593 B `f6afe43bd00ef7f2`; png 545591 B `b7fb8bb75fd70c30` |
| `chroma-artistic-nude-start`       | an artistic-nude start still (UnGloryHail, seed 1402) | `570d69cd-fa61-4bfd-80b9-924d4981dbf3-e1` | 7.4     | 140.6  | 0.0617 | png 1558704 B `bbd82323fba4a8c6`                                 |
| `wan22-i2v-lora-posing-nude-start` | the pair from that still, 4-step, 33 fr, seed 1495    | `b5e76dca-5b23-4bdd-91d5-7481e3dcd3d5-e2` | 267.3   | 370.1  | 0.6096 | mp4 237147 B `739e850687f20bf0`; png 612233 B `a1089e9e83321c55` |
| `wan22-i2v-baseline-nude-start`    | the same without the pair                             | `08b06a57-ec33-4321-a8f7-26d1fa9e20b0-e2` | 54.0    | 381.2  | 0.6278 | mp4 277506 B `5961746db79dd4ab`; png 617620 B `8fa20230940d66eb` |

**Total:** 2.2509 USD, above the 1 to 2 USD quoted, because the last two jobs
each landed on a cold worker and reloaded both experts.

**Findings:**

- The pair loads completely. The worker log of the first job (327 lines,
  streamed while it ran, kept beside the outputs) has no `lora key not loaded`
  line, which ComfyUI prints for every LoRA key that matches nothing; the files
  are in the kohya/musubi `lora_unet_blocks_N_*` format, 40 blocks, rank 32,
  which ComfyUI maps onto the Wan model.
- Neither the schedule nor the length is the cause: on the full 20-step schedule
  and at 81 frames the subject stays clothed from the clothed start still, as in
  the 4-step 33-frame clips.
- The start frame is the cause. Image-to-video holds the first frame's clothing;
  nothing in these runs removes it, with or without the pair.
- From the artistic-nude start still, both clips animate the figure naturally
  (turning from the back view to face the camera, weight shifting, a hand
  raised), with no visible anatomical artefacts at frames 0, 16 and 32. The clip
  with the pair and the one without differ by 23.0 dB PSNR and move by about the
  same amount (mean frame-to-frame luma change 2.00 against 2.56); the pair's
  posing is not recognisably different from what the prompt alone produces.

**Conclusion:** the explicit pairs work mechanically and change the render, but
no clip shows a result the pair adds over the base model: they cannot turn a
clothed start into a nude one, and from a nude start the base model already
poses the figure. A.03.04 is reopened on that basis.

## A.05.03 Retirements executed (2026-09-14)

User instruction on 2026-09-14: "proceed on all", in answer to deleting the
retired files from the Hetzner library and the cache volume. Each replacement
had rendered first (A.01.02/A.01.03 LTX-2.5, A.01.06 Animate-2, A.01.08 SeedVR2
3B, A.01.09 SAM 3.1 on core nodes, A.02.07 Depth Anything 3), and each file had
already left the catalog, the manifest and the register in its own commit.
Deleted at 20:40 to 20:44Z by exact key (a scratch script listing each manifest
`destPath` and refusing any other key than the file, its `.sha256` sidecar or a
`.partial` leftover), after a dry run that listed the same keys and sizes. The
first run hit a Hetzner `GatewayTimeout` after three deletions; the rerun with
retries finished, and a final listing of every key returned nothing. The cache
database's realesrgan, sam3 and rmbg-2.0 families, which still read `cached`,
were moved to `absent` with `evicted` events through the store's own eviction
methods.

| Retired row                   | Item    | Library key under `isis/models/`                                                   | Bytes          | Cache copy deleted                               |
| ----------------------------- | ------- | ---------------------------------------------------------------------------------- | -------------- | ------------------------------------------------ |
| `ltx23-22b-dev`               | A.01.04 | `checkpoints/ltx-2.3-22b-dev.safetensors`                                          | 46,149,344,974 | none                                             |
| `gemma3-12b-it-bf16`          | A.01.04 | `text_encoders/gemma_3_12B_it.safetensors`                                         | 24,379,468,890 | a `.partial` of 8,724,152,320 B from an old pull |
| `ltx23-distilled-lora-1.1`    | A.01.04 | `loras/ltx-2.3-22b-distilled-lora-384-1.1.safetensors`                             | 7,605,507,256  | none                                             |
| `ltx23-spatial-x2`            | A.01.04 | `latent_upscale_models/ltx-2.3-spatial-upscaler-x2-1.1.safetensors`                | 995,743,560    | none                                             |
| `ltx23-ic-lora-union-control` | A.01.04 | `loras/ltx-2.3-22b-ic-lora-union-control-ref0.5.safetensors`                       | 654,465,352    | none                                             |
| `ltx23-ic-lora-motion-track`  | A.01.04 | `loras/ltx-2.3-22b-ic-lora-motion-track-control-ref0.5.safetensors`                | 327,309,314    | none                                             |
| `ltx23-temporal-x2`           | A.01.04 | `latent_upscale_models/ltx-2.3-temporal-upscaler-x2-1.0.safetensors`               | 261,944,000    | none                                             |
| `wan22-animate-bf16`          | A.01.06 | `diffusion_models/wan2.2_animate_14B_bf16.safetensors`                             | 34,549,787,368 | none                                             |
| `wan22-animate-relight-lora`  | A.01.06 | `loras/wan2.2_animate_14B_relight_lora_bf16.safetensors`                           | 1,436,673,432  | none                                             |
| `sam3`                        | A.01.09 | `sam3/sam3.pt`                                                                     | 3,450,062,241  | the file and its sidecar                         |
| `sam3-safetensors`            | A.01.09 | `sam3/sam3.safetensors`                                                            | 3,449,860,632  | the file and its sidecar                         |
| `depth-anything-v2-vitl`      | A.02.07 | `controlnet_aux/depth-anything/Depth-Anything-V2-Large/depth_anything_v2_vitl.pth` | 1,341,395,338  | none                                             |
| `realesrgan-x4plus`           | A.01.08 | `upscale_models/RealESRGAN_x4plus.pth`                                             | 67,040,989     | the file and its sidecar                         |

Library bytes deleted: 124,668,603,346 (124.67 GB); cache bytes deleted:
15,691,116,377 (15.69 GB, 6.97 GB of retired weights and the 8.72 GB partial).
RMBG-2.0 (A.02.08) was deleted earlier the same day (above).

**Before and after:**

| Measure               | Morning of 2026-09-14                   | After the retirements                          |
| --------------------- | --------------------------------------- | ---------------------------------------------- |
| Manifest rows         | 111 (825.06 GB), 31 families, 11 pinned | 98 (700.39 GB), 27 families, 9 pinned          |
| Pinned closure        | 246.00 GB                               | 239.03 GB                                      |
| Hetzner library       | 117 objects, 813.38 GB                  | 100 objects, 687.82 GB                         |
| Cache volume (400 GB) | varied with the proof syncs             | 136 objects, 331.22 GB (pinned core + LTX-2.5) |

Lint: `runpod-volume-manifest-lint.mjs --strict` 0 errors, 0 warnings, and the
cache invariant holds (239.03 + 92.19 = 331.22 GB of 380 GB allowed). What stays
in the library although no catalog workflow requires it: the Animate-2 base
transformer and the Gemma 4 E4B prompt-refine encoder (user decisions recorded
under A.01.06 and A.02.03), and the YOLOv10m and ViTPose pose models only
Animate-1 loaded (1.30 GB, not asked about).

## C.12.01 — four Chroma LoRAs, rendered (2026-09-13)

User picks, all CivitAI with every commercial flag: Lenovo UltraReal (1662740 /
2299345 / file 2190290, trigger `l3n0v0`), Chroma Realistic Slider (1995853 /
2259125 / 2151331), Chroma Defluxer (2204954 / 2482626 / 2371171), Style > Muk
(1944975 / 2715471 / 2601052). Downloaded with `CIVITAI_API_KEY`, sha256 checked
against CivitAI's published hash, uploaded to `loras/` over the volume S3 API.

| LoRA                    | Job                                                | Exec    | USD     | Output                                    |
| ----------------------- | -------------------------------------------------- | ------- | ------- | ----------------------------------------- |
| lenovo-ultrareal        | `74b3a8e1-2477-454d-99ab-79e3b3ab7f07-e1` (re-run) | 211.3 s | $0.0927 | `chroma-txt2img-0-8d63d12f.png` 1024×1024 |
| chroma-realistic-slider | `8dadc8a6-562c-4bde-935d-815eedcab3f0-e2`          | 139.6 s | $0.0613 | `…-9ed5aeb4.png` 1024×1024                |
| chroma-defluxer         | `e7674db2-863d-488b-ada9-4d6bffa78987-e1`          | 28.4 s  | $0.0125 | `…-4f2d611f.png` 1024×1024                |
| style-muk               | `ed8c8307-318f-4982-9bf0-3c56daaff364-e1`          | 29.9 s  | $0.0131 | `…-fd6106f6.png` 1024×1024                |

Same prompt, `chroma1-hd`, seed 1213, strength 1.0, worker `1u0p43eprdxnmz`.
First Lenovo attempt `c4e059d5-…-e2` FAILED in validation (`value_not_in_list`
for `lora_name`, listing the LoRA files present) because its upload had not
landed — the multipart completion had failed
(`InvalidPart: part 1 ETag mismatch`) and a single PUT then succeeded. Pairwise
mean |Δ| 26.5–44.1 levels, 40–64 % of pixels differing by more than 32. Limits:
single-LoRA enum, no stacking; Muk's illustration style not visible on a
photographic prompt at 1.0.

## LTX weights via a reduced-manifest loader pod (2026-09-13)

Three rows (distilled LoRA 1.1, union-control, motion-track) pulled by
`pod create-loader --manifest <reduced>`; `.oshun/manifest-result.json` →
`finishedAt 09:47:01Z`, `failures 0`, each row `status ok` with the manifest
sha256 (`f5d4953f…`, `a1b888a8…`, `e279807e…`) and exact byte counts (7 605 507
256, 654 465 352, 327 309 314). Pod `8tsjt6bhvoe3e9` deleted; spend back to
$0.049/h. Note: that result file now describes only these three rows. The local
HTTP downloads of the same files kept dropping
(`HTTP/2 stream was not closed cleanly`); the pod pulled 8.6 GB in about two
minutes.

## Worker uploader: the start-up hang, found and fixed (2026-09-13)

Build `fd960d7c` hit RunPod's 30-minute build limit inside the smoke step with
nothing in the log. Reproduced locally (python 3.12, runpod 1.7.13, the module
installed through a `.pth` hook): with `BUCKET_ENDPOINT_URL` set the interpreter
never finished starting; `faulthandler` showed
`runpod/serverless/utils/rp_debugger.py:20` → `cpuinfo.get_cpu_info()` →
`subprocess.communicate`, i.e. the SDK import starts a child python which
processed the same hook and recursed. The hook now registers a meta-path finder
that patches `rp_upload.upload_image` when the handler imports it; start-up 1.1
s with and without the env. A real round trip through the patched call against
Hetzner (`app-dev-server`, `isis/outputs/_selftest-job/20cbb23d.txt`, 46 B)
returned identical bytes through the presigned URL and was deleted (GET then
404). The smoke step is now tee'd to the build log and capped at 25 minutes. An
earlier attempt with `sitecustomize.py` failed the install check because
Ubuntu's python3.12 ships its own `sitecustomize` ahead of site-packages (build
`ea83e467`).

## L.00 Model library and cache — measurement

Option B (user decision 2026-09-13). Every pod run records wall clock, pod id,
cost and the loader's result file.

### Hetzner multipart and the library modes, live from the Mac (2026-09-13, $0)

Loader modes against `app-dev-server` under the throwaway prefix
`isis/models/_livetest/`, parts forced small (16 MiB, single-PUT ceiling 20 MiB)
so three of four files took the multipart path: `library-seed` 4/4 ok (Lenovo
112 125 552 B in 7 parts, Slider 28 074 280 B in 2, Muk 74 820 484 B in 5,
Defluxer 6 816 088 B single PUT); a second `library-seed` skipped all four (size
and `x-amz-meta-sha256` both matched) with no PUT or POST; `library-verify`
re-hashed every byte from the bucket, 4/4 ok; `cache-sync` into an empty
directory pulled all four through `.partial`, verified and renamed them, 0
failures. All `_livetest/` objects deleted (listing of `isis/` then empty).
Rates of 1–3 MB/s are this home connection and say nothing about the pod path.

### L.00.01 transfer probe

Pod `b1k0jr7fdbvror` (CPU `cpu3c`, 2 vCPU, EU-RO-1, `ax0oj250j3` mounted,
$0.06/h), 13:48Z to 14:42:02Z, wall 3237.3 s, about $0.054, deleted after the
read. Result file `.oshun/library-probe-result.json`, 0 failures, row
`chroma1-hd` (17 800 038 288 B, 266 parts of 64 MiB):

| Leg                 | Seconds | MiB/s | Check               |
| ------------------- | ------- | ----- | ------------------- |
| Up, single stream   | 709.0   | 23.9  | multipart completed |
| Up, 8 parts         | 449.0   | 37.8  | multipart completed |
| Down, single stream | 1259.1  | 13.5  | sha256 matches      |
| Down, 8 parts       | 816.5   | 20.8  | sha256 matches      |

Full-object checksum: a deliberately wrong `x-amz-checksum-sha256` and the right
one were both **accepted**, so Hetzner does not validate full-object sha256 and
L.01.05's download-side verification stays in full scope. No probe objects left
behind.

## L.01 The library on Hetzner

### L.01.03 seed, four launches (2026-09-13)

1. Pod `2nz4kgp6emde9p`, `--parallel 16`, 14:43:26Z. The first row's multipart
   reached 236 of 266 parts by 14:48:04Z at 60.6 MB/s, then nothing for 16
   minutes: parts 233 and 236 never landed while 237 and 238 did. The loader's
   socket timeout was 600 s with 4 retries, so one silent connection could hold
   the ordered part queue for up to 50 minutes. Container telemetry read 0 %
   CPU. Deleted 15:05Z (HTTP 204); its open upload aborted (204,
   `ListMultipartUploads` then empty).
2. Pod `xws2eg6x6tmkw2`, `--parallel 8`, stall timeout 120 s, 15:06:40Z. Stalls
   recurred about one part in fifteen (part 44, then 59), each costing the full
   120 s: 15.8 MB/s average. Deleted 15:12Z; upload aborted.
3. Pod `75ml7zmu2t2q1z`, `--parallel 8`, stall timeout 30 s (commit
   `a94f2832105`), 15:12:20Z. Ran 35 to 96 MB/s until 16:05:02Z, then stopped
   reading `wan2.2_animate_14B_bf16.safetensors` (34.55 GB) from the volume
   after 504 of 515 parts: 40 minutes at 0 % CPU and 1 % memory, no parts
   pending, no error. Deleted 16:46Z; upload aborted.
4. Pod `wkd9odg71h3tls`, same settings, 16:46:25Z to 17:27:45Z, wall 2478 s.
   Result file `.oshun/library-seed-result.json`: 50 rows ok, 0 failures.
   Library listing: 50 objects, 367.17 GB, every size and every
   `x-amz-meta-sha256` matching the manifest. Deleted 17:28Z.

Pod time across the four launches was about 2.7 hours, about 0.16 USD. The
connection stalls are on Hetzner's side of individual connections (other parts
on other connections kept landing, and a fresh connection always succeeded). The
volume-read hang led to the loader watchdog (`111a49e7518`).

## L.02 Families and the pinned core

Manifest `families` (15, 9 pinned) and `library`/`cache` blocks committed in
`28083a4c432`; lint prints
`cache 400 GB: pinned 212.26 GB + largest unpinned ltx23 80.37 GB = 292.64 GB of 380 GB allowed`.
`requires_families` on all 17 workflows and `MODEL_NOT_IN_FAMILIES` in the
static validator committed in `02be697dd61` (sweep over defaults, every enum
value and both booleans; library suite 694 passed).

## L.03 Cache state, leases and eviction

Service `apps/isis/generation-api/src/services/runpod-model-cache/` committed in
`9b31b96481a`: 35 specs, the store and race specs against real Postgres (forced
lease-versus-eviction race in both orders, and 25 concurrent rounds with exactly
one winner each).

## L.04 Sync executor

Service and routes committed in `0fe7d06ba70` (L.04.02, L.04.03). Every loader
pod of 2026-09-13, CPU `cpu3c` in EU-RO-1 at 0.06 USD per hour (cost is pod time
at that rate, or the API's booked `cache_sync` line where it drove the pod).
Result files are copied to `docs/agents/evidence/isis-chroma-runpod/l-results/`.

| Pod              | Mode, run                             | Window (UTC)   | Wall                        | Cost USD      | Result file, outcome                                              |
| ---------------- | ------------------------------------- | -------------- | --------------------------- | ------------- | ----------------------------------------------------------------- |
| `b1k0jr7fdbvror` | library-probe (L.00.01)               | 13:48 to 14:42 | 3237 s                      | 0.054         | `library-probe-result.json`, 0 failures                           |
| `2nz4kgp6emde9p` | library-seed, parallel 16             | 14:43 to 15:05 | hung                        | 0.022         | none; deleted, upload aborted                                     |
| `xws2eg6x6tmkw2` | library-seed, 120 s stall             | 15:06 to 15:12 | stalls                      | 0.006         | none; deleted, upload aborted                                     |
| `75ml7zmu2t2q1z` | library-seed, 30 s stall              | 15:12 to 16:46 | volume read hung            | 0.094         | none; deleted, upload aborted                                     |
| `wkd9odg71h3tls` | library-seed                          | 16:46 to 17:28 | 2478 s                      | 0.042         | `library-seed-result.json`, 50 ok                                 |
| `ktbpl2bdda9vgf` | library-verify, 16 non-core rows      | 17:31 to 18:08 | 2245 s                      | 0.037         | `library-verify-result.json`, 14 ok, 2 timeouts                   |
| `9t25i5c9aoe0j6` | library-verify retry, 2 rows          | 18:09 to 18:41 | 1080 s run                  | 0.032         | `library-verify-retry-result.json`, Gemma ok, VACE timeout        |
| `uzcxhv22fkl5na` | cache-sync `6e9cc338`, pinned core    | 17:37 to 18:40 | 3631 s                      | 0.063 booked  | `cache-sync-core-result.json`, 31 of 34; reconcile hit a 524      |
| `1q4axik9vvhuv6` | cache-sync `7620b385`, core remainder | 18:46 to 19:01 | 739 s                       | 0.015 booked  | `cache-sync-core-followup-result.json`, 6 of 6                    |
| `gr5anr0ft8vc5r` | cache-sync `eb61d6df`, VACE miss      | 19:10 to 19:35 | 1369 s                      | 0.024 booked  | `cache-sync-vace-result.json`, 4 of 4                             |
| `tcjz2pz3tav1je` | cache-sync `015b7753`, rmbg-2.0       | 20:03 to 20:04 | about 25 s                  | 0.0004 booked | succeeded; file overwritten by the next sync before it was copied |
| `904j331z943wbd` | cache-sync `7b013797`, eviction proof | 20:05 to 20:46 | eviction done, pull stalled | 0.046 booked  | none; pod deleted, sync recorded failed by takeover               |
| `z3uhi38hc9hx76` | cache-sync `4d053b49`, LTX-2.3 resume | 20:52 to 22:55 | Gemma pull froze at 22:13   | 0.123 booked  | none; 3 of 7 files landed, pod deleted, sync failed by takeover   |

### L.04 defect found in use: a live sync taken over (2026-09-14)

Cache sync `b80de934` (the Qwen-Image edit families, 106.33 GB) spent minutes in
its `applying` phase listing a 361 GB volume. The driver refreshed its heartbeat
only in its result-polling loop, so the heartbeat went stale; the cache status
route calls `resumeStaleSync()`, and the next status read at 16:00:27Z claimed
the sync and launched a second driver in the same process. The second driver
tried to complete an eviction the first had finished ("cannot complete eviction
of wan22-animate2 ... it is not evicting") and recorded the sync failed with
0.112043 USD booked; the first then finished it as succeeded and booked 0.113316
USD, so the pod's cost was booked twice. The families were cached correctly.

Fix: a running driver refreshes its heartbeat on a timer in every phase
(`touchSyncHeartbeat`), and `resumeStaleSync()` never takes over a sync this
process is still driving. Specs: "never takes over a sync its own process is
still driving, so it is finalized once" and "keeps a running driver
heart-beating in every phase", both shown to fail with the fix removed. The sync
spec had also been failing unnoticed since L.04 moved the plan to `PLAN_GZ_B64`
(its fake pod still decoded `PLAN_B64`, and the spec only runs with
`ISIS_TEST_DATABASE_URL` set); the harness now decodes the compressed plan.
`ISIS_TEST_DATABASE_URL=postgresql://oshun:oshun_dev@127.0.0.1:5432/oshun_dev npx vitest run src/services/runpod-model-cache`
→ 60 passed.

## L.05 Dashboard

Cache panel committed in `52d7fd1fbb9`. Chromium
`apps/isis/web/e2e/runpod-model-cache-panel.spec.ts` 2/2 with the API mocked at
the network boundary (re-run by the reviewing session, not only the building
agent); view logic 7 tests, client 3, API/client/OpenAPI projection 11,
`RunPodPage.spec.tsx` 31. No listener left on 3010, 3020, 4010 or 5180.

## L.06 Migration of the volumes

### L.06.01 cache volume

Balance first (GraphQL `myself`, 15:26Z): `clientBalance` $17.94,
`currentSpendPerHr` $0.132, `spendLimit` $80. `volume create --dry-run` showed
`{name: isis-model-cache, size: 400, dataCenterId: EU-RO-1}`; the real call at
15:26:33Z returned id `5bst2pgw7y`. Storage is now $71.20 + $28.00 a month until
L.06.05.

### L.06.02 to L.06.05 (2026-09-13)

- **Core sync.** Sync `6e9cc338`, pod `uzcxhv22fkl5na`, 17:37 to 18:37:53Z: 31
  of 34 files, 3 lost to ranged reads that stalled five times on one range, then
  the API's reconcile failed on a 1000-key volume listing (Cloudflare 524).
  Follow-up sync `7620b385`, pod `1q4axik9vvhuv6`, wall 739 s: the 3 files plus
  the 3 BiRefNet companions, 0 failures. Booked cost 0.063 and 0.015 USD. The
  API then reads the 9 pinned families cached, 212.26 GB used, 0 unmanaged.
- **Repoint.** Console edit on both endpoints, 18:12Z; GraphQL and the drift
  checker read `networkVolumeId 5bst2pgw7y`.
- **Deletion.** `ax0oj250j3` deleted 18:13:21Z on the user's explicit go;
  `currentSpendPerHr` 0.222 before, 0.162 after with the same two CPU pods, and
  0.039 once the pods ended (the 400 GB volume alone).
- **Verification (L.01.05).** First pass over the 16 non-core rows: 14 ok, VACE
  and Gemma 3 hit read timeouts while three transfers shared Hetzner. Retry pod
  `9t25i5c9aoe0j6`: Gemma 3 ok (24.38 GB at 25.8 MiB/s), VACE timed out again;
  VACE is verified by its cache sync below.

### L.06.04 proofs on the cache volume

| Proof                              | Job                                        | Result                                                                                                            |
| ---------------------------------- | ------------------------------------------ | ----------------------------------------------------------------------------------------------------------------- |
| Model-free probe                   | RunPod `224dcfc6`, worker `uv0s0976i8isy9` | delay 121.1 s, execution 8.9 s, output `s3_url` on Hetzner                                                        |
| Chroma still (first)               | Isis `f792b235`                            | rendered, but **unwatermarked**: the still came back as a signed URL and three mappings dropped `requiresSigning` |
| Chroma still (after `6e3691ad7b9`) | Isis `9f0b2633`                            | 1024x1024, execution 177.1 s, 0.098 USD, `watermark.applied`, payload decoded with 512 of 512 copies intact       |
| Wan 2.2 I2V, 4-step, 33 frames     | Isis `b964e556`                            | 832x480, h264, 16 fps, 33 frames, execution 323.5 s, 0.450 USD; last-frame still watermarked                      |

Frames of the clip and the watermarked still are in
`docs/agents/evidence/isis-chroma-runpod/l0604/`. The VACE miss, sync and render
cycle, and the forced eviction, follow.

**VACE miss.** Submitting `wan21-vace-edit` at 19:10:25Z answered HTTP 409
`models_not_cached`: families `controlnet-aux` (1.69 GB) and `wan21-vace` (46.30
GB, of which 36.37 GB to pull because umt5 is shared with the cached
`wan-shared`), an estimated wait, and the exact sync body
`{"workflows": ["wan21-vace-edit"]}`. That sync (`eb61d6df`, pod
`gr5anr0ft8vc5r`) started at 19:10:37Z.

**VACE sync and render.** Sync `eb61d6df` finished 19:35:03Z, state `succeeded`,
0.024 USD, wall 1368.7 s: `wan2.1_vace_14B_fp16.safetensors` (34.68 GB at 24.9
MiB/s) and the three controlnet_aux files, all hashed ok. The resubmit first
failed with HTTP 500: the route accepted `type: video-to-video`, but the live
`jobs_type_check` constraint lists 18 types while the code's
`ISIS_GENERATION_TYPES` has more, so the insert was refused. That is a
pre-existing defect outside this section (recorded, not fixed here). Submitted
as `text-to-video` (what the dashboard's motion presets send), job `7bcfc7ea`
ran 19:35:57 to 20:02:42Z: 832x480, h264, 16 fps, 33 frames, execution 344.8 s,
0.480 USD, queue delay 1254.7 s waiting for a card. Frames 0, 16 and 32 follow
the source clip's boat and grate with the restyled palette
(`l0604/wan21-vace-frames-0-16-32.png`). **Miss to render: 52 min**, of which
24.4 sync, about 21 GPU queue and 5.7 render.

**Forced eviction, set-up.** The whole library (367 GB) fits the real cache's
380 GB usable, so no real request can force an eviction. The API was restarted
with `ISIS_RUNPOD_VOLUME_MANIFEST_FILE` pointing at a proof-only copy of the
manifest with `cache.sizeGb` 320 (usable 304 GB; the lint invariant still holds,
pinned 212.26 plus ltx23 80.37 = 292.64). Before, with the real manifest: used
249.52 GB; unpinned cached families `wan21-vace` (last used 20:02:42Z, the VACE
render), `controlnet-aux` (same) and `rmbg-2.0` (synced 20:04:09Z by a 30 s,
0.0004 USD sync done for this proof). Dry run of `{"workflows": ["ltx23-av"]}`
(needs `ltx23` and `controlnet-aux`): evict `wan21-vace`, 34.68 GB in one file
(umt5 stays, `wan-shared` is cached and pinned), pull `ltx23` 80.37 GB. Check:
249.52 - 34.68 + 80.37 = 295.21 GB fits 304; without the eviction 329.89 does
not. `rmbg-2.0`, touched later, is kept. The real sync `7b013797` (pod
`904j331z943wbd`) started 20:05:13Z.

**Forced eviction, result.** The loader evicts before it pulls, and the pull was
slow (Hetzner bursts and stalls; a watchdog rerun would also have restarted the
46 GB checkpoint from zero, fixed in `e0c6d74442e`), so after the eviction had
landed the pod was deleted at 20:46Z and the API restarted with a 60 s sync
timeout: it took the sync over at 20:51:15Z and recorded it `failed` ("no result
file 60 s after the pod was created"), 0.046 USD booked. Restarted on the real
400 GB manifest, the API reads: `wan21-vace` absent; `controlnet-aux` and
`rmbg-2.0` cached; used 228.80 GB of which 13.96 GB is the checkpoint's
`.partial`, so managed bytes 214.84 GB = 249.52 - 34.68, exactly what the plan
said it would free. A fresh `ltx23-av` sync (`4d053b49`, pod `z3uhi38hc9hx76`,
20:52:17Z, no evictions at 400 GB) resumes the partial.

That sync landed the 46.15 GB checkpoint and both latent upscalers by 21:55Z
(the resume worked: the checkpoint finished from its partial), then stopped on
the Gemma 3 encoder. Its `.partial` sat at 8,724,152,320 bytes, exactly 130
whole 64 MiB parts, from 22:13:52Z: the shape of a watchdog rerun cutting the
file back to whole parts and then fetching nothing. Forty minutes later (four
watchdog periods, and past the start loop's five attempts) it had not moved, so
at 22:54Z the API was restarted with a 60 s sync timeout, took the sync over,
recorded it failed ("no result file 60 s after the pod was created", 0.123 USD
booked) and deleted the pod; the API then restarted on its normal settings. The
likely cause is contention: the arsenal library pull (`ed3tappadomybi`, from
21:51Z) was uploading to the same Hetzner bucket at about 26 MB/s the whole
time, and Hetzner stalls connections under contention (L.00). The LTX-2.3 sync
is to be relaunched after that pull finishes; the partial and the three landed
files stay on the volume for it to resume from.

## L.07 Operation

### L.07.02 outputs retention and the usage alarm

Commit `dfbdea14099`. Live, 2026-09-13:

- 15:02Z, CLI from the Mac: `hetzner-usage` listed one bucket (`app-dev-server`,
  0 objects); `hetzner-sweep --days 7` dry run, 0 outputs.
- 15:20:55Z, the API guard's own runtime loader (`loadHetznerStorageOps`) with
  defaults: dry-run sweep of `isis/outputs/` (0 deleted, 0 kept); usage 17 800
  038 288 B in 1 object, all under `isis/models/`, not over 900 GB.
- Same live reading with the line set to 1 byte: the first check raised
  `storage.hetzner.usage_high` (payload date 2026-09-13, usedRatio as measured),
  the second check the same day logged the error line but sent no second
  webhook.

Found on the way: budget-alarm webhooks (C.04.06) read `url` from store records
that call it `endpointUrl`; fixed for both alarm paths with a regression spec.

## H.00 The hosted OpenRouter lane — policy and terms (2026-09-15)

The lane exists because the owner asked for the Meshy treatment to be extended:
"openrouter has a number of image and video models can we do the same as we are
doing with meshy … and only use runpod stuff when we want / need to?".

**The gate that had to be opened first.** The Eve SOTA provider/subprocessor
review (`docs/audits/eve-sota-provider-subprocessor-review/`) marks every hosted
media route `blocked-unattested`, OpenRouter image and video among them. The
owner chose "Attest Meshy + OpenRouter media" on 2026-09-15; the record is
`owner-attestations/2026-09-15-isis-hosted-media.json` (operator only,
operator-owned non-personal SFW inputs, vendor posture accepted as published).
Regenerate and verify with
`node tools/eve-everywhere/generate-provider-subprocessor-review.mjs` and its
`verify-` twin after touching any bound source file; the `node --test` suite
carries 58 semantic mutations.

**What is offered, and what is not.**
`libs/isis/workflows/src/hosted-media/openrouter-model-register.ts` holds one
row per model with the vendor terms that were read and the date they were read.

- Offered: the FLUX.2 family — klein 4B, flex, pro — for `txt2img` and
  `img-edit`, under Black Forest Labs' terms of 2026-08-01.
- Excluded, each with the reason on the row: OpenAI (terms answered 403),
  Alibaba Wan (Model Studio terms answered 404, and Article 4.48 is not served),
  Krea 2 and MiniMax H3 (already excluded for their weights), Google's image and
  video models (terms read, but the owner's proof approval excludes frontier
  vendors, so offering them needs a separate owner decision).
- **No video model is offered at all.** `OPENROUTER_VIDEO_BLOCKER` states it: no
  non-frontier vendor's video terms could be read. A video task is refused
  naming that blocker rather than half-served.

**Tenant exposure (H.00.02).** Asked and answered the same day: "Operator only
for now". `OPENROUTER_LANE_TENANT_EXPOSURE` carries the decision, its date and
its source, and a non-operator sees an empty catalog and is refused at submit.

### H.00.01 closed 2026-09-16 — the terms are fingerprinted and the two verdicts are fields

The register was built on 2026-09-15 and left open for three things. The first
(the catalog offering more than the register admits) was gone by 2026-09-16. The
other two are done here.

**Territory and training are fields now, not prose.** `OpenRouterVendorTerms`
gained `unitedKingdom` and `trainsOnInputs`, each a verdict plus the clause it
rests on, and `openRouterRegisterProblems()` refuses to offer a model whose UK
verdict is anything but `served` or `not_excluded`. The verdicts, read from the
documents:

- **Black Forest Labs** — UK `not_excluded` (12.1 Export Laws is the only
  territory clause: embargoed countries and restricted-parties lists, no
  served-territory list). Training `opt_out_held_by_openrouter`: 1.2(e) licenses
  Input, Output and Tasks to "train and improve our technologies", opt-out by
  writing to legal@blackforestlabs.ai — and that opt-out belongs to Black Forest
  Labs' customer, which is OpenRouter.
- **Google** — UK `conditional`, which is **not offerable**: "You may use only
  Paid Services when making API Clients available to users in the European
  Economic Area, Switzerland, or the United Kingdom", and Isis cannot see which
  tier OpenRouter buys. Training `depends_on_tier` for the same reason. This is
  now a second, independent reason the Google rows are excluded — until today
  the exclusion rested only on the owner's frontier-vendor decision.

**Two clauses the 2026-09-15 reading had missed**, found by re-reading the BFL
document in full and added to its `findings`:

- **1.3(n) forbids using Output "to train, distill or fine-tune any other AI
  models".** A hosted still may not become training data — which touches E.01.07
  (the body-physics LoRA) directly.
- **1.3(p) forbids removing or obscuring "any AI content marking or labelling or
  transparency metadata".** The lane's watermark and C2PA stages are a vendor
  requirement, not only Isis policy.

**The fingerprint, and why it is over text and not bytes.** Measured 2026-09-16
by fetching each document twice within a minute:

| document                         | bytes   | raw sha256 across two fetches                                       | text sha256                   |
| -------------------------------- | ------- | ------------------------------------------------------------------- | ----------------------------- |
| `bfl.ai/legal/terms-of-service`  | 321 843 | identical                                                           | `e98d98402c3ad246…`           |
| `ai.google.dev/gemini-api/terms` | 111 409 | **different** (3 149 bytes, all inside per-request CSP `nonce="…"`) | `7395f597b16af2ed…` identical |

So a raw-body hash would report drift on a document nobody touched. The drift
signal is `textSha256` under the named normalization `isis-terms-text-v1`
(`terms-document.ts`: drop `<script>`, `<style>`, `<noscript>`, comments and
tags, decode the entities a legal page uses, collapse whitespace; refuse an
extraction under 2 000 characters rather than hash a consent wall). `rawSha256`
is still recorded with a measured `rawStable` flag — it pins the exact bytes
that were read without pretending they are comparable.

**The check.** `tsx libs/isis/workflows/scripts/openrouter-terms-drift.ts`:

- `--check` (offline, and now a `dependsOn` of the lib's `lint` target via
  `hosted-terms:check`) runs the register's rules and re-hashes the checked-in
  normalized copies under `docs/agents/evidence/isis-chroma-runpod/h0001/`,
  proving the recorded hash is the hash of the recorded reading. Verified to
  fail: tampering one clause in the BFL evidence copy gave
  `evidence copy hashes 788c189206e180d4, row records e98d98402c3ad246 — one of them is stale`
  and exit 1; restoring it gave exit 0.
- `--check-remote` re-fetches and compares. Run live 2026-09-16:
  `Black Forest Labs: unchanged since 2026-09-15` and
  `Google (Gemini API terms): cosmetic only — body 111409 -> 111409 bytes, sha256 8c70ed744366c53f -> 4f82168dfe6225ef, same text`
  — a third distinct raw hash, same prose, correctly not an alarm.
- `--write` prints the digest literals and **refuses to edit the register**: a
  new hash without a fresh reading of the clauses would be a machine silently
  re-approving terms a person never read.

What it cannot do, written down because the approximation is real: it cannot
tell a reworded clause from a reformatted page, and a vendor that renders its
terms client-side would extract near-nothing. Both end in the same place — a
person re-reads the document.

Specs: 12 in `terms-document.spec.ts` (normalization, entity ordering,
`rawStable` measured rather than assumed, and the unchanged/cosmetic/changed/
incomparable verdicts), 14 new in `openrouter-model-register.spec.ts` (11 → 25),
seven of them negative cases: a `conditional` UK verdict refused, an `excluded`
one refused, a verdict with no clause behind it, a bad digest, a document too
short to be an agreement, an evidence path outside the tree, and two rows
quoting one URL with different hashes. 59 pass in `src/hosted-media/`.

## H.01 One OpenRouter media client (2026-09-15, moved 2026-09-16)

`libs/isis/ai-providers/src/providers/openrouter-media/` is the only place in
the repository that knows OpenRouter's media wire format: image create (buffered
and SSE), video create / get / wait / content download, both catalogues with
per-endpoint pricing, one typed error per documented failure, and the webhook
signature check (raw body, five-minute replay window in both directions,
constant-time compare, `@noble/hashes` because `node:crypto` is banned in
client-transpiled libs). The key is injected through `@oshun/config`; a grep for
`process.env` over the production files is empty. 38 client specs, 47 support
specs.

`openrouter-media-surface.snapshot.json` (fetched 2026-09-15) is the checked-in
catalogue. `scripts/openrouter-media-surface.ts --check` compares the client and
the register against it offline; `--check-remote` re-fetches and names added,
removed and repriced models. The negative cases are real: a named model or
parameter missing from the snapshot fails, a parameter the client and the
catalogue describe differently fails in both directions, and a route the client
calls that the snapshot does not list fails.

**The second copy is gone (H.01.03).** Oshun's BFF carried its own OpenRouter
video provider — its own poll loop, its own resolution and aspect-ratio tables,
its own HTTP error strings. On 2026-09-16 it moved onto this client and became
an adapter (364 → 271 lines). The move was possible because the boundary rule
already lets `scope:oshun` depend on `scope:isis` and the BFF already declared
`@isis/ai-providers`. The route gains three properties it did not have: the
bearer key reaches a content URL only when that URL is on the API host, failures
arrive typed, and a transient poll failure no longer abandons a job that is
running and billing. Its admission is unchanged — it stays behind
`isEveExtendedProviderAdmitted('video-openrouter')`, and sharing a transport
grants it nothing. Specs: 16 provider, 3 hero-gate, 12 executor, 20
provider-env, 85 openrouter-media; `tsc --noEmit` clean on both projects.

Using the client here taught it two things, both now in the shared code: an
optional poll backoff (`pollBackoffFactor` / `maxPollIntervalMs`, defaulting to
the documented fixed cadence) so a five-second clip is seen quickly without
polling a long job hard for its whole deadline, and `describeJobError`, so a
failed job's reason reaches the operator instead of `[object Object]`.

## H.06.01 — the OpenRouter lane, live (2026-09-16)

On the owner's approval of ~0.6 USD (asked and answered 2026-09-15: "Approve
both (~1.1 USD)", cheap models only) and under the 2026-09-15 hosted-media
attestation. Every job went through `POST /api/v1/jobs` on the local generation
API (port 4111, inline runner) as the operator tenant `tenant_isis_dev` with a
minted dev JWT, the Valkey ledger and the local outputs store. The daily budget
was pinned to `ISIS_OPENROUTER_DAILY_BUDGET_USD=0.6` so the approval was
enforced by the API, not only by intent.

**Spend: 0.043 USD of the 0.6 approved**, read live from the submit headers
after the runs: `x-openrouter-daily-spend-usd: 0.043`,
`x-openrouter-daily-remaining-usd: 0.557`, key
`isis:openrouter:cost:2026-09-16`.

| Group    | Job        | Model           | Actual | Estimate | Wall s | Output                                                                                                     |
| -------- | ---------- | --------------- | -----: | -------: | -----: | ---------------------------------------------------------------------------------------------------------- |
| txt2img  | `05bfced8` | flux.2-klein-4b |  0.014 |  0.01468 |    6.9 | PNG 1024×1024, 1 485 838 B, sha256 `3d6bdf6cd20694dd…`, bundle `prov_6533ce5e6b9314b2`                     |
| img-edit | `1d8db4d0` | flux.2-klein-4b |  0.015 |  0.01468 |   16.0 | PNG 1024×1024, 1 411 737 B, sha256 `22ff1732433075a6…`, bundle `prov_c83ff3282c7414ff`, one material input |

The edit was given the first image as its only reference (operator-owned, the
lane's own earlier output) and honoured it: same table, framing, window light,
notebook and lamp base; the shade is deep green enamel and a potted fern stands
behind the notebook, which is what the prompt asked for and nothing else.

**The two video groups the item asks for could not run and were not faked.** The
register offers no video model (H.00.01), and the lane says so live:
`POST /api/v1/jobs` with `openrouter:t2v:alibaba/wan-3.0` answered **422
`hosted_lane_video_unavailable`**, naming the blocker — Alibaba's terms 404,
ByteDance and Kuaishou unread, BFL video at 0.17 USD/s, Google excluded by the
owner's frontier rule.

Also proved live at no cost: a `racy` content rating on a hosted request is
refused 422 before anything is created, and the budget headers ride on the
refusal.

### Two defects the live run found, both fixed

1. **The first job was created, billed and then failed to complete.** Job
   `d5e98732` spent 0.014 USD, stored its PNG, and then
   `persistGenerationProvenance` refused the bundle: "Cannot complete job …
   without provenance modelVersion". The lane had set `modelVersion` to the
   model's `canonicalSlug`, which OpenRouter publishes only for **video**
   models; an image row carries no version at all. The lane now uses the model
   id — `flux.2-klein-4b` names the family and the size — and records
   `modelVersionSource` beside it so nobody reads it as a vendor-published
   version. That 0.014 USD bought a stored image and a red job; it is counted in
   the 0.043 above.
2. **The lane's own generation id was being overwritten.** The provenance bundle
   writes its own `generationId`, which is the JOB id, over whatever the lane
   put there — so the minted OpenRouter id survived only in the dispatch record,
   while `generationIdSource` still claimed to describe it. It is now recorded
   as `laneGenerationId` / `laneGenerationIdSource`, which the bundle does not
   touch: `img-1d8db4d0-…-0` is on the second job's output.

One smaller thing worth writing down: **OpenRouter answered `created: 0`** on
both image calls. The minted id's uniqueness never rested on that field — one
job creates at most one image, so the job id carries it — but the comment that
described it as "the answer's created second" was corrected to say so.

## H.02–H.05 — the lane through the API and the Studio (2026-09-16)

**The API.** A job naming `openrouter:<task>:<model>` runs on the hosted lane
with no GPU lease. The policy is judged at submit, on retry (which re-submits),
at every chain step and again in the runner before any HTTP call; the daily
budget is checked at submit and again before the runner creates. The catalog
carries `openRouterLanes` and `/workflows/hosted-lanes/openrouter` serves the
listing, both empty for a non-operator.

Refusals that cost nothing and say why: `hosted_lane_operator_only` (403),
`hosted_lane_video_unavailable` (422, naming the blocker),
`hosted_lane_runpod_only` (422, carrying `runpodOnlyReasons()` and the
self-hosted workflows that serve the task), `hosted_lane_parameter_unsupported`
(the catalog snapshot is the allowlist),
`hosted_lane_provenance_consent_required`, and
`OPENROUTER_DAILY_BUDGET_EXCEEDED` (429, or 503 when the ledger is unreachable).

**The Studio.** `HostedLanePanel` offers only the lanes the catalog served this
caller, builds its form from the snapshot's parameter descriptors, shows the
estimate from the API's own estimate route (never computed in the browser) and
the measured cost afterwards. 3 chromium Playwright specs cover the picker, a
refusal and the cost line.

**One rule worth stating plainly.** Every offered model's register row says the
vendor's acceptable-use policy binds its outputs, so a hosted output may not
feed an `adult_ok` workflow — and `wan22-i2v` is declared `adult_ok`. The
sfw_only motion workflow `ltx25-av` is the one that may take a hosted still.

### H.04.01 — a RunPod output now carries its own rating (2026-09-16, later)

The chain rule shipped earlier today depended on the **caller** telling it the
rating of material the caller did not make
(`inputs.hostedLane.inputContentRatings`). That is the half this closes.

**The judgement.** `judgeSubmitContentRating` sits in `@isis/workflows` beside
`evaluateContentPolicy`, because it answers the same question from the same
declaration. It gives three answers and `unknown` is a real one:

| the request                                                     | the judgement                       | why                                                                                                                                   |
| --------------------------------------------------------------- | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| `sfw_only` workflow, nothing non-neutral selected               | `neutral`                           | the submit gate refuses a racy or explicit option or preset on such a workflow, so reaching the recording IS the evidence             |
| anything selecting a weight or preset rated `racy` / `explicit` | that rating, most restrictive first | the declaration says so, and it names what carried it                                                                                 |
| `adult_ok` workflow that nothing rated                          | `unknown`                           | the prompt is unconstrained and no declaration rated it; recording `neutral` here is the exact guess the chain rule exists to prevent |

What it is a judgement **about** is written into the function's own doc comment:
the declared inputs of one request, not the pixels that come back. Nothing here
looks at an image. That is the same standard the hosted lane already holds its
own outputs to, and the vendor applies its own moderation to what it receives.

**Where it is recorded and read.** `judgeSubmitContent` runs in
`JobService.submit` immediately after `assertSubmitContentPolicy` — after the
gate on purpose, since the gate is what makes `neutral` an observation — and
stamps `metadata.contentJudgement`. The advancer reads it off the producing job
and it **outranks** the caller's declaration: a declaration is a claim about
someone else's material, a judgement is what that job's own gate concluded, so a
caller cannot declare its way past a rating Isis recorded. A rating this build
does not know reads back as no rating at all.

**What this does and does not unblock.** An `ltx25-av` or `klein4b` output now
crosses to a hosted step on its own record, with nothing declared. A
`chroma-txt2img` still is **still** refused without a declaration — chroma is
`adult_ok` and nothing rated it — and that is the correct answer, not a
remaining gap.

Specs: 6 in `content-policy.spec.ts` (113 in that file), 6 in
`catalog-content-policy.spec.ts`, 5 new in `job-chain-advancer.spec.ts` (21
there). The five advancer cases are the ones that matter: a recorded `neutral`
advances undeclared; a recorded `explicit` refuses even when the chain declares
`neutral`; a recorded `unknown` refuses and the message names the producing job
and its reason; a rating from a later build is not read as permission; and a
hosted output still cannot feed `adult_ok` whatever the judgement says. One
grammar fix in passing: the refusal read "A explicit input cannot be sent…".

H.04.01 stays open on its one remaining blocker, which is external: RunPod →
hosted in the video direction cannot exist while no video vendor's terms can be
read (`OPENROUTER_VIDEO_BLOCKER`).

## H.06.02 — the mixed-lane chain, attempted live and blocked (2026-09-16)

The chain half works: `POST /jobs/chain` accepted a hosted `txt2img` step
followed by `ltx25-av`, the still was created and stored, the chain advanced,
and the RunPod step received it. Two blockers stop the second half, and both are
written down rather than worked around.

**1. The volume input channel does not resolve on the live workers.** The
worker's own start-up log shows
`oshun-entry: /comfyui/input/volume -> /runpod-volume/inputs`, both endpoints
report `networkVolumeId: 5bst2pgw7y`, and the RunPod S3 API writes objects to
that volume and reads them back. ComfyUI nevertheless answered an
invalid-image-file error for every shape tried:

| Reference                     | Object on the volume          | Endpoint | Answer             |
| ----------------------------- | ----------------------------- | -------- | ------------------ |
| `volume/jobs/<jobId>/0.png`   | `inputs/jobs/<jobId>/0.png`   | video    | Invalid image file |
| `volume/h0602probe/still.png` | `inputs/h0602probe/still.png` | image    | Invalid image file |
| `volume/h0602-flat.png`       | `inputs/h0602-flat.png`       | image    | Invalid image file |

The cause is not isolated. The next step is a shell on a worker to look at
`/runpod-volume/inputs` while an object is known to be there. Until then, no
chain step can read a previous step's bytes in place, which also puts V.05.03's
large-media path in doubt.

**2. Budget.** The cheapest `ltx25-av` shape the catalog accepts estimates
0.7259 USD — above the ~0.5 USD approved for RunPod this round — so the video
budget gate refused before dispatch. That is the gate working.

**Three defects the attempt found, all fixed:** `workerPathForOutput` was
minting `volume/<jobId>/<name>` for objects under `outputs/`, which no symlink
reaches; an output the store WRITES to the volume carried no volume key at all;
and one transient S3 answer lost an already-billed generation, so a paid-for
write is now retried three times before it is called lost. A fourth, mine from
the same evening: the new cross-lane chain rule read only `metadata.lane`, so a
Meshy output — which records `provider` — looked like unrated self-hosted
material and stopped every Meshy chain.

## E.01 Benchmark suite (2026-09-16)

**E.01.01 — the suite, and the spec that checks it against the catalog.**
`libs/isis/workflows/src/benchmarks/benchmark-suite.ts` holds 14 entries across
the eight lanes E.01.01 names, 12 of them runnable today. Prompts, seeds and
shapes are checked-in literals; nothing is generated at run time, because a
benchmark whose inputs drift is not a benchmark.

| lane                  | entries | workflows                                  | runnable |
| --------------------- | ------- | ------------------------------------------ | -------- |
| photoreal-portrait    | 2       | `chroma-txt2img`, `zimage-txt2img`         | 2        |
| nude-yoga             | 2       | `chroma-txt2img`, `zimage-txt2img`         | 2        |
| figure-study          | 1       | `chroma-txt2img`                           | 1        |
| dance-and-movement    | 3       | `wan22-t2v`, `wan22-i2v`, `scail2-animate` | 2        |
| sets-and-environments | 2       | `qwen-txt2img`, `chroma-txt2img`           | 2        |
| costume-and-wardrobe  | 1       | `qwen-edit`                                | 1        |
| anime-and-stylised    | 2       | `hidream-o1-txt2img` ×2                    | 1        |
| typography            | 1       | `qwen-txt2img`                             | 1        |

**What the spec caught.** It validates every entry's workflow id, every pinned
input name and every enum value against the real catalog JSON on disk. On the
first run it failed **12 of 93** assertions, all of them real: invented aspect
values (`portrait`, `landscape`, `square`) where the catalog names
`portrait_896x1152`, `landscape_1152x896`, `square_1024`; a Z-Image entry using
Chroma's aspect vocabulary; a Qwen typography entry pinned to a HiDream aspect
(`portrait_3x4_1728x2304`, which Qwen does not have); and `resolution: '480p'`
where Wan names `480p_landscape_832x480`. Every one of those is either a submit
refusal or a silently different shape — the failure mode the item exists to
prevent, found for free instead of on a billed GPU minute.

**Policy is in the data, not in the runner.** The three adult-lane entries
(nude-yoga ×2, figure-study ×1) are `apiReference: false` and name `adult_ok`
workflows, and the spec asserts both — so an adult prompt cannot reach E.01.04's
external SFW reference through an edit to the runner. Nine entries are cleared
for that reference, all `neutral`.

**References, and the two entries that cannot run.** `wan22-i2v` and `qwen-edit`
take their start frame from an earlier entry in the same suite, so the material
is Isis's own and carries the operator self-consent C.12.14 settled as correct
for a typed prompt. Two entries are carried as `blocked` rather than dropped or
faked:

- `dance-and-movement/scail2-animate-driven` needs a driving clip Oshun owns or
  has cleared (**E.03.08**). Stock footage would put uncleared likeness through
  the lane and make the score meaningless.
- `anime-and-stylised/netayume` holds the prompt and seed for a dedicated anime
  base that is not in the catalog (**E.06.03**), so the eventual comparison is
  against the same input.

Dropping either would have made its lane read as covered in a results file while
measuring less than it claims. A further spec refuses a lane whose entries are
_all_ blocked, so this cannot hollow out a lane silently.

93 specs, all green. No spend: this item creates nothing and runs nothing.

**E.01.02 — the rubric and the aggregate (2026-09-16).** Built and specified;
the score store and the dashboard view are the remaining half, so the item stays
open.

`benchmark-rubric.ts` makes four decisions that are the difference between a
rubric and a mood:

- **Every point is anchored.** Six criteria, each with what a 1 and a 5 look
  like, so "3" means the same thing in March as in January.
- **Criteria are per lane.** `BENCHMARK_LANE_CRITERIA` gives each lane only what
  it can answer. Typography is scored on prompt adherence alone; asking it about
  soft-tissue motion would produce a number that means nothing and then get
  averaged into a routing decision.
- **Act-bleed is a defect count, not a quality.** Sexual pose, sexualised
  framing, unrequested second person — higher is worse, the opposite direction
  from every other column, so it is typed separately rather than left as a trap
  for whoever averages them. E.01.06 needs it reported per body-LoRA.
- **An unscored output stays unscored.** `summariseBenchmarkLane` reports
  `scored` and `unscored` beside every mean and returns `null` when nobody has
  looked — `0` would read as "scored, and terrible".

`validateBenchmarkScore` rejects rather than coerces, and each refusal is a
number that would otherwise enter E.01.05 without a person giving it: no rater,
a criterion the lane does not use, a value off the 1–5 scale, a score made under
an older `rubricVersion`, an `outputHash` that is not a sha256.

Cost per accepted output is priced against **everything the lane spent**, not
just the keepers, because the rejected renders are part of what the accepted one
cost. Blinding is `sample-<8 hex>` from the output bytes: stable across
sessions, and carrying no workflow, model or checkpoint — the rater sees the
lane, because they need it to know which criteria apply, and nothing else.

24 specs; `src/benchmarks/` is 117.

## E.04 Plates across content policies (2026-09-16)

**E.04.04 — the rule is now called, at both places it had to be.** The library
half landed 2026-09-15 (`libs/isis/workflows/src/output-chain-policy.ts`, which
reads a producing workflow's `models_required` against the volume manifest and
treats an output as bound when any of those licences carries an acceptable-use
clause). It had no callers. It has two now.

**At submit.** `JobService.submit` calls `assertSubmitOutputChainPolicy`
immediately after the content-policy gate and before the model-cache lease, so a
refusal leaves no lease, no job row and no outbox entry. The producing workflow
is read, not guessed: an input naming an Isis output carries the producing job
in its own URL (`/api/v1/outputs/<jobId>/<file>`, the shape
`output-store.service.ts` mints), so the job is looked up and its workflow read
off it using the executor's own override order (parameters first, then
metadata).

An input that is **not** an Isis output — an upload, an external URL, a data URI
— has no producing workflow here, and the rule does not apply to it. That is
deliberate and the specs pin it: E.04.04 is about what Isis made and whose
licence Isis therefore knows, not about every byte that reaches a graph.

**On each chain step.** The advancer answers it exactly where a RunPod output
feeds a RunPod step — which was previously an early `return null`, because no
_lane_ boundary is crossed there. A _licence_ boundary can be, and now is
checked, before the budget, so a step that must not run is never weighed against
money.

**Fail-closed in three places, each on purpose:**

- a manifest that cannot be read yields no artifacts, which makes every model
  unknown, which the library already treats as **bound** — so an unreadable
  manifest closes the gate rather than opening it;
- a producing job that cannot be found is `output_provenance_missing`, not
  "allowed";
- a producing workflow the catalog does not have (a rename) is the same.

**One real behaviour change, stated plainly.** An `adult_ok` step whose plate
has no readable producing workflow is now refused. Three existing advancer
fixtures tripped on it because they omitted a `workflowId` that a job which
actually rendered always carries — reference resolution, ledger choice and the
budget stop, none of them about content policy. The shared `completedJob`
fixture now supplies one unless a test sets it itself, and the provenance-
missing case is its own named test rather than an accident of an under-specified
fixture.

**Specs, both ways.** 17 in `output-chain-policy.service.spec.ts` against the
real catalog and the real manifest (`ltx25-av` into `wan22-i2v` refused with the
licence quoted; Qwen-Image, klein 4B and HiDream-O1 allowed; an `sfw_only`
consumer never even looks a job up; a non-Isis input never looked up; job-id
extraction and its negatives) and 4 new in `job-chain-advancer.spec.ts`, which
is now 25.

**Unrelated red noted, not fixed.** `catalog-content-policy.projection.spec.ts`
has 2 failures on this branch: the server's workflow-catalog interface carries
`modelFamilyPolicy` and the client projection does not. `git log -S` puts that
field in `74e9a68b437` (_sam 3d gated family and hunyuan exclusions_, T.01.03),
another session's commit that arrived here through a merge of `origin/main`.
Nothing in E.04.04 touches that route, that interface or the client. It is that
session's to close. The rest of the service suite is 589 passed, 35 skipped.
