# Evidence · Where a generation actually runs · configuration and tracker lines at c7a2f8fed166

Every excerpt below is copied from the repository at commit `c7a2f8fed166b8b63e679d8655af4886602319ac`. Nothing was run against RunPod to produce it. Lines longer than 1,600 characters are cut and marked.

## 1. The development environment of `infra/runpod/endpoints/desired-state.json`

Command: `git show c7a2f8fed166b8b63e679d8655af4886602319ac:infra/runpod/endpoints/desired-state.json`, then the `environments.development` object, re-indented.

```json
{
 "endpoints": [
  {
   "key": "comfyui-chroma",
   "managed": true,
   "endpointIdEnv": "RUNPOD_COMFYUI_ENDPOINT_ID",
   "canaryEndpointIdEnv": "RUNPOD_COMFYUI_CANARY_ENDPOINT_ID",
   "canaryTrafficPercentage": 0,
   "name": "isis-comfyui-chroma",
   "gpuIds": "NVIDIA GeForce RTX 4090,NVIDIA GeForce RTX 5090,NVIDIA RTX A5000,NVIDIA L4,NVIDIA GeForce RTX 3090",
   "templateName": "isis-comfyui-chroma-template",
   "templateId": "vfl3iu1dr6",
   "workersMin": 0,
   "workersMax": 1,
   "idleTimeout": 10,
   "executionTimeout": 300,
   "flashBoot": true,
   "dataCenterId": "EU-RO-1",
   "networkVolumeId": "5bst2pgw7y",
   "containerDiskInGb": 30,
   "image": null,
   "imageRepository": "ghcr.io/greychimp/oshun/runpod-comfyui-slim",
   "registryAuthName": "oshun-ghcr-pull",
   "workerKind": "worker-comfyui",
   "env": {
    "COMFY_POLLING_MAX_RETRIES": "1500",
    "COMFY_POLLING_INTERVAL_MS": "250",
    "COMFY_LOG_LEVEL": "INFO",
    "BUCKET_NAME": "app-dev-server",
    "BUCKET_KEY_PREFIX": "isis/outputs",
    "BUCKET_REGION": "hel1"
   },
   "notes": "Cost-bounded dev endpoint (ADR-0005). Image is built by RunPod from the GitHub repository (console: Serverless → New Endpoint → Import Git Repository; GitHub Actions is billing-blocked on this account as of 2026-09-11). `image` stays null for RunPod-built images; the live endpoint id is recorded after creation. `registryAuthId` is kept for the GHCR fallback path.",
   "registryAuthId": "cmtx2h80b001kiob5v4p8dtwc",
   "imageSource": {
    "kind": "runpod-github",
    "repository": "GreyChimp/oshun",
    "branch": "runpod-comfyui-slim",
    "dockerfilePath": "docker/runpod/comfyui-slim/Dockerfile",
    "rebuildTrigger": "github-release",
    "branchNote": "Quiet branch: every push to the source branch restarts the RunPod build and cancels the in-flight one, and main receives pushes from several agents. Fast-forward this branch deliberately when the worker image must change."
   },
   "endpointId": "6jye7vngg4i685",
   "allowedCudaVersions": [
    "12.8",
    "12.9",
    "13.0",
    "13.1",
    "13.2",
    "13.3"
   ],
   "gpuIdsNote": "RunPod GPU pools (saveEndpoint gpuIds): ADA_24 = 4090; AMPERE_24 = L4, A5000, 3090; BLACKWELL_32 = 5090, RTX PRO 4500 Blackwell. Widened 2026-09-11 because EU-RO-1 (the volume's datacenter) had no free 4090/A5000/3090 and workers sat throttled.",
   "envFromEnvironment": [
    "BUCKET_ENDPOINT_URL",
    "BUCKET_ACCESS_KEY_ID",
    "BUCKET_SECRET_ACCESS_KEY"
   ],
   "secretRefs": {
    "BUCKET_ACCESS_KEY_ID": "{{ RUNPOD_SECRET_oshun-hetzner-s3-access-key }}",
    "BUCKET_SECRET_ACCESS_KEY": "{{ RUNPOD_SECRET_oshun-hetzner-s3-secret-key }}"
   },
   "workerOutputNote": "C.12.06 / V.12.05 (user decision 2026-09-13): outputs go to Hetzner Object Storage, bucket app-dev-server, prefix isis/outputs, through the image's oshun_uploader .pth hook. Console env set 2026-09-13 15:4xZ with the two keys as RunPod secret references (secretRefs); first live s3_url output 15:51Z. BUCKET_ENDPOINT_URL is https://hel1.your-objectstorage.com; the two keys are HETZNER_APP_DEV_S3_ACCESS_KEY / _SECRET_KEY in env-master.env. A git-integration template's env is console-only; runbook §5b."
  },
  {
   "key": "comfyui-video",
   "managed": true,
   "endpointIdEnv": "RUNPOD_COMFYUI_VIDEO_ENDPOINT_ID",
   "name": "isis-comfyui-video",
   "gpuIds": "NVIDIA RTX PRO 6000 Blackwell Server Edition,NVIDIA RTX PRO 6000 Blackwell Workstation Edition,NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition,NVIDIA H100 PCIe,NVIDIA H100 80GB HBM3,NVIDIA H100 NVL,NVIDIA H200,NVIDIA A100 80GB PCIe,NVIDIA A100-SXM4-80GB",
   "gpuSelectionNote": "Verified live 2026-09-12. `gpuPools` returns AMPERE_16, AMPERE_24, ADA_24, BLACKWELL_32, ADA_32_PRO, AMPERE_48, ADA_48_PRO, AMPERE_80, ADA_80_PRO, BLACKWELL_96, HOPPER_141, BLACKWELL_180, BLACKWELL_288 — there is NO 80 GB Hopper pool, so the chroma endpoint's pool-id form (ADA_24,...) cannot express \"H100 80 GB\". These are the exact `gpuTypes` ids: \"NVIDIA H100 80GB HBM3\" is SXM and \"NVIDIA H100 PCIe\" is PCIe, both 80 GB (\"NVIDIA H100 NVL\" is 94 GB and is excluded as a different price class). V.03.04 must read the console back after creation and record what it actually shows — the console is the authority on placement (rule 9), and REST GET echoes a stale list.",
   "gpuFallbackNote": "If neither H100 type places a worker in EU-RO-1, the documented options are HOPPER_141 (H200 141 GB, a higher price class) or moving the video endpoint to a datacenter with H100 stock — which would need a second network volume, since volumes cannot move. Do NOT fall back to a smaller GPU: a 28.6 GB fp16 Wan expert does not fit 48 GB beside its encoders, so the job would OOM after the load was billed (V.03.05).",
   "templateName": "isis-comfyui-video-template",
   "templateId": "qcxiqduk01",
   "workersMin": 0,
   "workersMax": 1,
   "idleTimeout": 10,
   "executionTimeout": 900,
   "flashBoot": true,
   "dataCenterId": "EU-RO-1",
   "networkVolumeId": "5bst2pgw7y",
   "containerDiskInGb": 30,
   "image": null,
   "imageRepository": "ghcr.io/greychimp/oshun/runpod-comfyui-slim",
   "registryAuthName": "oshun-ghcr-pull",
   "registryAuthId": "cmtx2h80b001kiob5v4p8dtwc",
   "workerKind": "worker-comfyui",
   "env": {
    "COMFY_POLLING_MAX_RETRIES": "3600",
    "COMFY_POLLING_INTERVAL_MS": "250",
    "COMFY_LOG_LEVEL": "INFO",
    "BUCKET_NAME": "app-dev-server",
    "BUCKET_KEY_PREFIX": "isis/outputs",
    "BUCKET_REGION": "hel1"
   },
   "envFromEnvironment": [
    "BUCKET_ENDPOINT_URL",
    "BUCKET_ACCESS_KEY_ID",
    "BUCKET_SECRET_ACCESS_KEY"
   ],
   "notes": "Second endpoint for the `motion` catalog (decision D-V1): 80 GB so fp16 14B Wan experts, Animate, VACE and LTX-2.3 22B load without CPU offload, and a 900 s execution cap because a clip is minutes, not seconds. The image endpoint keeps its 24 GB / 300 s bounds untouched. Same worker image and the same network volume as comfyui-chroma (one image, D-V2). The three BUCKET_* names are listed by NAME only and injected by runpod-provision.mjs from the environment — never written into this file — so the worker can PUT video output straight to the volume's S3 endpoint instead of base64 (D-V4). NOT CREATED YET: V.03.04 creates it and records the live id.",
   "imageSource": {
    "kind": "runpod-github",
    "repository": "GreyChimp/oshun",
    "branch": "runpod-comfyui-slim",
    "dockerfilePath": "docker/runpod/comfyui-slim/Dockerfile",
    "rebuildTrigger": "github-release",
    "branchNote": "Quiet branch: every push to the source branch restarts the RunPod build and cancels the in-flight one, and main receives pushes from several agents. Fast-forward this branch deliberately when the worker image must change."
   },
   "endpointId": "2lezocymx2rgot",
   "secretRefs": {
    "BUCKET_ACCESS_KEY_ID": "{{ RUNPOD_SECRET_oshun-hetzner-s3-access-key }}",
    "BUCKET_SECRET_ACCESS_KEY": "{{ RUNPOD_SECRET_oshun-hetzner-s3-secret-key }}"
   },
   "workerOutputNote": "C.12.06 / V.12.05 (user decision 2026-09-13): outputs go to Hetzner Object Storage, bucket app-dev-server, prefix isis/outputs, through the image's oshun_uploader .pth hook. Console env set 2026-09-13 15:4xZ with the two keys as RunPod secret references (secretRefs); first live s3_url output 15:51Z. BUCKET_ENDPOINT_URL is https://hel1.your-objectstorage.com; the two keys are HETZNER_APP_DEV_S3_ACCESS_KEY / _SECRET_KEY in env-master.env. A git-integration template's env is console-only; runbook §5b."
  }
 ]
}
```

## 2. `docker/runpod/comfyui/RETIRED.md`, whole

Command: `git show c7a2f8fed166b8b63e679d8655af4886602319ac:docker/runpod/comfyui/RETIRED.md`

````text
# This image lineage is retired — do not revive it without a decision

**Status:** retired, never published. Recorded 2026-09-12 (tracker C.12.05).

`oshunai/runpod-comfyui` **does not exist on Docker Hub** (404, verified
2026-09-11) and never will: no Docker Hub account is being created (user
decision, 2026-09-11). Every image this directory builds is tagged
`oshunai/runpod-comfyui:<flavor>`, so nothing built here has ever been pushed
anywhere, and the workflow that pushes it cannot succeed.

The design is also the one the Chroma/RunPod work deliberately moved away from:
each flavor **bakes** its own model set into the image (the `flagship-image`
inventory alone lists 47 baked artifacts). The replacement keeps models on a
RunPod network volume and loads them on demand, from a single slim image built
by RunPod's own git integration — `docker/runpod/comfyui-slim/`, which is what
the live endpoints actually run.

## Why it is still here

C.12.05 says to retire this "once nothing references them". Things still do.
This is the caller scan, run 2026-09-12, so the next person does not have to
redo it:

| Reference                                                                            | What it does                                                                                                  |
| ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------- |
| `scripts/isis/generate_runpod_endpoint_image_manifests.mjs`                          | **Syncs INTO this directory** (`workflow_catalog`, `isis_nodes`) and is a CI gate over node/model consistency |
| `scripts/isis/validate_runpod_endpoint_image_manifests.mjs`                          | Validates the manifest the generator emits                                                                    |
| `infra/runpod/endpoints/image-build-manifest.json`                                   | **13 of its 17 endpoints** build from this context                                                            |
| `infra/runpod/endpoints/image-build-config.json`                                     | The generator's input; the flavors are declared there                                                         |
| `.github/workflows/deploy-runpod.yml`                                                | Path filter (line ~97) and a buildx job with this context (lines ~305–306)                                    |
| `docker/runpod/comfyui/tests/test_catalog_renderer.py`                               | Unit tests for `catalog_renderer.py`                                                                          |
| `DEPENDENCIES.md`, `V1/features/external-models-and-execution.md`, `ISIS_TODOS_2.md` | Prose references                                                                                              |

Roughly forty `scripts/isis/generate-v2-*`/`validate-v2-*` scripts read
`image-build-manifest.json` itself. They do not name this directory, but they do
consume what the generator produces from it, so the manifest cannot simply lose
13 endpoints without checking them.

## The order a removal has to happen in

Deleting this directory first would break a CI gate and a deploy workflow, which
is why C.12.05 is still open rather than done:

1. Decide what happens to the 13 flavor endpoints in
   `infra/runpod/endpoints/image-build-config.json` — removed, or marked retired
   the way `comfyui-vace` / `comfyui-ltx2` / `comfyui-flagship-video` already
   are in `infra/runpod/endpoints/inventory/`
   (`status: "retired-never-published"`).
2. Regenerate `image-build-manifest.json` and check every `generate-v2-*` /
   `validate-v2-*` consumer still passes against the smaller manifest.
3. Remove the comfyui build job and path filter from
   `.github/workflows/deploy-runpod.yml`.
4. Move `catalog_renderer.py` + its test, or drop them — the TypeScript port
   (`libs/isis/workflows/src/templates/catalog-renderer.ts`) is what the live
   path uses; this copy is the original the port was made from.
5. Then, and only then, delete this directory.

Until step 5, treat everything here as frozen: it is not the build path, it is
not deployed, and changes to it do not reach any running endpoint.

````

## 3. Tracker lines that open each cited item

File `ISIS_CHROMA_RUNPOD_MVP_TODOS_2026-09-11.md`. For each item, the lines where its id appears in bold as the item's own heading (`**<id>**`).

### C.12.12: 1 heading line(s)

```text
 2086  - [x] **C.12.12** ~~Slim worker image fails at container create on RunPod.~~
```

### V.03.04: 1 heading line(s)

```text
 3065  - [x] **V.03.04** Create the endpoint (`endpoint create --yes`), drift check
```

### V.01.09: 1 heading line(s)

```text
 2874  - [x] **V.01.09** ComfyUI's own validator, in CI, for $0: after the smoke test,
```

### V.01.07: 1 heading line(s)

```text
 2831  - [x] **V.01.07** Point BOTH endpoints (`comfyui-chroma`, `comfyui-video`) at
```

### A.05.01: 1 heading line(s)

```text
 5902  - [x] **A.05.01** One worker-image build after the new goldens exist: the
```

### C.12.18: 1 heading line(s)

```text
 2103  - [x] **C.12.18** _(found by C.11.08; fixed and proved 2026-09-12)_
```

### C.11.07: 1 heading line(s)

```text
 1699  - [x] **C.11.07** `matting-remove-background` × 4 models on the C.11.02 output;
```

### C.11.01: 1 heading line(s)

```text
 1665  - [x] **C.11.01** `chroma-txt2img` with `chroma1-hd`, square*1024, default steps
```

### V.01.04: 1 heading line(s)

```text
 2751  - [x] **V.01.04** README: add the video contract — video outputs arrive in
```

### L.06.03: 1 heading line(s)

```text
 6313  - [x] **L.06.03** Point both endpoints at the new volume. Try the console's
```

### V.12.05: 1 heading line(s)

```text
 5016  - [ ] **V.12.05** Webhook completion + direct S3 output are already the
```

### A.01.09: 1 heading line(s)

```text
 5428  - [x] **A.01.09** SAM 3.1 on core nodes: row
```

### C.12.05: 1 heading line(s)

```text
 1952  - [ ] **C.12.05** Retire `docker/runpod/comfyui/` flavor images formally (delete
```

