# ADR-0010: A Hetzner Model Library with a Fixed Least-Recently-Used Cache Volume on RunPod

**Status**: Accepted  
**Date**: 2026-09-13 (decided), 2026-09-18 (recorded)  
**Owner**: @GreyChimp (the decision and every live step are the owner's)  
**Authors**: Isis Runtime Engineering  
**Related ADRs**: [ADR-0003](ADR-0003-runpod-serverless-execution-strategy.md)
(pay-per-second serverless strategy),
[ADR-0004](ADR-0004-runpod-endpoint-provisioning-mechanism.md) (endpoint
lifecycle),
[ADR-0005 slim worker](ADR-0005-slim-worker-image-and-volume-models.md) (no
weights in the image; weights on a network volume) — this ADR supersedes that
ADR's "one network volume holds every weight" arrangement and keeps everything
else in it  
**Decision record**: the owner's decision block dated 2026-09-13 in
`ISIS_CHROMA_RUNPOD_MVP_TODOS_2026-09-11.md` ("Decision taken by the user on
2026-09-13 (model library on Hetzner, 400 GB LRU cache on RunPod)"), planned in
commit `c2d118585f7`  
**Evidence**: `docs/agents/isis-chroma-runpod-evidence.md` §§ L.00–L.07 (every
pod, its window, its wall clock, its cost and its result file)  
**Runbook**: `docs/domains/isis/runbooks/chroma-runpod-dev-stack.md` § 5c

## 1. Context and Problem Statement

ADR-0005 (slim worker) put every weight on a RunPod network volume and kept
images free of model files. That held while the manifest was 50 rows and 367 GB.
Two things then changed at once.

**The arsenal expansion** added the survey models to scope, taking the manifest
past 700 GB. Under the volume lint's 20 % headroom rule that sized a volume at
1024 GB, which RunPod prices in tiers at **$71.20 a month**.

**The spend profile made that the wrong shape.** A RunPod network volume bills
every hour it exists, whether or not a worker touches it. Generation in this
project happens in sessions, at certain hours. The whole eight-clip V.11 proof
matrix cost roughly **$3 of GPU** (evidence § V.11); storage had become ten to
twenty times the compute spend, for bytes that sit idle most of the week.

Meanwhile the account already pays for Hetzner Object Storage for another
purpose. Its base price includes **1 TB of storage and 1 TB of egress per
account**, ingress is free, and egress past the allowance is €1.00/TB. RunPod
charges nothing for data in. So the same 700 GB of weights costs approximately
nothing extra on Hetzner, and moving them there converts a standing monthly
charge into a per-miss **wait**, not a per-miss cost.

The problem to decide: where the weights live, what the RunPod side becomes, how
big it is, and what leaves it when it is full.

## 2. Decision

**The Hetzner bucket is the library and the source of truth for weights. The
RunPod network volume becomes a fixed 400 GB cache of it, holding a pinned core
plus whatever was used most recently, evicted least-recently-used first and only
when an incoming family does not fit.**

Concretely:

1. **Library.** `manifest.library` in
   `infra/runpod/volumes/isis-chroma-models.manifest.json`:
   `provider: hetzner-object-storage`, endpoint
   `https://hel1.your-objectstorage.com`, region `hel1`, bucket
   `app-dev-server`, prefix `isis/models/`, credentials named (never valued) as
   `HETZNER_APP_DEV_S3_ACCESS_KEY` and `HETZNER_APP_DEV_S3_SECRET_KEY`. **A
   row's library key is `prefix + destPath`** — the same string as its path
   under the cache's `modelsRoot`, so there is one name for a file everywhere.
2. **Cache.** `manifest.cache`: `isis-model-cache`, volume `5bst2pgw7y`, 400 GB,
   EU-RO-1, `marginRatio` 0.05, `invariantRatio` 0.95. Workers read from the
   cache; the cache is filled from the library; **nothing is copied to the cache
   by hand.**
3. **The unit of residency is a family, not a file.** `manifest.families` gives
   each family an id, a `pinned` flag and a tier; each artifact lists every
   family that needs it, so a shared file belongs to several. Each catalog
   workflow declares `requires_families`, and the static validator fails with
   `MODEL_NOT_IN_FAMILIES` when a rendered graph references a model outside the
   declared families — including inside `_condition` branches and enum values —
   so the declaration is checked rather than trusted.
4. **Admission at submit, before anything is billed.** A job whose families are
   not all cached is refused with `409 models_not_cached` carrying the families,
   their bytes, an estimated wait and the exact sync call to make; with
   `ISIS_RUNPOD_CACHE_AUTOSYNC=true` it is instead held as `awaiting_cache` and
   a sync is started or joined for it.
5. **Eviction is planned by a pure function** (`planCacheSync`) and applied by a
   CPU loader pod, never by the GPU worker.
6. **The old 1024 GB volume is deleted once the library is download-verified and
   the cache is proven**, not before.

## 3. Options Weighed

| Option                                                   | Monthly storage | Outcome                                                                                                                                                       |
| -------------------------------------------------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **A.** Keep the 1024 GB volume, every weight resident    | $71.20          | Rejected. Pays peak size every hour of every week for bytes touched in a few sessions.                                                                        |
| **B.** Hetzner library plus a fixed cache volume         | $28.00          | **Chosen.** Trades a standing charge for a first-use wait, which the miss log can then shrink by pinning.                                                     |
| **C.** Create and destroy a volume per session           | ~$0 idle        | Rejected. Both endpoints would need re-pointing every session — a path that has wedged an endpoint before — and every session pays a full pull of everything. |
| **D.** No volume; pull into each worker's container disk | $0              | Rejected. Every cold worker re-pulls **while GPU-billed**; the wait moves from free CPU-pod time onto the expensive meter.                                    |

Option D is the one worth keeping in mind, because it is what a naive reading of
"serverless" suggests. The measured cost of a miss under option B is CPU-pod
time at $0.06/hour plus wall clock; under option D the same bytes arrive on a
GPU worker at two orders of magnitude more per second.

## 4. The Size, and Why It Is 400 GB

Sized by the invariant, not by a headroom percentage: **the pinned core plus the
largest single unpinned family must fit inside `invariantRatio` × cache size,
and every family must fit beside the core** — a family that cannot fit beside
the pinned core can never load at all, which is a configuration bug the lint
catches rather than a runtime surprise.

At the decision (2026-09-13): pinned core 212.26 GB, largest unpinned family
Qwen-Image 106.32 GB, total 318.58 GB of the 380 GB allowed at 400 GB.

Live today (`node scripts/isis/runpod-volume-manifest-lint.mjs --strict`, 99
artifacts, library 700.39 GB):

```
cache 400 GB: pinned 239.03 GB + largest unpinned ltx25 92.20 GB = 331.22 GB of 380 GB allowed
```

Sizes rejected and why: **200 GB** cannot hold the core; **300 GB** holds the
core but can never load Qwen-Image beside it; **450 GB** would be needed only
for the one pairing of Qwen-Image and LTX on the same day. RunPod volumes **only
grow**, never shrink, so the rule is to start at 400 and grow on the evidence of
the miss log, never on a guess.

The pinned core is the set that must never make a member wait: at the decision,
the Chroma stack with its four LoRAs, both Wan 2.2 expert pairs, the Wan shared
encoders, SAM 3, BiRefNet, FlashVSR, GIMM-VFI and Real-ESRGAN — 212.26 GB. It is
not frozen: pinning is a `pinned` flag in the manifest plus a green lint, and
the set has already moved (SAM 3 → SAM 3.1, Real-ESRGAN out, SeedVR2 3B in).
Pinning pulls nothing by itself; the next sync that plans for the family keeps
it.

## 5. The Eviction Rules

The rules are implemented in
`apps/isis/generation-api/src/services/runpod-model-cache/planner.ts` as a pure
function over its input — no I/O — so they can be read and tested in one place.
`planCacheSync` returns a **hit**, a **plan** of pulls and evictions, or a
**typed refusal**, and a refusal moves no bytes.

1. **The core is never evicted, and neither is anything in use.** A pinned
   family, a family leased by a queued or running job, and the family being
   requested are all protected. Deleting weights under a live ComfyUI crashes
   it, and a crashed worker poisons every job after it — hence leases.
2. **Files are reference-counted; the eviction unit is the family.** A file
   leaves only when no family that remains cached, syncing, pinned, leased or
   requested still references it. This is not a detail: `umt5` serves Wan,
   Animate-2 and SCAIL-2; `ae.safetensors` serves Chroma and Z-Image; the Qwen
   encoder serves Z-Image and klein 4B. Evicting a family by deleting "its"
   files would silently break three others.
3. **Eviction only on demand.** Nothing is evicted when
   `used + bytesToPull + marginRatio × cacheBytes` already fits. Otherwise
   candidates are evicted **one at a time until the request fits, and no
   further** — a sync never tidies up.
4. **Least recently used first.** A family synced but never used counts from its
   `syncedAt`; ties break by family id, so a plan is deterministic.
5. **"Last used" is written when a job COMPLETES**, from the job ledger. A
   failed job does not refresh it, so a family that only ever fails ages out;
   filesystem access times are never consulted.
6. **If it cannot fit, it fails before any byte moves.** Four typed refusals,
   each naming what stands in the way: `exceeds_cache` (the pinned core plus the
   request can never fit at this size), `blocked_by_leases` (it would fit once
   running jobs release), `blocked_by_volume_state` (unmanaged bytes or families
   still `syncing` occupy the space), `unknown_family`.

Two cases the decision did not spell out, settled in the implementation and
recorded here because they are the ones that bite in operation:

- **Residue.** A sync that fails part-way leaves files on disk under an `absent`
  family, and a crashed eviction leaves a family `evicting`. Those bytes are
  real, so they count as used, and such a family is an eviction candidate
  **ahead of** healthy `cached` families — residue serves no job. Ordering is
  `evicting`, then `absent`, then `cached`, LRU inside each group. A `syncing`
  family is never a candidate: the sync executor owns it.
- **Unmanaged bytes.** Objects under `models/` that are not a manifest file at
  its manifest size — short copies, `.partial` leftovers, retired weights — are
  measured by the reconciler, count as used, and are never freed by a plan. When
  they are what stands in the way, the refusal is `blocked_by_volume_state` and
  says so.

**The table is never trusted on its own.** Before every plan, the reconciler
lists the volume through its S3 API; a family is `cached` only when every one of
its files is present at the manifest size, and a recorded `cached` family with a
missing or short file flips to `absent` with a logged event.

State lives in three tables plus the sync lock: `runpod_cache_families` (state
`absent | syncing | cached | evicting`, bytes, `last_used_at`, `synced_at`),
`runpod_cache_leases` (family, job, acquired at), append-only
`runpod_cache_events` (hit, miss, sync started, sync done, sync failed, evicted
— a trigger blocks UPDATE, DELETE and TRUNCATE), and `runpod_cache_syncs` with a
partial unique index allowing **one running sync per volume**. Leases are
acquired at submit in the same transaction as the cached check, which is what
makes the race between a submit and an eviction of the same family decidable; it
is tested against real Postgres in both forced orders and in 25 concurrent
rounds with exactly one winner each.

## 6. How Bytes Move

One loader program, `scripts/isis/runpod-volume-loader.py` (stdlib only,
including its own SigV4 signer checked against eight botocore vectors). Option B
added five modes beside its original `volume-seed`:

| Mode             | Direction           | Purpose                                                                                  |
| ---------------- | ------------------- | ---------------------------------------------------------------------------------------- |
| `library-seed`   | volume → library    | The one-time migration of what was already on the volume.                                |
| `library-pull`   | source → library    | Hugging Face or Civitai straight into the library, no local copy beyond one part buffer. |
| `library-verify` | library → /dev/null | Download and hash every row; writes `verifiedAt`.                                        |
| `library-probe`  | both                | The transfer measurement of L.00.01.                                                     |
| `cache-sync`     | library → cache     | Applies a plan of evictions and pulls to the mounted cache.                              |

Integrity is checked on every leg, because the storage will not do it for us.
The L.00.01 probe established that **Hetzner does not validate a full-object
`x-amz-checksum-sha256` on multipart completion** — a deliberately wrong
checksum was accepted — so download-side hashing stayed in full scope for every
row. Single PUTs up to 512 MiB carry the manifest sha256 as the signed payload
hash, so the server itself refuses wrong bytes; above that, multipart with a
per-part hash and a whole-file hash computed while reading, and a mismatch
**aborts the upload** rather than leaving a bad object. A cache-sync pull lands
through `.partial`, is verified, and only then renamed into place.

The sync orchestrator takes the volume's lock, reconciles, plans, marks the
planned evictions `evicting` and the pulls `syncing`, creates the CPU loader
pod, then in the background polls the result file, applies it, **deletes the pod
however the sync ended**, books the pod's wall-clock life as a `cache_sync` line
in the daily ledger, releases the lock and settles jobs held for the cache. A
sync that cannot start puts the families back. A pull family is `cached` only
when the loader reported it `cached`; anything else leaves it `absent` with a
`sync_failed` event.

## 7. Consequences

**Cost.** Storage falls from $71.20 to $28.00 a month; with the old volume gone
the cache alone is $0.039/hour. During the migration both volumes existed at
about $0.171/hour, which is why L.06.05's deletion was not left to wait.

**A miss now costs time.** A first use of an unpinned family waits for a sync.
Measured end to end on 2026-09-13 (evidence § L.06.04): submitting
`wan21-vace-edit` was refused at 19:10:25Z with `409 models_not_cached` naming
`controlnet-aux` (1.69 GB) and `wan21-vace` (46.30 GB, of which 36.37 GB to pull
because umt5 was already cached with `wan-shared`); the sync finished 24.4
minutes later; the clip rendered after about 21 minutes of GPU queue and 5.7 of
render. **Miss to render: 52 minutes.** Warming before a session is therefore
part of operation, not an optimisation — `POST /api/v1/runpod/cache/sync` with
`{"workflows": [...], "dryRun": true}` shows the pulls, the evictions and the
estimated wait, and the dashboard's RunPod page has the same picker.

**Every wait estimate is measured or absent.** The estimator computes from
`ISIS_RUNPOD_CACHE_SYNC_MB_PER_SECOND`, which the runbook sets to 40 MiB/s
(§ 5c). That is the L.06.02 measurement: a median of 40.6 MiB/s per file while
three transfers shared Hetzner, and about 75 MiB/s for a transfer alone. It is
not the L.00.01 probe's figure, which read 20.8 MiB/s on eight parallel parts
(`library-probe-result.json`). Unset, the 409 body and the dry-run plan return
`null` with a reason rather than a guess.

**The library is now the only full copy.** `ax0oj250j3` (1024 GB) was deleted at
18:13:21Z on 2026-09-13 on the owner's explicit go, ahead of both gates that
decision 6 sets: the library was not yet fully download-verified and the cache
had not rendered. L.06.05 records this. The loader result files in
`docs/agents/evidence/isis-chroma-runpod/l-results/` show what held at that
moment and when the rest followed:

- Both endpoints had been repointed to `5bst2pgw7y` (L.06.03), and the loader
  result files written so far had been copied into that directory.
- All 50 library rows were in the library (`library-seed-result.json`, finished
  17:27:45Z: 50 `ok`, 35 uploaded in that run and 15 already there). L.06.05
  records each as hash-checked against the manifest at upload.
- 14 of the 16 non-core rows were download-verified (`library-verify-result.json`,
  finished 18:08:27Z). The other two had timed out on Hetzner reads.
  `gemma3-12b-it-bf16` verified at 18:27:50Z (`library-verify-retry-result.json`).
  `wan21-vace-14b-fp16` timed out again, and was downloaded and hashed by the
  loader at 19:33:33Z (`cache-sync-vace-result.json`).
- The pinned core's download-side verification is the core sync, which hashes
  each file as it pulls it (L.01.05, L.06.02). It was still running: 26 of its
  34 files had been pulled (`cache-sync-core-result.json`). Five more followed by
  18:37:53Z, and the last three, which had timed out, at 18:58:33Z
  (`cache-sync-core-followup-result.json`).
- L.06.04 had not rendered. Its first clip rendered later that evening (the
  measurement under "A miss now costs time").

So every row was download-verified by 19:33:33Z, 80 minutes after the deletion,
and none was lost. The `deletionReason` in `infra/runpod/volumes/desired-state.json`
says 48 of 50 rows were download-verified at the moment of deletion. The result
files do not support that figure: they show 14 rows verified and the core sync in
progress.

A lost cache volume is rebuilt by syncing families from the library. A lost
library is not rebuildable from the cache.

**The Hetzner allowance is shared, per account.** Worker outputs under
`isis/outputs/` draw on the same 1 TB as `isis/models/`. The API's storage guard
sweeps outputs older than `ISIS_HETZNER_OUTPUTS_RETENTION_DAYS` (7) every
`ISIS_HETZNER_STORAGE_GUARD_INTERVAL_MS` (6 h), sums every bucket the
credentials list, and raises `storage.hetzner.usage_high` at
`ISIS_HETZNER_USAGE_ALARM_GB` (900), once per UTC day. The sweepable-prefix
allowlist lives in the S3 client (`HETZNER_SWEEPABLE_PREFIXES`), not in the
guard's env, so the guard cannot be configured toward the library — the sweeper
cannot delete weights. The manifest lint additionally warns past 1000 GB of
library.

**Adding a model is four steps, none of them a copy.** A manifest row with its
families (lint green, including the cache invariants it prints); the family
added to any workflow's `requires_families` (`model-families.spec.ts` fails
otherwise); the bytes into the library with `library-pull`; and the cache picks
it up the first time a job needs that family.

**Retirement means the library and the cache.** Retiring a model deletes it from
the library and evicts it from the cache; it no longer means deleting from a
volume. `authoring/generation-retirements.json` in the presentation center reads
these deletions from the commits that made them.

**Transfers are not reliable and the tooling assumes it.** The seed of 50 rows
took four launches: two pods stalled on S3 connections Hetzner stopped
answering, one hung 40 minutes at 0 % CPU on a volume read. The fixes are in the
loader — a 30 s stall timeout (`a94f2832105`), a watchdog that exits 75 with a
start loop that reruns (`111a49e7518`), a partial-file resume instead of a
restart (`e0c6d74442e`), and a source-connection resume by range request
(`d78f80df034`). The successful run was 2478 s for 367.17 GB, and a listing
confirmed 50 objects with every size and every `x-amz-meta-sha256` matching the
manifest. Contention is real: a cache sync and a library pull against the same
bucket at the same time stalled the sync.

**One defect this design produced, found in use and fixed.** A sync driver
refreshed its heartbeat only inside its result-polling loop, so during a long
`applying` phase the heartbeat went stale, `resumeStaleSync()` took the sync
over **inside the same process**, and the pod's cost was booked twice
(2026-09-14, sync `b80de934`). The driver now heartbeats on a timer in every
phase and `resumeStaleSync()` never takes over a sync this process is still
driving; both specs were shown to fail with the fix removed (`5ceea7b97c9`).

## 8. Status of the Implementation

Delivered under tracker sections L.00–L.07 and recorded item by item in
`docs/agents/isis-chroma-runpod-evidence.md`. The service is
`apps/isis/generation-api/src/services/runpod-model-cache/` (manifest, planner,
reconciler, store, sync service, admission, dashboard projection), with the
store and race specs run against real Postgres and every outside boundary —
volume listing, result file, pod API, spend gate, ledger, job service —
injected, so the orchestration is tested with the real planner and the real
store and nothing else doubled.
