# Stage graphs, budgets and the sandbox

A 3D request is not one render. It is a graph of stages, each one an ordinary
job, with money committed before work starts, a global slot per resource class,
typed errors, and every untrusted tool run behind four boundaries. This document
describes those four systems and how they fit together.

Code: `libs/isis/workflows/src/three-d-capabilities/stage-graph.ts` and
`request-contract.ts` (pure), and in
`apps/isis/generation-api/src/services/three-d-stages/`:
`stage-orchestrator.service.ts`, `stage-run.store.ts`, `budget-policy.ts`,
`budget-reservation.store.ts`, `resource-governor.ts`, `stage-errors.ts`,
`job-events.store.ts`, `event-delivery.service.ts`. Sandbox:
`apps/isis/gpu-worker/src/sandbox/`. Tracker: T.03.01–T.03.06, T.04.04.
Evidence: `docs/agents/isis-3d-studio-evidence.md` §§ T.03.

**State of the world.** The orchestrator dispatches work it cannot run today:
every 3D capability is `unavailable` or `candidate`
([capability states](./capability-states-and-admission.md)), so a stage job is
refused by the submit guard and the run fails with that refusal as its typed
error. The machinery below is real and tested; what it drives is not admitted
yet.

## 1. The graph

`parameters.threeD` names stages from a fixed list of fifteen:

`concept-image`, `geometry`, `texture`, `cleanup`, `remesh`, `uv-unwrap`,
`bake`, `lod`, `collision`, `rig`, `motion`, `retarget`, `scene-assembly`,
`render`, `export`.

`buildThreeDStageGraph` turns the requested stages into a directed acyclic graph
in topological order. Each stage may appear once. The shape:

- **The mesh chain is linear** — geometry → cleanup → remesh → uv-unwrap →
  texture → bake — and each step reads the latest mesh before it, skipping
  stages the request did not ask for.
- **LOD and collision branch off the final mesh**, and collision reads the LODs
  when both were asked for.
- **The animation chain** — rig → motion → retarget — reads the final mesh.
- **Scene assembly reads every asset stage**; **render** reads the scene, or the
  assets when there is no scene; **export** reads every stage but render.
- **A stage with nothing before it reads the request's source asset.**

A stage is **ready** only when every stage it depends on has succeeded. A failed
or cancelled dependency blocks it.

## 2. One stage, one job, one key

A parent job carrying `parameters.threeD` is **not rendered itself**. The
orchestrator persists one run per stage and submits each ready stage as an
ordinary job through the job service.

The idempotency key is derived from **the parent, the stage, the attempt and the
dependency hash** — the hash of what this stage's output would be made from.
Every guarantee rests on that key and on the database, never on memory:

- **A crash between dispatch and persistence.** The run is leased before the job
  is submitted and the job id recorded after. A dispatcher that dies in between
  leaves a lease that expires; the next dispatch recomputes **the same key**,
  and the job service returns the job that already exists through its unique
  idempotency index. No second job is created, and no second charge.
- **Duplicate and out-of-order callbacks.** Every callback is an event folded
  into the run under a row lock by `applyStageEvent`, which returns whether the
  event was applied, a duplicate, stale or rejected. A callback for another job
  than the run's, for an earlier attempt, or after the run is terminal never
  changes it. A completion repeated with **the same** output is a duplicate;
  with a **different** output it is rejected.
- **A parent never completes early.** Its status is recomputed from every run
  after each change, and it succeeds only when every stage succeeded with
  outputs whose bytes were measured. Anything else is running, failed or
  cancelled. **A partial result is never a completion.**

## 3. Steering a running request

Steering (T.04.04) changes the request while it runs, and the rule is that it
changes only what it affects.

- A **stage is affected** when its projection of the request changed, or when a
  stage it reads is affected.
- A **run is superseded** when what it was made from no longer matches — its
  dependency hash, recomputed against the steered request and the outputs it
  reads _now_.
- **Only superseded runs start a new attempt.** Every other run, finished or
  not, keeps its work; their jobs are stopped and re-attempted only where the
  hash moved.

The steered request is stored beside the runs, so an advance is decidable from
the database alone.

## 4. Budgets: money is committed before work starts

Three scopes, each with its own cap, **all required**:

| Scope                               | Cap                                                                                             |
| ----------------------------------- | ----------------------------------------------------------------------------------------------- |
| `3d-day:<UTC date>`                 | `ISIS_3D_DAILY_BUDGET_USD` — every 3D job together                                              |
| `3d-tenant-day:<tenant>:<UTC date>` | `ISIS_3D_TENANT_DAILY_BUDGET_USD`                                                               |
| `3d-project:<parent job id>`        | `ISIS_3D_PROJECT_BUDGET_USD`, **lowered — never raised** — by the request's own `budget.maxUsd` |

Every execution a parent job may pay for — a stage attempt, a controller call, a
repair — reserves a conservative amount against each scope it falls under
**before its job is submitted**, and the reservation is refused if any scope
would pass its cap counting everything already committed there. Amounts are
integer micro-USD.

A stage reserves the ceiling configured for it in `ISIS_3D_STAGE_CEILINGS_USD`
(JSON, stage to USD per attempt). **A stage with no ceiling has no known price,
and its execution is refused rather than run on a guess.** Nothing has a
default: a missing cap or ceiling fails closed with the reason.

### 4.1 What a scope counts

| Reservation state  | What it counts                                                                                                                                                                                     |
| ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `held`             | The reserved amount — work may be running.                                                                                                                                                         |
| `awaiting_billing` | The work ended but its charge is not known yet (a failed job, a provider that bills late). **The reserved amount keeps counting**, so delayed billing cannot free money that may already be spent. |
| `settled`          | The actual charge, which may exceed the reservation — recorded as an overrun and counted **in full**, so later reservations see it.                                                                |
| `released`         | Nothing. **Only work proven never to have started is released.**                                                                                                                                   |

A reservation's id is its parent, stage, attempt and purpose (`stage`,
`controller` or `repair`), so reserving again after a crash returns the
reservation already made instead of committing the money twice.

PostgreSQL serializes reservations by locking every scope row they touch, **in
key order**, for the transaction; the memory store does the same work with no
`await` between reading and writing.

## 5. Resource slots, and fairness

Every stage job holds **one slot of its resource class** from dispatch until its
job is terminal, and a class never has more holders than its cap however many
parents fan out at once. The classes are `gpu-image`, `gpu-video`, `gpu-3d`,
`cpu-dcc` and `controller`.

**A parent job holds nothing**, so a graph waiting on a CPU stage or a
controller call keeps no GPU slot.

When a class is full, a denied request is recorded as a **waiter**. A free slot
goes to the tenant whose turn it is — the one holding the fewest slots of that
class, and among those the one waiting longest. A request from a tenant whose
turn it is not is denied `fair_share_wait` **even when a slot is free**, and
keeps its place, so a tenant who fanned out a large graph cannot starve
another's first stage.

A lease is keyed by the stage run (parent, stage, attempt), so acquiring again
after a crash replays it.

## 6. Typed errors

Every error code a stage run can carry falls into exactly one of eleven
categories, and each category names **what to do next** and **whether trying
again unchanged can help**:

`request_invalid`, `revision_stale`, `asset_missing`, `licence_rejected`,
`budget_exhausted`, `queue_timeout`, `backend_unavailable`, `output_unverified`,
`cancelled`, `blocked`, `worker_failed`.

Each carries a `retryable` flag, the HTTP status an API control answers with,
and a one-sentence action. The six cases T.03.05 names are distinct from each
other _and_ from an invalid request, an unverified output, a cancellation and a
worker failure, so no two read the same to a client. **A code the catalogue does
not know is a worker failure reported with its raw code, never silently
dropped.**

Codes come from three places and keep their own spelling: submit refusals and
the orchestrator in lower case, the workflow executor, provider callbacks and
model cache in upper case.

Behaviour worth knowing: a stage job queued past its limit is cancelled, its
slot freed and **its reservation settled at zero**. A stage retry keyed by the
attempt it retries answers `202` once and `200` after. A stale
`expectedSequence` gets `409 revision_stale`.

## 7. Events, SSE and webhooks

A per-parent event log, deduped with a gapless sequence, holds **a snapshot of
the whole graph** in each event, read under the append lock. It feeds both SSE
and signed webhooks. SSE resumes from `Last-Event-ID`. Webhooks go through an
outbox with each subscription's own backoff, end `dead` when attempts run out,
and can be redelivered under the original event id.

A defect the tests found on the way: a failure retried within the same advance
was never published.

**Not claimed:** there has been no delivery to a real receiver, SSE polls rather
than listening, and the routes do not check job ownership.

## 8. The sandbox

Blender, the mesh pipeline and the upscaler open files a user supplied and run
code paths those files steer. Each runs through `runSandboxed` under four
boundaries.

### 8.1 Identity

Not as root. A worker running as root must name an unprivileged uid range
(`ISIS_SANDBOX_UID`, `ISIS_SANDBOX_UID_COUNT`) and an `ISIS_SANDBOX_GID`; each
run **leases a uid of its own**, which cannot read the worker's
`/proc/<pid>/environ` or another run's files. On Linux, every process still
holding that uid after the run — one that left the process group with `setsid` —
is killed before the uid is released. The job directory is made `0700` and
handed to that uid, hard-linked files excepted.

A non-root worker may run tools as itself only when
`ISIS_SANDBOX_ALLOW_SAME_USER=true` says so: a same-user tool can read the
worker's environment through `/proc`, so that is for development, never for a
worker holding storage keys.

### 8.2 Environment

The tool gets a **fixed** set of variables — PATH, locale, GPU selection, a home
and temp directory inside the job's work directory — not the worker's
environment. Storage keys, database and queue URLs never reach it. A variable
named in `ISIS_SANDBOX_PASS_ENV` that looks like a secret is refused.

### 8.3 Kernel limits

`bash` sets **soft and hard** rlimits before `exec`ing the tool, so neither it
nor anything it starts can raise them: CPU seconds, file size, open files,
processes (`RLIMIT_NPROC`, which counts every process of the _user_ — the reason
tools need their own uid) and, on Linux only, address space. **A limit that
cannot be applied stops the run before the tool starts.**

### 8.4 Watchdogs

Resident memory of the whole process group is sampled with `ps` and the group is
killed past `ISIS_SANDBOX_MEMORY_MB`; a wall-clock limit stops it. Sampling is a
**documented approximation** — a spike shorter than the interval can pass it,
which is why Linux also sets `RLIMIT_AS` where the tool tolerates it. CUDA
reserves large address ranges, so GPU tools leave it unset and rely on the
watchdog and the container's memory limit.

### 8.5 Network: `safeDownload`

A job names its inputs by URL, and before this every string starting with `http`
was fetched as given — including one pointing at the cloud metadata service, the
queue, or a service on the worker's private network, fetched **with the worker's
reach** and written into the job. `safeDownload` refuses that:

- https only unless `allowHttp`; no credentials in the URL; only allowed ports;
  only allowlisted hosts when an allowlist is set;
- **every address the host resolves to is checked at connection time, inside the
  socket's own `lookup`** — so a name that resolves to a public address when
  checked and a private one when connecting (DNS rebinding) is checked on the
  address actually used. Loopback, private, link-local (metadata), CGNAT,
  multicast, reserved and documentation ranges, and their IPv4-mapped IPv6
  forms, are refused unless a CIDR in `allowedPrivateCidrs` names them;
- redirects are followed by hand, at most `maxRedirects`, **each hop checked
  again from the start**;
- the body is capped while it streams, and a `Content-Length` over the cap is
  refused before reading;
- `timeoutMs` is a deadline for **the whole download**, redirects and body
  included, not only for a silent socket.

### 8.6 Archives

An uploaded bundle is read from its central directory and **every entry is
judged before one byte is written**:

- **Names.** Relative forward-slash paths only: no absolute path, drive letter,
  backslash, NUL, empty, `.` or `..` segment, and no two entries whose names
  differ only in case (they would overwrite each other on some file systems). A
  central-directory name must match its local header.
- **Kinds.** Regular files and directories only. A symlink, device or other
  special entry is refused, as is an encrypted entry, a method other than stored
  or deflate, and ZIP64.
- **Sizes.** At most `maxEntries` entries and `maxTotalBytes` declared in all;
  an entry whose declared expansion ratio passes `maxRatio` is refused as a
  bomb. Inflation is capped at the declared size, so an entry that lies about
  its size cannot expand past it, and its CRC-32 must match.
- **Overlap.** Two entries sharing a local header — the overlapping-file bomb —
  are refused.
- **Writing.** Files are created exclusively (`wx`) under a destination that
  must be empty, inside directories created by this extraction, and every path
  is re-checked against the destination's real path.

### 8.7 Asset references

A glTF names its buffers and images by URI, an OBJ names material libraries, a
material names texture files — and a loader follows those names. `trimesh` and
Blender read a relative path from disk and an absolute one from anywhere, so
`"uri": "../../../../proc/self/environ"` turns an import into a file read and an
`http://169.254…` URI into a fetch from inside the worker. Before any tool opens
an asset, every reference it holds is checked:

- `data:` URIs (glTF only) are allowed — they carry their bytes inline;
- any other scheme is refused;
- a path must be relative, without backslashes or NUL, and must resolve — **as
  written and percent-decoded, following links** — to an existing regular file
  inside the asset's root directory;
- OBJ, MTL and PLY references are judged in **every reading a loader might
  take** (the rest of the line, from any token, or one token), comments
  included.

That last rule is measured, not theoretical. On 2026-09-16, trimesh 4.12.2
followed **five** reference escapes into an exported GLB: a PLY `TextureFile`
named in a _comment_, a spaced `map_Kd` whose file name is the rest of its line,
a commented `mtllib`, a line-continued `mtllib`, and MTL maps resolved from the
OBJ's directory rather than the MTL's.

STL and OFF carry no references. FBX, DAE, USD and `.blend` are returned as
**unchecked, by name**, or refused when the caller asks — a format whose
references cannot be checked is never quietly treated as safe.

### 8.8 Blender specifically

Blender runs with `--factory-startup --disable-autoexec` and
`--python-exit-code`, and renders nothing that reads a file outside its scene's
directory. A stored auto-run script does not run; with auto-exec on, it does —
both asserted against real Blender 5.2.1.

### 8.9 Artifact URLs

Artifact URLs from the output registry are real Signature V4 presigned URLs
(60–3600 s), only for server-chosen keys under the owner's upload prefix, with
an upload's type and size signed. **Before this, the registry returned a
`storage.local` URL whose "signature" was base64 of the id and expiry** — a
forged signature, found and fixed by T.03.06.

### 8.10 Other defects this work found

Blender ignoring settings-script errors; job files readable across runs; hard
links chowned to the sandbox uid; `[::127.0.0.1]` allowed; downloads with no
overall deadline; a uid released after a failed sweep; whole archives held in
memory; and process-group runs hanging when macOS answered `EPERM` under load.

Mutations of 14 guards each fail their spec.

**Not claimed:** the tool inventory is not exhaustively sandboxed — the three
named tools are.
