A 3D request is not one render. It is a graph of stages, each one an ordinary job, with money committed before work starts, a global slot per resource class, typed errors, and every untrusted tool run behind four boundaries. This document describes those four systems and how they fit together.
Code: libs/isis/workflows/src/three-d-capabilities/stage-graph.ts and
request-contract.ts (pure), and in
apps/isis/generation-api/src/services/three-d-stages/:
stage-orchestrator.service.ts, stage-run.store.ts, budget-policy.ts,
budget-reservation.store.ts, resource-governor.ts, stage-errors.ts,
job-events.store.ts, event-delivery.service.ts. Sandbox:
apps/isis/gpu-worker/src/sandbox/. Tracker: T.03.01–T.03.06, T.04.04.
Evidence: docs/agents/isis-3d-studio-evidence.md §§ T.03.
State of the world. The orchestrator dispatches work it cannot run today:
every 3D capability is unavailable or candidate
(capability states), so a stage job is
refused by the submit guard and the run fails with that refusal as its typed
error. The machinery below is real and tested; what it drives is not admitted
yet.
1. The graph#
parameters.threeD names stages from a fixed list of fifteen:
concept-image, geometry, texture, cleanup, remesh, uv-unwrap,
bake, lod, collision, rig, motion, retarget, scene-assembly,
render, export.
buildThreeDStageGraph turns the requested stages into a directed acyclic graph
in topological order. Each stage may appear once. The shape:
- The mesh chain is linear — geometry → cleanup → remesh → uv-unwrap → texture → bake — and each step reads the latest mesh before it, skipping stages the request did not ask for.
- LOD and collision branch off the final mesh, and collision reads the LODs when both were asked for.
- The animation chain — rig → motion → retarget — reads the final mesh.
- Scene assembly reads every asset stage; render reads the scene, or the assets when there is no scene; export reads every stage but render.
- A stage with nothing before it reads the request's source asset.
A stage is ready only when every stage it depends on has succeeded. A failed or cancelled dependency blocks it.
2. One stage, one job, one key#
A parent job carrying parameters.threeD is not rendered itself. The
orchestrator persists one run per stage and submits each ready stage as an
ordinary job through the job service.
The idempotency key is derived from the parent, the stage, the attempt and the dependency hash — the hash of what this stage's output would be made from. Every guarantee rests on that key and on the database, never on memory:
- A crash between dispatch and persistence. The run is leased before the job is submitted and the job id recorded after. A dispatcher that dies in between leaves a lease that expires; the next dispatch recomputes the same key, and the job service returns the job that already exists through its unique idempotency index. No second job is created, and no second charge.
- Duplicate and out-of-order callbacks. Every callback is an event folded
into the run under a row lock by
applyStageEvent, which returns whether the event was applied, a duplicate, stale or rejected. A callback for another job than the run's, for an earlier attempt, or after the run is terminal never changes it. A completion repeated with the same output is a duplicate; with a different output it is rejected. - A parent never completes early. Its status is recomputed from every run after each change, and it succeeds only when every stage succeeded with outputs whose bytes were measured. Anything else is running, failed or cancelled. A partial result is never a completion.
3. Steering a running request#
Steering (T.04.04) changes the request while it runs, and the rule is that it changes only what it affects.
- A stage is affected when its projection of the request changed, or when a stage it reads is affected.
- A run is superseded when what it was made from no longer matches — its dependency hash, recomputed against the steered request and the outputs it reads now.
- Only superseded runs start a new attempt. Every other run, finished or not, keeps its work; their jobs are stopped and re-attempted only where the hash moved.
The steered request is stored beside the runs, so an advance is decidable from the database alone.
4. Budgets: money is committed before work starts#
Three scopes, each with its own cap, all required:
| Scope | Cap |
|---|---|
3d-day:<UTC date> |
ISIS_3D_DAILY_BUDGET_USD — every 3D job together |
3d-tenant-day:<tenant>:<UTC date> |
ISIS_3D_TENANT_DAILY_BUDGET_USD |
3d-project:<parent job id> |
ISIS_3D_PROJECT_BUDGET_USD, lowered — never raised — by the request's own budget.maxUsd |
Every execution a parent job may pay for — a stage attempt, a controller call, a repair — reserves a conservative amount against each scope it falls under before its job is submitted, and the reservation is refused if any scope would pass its cap counting everything already committed there. Amounts are integer micro-USD.
A stage reserves the ceiling configured for it in ISIS_3D_STAGE_CEILINGS_USD
(JSON, stage to USD per attempt). A stage with no ceiling has no known price,
and its execution is refused rather than run on a guess. Nothing has a
default: a missing cap or ceiling fails closed with the reason.
4.1 What a scope counts#
| Reservation state | What it counts |
|---|---|
held |
The reserved amount — work may be running. |
awaiting_billing |
The work ended but its charge is not known yet (a failed job, a provider that bills late). The reserved amount keeps counting, so delayed billing cannot free money that may already be spent. |
settled |
The actual charge, which may exceed the reservation — recorded as an overrun and counted in full, so later reservations see it. |
released |
Nothing. Only work proven never to have started is released. |
A reservation's id is its parent, stage, attempt and purpose (stage,
controller or repair), so reserving again after a crash returns the
reservation already made instead of committing the money twice.
PostgreSQL serializes reservations by locking every scope row they touch, in
key order, for the transaction; the memory store does the same work with no
await between reading and writing.
5. Resource slots, and fairness#
Every stage job holds one slot of its resource class from dispatch until its
job is terminal, and a class never has more holders than its cap however many
parents fan out at once. The classes are gpu-image, gpu-video, gpu-3d,
cpu-dcc and controller.
A parent job holds nothing, so a graph waiting on a CPU stage or a controller call keeps no GPU slot.
When a class is full, a denied request is recorded as a waiter. A free slot
goes to the tenant whose turn it is — the one holding the fewest slots of that
class, and among those the one waiting longest. A request from a tenant whose
turn it is not is denied fair_share_wait even when a slot is free, and
keeps its place, so a tenant who fanned out a large graph cannot starve
another's first stage.
A lease is keyed by the stage run (parent, stage, attempt), so acquiring again after a crash replays it.
6. Typed errors#
Every error code a stage run can carry falls into exactly one of eleven categories, and each category names what to do next and whether trying again unchanged can help:
request_invalid, revision_stale, asset_missing, licence_rejected,
budget_exhausted, queue_timeout, backend_unavailable, output_unverified,
cancelled, blocked, worker_failed.
Each carries a retryable flag, the HTTP status an API control answers with,
and a one-sentence action. The six cases T.03.05 names are distinct from each
other and from an invalid request, an unverified output, a cancellation and a
worker failure, so no two read the same to a client. A code the catalogue does
not know is a worker failure reported with its raw code, never silently
dropped.
Codes come from three places and keep their own spelling: submit refusals and the orchestrator in lower case, the workflow executor, provider callbacks and model cache in upper case.
Behaviour worth knowing: a stage job queued past its limit is cancelled, its
slot freed and its reservation settled at zero. A stage retry keyed by the
attempt it retries answers 202 once and 200 after. A stale
expectedSequence gets 409 revision_stale.
7. Events, SSE and webhooks#
A per-parent event log, deduped with a gapless sequence, holds a snapshot of
the whole graph in each event, read under the append lock. It feeds both SSE
and signed webhooks. SSE resumes from Last-Event-ID. Webhooks go through an
outbox with each subscription's own backoff, end dead when attempts run out,
and can be redelivered under the original event id.
A defect the tests found on the way: a failure retried within the same advance was never published.
Not claimed: there has been no delivery to a real receiver, SSE polls rather than listening, and the routes do not check job ownership.
8. The sandbox#
Blender, the mesh pipeline and the upscaler open files a user supplied and run
code paths those files steer. Each runs through runSandboxed under four
boundaries.
8.1 Identity#
Not as root. A worker running as root must name an unprivileged uid range
(ISIS_SANDBOX_UID, ISIS_SANDBOX_UID_COUNT) and an ISIS_SANDBOX_GID; each
run leases a uid of its own, which cannot read the worker's
/proc/<pid>/environ or another run's files. On Linux, every process still
holding that uid after the run — one that left the process group with setsid —
is killed before the uid is released. The job directory is made 0700 and
handed to that uid, hard-linked files excepted.
A non-root worker may run tools as itself only when
ISIS_SANDBOX_ALLOW_SAME_USER=true says so: a same-user tool can read the
worker's environment through /proc, so that is for development, never for a
worker holding storage keys.
8.2 Environment#
The tool gets a fixed set of variables — PATH, locale, GPU selection, a home
and temp directory inside the job's work directory — not the worker's
environment. Storage keys, database and queue URLs never reach it. A variable
named in ISIS_SANDBOX_PASS_ENV that looks like a secret is refused.
8.3 Kernel limits#
bash sets soft and hard rlimits before execing the tool, so neither it
nor anything it starts can raise them: CPU seconds, file size, open files,
processes (RLIMIT_NPROC, which counts every process of the user — the reason
tools need their own uid) and, on Linux only, address space. A limit that
cannot be applied stops the run before the tool starts.
8.4 Watchdogs#
Resident memory of the whole process group is sampled with ps and the group is
killed past ISIS_SANDBOX_MEMORY_MB; a wall-clock limit stops it. Sampling is a
documented approximation — a spike shorter than the interval can pass it,
which is why Linux also sets RLIMIT_AS where the tool tolerates it. CUDA
reserves large address ranges, so GPU tools leave it unset and rely on the
watchdog and the container's memory limit.
8.5 Network: safeDownload#
A job names its inputs by URL, and before this every string starting with http
was fetched as given — including one pointing at the cloud metadata service, the
queue, or a service on the worker's private network, fetched with the worker's
reach and written into the job. safeDownload refuses that:
- https only unless
allowHttp; no credentials in the URL; only allowed ports; only allowlisted hosts when an allowlist is set; - every address the host resolves to is checked at connection time, inside the
socket's own
lookup— so a name that resolves to a public address when checked and a private one when connecting (DNS rebinding) is checked on the address actually used. Loopback, private, link-local (metadata), CGNAT, multicast, reserved and documentation ranges, and their IPv4-mapped IPv6 forms, are refused unless a CIDR inallowedPrivateCidrsnames them; - redirects are followed by hand, at most
maxRedirects, each hop checked again from the start; - the body is capped while it streams, and a
Content-Lengthover the cap is refused before reading; timeoutMsis a deadline for the whole download, redirects and body included, not only for a silent socket.
8.6 Archives#
An uploaded bundle is read from its central directory and every entry is judged before one byte is written:
- Names. Relative forward-slash paths only: no absolute path, drive letter,
backslash, NUL, empty,
.or..segment, and no two entries whose names differ only in case (they would overwrite each other on some file systems). A central-directory name must match its local header. - Kinds. Regular files and directories only. A symlink, device or other special entry is refused, as is an encrypted entry, a method other than stored or deflate, and ZIP64.
- Sizes. At most
maxEntriesentries andmaxTotalBytesdeclared in all; an entry whose declared expansion ratio passesmaxRatiois refused as a bomb. Inflation is capped at the declared size, so an entry that lies about its size cannot expand past it, and its CRC-32 must match. - Overlap. Two entries sharing a local header — the overlapping-file bomb — are refused.
- Writing. Files are created exclusively (
wx) under a destination that must be empty, inside directories created by this extraction, and every path is re-checked against the destination's real path.
8.7 Asset references#
A glTF names its buffers and images by URI, an OBJ names material libraries, a
material names texture files — and a loader follows those names. trimesh and
Blender read a relative path from disk and an absolute one from anywhere, so
"uri": "../../../../proc/self/environ" turns an import into a file read and an
http://169.254… URI into a fetch from inside the worker. Before any tool opens
an asset, every reference it holds is checked:
data:URIs (glTF only) are allowed — they carry their bytes inline;- any other scheme is refused;
- a path must be relative, without backslashes or NUL, and must resolve — as written and percent-decoded, following links — to an existing regular file inside the asset's root directory;
- OBJ, MTL and PLY references are judged in every reading a loader might take (the rest of the line, from any token, or one token), comments included.
That last rule is measured, not theoretical. On 2026-09-16, trimesh 4.12.2
followed five reference escapes into an exported GLB: a PLY TextureFile
named in a comment, a spaced map_Kd whose file name is the rest of its line,
a commented mtllib, a line-continued mtllib, and MTL maps resolved from the
OBJ's directory rather than the MTL's.
STL and OFF carry no references. FBX, DAE, USD and .blend are returned as
unchecked, by name, or refused when the caller asks — a format whose
references cannot be checked is never quietly treated as safe.
8.8 Blender specifically#
Blender runs with --factory-startup --disable-autoexec and
--python-exit-code, and renders nothing that reads a file outside its scene's
directory. A stored auto-run script does not run; with auto-exec on, it does —
both asserted against real Blender 5.2.1.
8.9 Artifact URLs#
Artifact URLs from the output registry are real Signature V4 presigned URLs
(60–3600 s), only for server-chosen keys under the owner's upload prefix, with
an upload's type and size signed. Before this, the registry returned a
storage.local URL whose "signature" was base64 of the id and expiry — a
forged signature, found and fixed by T.03.06.
8.10 Other defects this work found#
Blender ignoring settings-script errors; job files readable across runs; hard
links chowned to the sandbox uid; [::127.0.0.1] allowed; downloads with no
overall deadline; a uid released after a failed sweep; whole archives held in
memory; and process-group runs hanging when macOS answered EPERM under load.
Mutations of 14 guards each fail their spec.
Not claimed: the tool inventory is not exhaustively sandboxed — the three named tools are.