# Oshun V1 — single-box Hetzner deployment

The V1 application platform (Oshun + Metis + Psyche — 20 application images)
runs on **one Hetzner Cloud CCX23** as docker-compose stacks behind a Caddy TLS
edge. A 21st deployable image performs encrypted off-box backup when explicitly
configured. A profile-gated private monitoring plane adds Prometheus,
Alertmanager, Grafana, answer-capable probes, dependency exporters, and backup
freshness metrics. This replaces the AWS ECS Fargate + RDS + ElastiCache + S3 +
ALB stack (`infra/terraform-v1`, now dormant) and the earlier Amplify hosting.
GPU inference stays off-box on RunPod, unchanged.

```
internet ──80/443──► caddy (ACME TLS, host routing)          [oshun-edge network]
                       │
   apex ────────────► web:8080          api.* ─────────► bff:8080
   admin.* ─────────► admin:8080        metis-api.* ───► metis-api-gateway:8080
   tenant.* ────────► tenant-admin      psyche-api.* ──► psyche-api-gateway:8000
   tg.* ────────────► telegram-miniapp  s3.* ──────────► minio:9000
   metis.* ─────────► metis-web         metis-admin.* ─► metis-admin
                       │
              [oshun-<stack> private network]
   postgres (pgvector/pg16, loopback-only host port) ── valkey 8 ── minio ── qdrant
   content-service · metis-worker · psyche-* (9 internal) · pg-backup (cron)

   [optional oshun-<stack> observability profile; private + loopback only]
   exporters/probes ──► prometheus ──► alertmanager ──► external incident receiver
                              │              └───────► independent dead-man receiver
                              └──────► grafana (SSH tunnel)
```

| AWS concern       | Here                                                        |
| ----------------- | ----------------------------------------------------------- |
| ECS Fargate       | docker compose project `oshun-<stack>` per environment      |
| ALB host routing  | Caddy edge (`edge/`), one per box, shared by all stacks     |
| ECR               | GHCR `ghcr.io/greychimp/oshun/<service>`                    |
| RDS Postgres      | `pgvector/pgvector:pg16` container + nightly pg_dump        |
| ElastiCache Redis | `valkey/valkey:8.1.10-alpine` by digest (AOF, volatile-lru) |
| S3                | MinIO (public S3 API at `s3.<domain>`, presigned-URL safe)  |
| Vector projection | Qdrant 1.19 (private; rebuilt from PostgreSQL at BFF boot)  |
| ACM / HTTPS       | Caddy automatic ACME (Let's Encrypt)                        |
| deploy-ecs.yml    | deploy-hetzner.yml (GHCR build + SSH deploy)                |

`telegram-bot` is intentionally absent: `@oshun/telegram-bot` is a library with
no server entrypoint — the live inbound path is the **BFF webhook**
(`OSHUN_TELEGRAM_*` env on `bff`). See "Telegram webhook" below.

---

## 1. One-time server setup

1. Create the server: Hetzner Cloud → **CCX23** (4 dedicated vCPU / 16 GB / 160
   GB NVMe), **Ubuntu 24.04**, your SSH key. Optionally enable Hetzner **server
   backups** (highly recommended — it is the whole-box DR layer).
2. Bootstrap (as root):

   ```bash
   scp infra/hetzner/scripts/setup-server.sh root@<ip>:/root/
   ssh root@<ip> 'bash /root/setup-server.sh'
   ```

   Idempotent. Installs Docker + compose, creates the unprivileged `deploy` user
   (docker group), ufw (22/80/443 only), a 4 GiB swapfile, fail2ban,
   unattended-upgrades, `/opt/oshun/{bundle,edge,staging,production}`,
   `/var/backups/oshun/<stack>`, and the shared `oshun-edge` network.

3. Configuration files (never committed; `chmod 600`):

   ```bash
   # on the box
   cp /opt/oshun/bundle/edge/.env.example /opt/oshun/edge/.env      # ACME email + domains
   cp /opt/oshun/bundle/.env.example        /opt/oshun/staging/.env        # app/runtime secrets
   cp /opt/oshun/bundle/backup.env.example /opt/oshun/staging/backup.env  # backup-only secrets
   chmod 600 /opt/oshun/staging/{.env,backup.env}
   ```

   Before enabling observability, create three more deploy-user-owned `0640`
   files. The two URLs must be distinct external HTTPS endpoints outside the
   Hetzner/platform failure domain; the dead-man service must page when its
   once-per-minute signal disappears:

   ```bash
   install -o deploy -g deploy -m 0640 /secure/alert-webhook-url /opt/oshun/staging/alert-webhook-url
   install -o deploy -g deploy -m 0640 /secure/deadman-webhook-url /opt/oshun/staging/deadman-webhook-url
   install -o deploy -g deploy -m 0640 /secure/grafana-admin-password /opt/oshun/staging/grafana-admin-password
   ```

   Set `OSHUN_OBSERVABILITY_ENABLED=1` only after those files exist. Production
   deployment refuses to proceed without the profile and validated files.

   (The bundle lands in `/opt/oshun/bundle` on the first CI deploy; for a fully
   manual first run, rsync `infra/hetzner/` there yourself plus
   `libs/oshun/persistence/prisma/` → `/opt/oshun/bundle/prisma/`.)

4. DNS: point the staging apex plus `api`, `metis-api`, `psyche-api`, `s3`,
   `admin`, `tenant`, `tg`, `metis`, `metis-admin` (or a wildcard) at the box.
   ACME issuance requires resolving DNS, so do this before the first deploy.
5. CI access — **one key per environment, locked to its stack**. Generate:

   ```bash
   ssh-keygen -t ed25519 -f oshun-ci-staging -N '' -C oshun-ci-staging
   ssh-keygen -t ed25519 -f oshun-ci-production -N '' -C oshun-ci-production
   ```

   Install each public key in `/home/deploy/.ssh/authorized_keys` with the
   forced-command wrapper `setup-server.sh` put at
   `/usr/local/lib/oshun/ssh-forced-command.sh` (root-owned so CI's bundle-write
   access can't alter its own guardrail):

   ```
   restrict,command="/usr/local/lib/oshun/ssh-forced-command.sh staging" ssh-ed25519 AAAA... oshun-ci-staging
   restrict,command="/usr/local/lib/oshun/ssh-forced-command.sh production" ssh-ed25519 AAAA... oshun-ci-production
   ```

   Each key can then do exactly two things: rsync the deploy bundle into
   `/opt/oshun/bundle`, and run `oshun-deploy <its own stack> …`. No shells, no
   scp, no other paths, not the other environment — a leaked staging key cannot
   deploy production. (It is containment, not hard isolation: both stacks still
   share one Docker daemon — see §4.) Your own operator key stays unrestricted;
   keep it as a separate `authorized_keys` line.

## 2. GitHub configuration

Repository **variables**:

| Variable                     | Example             |
| ---------------------------- | ------------------- |
| `STAGING_PLATFORM_DOMAIN`    | `staging.oshun.app` |
| `PRODUCTION_PLATFORM_DOMAIN` | `oshun.app`         |
| `HETZNER_SSH_USER` (opt)     | `deploy`            |
| `HETZNER_SSH_PORT` (opt)     | `22`                |

Per **environment** (`staging`, `production` — plus the `production-approval`
environment with required reviewers, same as the ECS setup) **secrets**:

| Secret                | Value                                                             |
| --------------------- | ----------------------------------------------------------------- |
| `HETZNER_HOST`        | the box IP (one observed + one unobserved stack can fit; see §6)  |
| `HETZNER_SSH_KEY`     | that environment's private key from step 5 (staging ≠ production) |
| `HETZNER_KNOWN_HOSTS` | optional `ssh-keyscan <ip>` output (pins host key)                |

Image pulls/pushes use the workflow's own `GITHUB_TOKEN` — no registry
credentials to manage. Images live at `ghcr.io/greychimp/oshun/<service>`.

## 3. Deploying

- **Staging** — automatic on every push to `main` that touches service code;
  only the **affected** services are built and rolled (shared inputs like
  `libs/`, the Dockerfiles, or this bundle trigger a full roll).
- **Production** — push a `v*.*.*` tag or publish a release; the workflow builds
  everything, waits on the `production-approval` environment, then deploys.
- **Manual** — `workflow_dispatch` with environment + optional single service.

Each deploy: builds images → rsyncs this directory to `/opt/oshun/bundle` → runs
`scripts/deploy.sh <stack> <tag> [services…]` over SSH, which pulls, starts the
data tier, applies the complete ordered migration history with pinned Prisma
7.8.0 `migrate deploy`, rolls the services, health-gates every container
(dumping logs on failure), smokes the public hosts through Caddy, and prunes
dangling images. Migration history is authoritative because schema diffs cannot
carry custom data backfills, triggers, advisory barriers, or invariant checks.
An existing database with absent or inconsistent `_prisma_migrations` history
fails before rollout and must be reconciled by an operator; deploy automation
never silently baselines it.

When `OSHUN_OBSERVABILITY_ENABLED=1`, the same deploy also renders a stack-bound
Prometheus configuration, starts the `observability` profile, and includes its
containers in the health gate. Production fails before pulling or rolling when
that setting or any external-receiver prerequisite is absent. Staging may stay
disabled while endpoints are being provisioned; its deploy log says explicitly
that Task 13.7 remains unproved.

**First deploy on a fresh box:** ACME certificates can't exist before Caddy has
served the domains once, and `docker compose ps` is empty — run the CI deploy
normally; if the edge smoke fails only on certificate timing, re-run, or run
manually with `SKIP_EDGE_CHECK=1`:

```bash
ssh deploy@<ip> 'SKIP_EDGE_CHECK=1 /opt/oshun/bundle/scripts/deploy.sh staging staging'
```

### Valkey cutover (T.01.08, first deploy after 2026-09-15)

The `redis` service now runs Valkey 8.1.10 (BSD-3) instead of `redis:7-alpine`
(Redis 7.4, RSALv2 or SSPLv1). The service name, port and every `REDIS_URL` are
unchanged, and `redis-cli` becomes `valkey-cli` inside the container.

**Valkey cannot read Redis 7.4's RDB or AOF files**, so the compose file mounts
a new `valkeydata` volume and the old `redisdata` stays untouched for rollback.
The owner chose to flush rather than replicate (2026-09-15), so the first deploy
starts with an empty keyspace. What that loses: BullMQ jobs still in the queue,
idempotency and rate-limit keys, session and login state, and the day's
cost-ledger counters. Nothing there is a system of record — PostgreSQL is.

Before that deploy:

1. Stop new work and let the queue drain
   (`docker compose -p oshun-<stack> exec redis redis-cli LLEN bull:<queue>:wait`
   on the old container, until it is 0).
2. Deploy as usual. The `redis` container is recreated on the new image and the
   new volume.
3. Check it: `docker compose -p oshun-<stack> exec redis valkey-cli ping` →
   `PONG`, and `valkey-cli info server | grep valkey_version` → `8.1.10`.
4. Users log in again where a session lived only in Redis.

To roll back, restore `image: redis:7-alpine` and the `redisdata` volume in
`docker-compose.yml`; the old data is still there.

### Rollback

Immutable tags (`sha-XXXXXXX`, release versions) stay in GHCR. To roll one
service (or several) back:

```bash
ssh deploy@<ip> '/opt/oshun/bundle/scripts/deploy.sh staging sha-abc1234 bff'
```

### Telegram webhook

**Nothing in a deploy does this, and until it is done Telegram delivers nothing
to the stack.** The whole Telegram surface — commands, room reads, reminders,
the Mini App's buttons — sits behind `POST /telegram/webhook`
(`apps/oshun/bff/src/routes/telegram.ts`), and an unregistered bot is
indistinguishable from a bot nobody has messaged. Do it once per environment,
after the BFF is live with `OSHUN_TELEGRAM_BOT_TOKEN` +
`OSHUN_TELEGRAM_WEBHOOK_SECRET` in the stack `.env`:

```bash
ssh deploy@<ip> '/opt/oshun/bundle/scripts/telegram-webhook.sh status   staging'
ssh deploy@<ip> '/opt/oshun/bundle/scripts/telegram-webhook.sh register staging'
```

`status` prints Telegram's own `getWebhookInfo` — the URL it delivers to, the
pending backlog, and the last delivery error. The BFF also reports this at boot:
grep its log for `telegram webhook`.

**One webhook per bot token.** Telegram keeps a single webhook per bot, so
registering staging with production's token silently takes production's updates
and neither side logs an error. Give each stack its own bot; the stack `.env`
carries its own `OSHUN_TELEGRAM_BOT_TOKEN` for exactly that reason. With
`OSHUN_TELEGRAM_WEBHOOK_URL` set (docker-compose.yml sets it from
`PLATFORM_DOMAIN`), the boot check names that case explicitly.

The equivalent by hand, if the script is not on the box:

```bash
curl -sS "https://api.telegram.org/bot<TOKEN>/setWebhook" \
  -d "url=https://api.<PLATFORM_DOMAIN>/telegram/webhook" \
  -d "secret_token=<OSHUN_TELEGRAM_WEBHOOK_SECRET>"
```

### 3.x The Presentation Center on the admin origin

The admin console serves the published Presentation Center read-only at
`/presentations/files/**`. It is **mounted, never baked into the image**: the
narration alone is **1.1 GB**, and the admin image is the shared
`docker/Dockerfile.web` which this feature leaves untouched.

Two things make it work, and both are in this bundle:

| Where                                 | What                                                                                                                                                                    |
| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `docker-compose.yml`, `admin` service | `OSHUN_PRESENTATION_CENTER_DIR: /srv/presentation-center` and a read-only mount of `${OSHUN_PRESENTATION_CENTER_HOST_DIR:-/srv/oshun/presentation-center}` at that path |
| `scripts/deploy.sh`, step 2b          | copies **only EI.1.02's allowlist** from `${OSHUN_PRESENTATION_CENTER_SOURCE:-/opt/oshun/presentation-center-source}` into the host path                                |

`deploy-hetzner.yml` rsyncs `docs/presentations/presentation-center/` to the
staging path — not into the bundle, which is mirrored with `--delete` on every
deploy and has no business carrying a gigabyte.

**The allowlist is applied on the host, not only in the route.** The source
directory holds the published site _and_ everything it was built from: the
authoring notes, the drafts, the inventory, the verification records and the
`content/*.json` the decks are generated out of. Copying the lot and relying on
the handler to refuse it would put all of it one bug away from being served.
Copying only what is published means a bug cannot reach what was never there.
The list in `deploy.sh` and the one in
`apps/oshun/admin/src/lib/presentation-center-files.ts` are the same list and
must be changed together.

**An absent center is a 503, not a crash.** If the staging path is empty the
sync is skipped with a log line, the mount stays empty, and the route answers
`503 {"error":"presentation_center_not_configured"}` with the directory named in
its reason. Unsetting `OSHUN_PRESENTATION_CENTER_DIR` does the same: inside the
container there is no checkout to walk up to, so the resolver refuses rather
than guessing.

To check the rendered config on the box:

```bash
docker compose -f /opt/oshun/bundle/docker-compose.yml config |
  grep -A2 -E 'OSHUN_PRESENTATION_CENTER_DIR|presentation-center'
```

## 4. Adding production (same box, one monitored stack maximum)

Each application environment is budgeted at **6.5 GiB** of steady-state
container limits. Its monitoring profile adds **672 MiB**, for about **7.1
GiB**. On a 16 GiB CCX23, co-host at most one observed environment with one
unobserved environment; never enable both monitoring profiles there. Two
observed stacks require separate boxes or CCX33+. Enabling production on the
same box:

1. `cp /opt/oshun/bundle/.env.example /opt/oshun/production/.env` and fill in:
   `STACK=production`, `NODE_ENV=production`, `IMAGE_TAG=production`,
   `POSTGRES_LOOPBACK_PORT=5434`, its own domain and **its own secrets** (never
   share JWT/DB/MinIO secrets across stacks). `chmod 600`. Production must have
   observability enabled; if staging already uses the monitoring profile on this
   16 GiB box, disable it there first or move/resize before deploying
   production. Use distinct loopback ports only on a larger host that supports
   both profiles.
2. Set `OSHUN_PRODUCTION_DOMAIN` in `/opt/oshun/edge/.env` and enable the site:
   `cp /opt/oshun/edge/sites/production.caddy.disabled /opt/oshun/edge/sites/production.caddy`
3. Point production DNS (apex + the same 9 subdomains, or a wildcard) at the
   box.
4. Generate a **separate** production CI key, install it with the
   production-locked forced command (§1 step 5), and set the `production` GitHub
   environment's `HETZNER_HOST`/`HETZNER_SSH_KEY` (same IP, that key).
5. Deploy: push a `v*.*.*` tag (or dispatch → production).

**Isolation honesty:** the two stacks are separated at the Docker level
(projects, networks, volumes, env files) and the CI keys are stack-locked, but
they share one kernel, disk, IP, and rootful Docker daemon — this contains
accidents and leaked-key blast radius, not a determined attacker. When
production has real users, promote it to its own box: run `setup-server.sh` on a
second CCX23 and repoint the `production` environment's `HETZNER_HOST`. Nothing
else changes — and the failure domains separate.

## 5. Backups & disaster recovery

- **Postgres** — the `pg-backup` sidecar dumps _every_ database nightly
  (`BACKUP_CRON`) to `/var/backups/oshun/<stack>/` as checksummed custom-format
  dumps, pruned after `BACKUP_RETENTION_DAYS`. Restore:

  ```bash
  sha256sum -c <dump>.sha256
  docker compose -p oshun-staging exec -T postgres \
    pg_restore -U oshun -d oshun --clean --if-exists < <dump>
  ```

- **Local baseline** — PostgreSQL dumps live on the same disk as the database;
  they are useful for point restores but do not survive box loss by themselves.
  The backup container mounts the `valkeydata` volume at `/input/redis`.
- **Encrypted off-box snapshots** — `offbox-backup` is isolated from application
  credentials by `/opt/oshun/<stack>/backup.env`. When enabled it authenticates
  every PostgreSQL dump, forces a Redis RDB checkpoint, exports every MinIO
  bucket through the S3 API, and snapshots content-service run state,
  computer-use snapshots, stack configuration/secrets, and the deployment bundle
  into an encrypted restic repository in a named independent failure domain. It
  checks a configured data subset after each run and writes a checksummed
  receipt under `/var/backups/oshun/<stack>/offbox-receipts`. Qdrant is not
  copied: its personalization collection is a derived projection rebuilt from
  the PostgreSQL personalization authority and code-pinned index metadata.
- **Capability objectives** — conversations, memory, vectors, workbench, ledger,
  schedules, watchers, task state, and audit have RPO 90,000 s; evidence
  manifests have RPO 21,600 s; derived index metadata has RPO 0 because the
  current schema is recreated from code. Every named capability has disaster RTO
  14,400 s. The authoritative per-capability table and restore acceptance
  procedure are in `docs/runbooks/database-backups.md`.
- **Post-backup deletion delta** — completed account erasures are encrypted into
  create-only receipts and synchronously replicated to the separately configured
  `OSHUN_RECOVERY_DELETION_REMOTE_*` S3 bucket. The bucket must be outside this
  box, have Object Lock enabled, and deny deletes/retention shortening to the
  BFF credential. The local BFF run-state copy is a cache and an additional
  restic input, not the only durable copy. On clean-host boot, BFF reconciles
  the remote inventory and replays it before exposing HTTP; remote/key/replay
  failure keeps the service down. Keep receipt and recovery-key retention at
  least as long as every backup that could contain erased data.
- **Fail-closed enablement** — keep `OSHUN_OFFBOX_BACKUP_ENABLED=0` until the
  remote repository, provider object lock/append-only control, restricted writer
  credential, and separately escrowed recovery password are configured. Local
  paths, localhost rest servers, implicit repository initialization, stale or
  corrupt dumps, missing state classes, and unnamed immutability all fail.
- **Clean-host drill** — on a replacement host with no Oshun containers, run:

  ```bash
  OSHUN_CLEAN_HOST_ID=<replacement-host-id> \
    /opt/oshun/bundle/backup/box-loss-restore-drill.sh production \
    /secure/recovered-backup.env /srv/oshun-drill \
    /srv/oshun-drill/restore-proof.json
  ```

  The drill verifies repository data and key recovery, restores PostgreSQL into
  clean pg16/pgvector instances, boots restored Redis, imports the MinIO export,
  verifies file/config state including the cached deletion-receipt census,
  measures RTO, and leaves BFF/autonomy stopped. Before starting BFF, configure
  the separately escrowed receipt key and independent remote receipt endpoint;
  BFF then imports any receipts newer than the restic snapshot and re-erases
  those subjects before traffic. Populate the restored/external inventories,
  start an empty Qdrant, and run the production reprojection command before
  admitting search:

  ```bash
  docker compose -p oshun-production \
    --env-file /opt/oshun/production/.env \
    -f /opt/oshun/bundle/docker-compose.yml \
    run --rm bff node dist/reproject-personalization-vector-index.js
  ```

  Then run
  `node scripts/operations/v1-restore-reconcile.mjs --input ... --output ...`.
  Autonomy may resume only when publications, provider effects, queues,
  approvals, invoices, artifacts, placements, and idempotency keys reconcile.

- **Defense in depth** — Hetzner whole-server backups/snapshots remain useful
  but are not a substitute for the independently encrypted snapshot or a second
  runtime/data failure domain.

## 6. Resource budget (6.5 GiB application; 7.1 GiB observed)

| Group                                  | Limit per env |
| -------------------------------------- | ------------- |
| postgres + redis + minio + pg-backup   | 1,440 MiB     |
| Qdrant personalization projection      | 256 MiB       |
| bff, content-service, metis-gw, worker | 1,376 MiB     |
| 6 front-ends                           | 1,216 MiB     |
| 10 psyche services                     | 2,112 MiB     |
| encrypted off-box backup worker        | 256 MiB       |
| **Application total per environment**  | **6,656 MiB** |
| optional monitoring profile            | 672 MiB       |
| **Observed environment total**         | **7,328 MiB** |

`minio-init` is a 64 MiB one-shot initializer, so the literal all-service
startup ceiling is 6,720 MiB before it exits.

Two unobserved environments total 13 GiB of ceilings. One observed plus one
unobserved environment totals 13.7 GiB, leaving tighter but usable headroom for
the OS, Caddy, page cache, and one-off migration containers, plus the 4 GiB
swapfile. Two observed environments total 14.3 GiB and are not supported on the
16 GiB host: move one to another box or resize before enabling the second
profile.

Notes on the 6.5 GiB ceiling and the levers when load arrives:

- Limits are **ceilings, not reservations** — idle RSS is far below them; the
  budget is what bounds the worst case, not what is consumed.
- psyche `api-gateway` and `orchestrator` ship 4 uvicorn workers in their image
  CMD; the compose file overrides both to **2 workers** (keep the overrides in
  sync with the Dockerfile CMDs if those change).
- `metis-worker`'s internal watchdog is set to 256 MB under its 320 MiB limit
  (`METIS_WORKER_MAX_MEMORY_MB` in the stack `.env` to change).
- Redis `maxmemory` is 160 MB (`REDIS_MAXMEMORY`) under its 256 MiB limit.
- Watch reality with `docker stats`; if a service keeps hitting its ceiling (OOM
  kills show in `docker inspect` / `compose ps`), raise that one limit — and
  when the sums no longer fit, resize to CCX33 (32 GiB) or move production to
  its own box (§4).

## 7. Operations quick reference

```bash
# status / logs (as deploy@box)
docker compose --profile observability -p oshun-staging \
  --env-file /opt/oshun/staging/.env -f /opt/oshun/bundle/docker-compose.yml ps
docker compose --profile observability -p oshun-staging \
  --env-file /opt/oshun/staging/.env -f /opt/oshun/bundle/docker-compose.yml \
  logs -f --tail 100 bff prometheus alertmanager

# private operator consoles (run on the operator workstation)
ssh -N -L 3000:127.0.0.1:3000 -L 9090:127.0.0.1:9090 \
  -L 9093:127.0.0.1:9093 deploy@<ip>

# psql into the stack DB (loopback publish)
psql postgresql://oshun:<pw>@127.0.0.1:5433/oshun

# reload edge after editing /opt/oshun/edge/sites/*.caddy
docker compose -p oshun-edge -f /opt/oshun/edge/docker-compose.yml \
  exec caddy caddy reload --config /etc/caddy/Caddyfile

# park a stack (frees its RAM; images/volumes stay)
docker compose -p oshun-staging --env-file /opt/oshun/staging/.env \
  -f /opt/oshun/bundle/docker-compose.yml stop
```

Alert response, dashboard interpretation, external receipt requirements, and the
guarded Task 13.7 drill are in
[`docs/runbooks/v1-observability-game-day.md`](../../docs/runbooks/v1-observability-game-day.md).
The game day is not a generic smoke test: it requires an announced actor-free
window, a distinct approver, fresh external incident and dead-man receipts,
durable reversal state, and exact source identity.

Known gaps carried over from the ECS model (not introduced here):
`metis-api-gateway` needs `METIS_PYTHON_BACKEND_URL` pointed at a Metis Python
backend that is not part of the 20 V1 application images (proxied routes 502
until it exists), and `apps/metis/mobile` still has no release workflow.
