The V1 application platform (Oshun + Metis + Psyche — 20 application images)
runs on one Hetzner Cloud CCX23 as docker-compose stacks behind a Caddy TLS
edge. A 21st deployable image performs encrypted off-box backup when explicitly
configured. A profile-gated private monitoring plane adds Prometheus,
Alertmanager, Grafana, answer-capable probes, dependency exporters, and backup
freshness metrics. This replaces the AWS ECS Fargate + RDS + ElastiCache + S3 +
ALB stack (infra/terraform-v1, now dormant) and the earlier Amplify hosting.
GPU inference stays off-box on RunPod, unchanged.
internet ──80/443──► caddy (ACME TLS, host routing) [oshun-edge network]
│
apex ────────────► web:8080 api.* ─────────► bff:8080
admin.* ─────────► admin:8080 metis-api.* ───► metis-api-gateway:8080
tenant.* ────────► tenant-admin psyche-api.* ──► psyche-api-gateway:8000
tg.* ────────────► telegram-miniapp s3.* ──────────► minio:9000
metis.* ─────────► metis-web metis-admin.* ─► metis-admin
│
[oshun-<stack> private network]
postgres (pgvector/pg16, loopback-only host port) ── valkey 8 ── minio ── qdrant
content-service · metis-worker · psyche-* (9 internal) · pg-backup (cron)
[optional oshun-<stack> observability profile; private + loopback only]
exporters/probes ──► prometheus ──► alertmanager ──► external incident receiver
│ └───────► independent dead-man receiver
└──────► grafana (SSH tunnel)
| AWS concern | Here |
|---|---|
| ECS Fargate | docker compose project oshun-<stack> per environment |
| ALB host routing | Caddy edge (edge/), one per box, shared by all stacks |
| ECR | GHCR ghcr.io/greychimp/oshun/<service> |
| RDS Postgres | pgvector/pgvector:pg16 container + nightly pg_dump |
| ElastiCache Redis | valkey/valkey:8.1.10-alpine by digest (AOF, volatile-lru) |
| S3 | MinIO (public S3 API at s3.<domain>, presigned-URL safe) |
| Vector projection | Qdrant 1.19 (private; rebuilt from PostgreSQL at BFF boot) |
| ACM / HTTPS | Caddy automatic ACME (Let's Encrypt) |
| deploy-ecs.yml | deploy-hetzner.yml (GHCR build + SSH deploy) |
telegram-bot is intentionally absent: @oshun/telegram-bot is a library with
no server entrypoint — the live inbound path is the BFF webhook
(OSHUN_TELEGRAM_* env on bff). See "Telegram webhook" below.
1. One-time server setup#
-
Create the server: Hetzner Cloud → CCX23 (4 dedicated vCPU / 16 GB / 160 GB NVMe), Ubuntu 24.04, your SSH key. Optionally enable Hetzner server backups (highly recommended — it is the whole-box DR layer).
-
Bootstrap (as root):
bashscp infra/hetzner/scripts/setup-server.sh root@<ip>:/root/ ssh root@<ip> 'bash /root/setup-server.sh'Idempotent. Installs Docker + compose, creates the unprivileged
deployuser (docker group), ufw (22/80/443 only), a 4 GiB swapfile, fail2ban, unattended-upgrades,/opt/oshun/{bundle,edge,staging,production},/var/backups/oshun/<stack>, and the sharedoshun-edgenetwork. -
Configuration files (never committed;
chmod 600):bash# on the box cp /opt/oshun/bundle/edge/.env.example /opt/oshun/edge/.env # ACME email + domains cp /opt/oshun/bundle/.env.example /opt/oshun/staging/.env # app/runtime secrets cp /opt/oshun/bundle/backup.env.example /opt/oshun/staging/backup.env # backup-only secrets chmod 600 /opt/oshun/staging/{.env,backup.env}Before enabling observability, create three more deploy-user-owned
0640files. The two URLs must be distinct external HTTPS endpoints outside the Hetzner/platform failure domain; the dead-man service must page when its once-per-minute signal disappears:bashinstall -o deploy -g deploy -m 0640 /secure/alert-webhook-url /opt/oshun/staging/alert-webhook-url install -o deploy -g deploy -m 0640 /secure/deadman-webhook-url /opt/oshun/staging/deadman-webhook-url install -o deploy -g deploy -m 0640 /secure/grafana-admin-password /opt/oshun/staging/grafana-admin-passwordSet
OSHUN_OBSERVABILITY_ENABLED=1only after those files exist. Production deployment refuses to proceed without the profile and validated files.(The bundle lands in
/opt/oshun/bundleon the first CI deploy; for a fully manual first run, rsyncinfra/hetzner/there yourself pluslibs/oshun/persistence/prisma/→/opt/oshun/bundle/prisma/.) -
DNS: point the staging apex plus
api,metis-api,psyche-api,s3,admin,tenant,tg,metis,metis-admin(or a wildcard) at the box. ACME issuance requires resolving DNS, so do this before the first deploy. -
CI access — one key per environment, locked to its stack. Generate:
bashssh-keygen -t ed25519 -f oshun-ci-staging -N '' -C oshun-ci-staging ssh-keygen -t ed25519 -f oshun-ci-production -N '' -C oshun-ci-productionInstall each public key in
/home/deploy/.ssh/authorized_keyswith the forced-command wrappersetup-server.shput at/usr/local/lib/oshun/ssh-forced-command.sh(root-owned so CI's bundle-write access can't alter its own guardrail):textrestrict,command="/usr/local/lib/oshun/ssh-forced-command.sh staging" ssh-ed25519 AAAA... oshun-ci-staging restrict,command="/usr/local/lib/oshun/ssh-forced-command.sh production" ssh-ed25519 AAAA... oshun-ci-productionEach key can then do exactly two things: rsync the deploy bundle into
/opt/oshun/bundle, and runoshun-deploy <its own stack> …. No shells, no scp, no other paths, not the other environment — a leaked staging key cannot deploy production. (It is containment, not hard isolation: both stacks still share one Docker daemon — see §4.) Your own operator key stays unrestricted; keep it as a separateauthorized_keysline.
2. GitHub configuration#
Repository variables:
| Variable | Example |
|---|---|
STAGING_PLATFORM_DOMAIN |
staging.oshun.app |
PRODUCTION_PLATFORM_DOMAIN |
oshun.app |
HETZNER_SSH_USER (opt) |
deploy |
HETZNER_SSH_PORT (opt) |
22 |
Per environment (staging, production — plus the production-approval
environment with required reviewers, same as the ECS setup) secrets:
| Secret | Value |
|---|---|
HETZNER_HOST |
the box IP (one observed + one unobserved stack can fit; see §6) |
HETZNER_SSH_KEY |
that environment's private key from step 5 (staging ≠ production) |
HETZNER_KNOWN_HOSTS |
optional ssh-keyscan <ip> output (pins host key) |
Image pulls/pushes use the workflow's own GITHUB_TOKEN — no registry
credentials to manage. Images live at ghcr.io/greychimp/oshun/<service>.
3. Deploying#
- Staging — automatic on every push to
mainthat touches service code; only the affected services are built and rolled (shared inputs likelibs/, the Dockerfiles, or this bundle trigger a full roll). - Production — push a
v*.*.*tag or publish a release; the workflow builds everything, waits on theproduction-approvalenvironment, then deploys. - Manual —
workflow_dispatchwith environment + optional single service.
Each deploy: builds images → rsyncs this directory to /opt/oshun/bundle → runs
scripts/deploy.sh <stack> <tag> [services…] over SSH, which pulls, starts the
data tier, applies the complete ordered migration history with pinned Prisma
7.8.0 migrate deploy, rolls the services, health-gates every container
(dumping logs on failure), smokes the public hosts through Caddy, and prunes
dangling images. Migration history is authoritative because schema diffs cannot
carry custom data backfills, triggers, advisory barriers, or invariant checks.
An existing database with absent or inconsistent _prisma_migrations history
fails before rollout and must be reconciled by an operator; deploy automation
never silently baselines it.
When OSHUN_OBSERVABILITY_ENABLED=1, the same deploy also renders a stack-bound
Prometheus configuration, starts the observability profile, and includes its
containers in the health gate. Production fails before pulling or rolling when
that setting or any external-receiver prerequisite is absent. Staging may stay
disabled while endpoints are being provisioned; its deploy log says explicitly
that Task 13.7 remains unproved.
First deploy on a fresh box: ACME certificates can't exist before Caddy has
served the domains once, and docker compose ps is empty — run the CI deploy
normally; if the edge smoke fails only on certificate timing, re-run, or run
manually with SKIP_EDGE_CHECK=1:
ssh deploy@<ip> 'SKIP_EDGE_CHECK=1 /opt/oshun/bundle/scripts/deploy.sh staging staging'
Valkey cutover (T.01.08, first deploy after 2026-09-15)#
The redis service now runs Valkey 8.1.10 (BSD-3) instead of redis:7-alpine
(Redis 7.4, RSALv2 or SSPLv1). The service name, port and every REDIS_URL are
unchanged, and redis-cli becomes valkey-cli inside the container.
Valkey cannot read Redis 7.4's RDB or AOF files, so the compose file mounts
a new valkeydata volume and the old redisdata stays untouched for rollback.
The owner chose to flush rather than replicate (2026-09-15), so the first deploy
starts with an empty keyspace. What that loses: BullMQ jobs still in the queue,
idempotency and rate-limit keys, session and login state, and the day's
cost-ledger counters. Nothing there is a system of record — PostgreSQL is.
Before that deploy:
- Stop new work and let the queue drain
(
docker compose -p oshun-<stack> exec redis redis-cli LLEN bull:<queue>:waiton the old container, until it is 0). - Deploy as usual. The
rediscontainer is recreated on the new image and the new volume. - Check it:
docker compose -p oshun-<stack> exec redis valkey-cli ping→PONG, andvalkey-cli info server | grep valkey_version→8.1.10. - Users log in again where a session lived only in Redis.
To roll back, restore image: redis:7-alpine and the redisdata volume in
docker-compose.yml; the old data is still there.
Rollback#
Immutable tags (sha-XXXXXXX, release versions) stay in GHCR. To roll one
service (or several) back:
ssh deploy@<ip> '/opt/oshun/bundle/scripts/deploy.sh staging sha-abc1234 bff'
Telegram webhook#
Nothing in a deploy does this, and until it is done Telegram delivers nothing
to the stack. The whole Telegram surface — commands, room reads, reminders,
the Mini App's buttons — sits behind POST /telegram/webhook
(apps/oshun/bff/src/routes/telegram.ts), and an unregistered bot is
indistinguishable from a bot nobody has messaged. Do it once per environment,
after the BFF is live with OSHUN_TELEGRAM_BOT_TOKEN +
OSHUN_TELEGRAM_WEBHOOK_SECRET in the stack .env:
ssh deploy@<ip> '/opt/oshun/bundle/scripts/telegram-webhook.sh status staging'
ssh deploy@<ip> '/opt/oshun/bundle/scripts/telegram-webhook.sh register staging'
status prints Telegram's own getWebhookInfo — the URL it delivers to, the
pending backlog, and the last delivery error. The BFF also reports this at boot:
grep its log for telegram webhook.
One webhook per bot token. Telegram keeps a single webhook per bot, so
registering staging with production's token silently takes production's updates
and neither side logs an error. Give each stack its own bot; the stack .env
carries its own OSHUN_TELEGRAM_BOT_TOKEN for exactly that reason. With
OSHUN_TELEGRAM_WEBHOOK_URL set (docker-compose.yml sets it from
PLATFORM_DOMAIN), the boot check names that case explicitly.
The equivalent by hand, if the script is not on the box:
curl -sS "https://api.telegram.org/bot<TOKEN>/setWebhook" \
-d "url=https://api.<PLATFORM_DOMAIN>/telegram/webhook" \
-d "secret_token=<OSHUN_TELEGRAM_WEBHOOK_SECRET>"
3.x The Presentation Center on the admin origin#
The admin console serves the published Presentation Center read-only at
/presentations/files/**. It is mounted, never baked into the image: the
narration alone is 1.1 GB, and the admin image is the shared
docker/Dockerfile.web which this feature leaves untouched.
Two things make it work, and both are in this bundle:
| Where | What |
|---|---|
docker-compose.yml, admin service |
OSHUN_PRESENTATION_CENTER_DIR: /srv/presentation-center and a read-only mount of ${OSHUN_PRESENTATION_CENTER_HOST_DIR:-/srv/oshun/presentation-center} at that path |
scripts/deploy.sh, step 2b |
copies only EI.1.02's allowlist from ${OSHUN_PRESENTATION_CENTER_SOURCE:-/opt/oshun/presentation-center-source} into the host path |
deploy-hetzner.yml rsyncs docs/presentations/presentation-center/ to the
staging path — not into the bundle, which is mirrored with --delete on every
deploy and has no business carrying a gigabyte.
The allowlist is applied on the host, not only in the route. The source
directory holds the published site and everything it was built from: the
authoring notes, the drafts, the inventory, the verification records and the
content/*.json the decks are generated out of. Copying the lot and relying on
the handler to refuse it would put all of it one bug away from being served.
Copying only what is published means a bug cannot reach what was never there.
The list in deploy.sh and the one in
apps/oshun/admin/src/lib/presentation-center-files.ts are the same list and
must be changed together.
An absent center is a 503, not a crash. If the staging path is empty the
sync is skipped with a log line, the mount stays empty, and the route answers
503 {"error":"presentation_center_not_configured"} with the directory named in
its reason. Unsetting OSHUN_PRESENTATION_CENTER_DIR does the same: inside the
container there is no checkout to walk up to, so the resolver refuses rather
than guessing.
To check the rendered config on the box:
docker compose -f /opt/oshun/bundle/docker-compose.yml config |
grep -A2 -E 'OSHUN_PRESENTATION_CENTER_DIR|presentation-center'
4. Adding production (same box, one monitored stack maximum)#
Each application environment is budgeted at 6.5 GiB of steady-state container limits. Its monitoring profile adds 672 MiB, for about 7.1 GiB. On a 16 GiB CCX23, co-host at most one observed environment with one unobserved environment; never enable both monitoring profiles there. Two observed stacks require separate boxes or CCX33+. Enabling production on the same box:
cp /opt/oshun/bundle/.env.example /opt/oshun/production/.envand fill in:STACK=production,NODE_ENV=production,IMAGE_TAG=production,POSTGRES_LOOPBACK_PORT=5434, its own domain and its own secrets (never share JWT/DB/MinIO secrets across stacks).chmod 600. Production must have observability enabled; if staging already uses the monitoring profile on this 16 GiB box, disable it there first or move/resize before deploying production. Use distinct loopback ports only on a larger host that supports both profiles.- Set
OSHUN_PRODUCTION_DOMAINin/opt/oshun/edge/.envand enable the site:cp /opt/oshun/edge/sites/production.caddy.disabled /opt/oshun/edge/sites/production.caddy - Point production DNS (apex + the same 9 subdomains, or a wildcard) at the box.
- Generate a separate production CI key, install it with the
production-locked forced command (§1 step 5), and set the
productionGitHub environment'sHETZNER_HOST/HETZNER_SSH_KEY(same IP, that key). - Deploy: push a
v*.*.*tag (or dispatch → production).
Isolation honesty: the two stacks are separated at the Docker level
(projects, networks, volumes, env files) and the CI keys are stack-locked, but
they share one kernel, disk, IP, and rootful Docker daemon — this contains
accidents and leaked-key blast radius, not a determined attacker. When
production has real users, promote it to its own box: run setup-server.sh on a
second CCX23 and repoint the production environment's HETZNER_HOST. Nothing
else changes — and the failure domains separate.
5. Backups & disaster recovery#
-
Postgres — the
pg-backupsidecar dumps every database nightly (BACKUP_CRON) to/var/backups/oshun/<stack>/as checksummed custom-format dumps, pruned afterBACKUP_RETENTION_DAYS. Restore:bashsha256sum -c <dump>.sha256 docker compose -p oshun-staging exec -T postgres \ pg_restore -U oshun -d oshun --clean --if-exists < <dump> -
Local baseline — PostgreSQL dumps live on the same disk as the database; they are useful for point restores but do not survive box loss by themselves. The backup container mounts the
valkeydatavolume at/input/redis. -
Encrypted off-box snapshots —
offbox-backupis isolated from application credentials by/opt/oshun/<stack>/backup.env. When enabled it authenticates every PostgreSQL dump, forces a Redis RDB checkpoint, exports every MinIO bucket through the S3 API, and snapshots content-service run state, computer-use snapshots, stack configuration/secrets, and the deployment bundle into an encrypted restic repository in a named independent failure domain. It checks a configured data subset after each run and writes a checksummed receipt under/var/backups/oshun/<stack>/offbox-receipts. Qdrant is not copied: its personalization collection is a derived projection rebuilt from the PostgreSQL personalization authority and code-pinned index metadata. -
Capability objectives — conversations, memory, vectors, workbench, ledger, schedules, watchers, task state, and audit have RPO 90,000 s; evidence manifests have RPO 21,600 s; derived index metadata has RPO 0 because the current schema is recreated from code. Every named capability has disaster RTO 14,400 s. The authoritative per-capability table and restore acceptance procedure are in
docs/runbooks/database-backups.md. -
Post-backup deletion delta — completed account erasures are encrypted into create-only receipts and synchronously replicated to the separately configured
OSHUN_RECOVERY_DELETION_REMOTE_*S3 bucket. The bucket must be outside this box, have Object Lock enabled, and deny deletes/retention shortening to the BFF credential. The local BFF run-state copy is a cache and an additional restic input, not the only durable copy. On clean-host boot, BFF reconciles the remote inventory and replays it before exposing HTTP; remote/key/replay failure keeps the service down. Keep receipt and recovery-key retention at least as long as every backup that could contain erased data. -
Fail-closed enablement — keep
OSHUN_OFFBOX_BACKUP_ENABLED=0until the remote repository, provider object lock/append-only control, restricted writer credential, and separately escrowed recovery password are configured. Local paths, localhost rest servers, implicit repository initialization, stale or corrupt dumps, missing state classes, and unnamed immutability all fail. -
Clean-host drill — on a replacement host with no Oshun containers, run:
bashOSHUN_CLEAN_HOST_ID=<replacement-host-id> \ /opt/oshun/bundle/backup/box-loss-restore-drill.sh production \ /secure/recovered-backup.env /srv/oshun-drill \ /srv/oshun-drill/restore-proof.jsonThe drill verifies repository data and key recovery, restores PostgreSQL into clean pg16/pgvector instances, boots restored Redis, imports the MinIO export, verifies file/config state including the cached deletion-receipt census, measures RTO, and leaves BFF/autonomy stopped. Before starting BFF, configure the separately escrowed receipt key and independent remote receipt endpoint; BFF then imports any receipts newer than the restic snapshot and re-erases those subjects before traffic. Populate the restored/external inventories, start an empty Qdrant, and run the production reprojection command before admitting search:
bashdocker compose -p oshun-production \ --env-file /opt/oshun/production/.env \ -f /opt/oshun/bundle/docker-compose.yml \ run --rm bff node dist/reproject-personalization-vector-index.jsThen run
node scripts/operations/v1-restore-reconcile.mjs --input ... --output .... Autonomy may resume only when publications, provider effects, queues, approvals, invoices, artifacts, placements, and idempotency keys reconcile. -
Defense in depth — Hetzner whole-server backups/snapshots remain useful but are not a substitute for the independently encrypted snapshot or a second runtime/data failure domain.
6. Resource budget (6.5 GiB application; 7.1 GiB observed)#
| Group | Limit per env |
|---|---|
| postgres + redis + minio + pg-backup | 1,440 MiB |
| Qdrant personalization projection | 256 MiB |
| bff, content-service, metis-gw, worker | 1,376 MiB |
| 6 front-ends | 1,216 MiB |
| 10 psyche services | 2,112 MiB |
| encrypted off-box backup worker | 256 MiB |
| Application total per environment | 6,656 MiB |
| optional monitoring profile | 672 MiB |
| Observed environment total | 7,328 MiB |
minio-init is a 64 MiB one-shot initializer, so the literal all-service
startup ceiling is 6,720 MiB before it exits.
Two unobserved environments total 13 GiB of ceilings. One observed plus one unobserved environment totals 13.7 GiB, leaving tighter but usable headroom for the OS, Caddy, page cache, and one-off migration containers, plus the 4 GiB swapfile. Two observed environments total 14.3 GiB and are not supported on the 16 GiB host: move one to another box or resize before enabling the second profile.
Notes on the 6.5 GiB ceiling and the levers when load arrives:
- Limits are ceilings, not reservations — idle RSS is far below them; the budget is what bounds the worst case, not what is consumed.
- psyche
api-gatewayandorchestratorship 4 uvicorn workers in their image CMD; the compose file overrides both to 2 workers (keep the overrides in sync with the Dockerfile CMDs if those change). metis-worker's internal watchdog is set to 256 MB under its 320 MiB limit (METIS_WORKER_MAX_MEMORY_MBin the stack.envto change).- Redis
maxmemoryis 160 MB (REDIS_MAXMEMORY) under its 256 MiB limit. - Watch reality with
docker stats; if a service keeps hitting its ceiling (OOM kills show indocker inspect/compose ps), raise that one limit — and when the sums no longer fit, resize to CCX33 (32 GiB) or move production to its own box (§4).
7. Operations quick reference#
# status / logs (as deploy@box)
docker compose --profile observability -p oshun-staging \
--env-file /opt/oshun/staging/.env -f /opt/oshun/bundle/docker-compose.yml ps
docker compose --profile observability -p oshun-staging \
--env-file /opt/oshun/staging/.env -f /opt/oshun/bundle/docker-compose.yml \
logs -f --tail 100 bff prometheus alertmanager
# private operator consoles (run on the operator workstation)
ssh -N -L 3000:127.0.0.1:3000 -L 9090:127.0.0.1:9090 \
-L 9093:127.0.0.1:9093 deploy@<ip>
# psql into the stack DB (loopback publish)
psql postgresql://oshun:<pw>@127.0.0.1:5433/oshun
# reload edge after editing /opt/oshun/edge/sites/*.caddy
docker compose -p oshun-edge -f /opt/oshun/edge/docker-compose.yml \
exec caddy caddy reload --config /etc/caddy/Caddyfile
# park a stack (frees its RAM; images/volumes stay)
docker compose -p oshun-staging --env-file /opt/oshun/staging/.env \
-f /opt/oshun/bundle/docker-compose.yml stop
Alert response, dashboard interpretation, external receipt requirements, and the
guarded Task 13.7 drill are in
docs/runbooks/v1-observability-game-day.md.
The game day is not a generic smoke test: it requires an announced actor-free
window, a distinct approver, fresh external incident and dead-man receipts,
durable reversal state, and exact source identity.
Known gaps carried over from the ECS model (not introduced here):
metis-api-gateway needs METIS_PYTHON_BACKEND_URL pointed at a Metis Python
backend that is not part of the 20 V1 application images (proxied routes 502
until it exists), and apps/metis/mobile still has no release workflow.