Eve library

Oshun V1 — single-box Hetzner deployment

Other

Revision 1

On this page

The V1 application platform (Oshun + Metis + Psyche — 20 application images) runs on one Hetzner Cloud CCX23 as docker-compose stacks behind a Caddy TLS edge. A 21st deployable image performs encrypted off-box backup when explicitly configured. A profile-gated private monitoring plane adds Prometheus, Alertmanager, Grafana, answer-capable probes, dependency exporters, and backup freshness metrics. This replaces the AWS ECS Fargate + RDS + ElastiCache + S3 + ALB stack (infra/terraform-v1, now dormant) and the earlier Amplify hosting. GPU inference stays off-box on RunPod, unchanged.

text
internet ──80/443──► caddy (ACME TLS, host routing)          [oshun-edge network]
                       │
   apex ────────────► web:8080          api.* ─────────► bff:8080
   admin.* ─────────► admin:8080        metis-api.* ───► metis-api-gateway:8080
   tenant.* ────────► tenant-admin      psyche-api.* ──► psyche-api-gateway:8000
   tg.* ────────────► telegram-miniapp  s3.* ──────────► minio:9000
   metis.* ─────────► metis-web         metis-admin.* ─► metis-admin
                       │
              [oshun-<stack> private network]
   postgres (pgvector/pg16, loopback-only host port) ── valkey 8 ── minio ── qdrant
   content-service · metis-worker · psyche-* (9 internal) · pg-backup (cron)

   [optional oshun-<stack> observability profile; private + loopback only]
   exporters/probes ──► prometheus ──► alertmanager ──► external incident receiver
                              │              └───────► independent dead-man receiver
                              └──────► grafana (SSH tunnel)
AWS concern Here
ECS Fargate docker compose project oshun-<stack> per environment
ALB host routing Caddy edge (edge/), one per box, shared by all stacks
ECR GHCR ghcr.io/greychimp/oshun/<service>
RDS Postgres pgvector/pgvector:pg16 container + nightly pg_dump
ElastiCache Redis valkey/valkey:8.1.10-alpine by digest (AOF, volatile-lru)
S3 MinIO (public S3 API at s3.<domain>, presigned-URL safe)
Vector projection Qdrant 1.19 (private; rebuilt from PostgreSQL at BFF boot)
ACM / HTTPS Caddy automatic ACME (Let's Encrypt)
deploy-ecs.yml deploy-hetzner.yml (GHCR build + SSH deploy)

telegram-bot is intentionally absent: @oshun/telegram-bot is a library with no server entrypoint — the live inbound path is the BFF webhook (OSHUN_TELEGRAM_* env on bff). See "Telegram webhook" below.


1. One-time server setup#

  1. Create the server: Hetzner Cloud → CCX23 (4 dedicated vCPU / 16 GB / 160 GB NVMe), Ubuntu 24.04, your SSH key. Optionally enable Hetzner server backups (highly recommended — it is the whole-box DR layer).

  2. Bootstrap (as root):

    bash
    scp infra/hetzner/scripts/setup-server.sh root@<ip>:/root/
    ssh root@<ip> 'bash /root/setup-server.sh'
    

    Idempotent. Installs Docker + compose, creates the unprivileged deploy user (docker group), ufw (22/80/443 only), a 4 GiB swapfile, fail2ban, unattended-upgrades, /opt/oshun/{bundle,edge,staging,production}, /var/backups/oshun/<stack>, and the shared oshun-edge network.

  3. Configuration files (never committed; chmod 600):

    bash
    # on the box
    cp /opt/oshun/bundle/edge/.env.example /opt/oshun/edge/.env      # ACME email + domains
    cp /opt/oshun/bundle/.env.example        /opt/oshun/staging/.env        # app/runtime secrets
    cp /opt/oshun/bundle/backup.env.example /opt/oshun/staging/backup.env  # backup-only secrets
    chmod 600 /opt/oshun/staging/{.env,backup.env}
    

    Before enabling observability, create three more deploy-user-owned 0640 files. The two URLs must be distinct external HTTPS endpoints outside the Hetzner/platform failure domain; the dead-man service must page when its once-per-minute signal disappears:

    bash
    install -o deploy -g deploy -m 0640 /secure/alert-webhook-url /opt/oshun/staging/alert-webhook-url
    install -o deploy -g deploy -m 0640 /secure/deadman-webhook-url /opt/oshun/staging/deadman-webhook-url
    install -o deploy -g deploy -m 0640 /secure/grafana-admin-password /opt/oshun/staging/grafana-admin-password
    

    Set OSHUN_OBSERVABILITY_ENABLED=1 only after those files exist. Production deployment refuses to proceed without the profile and validated files.

    (The bundle lands in /opt/oshun/bundle on the first CI deploy; for a fully manual first run, rsync infra/hetzner/ there yourself plus libs/oshun/persistence/prisma/ → /opt/oshun/bundle/prisma/.)

  4. DNS: point the staging apex plus api, metis-api, psyche-api, s3, admin, tenant, tg, metis, metis-admin (or a wildcard) at the box. ACME issuance requires resolving DNS, so do this before the first deploy.

  5. CI access — one key per environment, locked to its stack. Generate:

    bash
    ssh-keygen -t ed25519 -f oshun-ci-staging -N '' -C oshun-ci-staging
    ssh-keygen -t ed25519 -f oshun-ci-production -N '' -C oshun-ci-production
    

    Install each public key in /home/deploy/.ssh/authorized_keys with the forced-command wrapper setup-server.sh put at /usr/local/lib/oshun/ssh-forced-command.sh (root-owned so CI's bundle-write access can't alter its own guardrail):

    text
    restrict,command="/usr/local/lib/oshun/ssh-forced-command.sh staging" ssh-ed25519 AAAA... oshun-ci-staging
    restrict,command="/usr/local/lib/oshun/ssh-forced-command.sh production" ssh-ed25519 AAAA... oshun-ci-production
    

    Each key can then do exactly two things: rsync the deploy bundle into /opt/oshun/bundle, and run oshun-deploy <its own stack> …. No shells, no scp, no other paths, not the other environment — a leaked staging key cannot deploy production. (It is containment, not hard isolation: both stacks still share one Docker daemon — see §4.) Your own operator key stays unrestricted; keep it as a separate authorized_keys line.

2. GitHub configuration#

Repository variables:

Variable Example
STAGING_PLATFORM_DOMAIN staging.oshun.app
PRODUCTION_PLATFORM_DOMAIN oshun.app
HETZNER_SSH_USER (opt) deploy
HETZNER_SSH_PORT (opt) 22

Per environment (staging, production — plus the production-approval environment with required reviewers, same as the ECS setup) secrets:

Secret Value
HETZNER_HOST the box IP (one observed + one unobserved stack can fit; see §6)
HETZNER_SSH_KEY that environment's private key from step 5 (staging ≠ production)
HETZNER_KNOWN_HOSTS optional ssh-keyscan <ip> output (pins host key)

Image pulls/pushes use the workflow's own GITHUB_TOKEN — no registry credentials to manage. Images live at ghcr.io/greychimp/oshun/<service>.

3. Deploying#

  • Staging — automatic on every push to main that touches service code; only the affected services are built and rolled (shared inputs like libs/, the Dockerfiles, or this bundle trigger a full roll).
  • Production — push a v*.*.* tag or publish a release; the workflow builds everything, waits on the production-approval environment, then deploys.
  • Manual — workflow_dispatch with environment + optional single service.

Each deploy: builds images → rsyncs this directory to /opt/oshun/bundle → runs scripts/deploy.sh <stack> <tag> [services…] over SSH, which pulls, starts the data tier, applies the complete ordered migration history with pinned Prisma 7.8.0 migrate deploy, rolls the services, health-gates every container (dumping logs on failure), smokes the public hosts through Caddy, and prunes dangling images. Migration history is authoritative because schema diffs cannot carry custom data backfills, triggers, advisory barriers, or invariant checks. An existing database with absent or inconsistent _prisma_migrations history fails before rollout and must be reconciled by an operator; deploy automation never silently baselines it.

When OSHUN_OBSERVABILITY_ENABLED=1, the same deploy also renders a stack-bound Prometheus configuration, starts the observability profile, and includes its containers in the health gate. Production fails before pulling or rolling when that setting or any external-receiver prerequisite is absent. Staging may stay disabled while endpoints are being provisioned; its deploy log says explicitly that Task 13.7 remains unproved.

First deploy on a fresh box: ACME certificates can't exist before Caddy has served the domains once, and docker compose ps is empty — run the CI deploy normally; if the edge smoke fails only on certificate timing, re-run, or run manually with SKIP_EDGE_CHECK=1:

bash
ssh deploy@<ip> 'SKIP_EDGE_CHECK=1 /opt/oshun/bundle/scripts/deploy.sh staging staging'

Valkey cutover (T.01.08, first deploy after 2026-09-15)#

The redis service now runs Valkey 8.1.10 (BSD-3) instead of redis:7-alpine (Redis 7.4, RSALv2 or SSPLv1). The service name, port and every REDIS_URL are unchanged, and redis-cli becomes valkey-cli inside the container.

Valkey cannot read Redis 7.4's RDB or AOF files, so the compose file mounts a new valkeydata volume and the old redisdata stays untouched for rollback. The owner chose to flush rather than replicate (2026-09-15), so the first deploy starts with an empty keyspace. What that loses: BullMQ jobs still in the queue, idempotency and rate-limit keys, session and login state, and the day's cost-ledger counters. Nothing there is a system of record — PostgreSQL is.

Before that deploy:

  1. Stop new work and let the queue drain (docker compose -p oshun-<stack> exec redis redis-cli LLEN bull:<queue>:wait on the old container, until it is 0).
  2. Deploy as usual. The redis container is recreated on the new image and the new volume.
  3. Check it: docker compose -p oshun-<stack> exec redis valkey-cli ping → PONG, and valkey-cli info server | grep valkey_version → 8.1.10.
  4. Users log in again where a session lived only in Redis.

To roll back, restore image: redis:7-alpine and the redisdata volume in docker-compose.yml; the old data is still there.

Rollback#

Immutable tags (sha-XXXXXXX, release versions) stay in GHCR. To roll one service (or several) back:

bash
ssh deploy@<ip> '/opt/oshun/bundle/scripts/deploy.sh staging sha-abc1234 bff'

Telegram webhook#

Nothing in a deploy does this, and until it is done Telegram delivers nothing to the stack. The whole Telegram surface — commands, room reads, reminders, the Mini App's buttons — sits behind POST /telegram/webhook (apps/oshun/bff/src/routes/telegram.ts), and an unregistered bot is indistinguishable from a bot nobody has messaged. Do it once per environment, after the BFF is live with OSHUN_TELEGRAM_BOT_TOKEN + OSHUN_TELEGRAM_WEBHOOK_SECRET in the stack .env:

bash
ssh deploy@<ip> '/opt/oshun/bundle/scripts/telegram-webhook.sh status   staging'
ssh deploy@<ip> '/opt/oshun/bundle/scripts/telegram-webhook.sh register staging'

status prints Telegram's own getWebhookInfo — the URL it delivers to, the pending backlog, and the last delivery error. The BFF also reports this at boot: grep its log for telegram webhook.

One webhook per bot token. Telegram keeps a single webhook per bot, so registering staging with production's token silently takes production's updates and neither side logs an error. Give each stack its own bot; the stack .env carries its own OSHUN_TELEGRAM_BOT_TOKEN for exactly that reason. With OSHUN_TELEGRAM_WEBHOOK_URL set (docker-compose.yml sets it from PLATFORM_DOMAIN), the boot check names that case explicitly.

The equivalent by hand, if the script is not on the box:

bash
curl -sS "https://api.telegram.org/bot<TOKEN>/setWebhook" \
  -d "url=https://api.<PLATFORM_DOMAIN>/telegram/webhook" \
  -d "secret_token=<OSHUN_TELEGRAM_WEBHOOK_SECRET>"

3.x The Presentation Center on the admin origin#

The admin console serves the published Presentation Center read-only at /presentations/files/**. It is mounted, never baked into the image: the narration alone is 1.1 GB, and the admin image is the shared docker/Dockerfile.web which this feature leaves untouched.

Two things make it work, and both are in this bundle:

Where What
docker-compose.yml, admin service OSHUN_PRESENTATION_CENTER_DIR: /srv/presentation-center and a read-only mount of ${OSHUN_PRESENTATION_CENTER_HOST_DIR:-/srv/oshun/presentation-center} at that path
scripts/deploy.sh, step 2b copies only EI.1.02's allowlist from ${OSHUN_PRESENTATION_CENTER_SOURCE:-/opt/oshun/presentation-center-source} into the host path

deploy-hetzner.yml rsyncs docs/presentations/presentation-center/ to the staging path — not into the bundle, which is mirrored with --delete on every deploy and has no business carrying a gigabyte.

The allowlist is applied on the host, not only in the route. The source directory holds the published site and everything it was built from: the authoring notes, the drafts, the inventory, the verification records and the content/*.json the decks are generated out of. Copying the lot and relying on the handler to refuse it would put all of it one bug away from being served. Copying only what is published means a bug cannot reach what was never there. The list in deploy.sh and the one in apps/oshun/admin/src/lib/presentation-center-files.ts are the same list and must be changed together.

An absent center is a 503, not a crash. If the staging path is empty the sync is skipped with a log line, the mount stays empty, and the route answers 503 {"error":"presentation_center_not_configured"} with the directory named in its reason. Unsetting OSHUN_PRESENTATION_CENTER_DIR does the same: inside the container there is no checkout to walk up to, so the resolver refuses rather than guessing.

To check the rendered config on the box:

bash
docker compose -f /opt/oshun/bundle/docker-compose.yml config |
  grep -A2 -E 'OSHUN_PRESENTATION_CENTER_DIR|presentation-center'

4. Adding production (same box, one monitored stack maximum)#

Each application environment is budgeted at 6.5 GiB of steady-state container limits. Its monitoring profile adds 672 MiB, for about 7.1 GiB. On a 16 GiB CCX23, co-host at most one observed environment with one unobserved environment; never enable both monitoring profiles there. Two observed stacks require separate boxes or CCX33+. Enabling production on the same box:

  1. cp /opt/oshun/bundle/.env.example /opt/oshun/production/.env and fill in: STACK=production, NODE_ENV=production, IMAGE_TAG=production, POSTGRES_LOOPBACK_PORT=5434, its own domain and its own secrets (never share JWT/DB/MinIO secrets across stacks). chmod 600. Production must have observability enabled; if staging already uses the monitoring profile on this 16 GiB box, disable it there first or move/resize before deploying production. Use distinct loopback ports only on a larger host that supports both profiles.
  2. Set OSHUN_PRODUCTION_DOMAIN in /opt/oshun/edge/.env and enable the site: cp /opt/oshun/edge/sites/production.caddy.disabled /opt/oshun/edge/sites/production.caddy
  3. Point production DNS (apex + the same 9 subdomains, or a wildcard) at the box.
  4. Generate a separate production CI key, install it with the production-locked forced command (§1 step 5), and set the production GitHub environment's HETZNER_HOST/HETZNER_SSH_KEY (same IP, that key).
  5. Deploy: push a v*.*.* tag (or dispatch → production).

Isolation honesty: the two stacks are separated at the Docker level (projects, networks, volumes, env files) and the CI keys are stack-locked, but they share one kernel, disk, IP, and rootful Docker daemon — this contains accidents and leaked-key blast radius, not a determined attacker. When production has real users, promote it to its own box: run setup-server.sh on a second CCX23 and repoint the production environment's HETZNER_HOST. Nothing else changes — and the failure domains separate.

5. Backups & disaster recovery#

  • Postgres — the pg-backup sidecar dumps every database nightly (BACKUP_CRON) to /var/backups/oshun/<stack>/ as checksummed custom-format dumps, pruned after BACKUP_RETENTION_DAYS. Restore:

    bash
    sha256sum -c <dump>.sha256
    docker compose -p oshun-staging exec -T postgres \
      pg_restore -U oshun -d oshun --clean --if-exists < <dump>
    
  • Local baseline — PostgreSQL dumps live on the same disk as the database; they are useful for point restores but do not survive box loss by themselves. The backup container mounts the valkeydata volume at /input/redis.

  • Encrypted off-box snapshots — offbox-backup is isolated from application credentials by /opt/oshun/<stack>/backup.env. When enabled it authenticates every PostgreSQL dump, forces a Redis RDB checkpoint, exports every MinIO bucket through the S3 API, and snapshots content-service run state, computer-use snapshots, stack configuration/secrets, and the deployment bundle into an encrypted restic repository in a named independent failure domain. It checks a configured data subset after each run and writes a checksummed receipt under /var/backups/oshun/<stack>/offbox-receipts. Qdrant is not copied: its personalization collection is a derived projection rebuilt from the PostgreSQL personalization authority and code-pinned index metadata.

  • Capability objectives — conversations, memory, vectors, workbench, ledger, schedules, watchers, task state, and audit have RPO 90,000 s; evidence manifests have RPO 21,600 s; derived index metadata has RPO 0 because the current schema is recreated from code. Every named capability has disaster RTO 14,400 s. The authoritative per-capability table and restore acceptance procedure are in docs/runbooks/database-backups.md.

  • Post-backup deletion delta — completed account erasures are encrypted into create-only receipts and synchronously replicated to the separately configured OSHUN_RECOVERY_DELETION_REMOTE_* S3 bucket. The bucket must be outside this box, have Object Lock enabled, and deny deletes/retention shortening to the BFF credential. The local BFF run-state copy is a cache and an additional restic input, not the only durable copy. On clean-host boot, BFF reconciles the remote inventory and replays it before exposing HTTP; remote/key/replay failure keeps the service down. Keep receipt and recovery-key retention at least as long as every backup that could contain erased data.

  • Fail-closed enablement — keep OSHUN_OFFBOX_BACKUP_ENABLED=0 until the remote repository, provider object lock/append-only control, restricted writer credential, and separately escrowed recovery password are configured. Local paths, localhost rest servers, implicit repository initialization, stale or corrupt dumps, missing state classes, and unnamed immutability all fail.

  • Clean-host drill — on a replacement host with no Oshun containers, run:

    bash
    OSHUN_CLEAN_HOST_ID=<replacement-host-id> \
      /opt/oshun/bundle/backup/box-loss-restore-drill.sh production \
      /secure/recovered-backup.env /srv/oshun-drill \
      /srv/oshun-drill/restore-proof.json
    

    The drill verifies repository data and key recovery, restores PostgreSQL into clean pg16/pgvector instances, boots restored Redis, imports the MinIO export, verifies file/config state including the cached deletion-receipt census, measures RTO, and leaves BFF/autonomy stopped. Before starting BFF, configure the separately escrowed receipt key and independent remote receipt endpoint; BFF then imports any receipts newer than the restic snapshot and re-erases those subjects before traffic. Populate the restored/external inventories, start an empty Qdrant, and run the production reprojection command before admitting search:

    bash
    docker compose -p oshun-production \
      --env-file /opt/oshun/production/.env \
      -f /opt/oshun/bundle/docker-compose.yml \
      run --rm bff node dist/reproject-personalization-vector-index.js
    

    Then run node scripts/operations/v1-restore-reconcile.mjs --input ... --output .... Autonomy may resume only when publications, provider effects, queues, approvals, invoices, artifacts, placements, and idempotency keys reconcile.

  • Defense in depth — Hetzner whole-server backups/snapshots remain useful but are not a substitute for the independently encrypted snapshot or a second runtime/data failure domain.

6. Resource budget (6.5 GiB application; 7.1 GiB observed)#

Group Limit per env
postgres + redis + minio + pg-backup 1,440 MiB
Qdrant personalization projection 256 MiB
bff, content-service, metis-gw, worker 1,376 MiB
6 front-ends 1,216 MiB
10 psyche services 2,112 MiB
encrypted off-box backup worker 256 MiB
Application total per environment 6,656 MiB
optional monitoring profile 672 MiB
Observed environment total 7,328 MiB

minio-init is a 64 MiB one-shot initializer, so the literal all-service startup ceiling is 6,720 MiB before it exits.

Two unobserved environments total 13 GiB of ceilings. One observed plus one unobserved environment totals 13.7 GiB, leaving tighter but usable headroom for the OS, Caddy, page cache, and one-off migration containers, plus the 4 GiB swapfile. Two observed environments total 14.3 GiB and are not supported on the 16 GiB host: move one to another box or resize before enabling the second profile.

Notes on the 6.5 GiB ceiling and the levers when load arrives:

  • Limits are ceilings, not reservations — idle RSS is far below them; the budget is what bounds the worst case, not what is consumed.
  • psyche api-gateway and orchestrator ship 4 uvicorn workers in their image CMD; the compose file overrides both to 2 workers (keep the overrides in sync with the Dockerfile CMDs if those change).
  • metis-worker's internal watchdog is set to 256 MB under its 320 MiB limit (METIS_WORKER_MAX_MEMORY_MB in the stack .env to change).
  • Redis maxmemory is 160 MB (REDIS_MAXMEMORY) under its 256 MiB limit.
  • Watch reality with docker stats; if a service keeps hitting its ceiling (OOM kills show in docker inspect / compose ps), raise that one limit — and when the sums no longer fit, resize to CCX33 (32 GiB) or move production to its own box (§4).

7. Operations quick reference#

bash
# status / logs (as deploy@box)
docker compose --profile observability -p oshun-staging \
  --env-file /opt/oshun/staging/.env -f /opt/oshun/bundle/docker-compose.yml ps
docker compose --profile observability -p oshun-staging \
  --env-file /opt/oshun/staging/.env -f /opt/oshun/bundle/docker-compose.yml \
  logs -f --tail 100 bff prometheus alertmanager

# private operator consoles (run on the operator workstation)
ssh -N -L 3000:127.0.0.1:3000 -L 9090:127.0.0.1:9090 \
  -L 9093:127.0.0.1:9093 deploy@<ip>

# psql into the stack DB (loopback publish)
psql postgresql://oshun:<pw>@127.0.0.1:5433/oshun

# reload edge after editing /opt/oshun/edge/sites/*.caddy
docker compose -p oshun-edge -f /opt/oshun/edge/docker-compose.yml \
  exec caddy caddy reload --config /etc/caddy/Caddyfile

# park a stack (frees its RAM; images/volumes stay)
docker compose -p oshun-staging --env-file /opt/oshun/staging/.env \
  -f /opt/oshun/bundle/docker-compose.yml stop

Alert response, dashboard interpretation, external receipt requirements, and the guarded Task 13.7 drill are in docs/runbooks/v1-observability-game-day.md. The game day is not a generic smoke test: it requires an announced actor-free window, a distinct approver, fresh external incident and dead-man receipts, durable reversal state, and exact source identity.

Known gaps carried over from the ECS model (not introduced here): metis-api-gateway needs METIS_PYTHON_BACKEND_URL pointed at a Metis Python backend that is not part of the 20 V1 application images (proxied routes 502 until it exists), and apps/metis/mobile still has no release workflow.