# Hetzner Reference Deployment

Status: Phase 180.C.14.07 reference procedure.

This document defines the first Hetzner Cloud reference deployment for Bellona
Remote Gateway. It is intentionally narrower than the later production
deployment tasks. The goal for this stage is to document a repeatable gateway
deployment shape with TLS, a TURN placeholder, environment variables, logs, and
a smoke-test checklist while keeping command authority inside the Bellona
gateway path.

## Sources Reviewed

Primary references verified on 2026-04-23:

- Hetzner Cloud Firewall overview:
  <https://docs.hetzner.com/cloud/firewalls/overview>
- Hetzner Cloud Firewall FAQ: <https://docs.hetzner.com/cloud/firewalls/faq/>
- Hetzner Cloud server FAQ, including cloud-init user data:
  <https://docs.hetzner.com/cloud/servers/faq>
- Caddy Automatic HTTPS: <https://caddyserver.com/docs/automatic-https>
- Caddy `reverse_proxy` directive:
  <https://caddyserver.com/docs/caddyfile/directives/reverse_proxy>
- coturn `turnserver` reference:
  <https://github.com/coturn/coturn/wiki/turnserver>

Repo-local implementation references:

- `apps/bellona/remote-gateway/README.md`
- `apps/bellona/remote-gateway/src/config.ts`
- `apps/bellona/remote-host/README.md`
- `libs/bellona/mcp-gateway/README.md`
- `docs/domains/bellona/extras/remote-control/local-cloud-agent-simulation.md`
- `docs/domains/bellona/extras/remote-control/external-implementation-research-2026-04-21.md`

## Current Scope

This stage documents a single-node Hetzner gateway edge:

1. Hetzner Cloud VM.
2. Caddy terminates public HTTPS/WSS on ports `80` and `443`.
3. Bellona Remote Gateway binds to `127.0.0.1:4070`.
4. Remote Host connects outbound over `wss://<gateway-domain>/v1/hosts/connect`.
5. Cloud agents use the gateway-owned MCP path when a remote/cloud MCP HTTP
   transport is added. Until then, the local stdio MCP simulation remains the
   executable MCP proof from `180.C.14.06`.

Do not expose Blender, Chrome/CDP, desktop helpers, host health ports, gateway
port `4070`, or MCP stdio over the public internet. Caddy is the only public
HTTP entrypoint in this reference design.

## Baseline Topology

| Component        | Hetzner Reference Value                                  | Notes                                         |
| ---------------- | -------------------------------------------------------- | --------------------------------------------- |
| Region           | `fsn1`, `nbg1`, `hel1`, `ash`, or `hil`                  | Pick closest trusted operator/cloud-agent POP |
| OS               | Ubuntu LTS image                                         | Keep packages unattended-upgrade eligible     |
| VM size          | `cx22` or larger                                         | MVP gateway is CPU-light                      |
| Public DNS       | `bellona-gateway.example.com` A/AAAA to server addresses | Required for public ACME HTTP challenge       |
| Public ingress   | `80/tcp`, `443/tcp`, restricted `22/tcp`                 | No direct app port exposure                   |
| Gateway bind     | `127.0.0.1:4070`                                         | Reverse-proxied by Caddy                      |
| Store            | `/var/lib/bellona/remote-gateway/store.json`             | First-stage JSON store only                   |
| Logs             | `journald` for gateway and Caddy                         | Audit remains gateway-owned                   |
| TURN placeholder | coturn package/config held disabled                      | Open TURN ports only for later media testing  |

## Provisioning Notes

Use a Hetzner Cloud Firewall from the start. Hetzner Cloud Firewalls have
implicit deny for inbound traffic unless a rule allows it, while outbound
traffic remains allowed if no outbound rules are configured. Keep that default
outbound behavior for this MVP unless a later hardening task adds explicit
egress rules.

Minimum firewall rules:

| Direction | Protocol | Port           | Source              | Purpose                                 |
| --------- | -------- | -------------- | ------------------- | --------------------------------------- |
| Inbound   | TCP      | `22`           | Operator CIDR       | SSH only from trusted operator networks |
| Inbound   | TCP      | `80`           | `0.0.0.0/0`, `::/0` | ACME HTTP challenge and redirect        |
| Inbound   | TCP      | `443`          | `0.0.0.0/0`, `::/0` | HTTPS/WSS gateway edge                  |
| Inbound   | TCP/UDP  | `3478`, `5349` | Closed by default   | TURN placeholder only                   |
| Inbound   | UDP      | `49152-49252`  | Closed by default   | TURN relay placeholder only             |
| Outbound  | Any      | Any            | Default allow       | Host callbacks, package installs, ACME  |

Cloud-init can install the base packages, but do not put production secrets in
cloud-init user data. Use a secrets manager, operator-injected environment file,
or an encrypted provisioning channel for values such as
`BELLONA_REMOTE_GATEWAY_HOST_TOKEN_SIGNING_SECRET`.

## Host Packages

Install Node and pnpm according to the workspace's supported runtime. The
reference service assumes a checked-out repo at `/opt/oshun/current`.

```sh
sudo apt-get update
sudo apt-get install -y caddy git jq curl
sudo mkdir -p /opt/oshun /var/lib/bellona/remote-gateway
sudo useradd --system --home /var/lib/bellona --shell /usr/sbin/nologin bellona
sudo chown -R bellona:bellona /var/lib/bellona
```

Deploy the repo with the same package manager used by the workspace:

```sh
cd /opt/oshun/current
pnpm install --frozen-lockfile
pnpm nx build @bellona/remote-gateway --skip-nx-cache --verbose
sudo chown -R bellona:bellona /opt/oshun/current /var/lib/bellona/remote-gateway
```

The current Nx `start` target runs `tsx src/main.ts`; this reference keeps that
developer-compatible command until a later packaging task adds a production
bundle runner.

## Gateway Environment

Create `/etc/bellona/remote-gateway.env` readable by root and the service user
only.

```env
NODE_ENV=production
BELLONA_REMOTE_GATEWAY_HOST=127.0.0.1
BELLONA_REMOTE_GATEWAY_PORT=4070
BELLONA_REMOTE_GATEWAY_LOG_LEVEL=info
BELLONA_REMOTE_GATEWAY_STORE_FILE=/var/lib/bellona/remote-gateway/store.json
BELLONA_REMOTE_GATEWAY_HOST_TOKEN_SIGNING_SECRET=replace-with-32-plus-byte-secret
BELLONA_REMOTE_GATEWAY_HOST_TOKEN_TTL_MS=7776000000
BELLONA_REMOTE_GATEWAY_PAIRING_CODE_TTL_MS=600000
BELLONA_REMOTE_GATEWAY_APPROVAL_REQUEST_TTL_MS=60000
BELLONA_REMOTE_GATEWAY_HOST_HEARTBEAT_INTERVAL_MS=30000
BELLONA_REMOTE_GATEWAY_HOST_HEARTBEAT_TIMEOUT_MS=90000
BELLONA_REMOTE_GATEWAY_SHUTDOWN_GRACE_MS=5000
```

Required production changes from local defaults:

- `BELLONA_REMOTE_GATEWAY_HOST=127.0.0.1`, so only Caddy can reach the gateway.
- `BELLONA_REMOTE_GATEWAY_HOST_TOKEN_SIGNING_SECRET` must be unique, high
  entropy, at least 32 bytes, and never committed.
- `BELLONA_REMOTE_GATEWAY_STORE_FILE` must be on persistent disk and backed up
  before destructive maintenance.
- `BELLONA_REMOTE_GATEWAY_LOG_LEVEL=debug` is allowed only for bounded incident
  windows because logs include operational metadata.

## systemd Unit

Create `/etc/systemd/system/bellona-remote-gateway.service`:

```ini
[Unit]
Description=Bellona Remote Gateway
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=bellona
Group=bellona
WorkingDirectory=/opt/oshun/current
EnvironmentFile=/etc/bellona/remote-gateway.env
ExecStart=/usr/bin/env pnpm nx start @bellona/remote-gateway
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=full
ProtectHome=true
ReadWritePaths=/var/lib/bellona /opt/oshun/current

[Install]
WantedBy=multi-user.target
```

Enable and start:

```sh
sudo systemctl daemon-reload
sudo systemctl enable --now bellona-remote-gateway
sudo systemctl status bellona-remote-gateway --no-pager
```

## TLS And Reverse Proxy

Caddy is the reference TLS edge because it can activate Automatic HTTPS when a
site block uses a real domain name, and its `reverse_proxy` directive supports
HTTP and streaming requests such as WebSockets.

Create `/etc/caddy/Caddyfile`:

```caddyfile
{
	email ops@example.com
}

bellona-gateway.example.com {
	encode zstd gzip

	header {
		Strict-Transport-Security "max-age=31536000; includeSubDomains"
		X-Content-Type-Options "nosniff"
		Referrer-Policy "no-referrer"
	}

	handle /health {
		reverse_proxy 127.0.0.1:4070
	}

	handle /v1/health {
		reverse_proxy 127.0.0.1:4070
	}

	handle /ready {
		reverse_proxy 127.0.0.1:4070
	}

	handle /v1/ready {
		reverse_proxy 127.0.0.1:4070
	}

	handle /v1/* {
		reverse_proxy 127.0.0.1:4070
	}

	respond 404
}
```

Reload Caddy:

```sh
sudo caddy validate --config /etc/caddy/Caddyfile
sudo systemctl reload caddy
```

## TURN Placeholder

TURN is not activated by this `180.C.14.07` runbook because the current phase
has not implemented live WebRTC relay, signaling, TURN credentials, or
cloud-agent stream smoke coverage. Keep TURN ports closed until a later WebRTC
task explicitly opens them.

Placeholder package/config for later `coturn` work:

```sh
sudo apt-get install -y coturn
sudo systemctl disable --now coturn
```

Draft `/etc/turnserver.conf` values for later activation:

```conf
listening-port=3478
tls-listening-port=5349
min-port=49152
max-port=49252
fingerprint
lt-cred-mech
realm=bellona-gateway.example.com
server-name=bellona-gateway.example.com
no-cli
no-multicast-peers
no-loopback-peers
```

Before enabling TURN in a later task, add credential rotation, TLS certificates,
firewall openings for `3478`, `5349`, and the relay range, and a smoke that
proves TURN-only relay plus teardown.

## Logs

Gateway logs:

```sh
journalctl -u bellona-remote-gateway -f
journalctl -u bellona-remote-gateway --since "15 minutes ago" --no-pager
```

Caddy logs:

```sh
journalctl -u caddy -f
journalctl -u caddy --since "15 minutes ago" --no-pager
```

Expected gateway event names for MVP triage:

- `remote gateway listening`
- `remote host handshake accepted`
- `remote gateway command received`
- `remote gateway command dispatched`
- `remote gateway command result received`
- `remote gateway graceful shutdown failed`, only on shutdown failure

Operational logs are not the security record. Audit entries and timeline
evidence remain owned by the Remote Gateway services and persistent store.

## Smoke Test

Run this after DNS, firewall, Caddy, and the gateway service are live:

```sh
COMMIT_SHA="$(git -C /opt/oshun/current rev-parse HEAD)"
echo "Bellona Hetzner smoke commit: ${COMMIT_SHA}"

curl -fsS https://bellona-gateway.example.com/health | jq .
curl -fsS https://bellona-gateway.example.com/ready | jq .

sudo journalctl -u bellona-remote-gateway --since "5 minutes ago" --no-pager
sudo journalctl -u caddy --since "5 minutes ago" --no-pager
```

Expected results:

- `/health` returns `status: "healthy"`.
- `/ready` returns `status: "ready"` with `checks.config`,
  `checks.deviceRegistry`, and `checks.shuttingDown`.
- Gateway logs show `remote gateway listening`.
- Caddy logs show successful HTTPS requests to `/health` and `/ready`.
- No process listens publicly on `4070`.

Check local binding from the server:

```sh
ss -ltnp | grep ':4070'
curl -fsS http://127.0.0.1:4070/health | jq .
```

From the Remote Host MacBook, configure:

```env
BELLONA_REMOTE_HOST_GATEWAY_URL=wss://bellona-gateway.example.com/v1/hosts/connect
BELLONA_REMOTE_HOST_PAIRING_EXCHANGE_URL=https://bellona-gateway.example.com/v1/pairing/exchange
```

The full MVP cloud-agent smoke is deferred to `180.C.14.09`; this task's smoke
only proves the Hetzner gateway edge, TLS, readiness, logging, and host-facing
WebSocket route are in the documented path.

## Evidence To Record

For each Hetzner reference deployment trial, record:

- Hetzner region, VM type, OS image, and server id.
- Gateway public domain and DNS record type, without private secrets.
- Git commit SHA deployed from `/opt/oshun/current`.
- Cloud Firewall rules, including confirmation that `4070` is not public.
- Caddy validation output and service status.
- Gateway service status and last startup logs.
- `/health` and `/ready` JSON outputs over public HTTPS.
- `ss` output proving gateway bind is local-only.
- TURN placeholder status, including whether coturn is disabled.
- Any failed smoke command and the exact remediation used.

## Limitations

- This is not the production-ready Hetzner deployment from `180.C.25.08`.
- The current Remote Gateway still uses the first-stage JSON store, not
  Postgres/Redis/object storage.
- The current MCP package exposes stdio locally; remote/cloud MCP HTTP transport
  remains a later implementation task.
- TURN is documented as a placeholder only; no WebRTC media relay is claimed by
  this task.
- No live Blender, browser, desktop, or remote MacBook command flow is certified
  until `180.C.14.09` runs the cloud-agent MVP smoke through the deployed path.
