# Bellona MVP Alpha Operations Runbooks

Status: `180.C.16.07` alpha runbook pack.

Runbook date: 2026-04-23.

These runbooks consolidate the current Phase 180 operating procedures for MVP
alpha hardening. They do not mark any live hardware, cloud, WebRTC, plugin, or
uninstall behavior as production-certified. Each runbook names the automated or
manual evidence that must be attached when the procedure is exercised.

## Sources Reviewed

- `docs/domains/bellona/extras/remote-control/blender-read-only-mvp-second-macbook.md`
- `docs/domains/bellona/extras/remote-control/hetzner-reference-deployment.md`
- `docs/domains/bellona/extras/remote-control/aws-reference-deployment.md`
- `docs/domains/bellona/extras/remote-control/session-timeline-and-audit-bundle.md`
- `docs/domains/bellona/extras/remote-control/macos-packaging-and-notarization-plan.md`
- `docs/domains/bellona/extras/remote-control/network-impairment-test-plan.md`
- `docs/domains/bellona/extras/remote-control/security-regression-test-plan.md`
- `docs/domains/bellona/extras/remote-control/performance-budget-test-plan.md`
- `apps/bellona/remote-host/README.md`
- `apps/bellona/remote-gateway/README.md`

## Common Evidence Rules

For every live run, capture:

- repository commit SHA on every participating machine
- operator, host, gateway, and cloud environment labels
- absolute timestamps with timezone for start, failure, remediation, and finish
- relevant `/health`, `/ready`, timeline, audit, artifact, and command IDs
- redacted logs only; never paste host tokens, API keys, cookies, passwords, or
  personal filesystem paths into evidence
- final status: `passed`, `failed`, `blocked`, or `skipped-with-reason`

When a runbook says "stop," do not continue the scenario until the failed check
is understood and documented.

## Two-MacBook LAN Setup

Use this when `MAC-OP-01` runs the gateway and `MAC-HOST-01` runs Remote Host
plus local adapters on the same trusted LAN.

Prerequisites:

- same repository commit on both machines
- `pnpm install` completed on both machines
- gateway, host, and adapter ports reserved locally
- no inbound command, browser, Blender, Unreal, or MCP port exposed on
  `MAC-HOST-01`
- optional `jq`, `lsof`, `nc`, and `curl`

Procedure:

1. On `MAC-OP-01`, find the LAN address:

   ```bash
   export BELLONA_OPERATOR_LAN_IP="$(ipconfig getifaddr en0 || ipconfig getifaddr en1)"
   ```

2. Start the gateway with a durable first-stage store:

   ```bash
   mkdir -p .local/bellona

   BELLONA_REMOTE_GATEWAY_HOST=0.0.0.0 \
   BELLONA_REMOTE_GATEWAY_PORT=4097 \
   BELLONA_REMOTE_GATEWAY_STORE_FILE=.local/bellona/remote-gateway.lan.json \
   BELLONA_REMOTE_GATEWAY_LOG_LEVEL=debug \
   pnpm nx start @bellona/remote-gateway
   ```

3. From `MAC-OP-01`, verify gateway readiness:

   ```bash
   curl -fsS "http://${BELLONA_OPERATOR_LAN_IP}:4097/health" | jq .
   curl -fsS "http://${BELLONA_OPERATOR_LAN_IP}:4097/ready" | jq .
   ```

4. On `MAC-HOST-01`, point the host at the LAN gateway:

   ```bash
   export BELLONA_REMOTE_HOST_GATEWAY_URL="ws://${BELLONA_OPERATOR_LAN_IP}:4097/v1/hosts/connect"
   export BELLONA_REMOTE_HOST_PAIRING_EXCHANGE_URL="http://${BELLONA_OPERATOR_LAN_IP}:4097/v1/pairing/exchange"
   ```

5. Verify local adapters before starting a host-to-gateway run. For Blender:

   ```bash
   pnpm nx blender:scene-info-smoke @bellona/remote-host -- --required
   ```

6. Verify the routed path with the available smoke:

   ```bash
   pnpm nx blender:scene-info-remote-smoke @bellona/remote-host
   ```

7. Confirm timeline and audit continuity from `MAC-OP-01`:

   ```bash
   curl -fsS "http://${BELLONA_OPERATOR_LAN_IP}:4097/v1/timeline?limit=200" | jq .
   ```

Evidence to attach:

- LAN IP used by `MAC-OP-01`
- gateway `/health` and `/ready` JSON
- host connection/device ID and adapter capability report
- routed smoke command ID, result ID, timeline event count, and audit entry
  types
- proof that Blender/Unreal/browser/debug ports are loopback-only on
  `MAC-HOST-01`

Stop conditions:

- any controlled-host service binds to a LAN address unexpectedly
- gateway readiness is not `ready`
- timeline lacks command receipt, dispatch, and completion/denial records

## Cloud Gateway Setup

Use this when the gateway runs on Hetzner, AWS, or another trusted cloud edge
and Remote Host connects outbound over WSS 443.

Prerequisites:

- DNS name, TLS certificate, and public HTTPS path are ready
- gateway process binds only to localhost behind the reverse proxy or load
  balancer
- public firewall/security group exposes only approved ingress such as 80/443
- host token and pairing secrets are injected through a secret manager or local
  environment file, never cloud-init user data
- TURN is documented as enabled, disabled, or blocked; do not claim TURN relay
  without a later WebRTC smoke

Procedure:

1. Deploy the gateway at the documented commit.
2. Verify public readiness:

   ```bash
   curl -fsS https://bellona-gateway.example.com/health | jq .
   curl -fsS https://bellona-gateway.example.com/ready | jq .
   ```

3. Verify the local gateway bind on the cloud host. For a Linux VM:

   ```sh
   ss -ltnp | grep ':4070'
   curl -fsS http://127.0.0.1:4070/health | jq .
   ```

4. Inspect recent gateway and proxy/load-balancer logs. For systemd/Caddy:

   ```sh
   sudo journalctl -u bellona-remote-gateway --since "10 minutes ago" --no-pager
   sudo journalctl -u caddy --since "10 minutes ago" --no-pager
   ```

5. Configure the Remote Host:

   ```env
   BELLONA_REMOTE_HOST_GATEWAY_URL=wss://bellona-gateway.example.com/v1/hosts/connect
   BELLONA_REMOTE_HOST_PAIRING_EXCHANGE_URL=https://bellona-gateway.example.com/v1/pairing/exchange
   ```

Evidence to attach:

- provider, region, VM/instance type, OS image, and commit SHA
- DNS record and certificate identifier without private keys
- firewall/security group rules proving the internal gateway port is not public
- public `/health` and `/ready` JSON
- recent gateway/proxy logs with secrets redacted
- host-facing WSS URL and pairing URL
- TURN status and the reason if TURN is not yet exercised

Stop conditions:

- internal gateway port is publicly reachable
- `/ready` is not ready
- logs expose secrets
- host requires inbound connectivity

## Permissions Troubleshooting

Use this when desktop fallback, screenshot capture, window list/focus, click, or
type commands report missing macOS permissions.

Checks:

- Screen Recording maps to `desktop.observe`
- Accessibility maps to `desktop.input`
- granted TCC state does not bypass gateway policy or operator approval
- missing/unknown TCC state must be advertised as denied host permissions with
  remediation text

Procedure:

1. Run host diagnostics and capture the macOS permission report:

   ```bash
   pnpm nx desktop-fallback-remote-smoke @bellona/remote-host
   ```

2. On `MAC-HOST-01`, open System Settings > Privacy & Security.
3. Grant Screen Recording to the signed Bellona Remote Host or current dev
   terminal host process for observe/screenshot flows.
4. Grant Accessibility only when input/focus/type/click flows are being tested.
5. Restart the host process after changing TCC grants.
6. Rerun the diagnostic smoke and compare permission decisions.

Evidence to attach:

- before/after permission report
- command IDs blocked by missing permissions
- exact TCC grant changed
- confirmation that gateway policy still reports `requires-approval` for desktop
  input after the macOS grant

Stop conditions:

- desktop input is allowed without an Accessibility grant
- screenshot/window capture proceeds without Screen Recording
- a test bypasses gateway policy or approval because TCC is granted

## Plugin Repair

Use this when Blender addon or Unreal plugin detection fails.

Blender repair:

1. Confirm the addon is linked into the installed Blender version:

   ```bash
   BLENDER_VERSION=<installed-major-minor>
   BLENDER_ADDONS_DIR="${HOME}/Library/Application Support/Blender/${BLENDER_VERSION}/scripts/addons"
   ls -la "${BLENDER_ADDONS_DIR}/bellona_addon"
   ```

2. Ensure the addon host is `localhost` and port is `9876`.
3. Confirm the port is loopback-only:

   ```bash
   lsof -nP -iTCP:9876 -sTCP:LISTEN
   nc -vz 127.0.0.1 9876
   ```

4. Run:

   ```bash
   pnpm nx blender:scene-info-smoke @bellona/remote-host -- --required
   ```

Unreal repair:

1. Confirm the checked-in plugin descriptor exists:

   ```bash
   test -f libs/bellona/unreal/plugin/BellonaUnrealEditor/BellonaUnrealEditor.uplugin
   ```

2. Confirm the project has the plugin installed or symlinked according to the
   Unreal plugin architecture doc.
3. Confirm the Remote Control endpoint remains localhost-only.
4. Run the available deterministic remote smoke:

   ```bash
   pnpm nx unreal:readonly-remote-smoke @bellona/remote-host --skip-nx-cache --verbose
   ```

Evidence to attach:

- Blender version, addon path, addon endpoint, and local smoke output
- Unreal Engine version, project path, plugin descriptor path, and smoke output
- proof that addon/plugin/debug endpoints are not LAN-exposed

Stop conditions:

- Blender addon binds to a LAN address
- Unreal Remote Control or plugin transport is exposed beyond localhost
- repair requires arbitrary Python, shell execution, console commands, or policy
  bypass

## Stream Failure

Use this when screenshot-first stream status, WebRTC loopback, same-LAN media,
or TURN media fails.

Procedure:

1. Classify the stream path: screenshot placeholder, same-LAN WebRTC, or TURN.
2. Record stream descriptor ID, session ID, device ID, and command ID.
3. Check gateway timeline for stream evidence and command state:

   ```bash
   curl -fsS "${GATEWAY_URL}/v1/timeline?sessionId=${SESSION_ID}&kind=stream,command&limit=200" | jq .
   ```

4. If same-LAN or TURN is in scope, collect ICE candidate type, selected route,
   startup timing, disconnect reason, and teardown status.
5. If screenshot fallback is in scope, verify artifact registration, redaction
   status, and capture timing.
6. Mark the stream degraded or failed; do not silently fall back without audit
   evidence.

Evidence to attach:

- stream descriptor and timeline records
- startup duration from the performance budget plan
- selected path: screenshot, direct ICE, or TURN relay
- degradation reason and user-visible state
- teardown/cleanup result

Stop conditions:

- media or screenshots continue after stop/revocation
- stream evidence lacks audit/timeline records
- TURN is claimed without a TURN-only relay smoke

## Revocation

Use this when an operator removes trust from a device or suspects host
compromise.

Procedure:

1. Pause or stop active sessions for the device if possible.
2. Revoke the device through the gateway device revocation service or the
   operator surface that wraps it.
3. Confirm the device snapshot status is `blocked`.
4. Confirm active host connections were disconnected with reason
   `device-revoked`.
5. Attempt a read-only diagnostic command and confirm policy denies it.
6. Attempt host reconnect and confirm it is rejected or remains blocked.
7. Export the session/device audit bundle.

Evidence to attach:

- device ID, revocation reason, revoking participant ID
- audit entry type `device.revoked`
- disconnected connection IDs
- denied command ID after revocation
- failed reconnect evidence
- audit bundle path/checksum

Stop conditions:

- blocked device can dispatch any command
- host can reconnect without re-pairing/explicit unblock
- revocation audit entry is missing

## Audit Export

Use this when a session needs review, incident evidence, release evidence, or
handoff to another operator.

Prerequisites:

- gateway started with `BELLONA_REMOTE_GATEWAY_STORE_FILE`
- session is stopped, paused, or otherwise quiescent when an exact export is
  required
- `jq`, `shasum`, and `zip` installed

Procedure:

1. Capture the timeline:

   ```bash
   SESSION_ID=session:example-remote-control
   GATEWAY_URL=http://127.0.0.1:4097
   STORE_FILE=.local/bellona/remote-gateway.store.json
   EXPORT_DIR=.local/bellona/audit-export/${SESSION_ID}

   mkdir -p "${EXPORT_DIR}"

   curl -fsS "${GATEWAY_URL}/v1/timeline?sessionId=${SESSION_ID}&limit=1000" \
     > "${EXPORT_DIR}/timeline.json"
   ```

2. Build the audit bundle from the durable store using the procedure in
   `session-timeline-and-audit-bundle.md`.
3. Generate a SHA-256 manifest and ZIP archive.
4. Register the bundle as an `audit-export` artifact.
5. Store the bundle in the release/incident evidence location.

Evidence to attach:

- timeline query URL
- bundle file path or object-store URI
- manifest SHA-256
- registered artifact ID
- redaction status

Stop conditions:

- gateway store is missing when durable audit is required
- exported bundle contains plaintext secrets
- manifest does not verify

## Uninstall

Use this when removing the Remote Host, LaunchAgent, helpers, addon/plugin
payloads, local identity, or development fixtures from a Mac.

Current status:

- signed installer/uninstaller and LaunchAgent lifecycle are planned in Stage
  26; this runbook is a controlled development cleanup path
- do not delete Keychain identity or audit evidence unless the operator request
  explicitly includes identity/evidence removal

Procedure:

1. Stop active sessions and export needed audit bundles.
2. Stop the development Remote Host process.
3. If a LaunchAgent exists, unload it:

   ```bash
   launchctl bootout gui/$(id -u) ~/Library/LaunchAgents/com.oshun.bellona.remote-host.plist
   ```

4. Remove development LaunchAgent and helper payloads only after recording their
   paths:

   ```bash
   rm -f ~/Library/LaunchAgents/com.oshun.bellona.remote-host.plist
   ```

5. Remove development identity fixture files only when they are not needed:

   ```bash
   rm -f ~/.oshun/bellona/remote-host*.identity.dev.json
   ```

6. Remove Blender addon symlinks created by the development runbook:

   ```bash
   BLENDER_VERSION=<installed-major-minor>
   rm -f "${HOME}/Library/Application Support/Blender/${BLENDER_VERSION}/scripts/addons/bellona_addon"
   ```

7. Remove Unreal project plugin symlinks or copied plugin payloads created by
   the repair/install run; leave repository source files intact.
8. Revoke the device in the gateway if the host should no longer be trusted.
9. Optionally reset macOS TCC grants through System Settings when the operator
   wants permission cleanup.

Evidence to attach:

- sessions stopped and audit bundle exported
- LaunchAgent unload result
- removed file paths
- identity retention/removal decision
- device revocation result when applicable
- final `ps`, `launchctl`, and `lsof` checks showing no Remote Host process or
  unintended local listener remains

Stop conditions:

- audit evidence is deleted before export
- Keychain identity is deleted without explicit instruction
- repository plugin source is removed instead of only installed/symlinked
  payloads
