Eve library

Bellona MVP Alpha Operations Runbooks

Guide

Revision 1

On this page

Status: 180.C.16.07 alpha runbook pack.

Runbook date: 2026-04-23.

These runbooks consolidate the current Phase 180 operating procedures for MVP alpha hardening. They do not mark any live hardware, cloud, WebRTC, plugin, or uninstall behavior as production-certified. Each runbook names the automated or manual evidence that must be attached when the procedure is exercised.

Sources Reviewed#

  • docs/domains/bellona/extras/remote-control/blender-read-only-mvp-second-macbook.md
  • docs/domains/bellona/extras/remote-control/hetzner-reference-deployment.md
  • docs/domains/bellona/extras/remote-control/aws-reference-deployment.md
  • docs/domains/bellona/extras/remote-control/session-timeline-and-audit-bundle.md
  • docs/domains/bellona/extras/remote-control/macos-packaging-and-notarization-plan.md
  • docs/domains/bellona/extras/remote-control/network-impairment-test-plan.md
  • docs/domains/bellona/extras/remote-control/security-regression-test-plan.md
  • docs/domains/bellona/extras/remote-control/performance-budget-test-plan.md
  • apps/bellona/remote-host/README.md
  • apps/bellona/remote-gateway/README.md

Common Evidence Rules#

For every live run, capture:

  • repository commit SHA on every participating machine
  • operator, host, gateway, and cloud environment labels
  • absolute timestamps with timezone for start, failure, remediation, and finish
  • relevant /health, /ready, timeline, audit, artifact, and command IDs
  • redacted logs only; never paste host tokens, API keys, cookies, passwords, or personal filesystem paths into evidence
  • final status: passed, failed, blocked, or skipped-with-reason

When a runbook says "stop," do not continue the scenario until the failed check is understood and documented.

Two-MacBook LAN Setup#

Use this when MAC-OP-01 runs the gateway and MAC-HOST-01 runs Remote Host plus local adapters on the same trusted LAN.

Prerequisites:

  • same repository commit on both machines
  • pnpm install completed on both machines
  • gateway, host, and adapter ports reserved locally
  • no inbound command, browser, Blender, Unreal, or MCP port exposed on MAC-HOST-01
  • optional jq, lsof, nc, and curl

Procedure:

  1. On MAC-OP-01, find the LAN address:

    bash
    export BELLONA_OPERATOR_LAN_IP="$(ipconfig getifaddr en0 || ipconfig getifaddr en1)"
    
  2. Start the gateway with a durable first-stage store:

    bash
    mkdir -p .local/bellona
    
    BELLONA_REMOTE_GATEWAY_HOST=0.0.0.0 \
    BELLONA_REMOTE_GATEWAY_PORT=4097 \
    BELLONA_REMOTE_GATEWAY_STORE_FILE=.local/bellona/remote-gateway.lan.json \
    BELLONA_REMOTE_GATEWAY_LOG_LEVEL=debug \
    pnpm nx start @bellona/remote-gateway
    
  3. From MAC-OP-01, verify gateway readiness:

    bash
    curl -fsS "http://${BELLONA_OPERATOR_LAN_IP}:4097/health" | jq .
    curl -fsS "http://${BELLONA_OPERATOR_LAN_IP}:4097/ready" | jq .
    
  4. On MAC-HOST-01, point the host at the LAN gateway:

    bash
    export BELLONA_REMOTE_HOST_GATEWAY_URL="ws://${BELLONA_OPERATOR_LAN_IP}:4097/v1/hosts/connect"
    export BELLONA_REMOTE_HOST_PAIRING_EXCHANGE_URL="http://${BELLONA_OPERATOR_LAN_IP}:4097/v1/pairing/exchange"
    
  5. Verify local adapters before starting a host-to-gateway run. For Blender:

    bash
    pnpm nx blender:scene-info-smoke @bellona/remote-host -- --required
    
  6. Verify the routed path with the available smoke:

    bash
    pnpm nx blender:scene-info-remote-smoke @bellona/remote-host
    
  7. Confirm timeline and audit continuity from MAC-OP-01:

    bash
    curl -fsS "http://${BELLONA_OPERATOR_LAN_IP}:4097/v1/timeline?limit=200" | jq .
    

Evidence to attach:

  • LAN IP used by MAC-OP-01
  • gateway /health and /ready JSON
  • host connection/device ID and adapter capability report
  • routed smoke command ID, result ID, timeline event count, and audit entry types
  • proof that Blender/Unreal/browser/debug ports are loopback-only on MAC-HOST-01

Stop conditions:

  • any controlled-host service binds to a LAN address unexpectedly
  • gateway readiness is not ready
  • timeline lacks command receipt, dispatch, and completion/denial records

Cloud Gateway Setup#

Use this when the gateway runs on Hetzner, AWS, or another trusted cloud edge and Remote Host connects outbound over WSS 443.

Prerequisites:

  • DNS name, TLS certificate, and public HTTPS path are ready
  • gateway process binds only to localhost behind the reverse proxy or load balancer
  • public firewall/security group exposes only approved ingress such as 80/443
  • host token and pairing secrets are injected through a secret manager or local environment file, never cloud-init user data
  • TURN is documented as enabled, disabled, or blocked; do not claim TURN relay without a later WebRTC smoke

Procedure:

  1. Deploy the gateway at the documented commit.

  2. Verify public readiness:

    bash
    curl -fsS https://bellona-gateway.example.com/health | jq .
    curl -fsS https://bellona-gateway.example.com/ready | jq .
    
  3. Verify the local gateway bind on the cloud host. For a Linux VM:

    sh
    ss -ltnp | grep ':4070'
    curl -fsS http://127.0.0.1:4070/health | jq .
    
  4. Inspect recent gateway and proxy/load-balancer logs. For systemd/Caddy:

    sh
    sudo journalctl -u bellona-remote-gateway --since "10 minutes ago" --no-pager
    sudo journalctl -u caddy --since "10 minutes ago" --no-pager
    
  5. Configure the Remote Host:

    env
    BELLONA_REMOTE_HOST_GATEWAY_URL=wss://bellona-gateway.example.com/v1/hosts/connect
    BELLONA_REMOTE_HOST_PAIRING_EXCHANGE_URL=https://bellona-gateway.example.com/v1/pairing/exchange
    

Evidence to attach:

  • provider, region, VM/instance type, OS image, and commit SHA
  • DNS record and certificate identifier without private keys
  • firewall/security group rules proving the internal gateway port is not public
  • public /health and /ready JSON
  • recent gateway/proxy logs with secrets redacted
  • host-facing WSS URL and pairing URL
  • TURN status and the reason if TURN is not yet exercised

Stop conditions:

  • internal gateway port is publicly reachable
  • /ready is not ready
  • logs expose secrets
  • host requires inbound connectivity

Permissions Troubleshooting#

Use this when desktop fallback, screenshot capture, window list/focus, click, or type commands report missing macOS permissions.

Checks:

  • Screen Recording maps to desktop.observe
  • Accessibility maps to desktop.input
  • granted TCC state does not bypass gateway policy or operator approval
  • missing/unknown TCC state must be advertised as denied host permissions with remediation text

Procedure:

  1. Run host diagnostics and capture the macOS permission report:

    bash
    pnpm nx desktop-fallback-remote-smoke @bellona/remote-host
    
  2. On MAC-HOST-01, open System Settings > Privacy & Security.

  3. Grant Screen Recording to the signed Bellona Remote Host or current dev terminal host process for observe/screenshot flows.

  4. Grant Accessibility only when input/focus/type/click flows are being tested.

  5. Restart the host process after changing TCC grants.

  6. Rerun the diagnostic smoke and compare permission decisions.

Evidence to attach:

  • before/after permission report
  • command IDs blocked by missing permissions
  • exact TCC grant changed
  • confirmation that gateway policy still reports requires-approval for desktop input after the macOS grant

Stop conditions:

  • desktop input is allowed without an Accessibility grant
  • screenshot/window capture proceeds without Screen Recording
  • a test bypasses gateway policy or approval because TCC is granted

Plugin Repair#

Use this when Blender addon or Unreal plugin detection fails.

Blender repair:

  1. Confirm the addon is linked into the installed Blender version:

    bash
    BLENDER_VERSION=<installed-major-minor>
    BLENDER_ADDONS_DIR="${HOME}/Library/Application Support/Blender/${BLENDER_VERSION}/scripts/addons"
    ls -la "${BLENDER_ADDONS_DIR}/bellona_addon"
    
  2. Ensure the addon host is localhost and port is 9876.

  3. Confirm the port is loopback-only:

    bash
    lsof -nP -iTCP:9876 -sTCP:LISTEN
    nc -vz 127.0.0.1 9876
    
  4. Run:

    bash
    pnpm nx blender:scene-info-smoke @bellona/remote-host -- --required
    

Unreal repair:

  1. Confirm the checked-in plugin descriptor exists:

    bash
    test -f libs/bellona/unreal/plugin/BellonaUnrealEditor/BellonaUnrealEditor.uplugin
    
  2. Confirm the project has the plugin installed or symlinked according to the Unreal plugin architecture doc.

  3. Confirm the Remote Control endpoint remains localhost-only.

  4. Run the available deterministic remote smoke:

    bash
    pnpm nx unreal:readonly-remote-smoke @bellona/remote-host --skip-nx-cache --verbose
    

Evidence to attach:

  • Blender version, addon path, addon endpoint, and local smoke output
  • Unreal Engine version, project path, plugin descriptor path, and smoke output
  • proof that addon/plugin/debug endpoints are not LAN-exposed

Stop conditions:

  • Blender addon binds to a LAN address
  • Unreal Remote Control or plugin transport is exposed beyond localhost
  • repair requires arbitrary Python, shell execution, console commands, or policy bypass

Stream Failure#

Use this when screenshot-first stream status, WebRTC loopback, same-LAN media, or TURN media fails.

Procedure:

  1. Classify the stream path: screenshot placeholder, same-LAN WebRTC, or TURN.

  2. Record stream descriptor ID, session ID, device ID, and command ID.

  3. Check gateway timeline for stream evidence and command state:

    bash
    curl -fsS "${GATEWAY_URL}/v1/timeline?sessionId=${SESSION_ID}&kind=stream,command&limit=200" | jq .
    
  4. If same-LAN or TURN is in scope, collect ICE candidate type, selected route, startup timing, disconnect reason, and teardown status.

  5. If screenshot fallback is in scope, verify artifact registration, redaction status, and capture timing.

  6. Mark the stream degraded or failed; do not silently fall back without audit evidence.

Evidence to attach:

  • stream descriptor and timeline records
  • startup duration from the performance budget plan
  • selected path: screenshot, direct ICE, or TURN relay
  • degradation reason and user-visible state
  • teardown/cleanup result

Stop conditions:

  • media or screenshots continue after stop/revocation
  • stream evidence lacks audit/timeline records
  • TURN is claimed without a TURN-only relay smoke

Revocation#

Use this when an operator removes trust from a device or suspects host compromise.

Procedure:

  1. Pause or stop active sessions for the device if possible.
  2. Revoke the device through the gateway device revocation service or the operator surface that wraps it.
  3. Confirm the device snapshot status is blocked.
  4. Confirm active host connections were disconnected with reason device-revoked.
  5. Attempt a read-only diagnostic command and confirm policy denies it.
  6. Attempt host reconnect and confirm it is rejected or remains blocked.
  7. Export the session/device audit bundle.

Evidence to attach:

  • device ID, revocation reason, revoking participant ID
  • audit entry type device.revoked
  • disconnected connection IDs
  • denied command ID after revocation
  • failed reconnect evidence
  • audit bundle path/checksum

Stop conditions:

  • blocked device can dispatch any command
  • host can reconnect without re-pairing/explicit unblock
  • revocation audit entry is missing

Audit Export#

Use this when a session needs review, incident evidence, release evidence, or handoff to another operator.

Prerequisites:

  • gateway started with BELLONA_REMOTE_GATEWAY_STORE_FILE
  • session is stopped, paused, or otherwise quiescent when an exact export is required
  • jq, shasum, and zip installed

Procedure:

  1. Capture the timeline:

    bash
    SESSION_ID=session:example-remote-control
    GATEWAY_URL=http://127.0.0.1:4097
    STORE_FILE=.local/bellona/remote-gateway.store.json
    EXPORT_DIR=.local/bellona/audit-export/${SESSION_ID}
    
    mkdir -p "${EXPORT_DIR}"
    
    curl -fsS "${GATEWAY_URL}/v1/timeline?sessionId=${SESSION_ID}&limit=1000" \
      > "${EXPORT_DIR}/timeline.json"
    
  2. Build the audit bundle from the durable store using the procedure in session-timeline-and-audit-bundle.md.

  3. Generate a SHA-256 manifest and ZIP archive.

  4. Register the bundle as an audit-export artifact.

  5. Store the bundle in the release/incident evidence location.

Evidence to attach:

  • timeline query URL
  • bundle file path or object-store URI
  • manifest SHA-256
  • registered artifact ID
  • redaction status

Stop conditions:

  • gateway store is missing when durable audit is required
  • exported bundle contains plaintext secrets
  • manifest does not verify

Uninstall#

Use this when removing the Remote Host, LaunchAgent, helpers, addon/plugin payloads, local identity, or development fixtures from a Mac.

Current status:

  • signed installer/uninstaller and LaunchAgent lifecycle are planned in Stage 26; this runbook is a controlled development cleanup path
  • do not delete Keychain identity or audit evidence unless the operator request explicitly includes identity/evidence removal

Procedure:

  1. Stop active sessions and export needed audit bundles.

  2. Stop the development Remote Host process.

  3. If a LaunchAgent exists, unload it:

    bash
    launchctl bootout gui/$(id -u) ~/Library/LaunchAgents/com.oshun.bellona.remote-host.plist
    
  4. Remove development LaunchAgent and helper payloads only after recording their paths:

    bash
    rm -f ~/Library/LaunchAgents/com.oshun.bellona.remote-host.plist
    
  5. Remove development identity fixture files only when they are not needed:

    bash
    rm -f ~/.oshun/bellona/remote-host*.identity.dev.json
    
  6. Remove Blender addon symlinks created by the development runbook:

    bash
    BLENDER_VERSION=<installed-major-minor>
    rm -f "${HOME}/Library/Application Support/Blender/${BLENDER_VERSION}/scripts/addons/bellona_addon"
    
  7. Remove Unreal project plugin symlinks or copied plugin payloads created by the repair/install run; leave repository source files intact.

  8. Revoke the device in the gateway if the host should no longer be trusted.

  9. Optionally reset macOS TCC grants through System Settings when the operator wants permission cleanup.

Evidence to attach:

  • sessions stopped and audit bundle exported
  • LaunchAgent unload result
  • removed file paths
  • identity retention/removal decision
  • device revocation result when applicable
  • final ps, launchctl, and lsof checks showing no Remote Host process or unintended local listener remains

Stop conditions:

  • audit evidence is deleted before export
  • Keychain identity is deleted without explicit instruction
  • repository plugin source is removed instead of only installed/symlinked payloads