Eve library

Bellona MVP Alpha Soak-Test Plan

Guide

Revision 1

On this page

Status: 180.C.16.13 deterministic multi-day soak-plan artifact.

Implementation date: 2026-04-23.

This plan defines how Bellona should collect MVP alpha evidence for multi-day Remote Host uptime. The intent is not a synthetic benchmark. The intent is to prove that Bellona can survive realistic long-running operator use across:

  • host sleep and wake cycles
  • LaunchAgent restart and bounded crash-loop recovery
  • gateway reconnect stability
  • audit-store growth under sustained command traffic
  • process memory drift across a 72+ hour run

Sources Reviewed#

Local implementation references:

  • testing/bellona/remote-control/mvp-alpha-soak-plan.ts
  • testing/bellona/remote-control/network-impairment.ts
  • docs/domains/bellona/extras/remote-control/network-impairment-test-plan.md
  • docs/domains/bellona/extras/remote-control/remote-host-update-channel-and-rollback-plan.md
  • docs/domains/bellona/extras/remote-control/session-timeline-and-audit-bundle.md
  • apps/bellona/remote-host/src/app.ts
  • apps/bellona/remote-host/src/adapter-registry.ts
  • apps/bellona/remote-host/src/macos-packaging-plan.ts

Existing implementation patterns analyzed:

  • The Stage 16 network-impairment plan for the already-defined sleep/wake and reconnect evidence Bellona expects later hardware runs to provide.
  • The Remote Host update and macOS packaging plans for the current LaunchAgent verification and rollback surfaces.
  • The current host health/readiness payloads and state snapshot resource telemetry for uptime and memory-pressure observations.

Run Envelope#

  • minimum duration: 72 continuous hours
  • preferred duration: 84 hours so the last day still has room for export and review without truncating the 72-hour evidence window
  • required rigs: MAC-HOST-01, MAC-OP-01, LAN-GW-01, and NET-IMPAIR-01

The soak run should remain on the real macOS host. VM-only evidence is not good enough for sleep/wake, LaunchAgent, and multi-day resource-behavior claims.

Probe Cadence#

The plan uses these recurring probes:

Probe Cadence Source Expected signal
Host health every 15m curl /health monotonic uptimeMs, stable adapter counts, expected gateway state
Host readiness every 15m curl /readiness returns ready outside planned drill windows
Low-risk command path every 30m agent.readiness, state.snapshot, or blender.scene.info through the real gateway-host path success result plus audit/timeline evidence
RSS and memory pressure every 15m ps -o rss= plus Bellona state.snapshot RSS stays within leak budget and memory pressure stays below critical
Audit-store growth every 6h store size, entry count, export size, write-failure count monotonic growth with zero write failures
Audit export validation every 24h audit export and hash-chain validation export stays valid and replayable throughout the run

Phase Plan#

1. Baseline Warmup (0h-4h)#

  • capture baseline health/readiness payloads
  • capture the hour-4 RSS baseline after the system warms up
  • capture baseline audit-store size and the initial export bundle
  • confirm the recurring read-only command probe succeeds before any fault drill

2. Day 1 Uptime (4h-24h)#

  • run two sleep/wake drills around hours 8 and 16
  • record wake-to-ready recovery time after each drill
  • perform one gateway reconnect drill after the host is healthy
  • keep the recurring probes continuous outside the planned drill windows

3. Day 2 Recovery (24h-48h)#

  • force one LaunchAgent restart around hour 28
  • repeat sleep/wake around hour 32
  • run another gateway reconnect drill around hour 36
  • keep collecting audit-growth checkpoints and RSS deltas through the second day

4. Day 3 Crash-Loop And Export (48h-72h)#

  • inject a bounded crash-loop drill: three forced exits within 10 minutes around hour 52
  • repeat sleep/wake around hours 56 and 64
  • run the final low-risk command probe and final audit export at the end of the 72-hour window
  • record the final RSS delta against the hour-4 baseline

Pass / Fail Rules#

Success requires all of the following:

  • at least 72 continuous hours of evidence
  • at least six planned sleep/wake cycles with successful post-wake probes
  • at least one forced LaunchAgent restart and one bounded crash-loop drill that recover without undocumented manual repair
  • gateway reconnect probes remain within the existing 10s reconnect hard max
  • remote_gateway.audit.write_failure.count stays at 0
  • every daily and final audit export validates successfully
  • hour-72 RSS remains within +25% of the hour-4 baseline for the same workload, and Bellona memory pressure never reaches critical

Fail fast and stop the run when any of these happen:

  • any audit write failure
  • any planned sleep/wake or LaunchAgent drill fails to return to ready within the documented threshold
  • two consecutive low-risk command probes fail outside a planned perturbation window
  • memory pressure becomes critical or RSS stays above the +25% budget for more than 12 continuous hours
  • any audit export validation or replay failure

Required Evidence Bundle#

The final soak bundle should include:

  • 72+ hour probe logs
  • sleep/wake timestamps and recovery timings
  • LaunchAgent restart and crash-loop drill logs
  • gateway reconnect latency samples
  • audit-store growth checkpoints
  • daily and final audit export bundles with validation output
  • RSS and memory-pressure trend chart
  • operator signoff note for any anomaly, even if the run is still considered a pass

Verification#

The committed automated verification for the plan artifact is:

bash
pnpm exec vitest run --config vitest.config.ts tests/mvp-alpha-soak-plan.test.ts --reporter=verbose
pnpm nx run-many -p @bellona/remote-control-testing -t lint,typecheck,test,build --skip-nx-cache --verbose

The first command proves the plan structure, duration, phases, probes, and fail-fast expectations. The second command keeps the reusable soak-plan module buildable in the Bellona remote-control testing package.