Status: 180.C.16.13 deterministic multi-day soak-plan artifact.
Implementation date: 2026-04-23.
This plan defines how Bellona should collect MVP alpha evidence for multi-day Remote Host uptime. The intent is not a synthetic benchmark. The intent is to prove that Bellona can survive realistic long-running operator use across:
- host sleep and wake cycles
- LaunchAgent restart and bounded crash-loop recovery
- gateway reconnect stability
- audit-store growth under sustained command traffic
- process memory drift across a 72+ hour run
Sources Reviewed#
Local implementation references:
testing/bellona/remote-control/mvp-alpha-soak-plan.tstesting/bellona/remote-control/network-impairment.tsdocs/domains/bellona/extras/remote-control/network-impairment-test-plan.mddocs/domains/bellona/extras/remote-control/remote-host-update-channel-and-rollback-plan.mddocs/domains/bellona/extras/remote-control/session-timeline-and-audit-bundle.mdapps/bellona/remote-host/src/app.tsapps/bellona/remote-host/src/adapter-registry.tsapps/bellona/remote-host/src/macos-packaging-plan.ts
Existing implementation patterns analyzed:
- The Stage 16 network-impairment plan for the already-defined sleep/wake and reconnect evidence Bellona expects later hardware runs to provide.
- The Remote Host update and macOS packaging plans for the current LaunchAgent verification and rollback surfaces.
- The current host health/readiness payloads and state snapshot resource telemetry for uptime and memory-pressure observations.
Run Envelope#
- minimum duration:
72continuous hours - preferred duration:
84hours so the last day still has room for export and review without truncating the 72-hour evidence window - required rigs:
MAC-HOST-01,MAC-OP-01,LAN-GW-01, andNET-IMPAIR-01
The soak run should remain on the real macOS host. VM-only evidence is not good enough for sleep/wake, LaunchAgent, and multi-day resource-behavior claims.
Probe Cadence#
The plan uses these recurring probes:
| Probe | Cadence | Source | Expected signal |
|---|---|---|---|
| Host health | every 15m |
curl /health |
monotonic uptimeMs, stable adapter counts, expected gateway state |
| Host readiness | every 15m |
curl /readiness |
returns ready outside planned drill windows |
| Low-risk command path | every 30m |
agent.readiness, state.snapshot, or blender.scene.info through the real gateway-host path |
success result plus audit/timeline evidence |
| RSS and memory pressure | every 15m |
ps -o rss= plus Bellona state.snapshot |
RSS stays within leak budget and memory pressure stays below critical |
| Audit-store growth | every 6h |
store size, entry count, export size, write-failure count | monotonic growth with zero write failures |
| Audit export validation | every 24h |
audit export and hash-chain validation | export stays valid and replayable throughout the run |
Phase Plan#
1. Baseline Warmup (0h-4h)#
- capture baseline health/readiness payloads
- capture the hour-4 RSS baseline after the system warms up
- capture baseline audit-store size and the initial export bundle
- confirm the recurring read-only command probe succeeds before any fault drill
2. Day 1 Uptime (4h-24h)#
- run two sleep/wake drills around hours
8and16 - record wake-to-ready recovery time after each drill
- perform one gateway reconnect drill after the host is healthy
- keep the recurring probes continuous outside the planned drill windows
3. Day 2 Recovery (24h-48h)#
- force one LaunchAgent restart around hour
28 - repeat sleep/wake around hour
32 - run another gateway reconnect drill around hour
36 - keep collecting audit-growth checkpoints and RSS deltas through the second day
4. Day 3 Crash-Loop And Export (48h-72h)#
- inject a bounded crash-loop drill: three forced exits within
10minutes around hour52 - repeat sleep/wake around hours
56and64 - run the final low-risk command probe and final audit export at the end of the 72-hour window
- record the final RSS delta against the hour-4 baseline
Pass / Fail Rules#
Success requires all of the following:
- at least
72continuous hours of evidence - at least six planned sleep/wake cycles with successful post-wake probes
- at least one forced LaunchAgent restart and one bounded crash-loop drill that recover without undocumented manual repair
- gateway reconnect probes remain within the existing
10sreconnect hard max remote_gateway.audit.write_failure.countstays at0- every daily and final audit export validates successfully
- hour-72 RSS remains within
+25%of the hour-4 baseline for the same workload, and Bellona memory pressure never reachescritical
Fail fast and stop the run when any of these happen:
- any audit write failure
- any planned sleep/wake or LaunchAgent drill fails to return to
readywithin the documented threshold - two consecutive low-risk command probes fail outside a planned perturbation window
- memory pressure becomes
criticalor RSS stays above the+25%budget for more than12continuous hours - any audit export validation or replay failure
Required Evidence Bundle#
The final soak bundle should include:
- 72+ hour probe logs
- sleep/wake timestamps and recovery timings
- LaunchAgent restart and crash-loop drill logs
- gateway reconnect latency samples
- audit-store growth checkpoints
- daily and final audit export bundles with validation output
- RSS and memory-pressure trend chart
- operator signoff note for any anomaly, even if the run is still considered a pass
Verification#
The committed automated verification for the plan artifact is:
pnpm exec vitest run --config vitest.config.ts tests/mvp-alpha-soak-plan.test.ts --reporter=verbose
pnpm nx run-many -p @bellona/remote-control-testing -t lint,typecheck,test,build --skip-nx-cache --verbose
The first command proves the plan structure, duration, phases, probes, and fail-fast expectations. The second command keeps the reusable soak-plan module buildable in the Bellona remote-control testing package.