# Bellona MVP Alpha Soak-Test Plan

Status: `180.C.16.13` deterministic multi-day soak-plan artifact.

Implementation date: 2026-04-23.

This plan defines how Bellona should collect MVP alpha evidence for multi-day
Remote Host uptime. The intent is not a synthetic benchmark. The intent is to
prove that Bellona can survive realistic long-running operator use across:

- host sleep and wake cycles
- LaunchAgent restart and bounded crash-loop recovery
- gateway reconnect stability
- audit-store growth under sustained command traffic
- process memory drift across a 72+ hour run

## Sources Reviewed

Local implementation references:

- `testing/bellona/remote-control/mvp-alpha-soak-plan.ts`
- `testing/bellona/remote-control/network-impairment.ts`
- `docs/domains/bellona/extras/remote-control/network-impairment-test-plan.md`
- `docs/domains/bellona/extras/remote-control/remote-host-update-channel-and-rollback-plan.md`
- `docs/domains/bellona/extras/remote-control/session-timeline-and-audit-bundle.md`
- `apps/bellona/remote-host/src/app.ts`
- `apps/bellona/remote-host/src/adapter-registry.ts`
- `apps/bellona/remote-host/src/macos-packaging-plan.ts`

Existing implementation patterns analyzed:

- The Stage 16 network-impairment plan for the already-defined sleep/wake and
  reconnect evidence Bellona expects later hardware runs to provide.
- The Remote Host update and macOS packaging plans for the current LaunchAgent
  verification and rollback surfaces.
- The current host health/readiness payloads and state snapshot resource
  telemetry for uptime and memory-pressure observations.

## Run Envelope

- minimum duration: `72` continuous hours
- preferred duration: `84` hours so the last day still has room for export and
  review without truncating the 72-hour evidence window
- required rigs: `MAC-HOST-01`, `MAC-OP-01`, `LAN-GW-01`, and `NET-IMPAIR-01`

The soak run should remain on the real macOS host. VM-only evidence is not good
enough for sleep/wake, LaunchAgent, and multi-day resource-behavior claims.

## Probe Cadence

The plan uses these recurring probes:

| Probe                   | Cadence     | Source                                                                                          | Expected signal                                                       |
| ----------------------- | ----------- | ----------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- |
| Host health             | every `15m` | `curl /health`                                                                                  | monotonic `uptimeMs`, stable adapter counts, expected gateway state   |
| Host readiness          | every `15m` | `curl /readiness`                                                                               | returns `ready` outside planned drill windows                         |
| Low-risk command path   | every `30m` | `agent.readiness`, `state.snapshot`, or `blender.scene.info` through the real gateway-host path | success result plus audit/timeline evidence                           |
| RSS and memory pressure | every `15m` | `ps -o rss=` plus Bellona `state.snapshot`                                                      | RSS stays within leak budget and memory pressure stays below critical |
| Audit-store growth      | every `6h`  | store size, entry count, export size, write-failure count                                       | monotonic growth with zero write failures                             |
| Audit export validation | every `24h` | audit export and hash-chain validation                                                          | export stays valid and replayable throughout the run                  |

## Phase Plan

### 1. Baseline Warmup (`0h`-`4h`)

- capture baseline health/readiness payloads
- capture the hour-4 RSS baseline after the system warms up
- capture baseline audit-store size and the initial export bundle
- confirm the recurring read-only command probe succeeds before any fault drill

### 2. Day 1 Uptime (`4h`-`24h`)

- run two sleep/wake drills around hours `8` and `16`
- record wake-to-ready recovery time after each drill
- perform one gateway reconnect drill after the host is healthy
- keep the recurring probes continuous outside the planned drill windows

### 3. Day 2 Recovery (`24h`-`48h`)

- force one LaunchAgent restart around hour `28`
- repeat sleep/wake around hour `32`
- run another gateway reconnect drill around hour `36`
- keep collecting audit-growth checkpoints and RSS deltas through the second day

### 4. Day 3 Crash-Loop And Export (`48h`-`72h`)

- inject a bounded crash-loop drill: three forced exits within `10` minutes
  around hour `52`
- repeat sleep/wake around hours `56` and `64`
- run the final low-risk command probe and final audit export at the end of the
  72-hour window
- record the final RSS delta against the hour-4 baseline

## Pass / Fail Rules

Success requires all of the following:

- at least `72` continuous hours of evidence
- at least six planned sleep/wake cycles with successful post-wake probes
- at least one forced LaunchAgent restart and one bounded crash-loop drill that
  recover without undocumented manual repair
- gateway reconnect probes remain within the existing `10s` reconnect hard max
- `remote_gateway.audit.write_failure.count` stays at `0`
- every daily and final audit export validates successfully
- hour-72 RSS remains within `+25%` of the hour-4 baseline for the same
  workload, and Bellona memory pressure never reaches `critical`

Fail fast and stop the run when any of these happen:

- any audit write failure
- any planned sleep/wake or LaunchAgent drill fails to return to `ready` within
  the documented threshold
- two consecutive low-risk command probes fail outside a planned perturbation
  window
- memory pressure becomes `critical` or RSS stays above the `+25%` budget for
  more than `12` continuous hours
- any audit export validation or replay failure

## Required Evidence Bundle

The final soak bundle should include:

- 72+ hour probe logs
- sleep/wake timestamps and recovery timings
- LaunchAgent restart and crash-loop drill logs
- gateway reconnect latency samples
- audit-store growth checkpoints
- daily and final audit export bundles with validation output
- RSS and memory-pressure trend chart
- operator signoff note for any anomaly, even if the run is still considered a
  pass

## Verification

The committed automated verification for the plan artifact is:

```bash
pnpm exec vitest run --config vitest.config.ts tests/mvp-alpha-soak-plan.test.ts --reporter=verbose
pnpm nx run-many -p @bellona/remote-control-testing -t lint,typecheck,test,build --skip-nx-cache --verbose
```

The first command proves the plan structure, duration, phases, probes, and
fail-fast expectations. The second command keeps the reusable soak-plan module
buildable in the Bellona remote-control testing package.
