# Chat and TTS Load Testing Playbook

This playbook codifies how Lilith models throughput requirements for interactive
chat and low-latency text-to-speech (TTS) workloads. It defines target TPS
(transactions per second) envelopes, burst behaviour, and long-haul soak
expectations so that teams can repeatedly validate service capacity, spot
regressions, and catch memory leaks.

---

## Objectives

1. **Dimension concurrency** for conversational APIs and streaming TTS so that
   sustained traffic meets SLOs with ≥20% headroom.
2. **Exercise burst tolerance** that mirrors production spikes (live events,
   meditation sessions, or marketing campaigns).
3. **Detect long-haul regressions** via soak tests that track latency drift,
   error accumulation, and heap growth.

---

## TPS Modeling

| Profile | Target TPS | Concurrency Formula                                               | Latency Targets              | Error Budget |
| ------- | ---------- | ----------------------------------------------------------------- | ---------------------------- | ------------ |
| Chat    | 120 p95    | `ceil(TPS * p95_latency_ms / 1000)` with baseline latency 900 ms  | p95 ≤ 900 ms, p99 ≤ 1500 ms  | 0.5%         |
| TTS     | 60 p95     | `ceil(TPS * p95_latency_ms / 1000)` with baseline latency 1400 ms | p95 ≤ 1400 ms, p99 ≤ 2000 ms | 0.5%         |

The canonical flow is:

1. Stand the Lilith BFF up locally (`pnpm nx run @lilith/bff:serve`) or target a
   staging BFF URL.
2. Run the script against a real endpoint. `--dry-run` is a plan-only mode and
   its latency / TPS numbers are **synthesised** from the declared
   `p95LatencyMs`; a dry-run report must never be used as a measurement of
   record:
   ```bash
   scripts/lilith/performance/run-load-profile.ts \
     --mode chat --target-tps 120 --duration 300 \
     --target-url http://127.0.0.1:3001/v1/chat \
     --report reports/performance/chat-tps-$(date +%Y%m%d).json
   ```
3. Customize latency assumptions via `--p95-latency-ms` when new baseline data
   is available.

The script automatically computes:

- **Steady-state concurrency** (baseline load)
- **Burst concurrency** using `burst_multiplier`
- **Request cadence** (warmup, steady, burst, cooldown phases)

> **Synthetic vs measured.** When `--dry-run` is passed OR no `--target-url` is
> provided, every latency, TPS, and error-rate number in the emitted report is
> synthesised. A report with an empty `parameters.targetUrl` or whose
> `metrics.achieved_tps` equals the planned TPS exactly is a dry-run and MUST
> NOT be used as a measurement of record. The CI spec
> `testing/performance/benchmarks/tps-models.spec.ts` guards this invariant:
> dry-run reports go one place (planning, phase review), measured reports go
> another (baselines).

---

## Burst Testing

### Phases

1. **Warmup** (10% of total duration) ramping from 30% → baseline TPS.
2. **Steady-state** maintaining baseline TPS for 60% of the window.
3. **Burst** applying the configured multiplier (default 2.5× for chat, 3× for
   TTS) over a 90-second window.
4. **Stabilisation** ramping back to baseline and verifying recovery.

### Execution

```bash
scripts/performance/run-load-profile.ts \
  --mode chat \
  --target-tps 120 \
  --duration 420 \
  --burst-multiplier 2.5 \
  --target-url https://bff.dev.lilith.ai/v1/chat \
  --report reports/performance/chat-burst-$(date +%Y%m%d).json
```

The generated report captures:

- Achieved TPS and latency distribution (synthetic in dry-run mode, measured
  when a target URL is provided)
- Error/non-200 counts
- Memory delta (if `--profile-memory` is enabled)

---

## Long-Haul Soak

Long-haul drills run for ≥4 hours to catch slow leaks.

1. **Schedule** via CI or chaos workflow to run nightly against staging.
2. **Use** `--duration 14400 --sample-interval 60 --leak-threshold-mb 50`.
3. **Capture metrics** into `reports/performance/<profile>-soak-*.json` and
   upload to the observability bucket.
4. **Alert** when:
   - Heap growth exceeds threshold
   - Error rate > 0.5%
   - Latency drift > 15% vs. baseline

`run-load-profile.ts` exposes optional flags:

- `--monitor-memory` – pull RSS for the injector process
- `--log-interval` – JSON line output for dashboards

---

## Reports & Baselines

Baseline JSON artifacts are stored in `reports/performance/` and validated by
Playwright compliance tests.

| File                     | Description                                                                                                    |
| ------------------------ | -------------------------------------------------------------------------------------------------------------- |
| `chat-tps-baseline.json` | Captures latest validated chat TPS run (target & achieved TPS, latency percentiles, error rate, memory delta). |
| `tts-tps-baseline.json`  | Same for TTS streaming profile.                                                                                |

These reports are regenerated whenever the load script runs with `--report`.
They keep enough metadata to trend results across releases (commit hash,
environment, hardware footprint).

---

## Automation Workflow

1. `npm run perf:chat:dry` → invokes the script in dry-run mode to produce the
   plan without load.
2. `npm run perf:chat:burst` / `npm run perf:tts:burst` → targeted burst tests
   against staging.
3. `npm run perf:soak` → 4-hour soak with periodic memory sampling.
4. Vitest spec `testing/performance/benchmarks/tps-models.spec.ts` guards the
   documentation, automation scripts, and target-baseline alignment for this
   pipeline.

---

## Incident Response

- When burst/soak tests regress, update the baseline report and file a JIRA
  ticket tagged `perf-regression`.
- For leaks, capture heap snapshots (Node: `NODE_OPTIONS=--inspect`) and attach
  to the ticket.
- Update this playbook whenever latency targets or throughput goals change.

---

## References

- `scripts/performance/run-load-profile.ts`
- `tests/performance.tps-models.spec.ts`
- `reports/performance/chat-tps-baseline.json`
- `reports/performance/tts-tps-baseline.json`
- Observability dashboard: `dashboards/performance/chat-tts`
