Eve library

Chat and TTS Load Testing Playbook

Guide

Revision 1

On this page

This playbook codifies how Lilith models throughput requirements for interactive chat and low-latency text-to-speech (TTS) workloads. It defines target TPS (transactions per second) envelopes, burst behaviour, and long-haul soak expectations so that teams can repeatedly validate service capacity, spot regressions, and catch memory leaks.


Objectives#

  1. Dimension concurrency for conversational APIs and streaming TTS so that sustained traffic meets SLOs with ≥20% headroom.
  2. Exercise burst tolerance that mirrors production spikes (live events, meditation sessions, or marketing campaigns).
  3. Detect long-haul regressions via soak tests that track latency drift, error accumulation, and heap growth.

TPS Modeling#

Profile Target TPS Concurrency Formula Latency Targets Error Budget
Chat 120 p95 ceil(TPS * p95_latency_ms / 1000) with baseline latency 900 ms p95 ≤ 900 ms, p99 ≤ 1500 ms 0.5%
TTS 60 p95 ceil(TPS * p95_latency_ms / 1000) with baseline latency 1400 ms p95 ≤ 1400 ms, p99 ≤ 2000 ms 0.5%

The canonical flow is:

  1. Stand the Lilith BFF up locally (pnpm nx run @lilith/bff:serve) or target a staging BFF URL.
  2. Run the script against a real endpoint. --dry-run is a plan-only mode and its latency / TPS numbers are synthesised from the declared p95LatencyMs; a dry-run report must never be used as a measurement of record:
    bash
    scripts/lilith/performance/run-load-profile.ts \
      --mode chat --target-tps 120 --duration 300 \
      --target-url http://127.0.0.1:3001/v1/chat \
      --report reports/performance/chat-tps-$(date +%Y%m%d).json
    
  3. Customize latency assumptions via --p95-latency-ms when new baseline data is available.

The script automatically computes:

  • Steady-state concurrency (baseline load)
  • Burst concurrency using burst_multiplier
  • Request cadence (warmup, steady, burst, cooldown phases)

Synthetic vs measured. When --dry-run is passed OR no --target-url is provided, every latency, TPS, and error-rate number in the emitted report is synthesised. A report with an empty parameters.targetUrl or whose metrics.achieved_tps equals the planned TPS exactly is a dry-run and MUST NOT be used as a measurement of record. The CI spec testing/performance/benchmarks/tps-models.spec.ts guards this invariant: dry-run reports go one place (planning, phase review), measured reports go another (baselines).


Burst Testing#

Phases#

  1. Warmup (10% of total duration) ramping from 30% → baseline TPS.
  2. Steady-state maintaining baseline TPS for 60% of the window.
  3. Burst applying the configured multiplier (default 2.5× for chat, 3× for TTS) over a 90-second window.
  4. Stabilisation ramping back to baseline and verifying recovery.

Execution#

bash
scripts/performance/run-load-profile.ts \
  --mode chat \
  --target-tps 120 \
  --duration 420 \
  --burst-multiplier 2.5 \
  --target-url https://bff.dev.lilith.ai/v1/chat \
  --report reports/performance/chat-burst-$(date +%Y%m%d).json

The generated report captures:

  • Achieved TPS and latency distribution (synthetic in dry-run mode, measured when a target URL is provided)
  • Error/non-200 counts
  • Memory delta (if --profile-memory is enabled)

Long-Haul Soak#

Long-haul drills run for ≥4 hours to catch slow leaks.

  1. Schedule via CI or chaos workflow to run nightly against staging.
  2. Use --duration 14400 --sample-interval 60 --leak-threshold-mb 50.
  3. Capture metrics into reports/performance/<profile>-soak-*.json and upload to the observability bucket.
  4. Alert when:
    • Heap growth exceeds threshold
    • Error rate > 0.5%
    • Latency drift > 15% vs. baseline

run-load-profile.ts exposes optional flags:

  • --monitor-memory – pull RSS for the injector process
  • --log-interval – JSON line output for dashboards

Reports & Baselines#

Baseline JSON artifacts are stored in reports/performance/ and validated by Playwright compliance tests.

File Description
chat-tps-baseline.json Captures latest validated chat TPS run (target & achieved TPS, latency percentiles, error rate, memory delta).
tts-tps-baseline.json Same for TTS streaming profile.

These reports are regenerated whenever the load script runs with --report. They keep enough metadata to trend results across releases (commit hash, environment, hardware footprint).


Automation Workflow#

  1. npm run perf:chat:dry → invokes the script in dry-run mode to produce the plan without load.
  2. npm run perf:chat:burst / npm run perf:tts:burst → targeted burst tests against staging.
  3. npm run perf:soak → 4-hour soak with periodic memory sampling.
  4. Vitest spec testing/performance/benchmarks/tps-models.spec.ts guards the documentation, automation scripts, and target-baseline alignment for this pipeline.

Incident Response#

  • When burst/soak tests regress, update the baseline report and file a JIRA ticket tagged perf-regression.
  • For leaks, capture heap snapshots (Node: NODE_OPTIONS=--inspect) and attach to the ticket.
  • Update this playbook whenever latency targets or throughput goals change.

References#

  • scripts/performance/run-load-profile.ts
  • tests/performance.tps-models.spec.ts
  • reports/performance/chat-tps-baseline.json
  • reports/performance/tts-tps-baseline.json
  • Observability dashboard: dashboards/performance/chat-tts