This playbook codifies how Lilith models throughput requirements for interactive chat and low-latency text-to-speech (TTS) workloads. It defines target TPS (transactions per second) envelopes, burst behaviour, and long-haul soak expectations so that teams can repeatedly validate service capacity, spot regressions, and catch memory leaks.
Objectives#
- Dimension concurrency for conversational APIs and streaming TTS so that sustained traffic meets SLOs with ≥20% headroom.
- Exercise burst tolerance that mirrors production spikes (live events, meditation sessions, or marketing campaigns).
- Detect long-haul regressions via soak tests that track latency drift, error accumulation, and heap growth.
TPS Modeling#
| Profile | Target TPS | Concurrency Formula | Latency Targets | Error Budget |
|---|---|---|---|---|
| Chat | 120 p95 | ceil(TPS * p95_latency_ms / 1000) with baseline latency 900 ms |
p95 ≤ 900 ms, p99 ≤ 1500 ms | 0.5% |
| TTS | 60 p95 | ceil(TPS * p95_latency_ms / 1000) with baseline latency 1400 ms |
p95 ≤ 1400 ms, p99 ≤ 2000 ms | 0.5% |
The canonical flow is:
- Stand the Lilith BFF up locally (
pnpm nx run @lilith/bff:serve) or target a staging BFF URL. - Run the script against a real endpoint.
--dry-runis a plan-only mode and its latency / TPS numbers are synthesised from the declaredp95LatencyMs; a dry-run report must never be used as a measurement of record:bashscripts/lilith/performance/run-load-profile.ts \ --mode chat --target-tps 120 --duration 300 \ --target-url http://127.0.0.1:3001/v1/chat \ --report reports/performance/chat-tps-$(date +%Y%m%d).json - Customize latency assumptions via
--p95-latency-mswhen new baseline data is available.
The script automatically computes:
- Steady-state concurrency (baseline load)
- Burst concurrency using
burst_multiplier - Request cadence (warmup, steady, burst, cooldown phases)
Synthetic vs measured. When
--dry-runis passed OR no--target-urlis provided, every latency, TPS, and error-rate number in the emitted report is synthesised. A report with an emptyparameters.targetUrlor whosemetrics.achieved_tpsequals the planned TPS exactly is a dry-run and MUST NOT be used as a measurement of record. The CI spectesting/performance/benchmarks/tps-models.spec.tsguards this invariant: dry-run reports go one place (planning, phase review), measured reports go another (baselines).
Burst Testing#
Phases#
- Warmup (10% of total duration) ramping from 30% → baseline TPS.
- Steady-state maintaining baseline TPS for 60% of the window.
- Burst applying the configured multiplier (default 2.5× for chat, 3× for TTS) over a 90-second window.
- Stabilisation ramping back to baseline and verifying recovery.
Execution#
scripts/performance/run-load-profile.ts \
--mode chat \
--target-tps 120 \
--duration 420 \
--burst-multiplier 2.5 \
--target-url https://bff.dev.lilith.ai/v1/chat \
--report reports/performance/chat-burst-$(date +%Y%m%d).json
The generated report captures:
- Achieved TPS and latency distribution (synthetic in dry-run mode, measured when a target URL is provided)
- Error/non-200 counts
- Memory delta (if
--profile-memoryis enabled)
Long-Haul Soak#
Long-haul drills run for ≥4 hours to catch slow leaks.
- Schedule via CI or chaos workflow to run nightly against staging.
- Use
--duration 14400 --sample-interval 60 --leak-threshold-mb 50. - Capture metrics into
reports/performance/<profile>-soak-*.jsonand upload to the observability bucket. - Alert when:
- Heap growth exceeds threshold
- Error rate > 0.5%
- Latency drift > 15% vs. baseline
run-load-profile.ts exposes optional flags:
--monitor-memory– pull RSS for the injector process--log-interval– JSON line output for dashboards
Reports & Baselines#
Baseline JSON artifacts are stored in reports/performance/ and validated by
Playwright compliance tests.
| File | Description |
|---|---|
chat-tps-baseline.json |
Captures latest validated chat TPS run (target & achieved TPS, latency percentiles, error rate, memory delta). |
tts-tps-baseline.json |
Same for TTS streaming profile. |
These reports are regenerated whenever the load script runs with --report.
They keep enough metadata to trend results across releases (commit hash,
environment, hardware footprint).
Automation Workflow#
npm run perf:chat:dry→ invokes the script in dry-run mode to produce the plan without load.npm run perf:chat:burst/npm run perf:tts:burst→ targeted burst tests against staging.npm run perf:soak→ 4-hour soak with periodic memory sampling.- Vitest spec
testing/performance/benchmarks/tps-models.spec.tsguards the documentation, automation scripts, and target-baseline alignment for this pipeline.
Incident Response#
- When burst/soak tests regress, update the baseline report and file a JIRA
ticket tagged
perf-regression. - For leaks, capture heap snapshots (Node:
NODE_OPTIONS=--inspect) and attach to the ticket. - Update this playbook whenever latency targets or throughput goals change.
References#
scripts/performance/run-load-profile.tstests/performance.tps-models.spec.tsreports/performance/chat-tps-baseline.jsonreports/performance/tts-tps-baseline.json- Observability dashboard:
dashboards/performance/chat-tts