Decision#
Default ASR: ONNX Whisper (tiny.en, base.en, small.en, medium.en)
loaded through the shared ML runtime, with INT8 / FP16 quantisation per device
tier. Default VAD: ONNX Silero VAD (v5). Reserve Deepgram (Nova-3) as the cloud
fallback for >5-minute streaming transcription where Whisper-medium cannot keep
up on the device.
VAD always runs locally — sending raw audio to a cloud just to detect silence wastes bytes and bandwidth. ASR routes through the local model by default; the cloud path is opt-in per call.
Constraints#
- Privacy-first. Voice should be transcribed locally for the default flow; no audio leaves the device unless the user explicitly opts into cloud-grade transcription.
- Streaming partial transcripts. First word must surface in < 400 ms after the speaker pauses; partial updates every 200 ms.
- Multilingual reach. At minimum the major Latin-script languages (en/es/fr/de/it/pt) and CJK (zh/ja/ko); cloud fallback handles long-tail languages.
- VAD recall ≥ 0.95 at < 1% false-trigger rate in noisy environments (fitness app, café).
Options considered#
ASR#
A. ONNX Whisper (chosen)#
- OpenAI's Whisper has the best zero-shot multilingual performance of any open
ASR model. ONNX exports from
transformers.jsandwhisper-cppare both validated. - Variant landscape:
tiny.en(~39 MB FP16, real-time on CPU),base.en(~74 MB),small.en(~244 MB; default mid-tier),medium.en(~769 MB; opt-in for accuracy). - Beam search is slow in JS; we ship the Rust
ml-postprocesskernel for greedy decoding + length-normalised log-prob ranking which matchestransformersreference output to within 1 word per 1000.
B. Deepgram Nova-3 (cloud fallback, kept)#
- Best WER on long-form English (~3.0 vs Whisper-medium's ~5.5).
- Streaming WebSocket API with sub-300 ms partials.
- Per-minute billing — fine for cloud-tier usage gated behind opt-in.
C. Distil-Whisper (deferred)#
- Faster-than-Whisper on the same architecture but the published ONNX exports are larger than the equivalent Whisper variant for the same WER. Revisit at V2.
VAD#
D. ONNX Silero VAD v5 (chosen)#
- 1.7 MB ONNX, runs in < 1 ms per 30 ms audio chunk on CPU.
- Recall 0.97 / false-trigger 0.4% on the Silero validation suite — comfortably inside our envelope.
- Stateful (RNN-based); shared loader exposes a
VadStreamhelper that maintains the hidden state across chunks.
E. WebRTC VAD — rejected#
- Energy-based; brittle in non-speech-but-stationary noise (HVAC, vehicle interior). Recall drops to ~0.85 at low false-trigger.
F. Cloud VAD — rejected#
- A round trip per 30 ms chunk is silly; no provider offers it this granular anyway.
Latency budget#
| component | target (p50) | target (p99) |
|---|---|---|
| Silero VAD per 30 ms chunk | 0.3 ms | 1.0 ms |
| Whisper-tiny.en short clip (<5s) | 180 ms | 400 ms |
| Whisper-base.en short clip (<5s) | 320 ms | 650 ms |
| Whisper-small.en short clip (<5s) | 550 ms | 1 100 ms |
| Whisper-medium.en short clip (<5s) | 1 600 ms | 2 800 ms |
| Deepgram Nova-3 streaming partial | 150 ms | 300 ms |
(All numbers measured on a Macbook Air M2; WebGPU + INT8 on Whisper.)
Action items#
- Pull Whisper ONNX exports (
tiny.en/base.en/small.en/medium.en) fromxenova/whisper-*; verify WER on the LibriSpeechtest-cleanset within ±0.3 of the PyTorch ref. - Pull Silero VAD v5 ONNX; verify recall + false-trigger on the Silero validation suite.
- Implement a Rust greedy decoder for Whisper at
libs/shared/crates/ml-postprocess/src/whisper.rs. - Wire the
libs/shared/src/ml/asr.tsandvad.tsnamespaces. - Add a Deepgram client at
libs/aje/agents/src/asr/deepgram.ts(cloud-tier opt-in path).
References: