Bridges the official MRT2 streaming engine to Euterpe's realtime route. The
engine is the MLX magenta_rt Python package — this repo does not
reimplement the model; the sidecar calls it.
GET /v1/generation/realtime (BFF)
→ SidecarOnDeviceRuntime (@euterpe/realtime-gen) spawns: python3 magenta_rt_sidecar.py
↕ newline-JSON over stdio
magenta_rt_sidecar.py → MagentaRT2Mlx.generate(style, notes, drums, frames, state)
→ 48 kHz stereo PCM frames back (one `step` per inferFrame ≈ 200 ms)
The model is loaded once (warm) at handshake and state is threaded across
steps, so live control (WebMIDI notes → 128-pianoroll, text → MusicCoCa style,
audio-ref, drum toggle) changes the stream in real time. It never fabricates
audio: engine/weights absent → not_ready; any generation failure → error.
Protocol (stdio, newline-delimited JSON)#
| → sidecar | sidecar → |
|---|---|
{"op":"handshake","tier":"230m"|"2.4b"} |
{"type":"ready","sampleRate":48000,"channels":2,"frameSeconds":0.04} / {"type":"not_ready","reason":…} |
{"op":"step","text":str|null,"notes":[int…]|null,"drums":-1|0|1,"frames":int} |
{"type":"frame","pcm":"<base64 f32le interleaved stereo>"} / {"type":"error","reason":…} |
{"op":"audioref","uri":str} |
(applied to the next frame) |
{"op":"stop"} |
(resets streaming state) |
The entire remaining runbook (this is all that's left)#
# 1. Install the real engine on an Apple Silicon Mac.
uv pip install "magenta-rt[mlx]" # provides `mrt` + the magenta_rt package
# 2. Download the weights (mrt2_small ≈ runs on a MacBook Air; mrt2_base needs M3 Pro/M2 Max).
mrt models download mrt2_small # → ~/Documents/Magenta/magenta-rt-v2/models/…
# 3. Point the BFF at the sidecar.
export OSHUN_MRT2_ENABLED=true
export OSHUN_MRT2_SIDECAR="$PWD/tools/euterpe-mrt2-sidecar/magenta_rt_sidecar.py"
export OSHUN_MRT2_TIER=230m # 230m | 2.4b
# optional: OSHUN_MRT2_PYTHON=python3 (interpreter that has magenta_rt)
That's it. With those done, GET /v1/generation/realtime streams real live
audio (WebMIDI / prompt / drums / style-ref → ~200 ms 48 kHz stereo). Without
them, every layer fails closed (not_ready → 503 → panel "engine not
configured"). Confirm the exact download name against mrt models download
(interactive picker if the name is omitted).
Verify it (deploy smoke)#
# engine-gated e2e (skips automatically unless magenta_rt + weights are present):
npx vitest run libs/euterpe/realtime-gen/src/runtime/sidecar-e2e.test.ts
Architecture note#
The Rust @euterpe/realtime-engine crate is the discovery /
device-feasibility / codec-math helper (it parses a downloaded bundle, counts
params, gates tiers). Generation runs in this sidecar, not in the Rust
crate. Weights live in the magenta_rt checkpoints dir (above), not
OSHUN_MRT2_WEIGHTS_DIR (which only fed the Rust discovery helper).
A CPU backend was tried, and cannot keep up in real time (2026-09-21)#
The realtime route streams audio as it is generated, so a backend has to make a
second of audio in under a second of compute (real-time factor, RTF, below 1).
MRT2's engine is MLX, which runs only on Apple silicon, so on the Linux dev
server the candidate was MusicGen-small through Hugging Face transformers, on
CPU. cpu_rtf_benchmark.py measures it:
python3 -m venv --system-site-packages .venv-euterpe-musicgen # system torch 2.11.0
.venv-euterpe-musicgen/bin/pip install "transformers==5.17.0" "scipy==1.16.2"
.venv-euterpe-musicgen/bin/python tools/euterpe-mrt2-sidecar/cpu_rtf_benchmark.py
Measured on 4 cores of an AMD EPYC-Milan at 2.0 GHz. The run generated 250 tokens (4.94 s of 32 kHz audio) from the prompt "calm lo-fi piano with soft drums". The model loaded in 3.5 s from cache (13.6 s on first download).
| variant | wall | RTF | RMS | spectral flatness |
|---|---|---|---|---|
| fp32 (as shipped) | 26.9 s | 5.45 | 0.095 | 0.0001 |
| int8 dynamic quantization of Linear layers | 8.4 s | 1.70 | 0.353 | 0.0001 |
Neither variant is real time. Int8 is three times faster and still 70% slower than playback. The flatness of both is far from noise (1.0), so the speed is not coming from broken output. A stream from either would fall further behind the listener every second, so neither is wired into the realtime route, which stays fail-closed.
Real time needs faster hardware: a GPU for MusicGen, or Apple silicon for MRT2. Neither has been measured here. Rerun the benchmark on the target host before wiring either in.