Eve library

Euterpe ↔ Magenta RealTime 2 sidecar

Other

Revision 2

On this page

Bridges the official MRT2 streaming engine to Euterpe's realtime route. The engine is the MLX magenta_rt Python package — this repo does not reimplement the model; the sidecar calls it.

text
GET /v1/generation/realtime (BFF)
  → SidecarOnDeviceRuntime (@euterpe/realtime-gen)  spawns: python3 magenta_rt_sidecar.py
      ↕ newline-JSON over stdio
    magenta_rt_sidecar.py → MagentaRT2Mlx.generate(style, notes, drums, frames, state)
      → 48 kHz stereo PCM frames back (one `step` per inferFrame ≈ 200 ms)

The model is loaded once (warm) at handshake and state is threaded across steps, so live control (WebMIDI notes → 128-pianoroll, text → MusicCoCa style, audio-ref, drum toggle) changes the stream in real time. It never fabricates audio: engine/weights absent → not_ready; any generation failure → error.

Protocol (stdio, newline-delimited JSON)#

→ sidecar sidecar →
{"op":"handshake","tier":"230m"|"2.4b"} {"type":"ready","sampleRate":48000,"channels":2,"frameSeconds":0.04} / {"type":"not_ready","reason":…}
{"op":"step","text":str|null,"notes":[int…]|null,"drums":-1|0|1,"frames":int} {"type":"frame","pcm":"<base64 f32le interleaved stereo>"} / {"type":"error","reason":…}
{"op":"audioref","uri":str} (applied to the next frame)
{"op":"stop"} (resets streaming state)

The entire remaining runbook (this is all that's left)#

bash
# 1. Install the real engine on an Apple Silicon Mac.
uv pip install "magenta-rt[mlx]"        # provides `mrt` + the magenta_rt package

# 2. Download the weights (mrt2_small ≈ runs on a MacBook Air; mrt2_base needs M3 Pro/M2 Max).
mrt models download mrt2_small          # → ~/Documents/Magenta/magenta-rt-v2/models/…

# 3. Point the BFF at the sidecar.
export OSHUN_MRT2_ENABLED=true
export OSHUN_MRT2_SIDECAR="$PWD/tools/euterpe-mrt2-sidecar/magenta_rt_sidecar.py"
export OSHUN_MRT2_TIER=230m             # 230m | 2.4b
# optional: OSHUN_MRT2_PYTHON=python3   (interpreter that has magenta_rt)

That's it. With those done, GET /v1/generation/realtime streams real live audio (WebMIDI / prompt / drums / style-ref → ~200 ms 48 kHz stereo). Without them, every layer fails closed (not_ready → 503 → panel "engine not configured"). Confirm the exact download name against mrt models download (interactive picker if the name is omitted).

Verify it (deploy smoke)#

bash
# engine-gated e2e (skips automatically unless magenta_rt + weights are present):
npx vitest run libs/euterpe/realtime-gen/src/runtime/sidecar-e2e.test.ts

Architecture note#

The Rust @euterpe/realtime-engine crate is the discovery / device-feasibility / codec-math helper (it parses a downloaded bundle, counts params, gates tiers). Generation runs in this sidecar, not in the Rust crate. Weights live in the magenta_rt checkpoints dir (above), not OSHUN_MRT2_WEIGHTS_DIR (which only fed the Rust discovery helper).

A CPU backend was tried, and cannot keep up in real time (2026-09-21)#

The realtime route streams audio as it is generated, so a backend has to make a second of audio in under a second of compute (real-time factor, RTF, below 1). MRT2's engine is MLX, which runs only on Apple silicon, so on the Linux dev server the candidate was MusicGen-small through Hugging Face transformers, on CPU. cpu_rtf_benchmark.py measures it:

bash
python3 -m venv --system-site-packages .venv-euterpe-musicgen   # system torch 2.11.0
.venv-euterpe-musicgen/bin/pip install "transformers==5.17.0" "scipy==1.16.2"
.venv-euterpe-musicgen/bin/python tools/euterpe-mrt2-sidecar/cpu_rtf_benchmark.py

Measured on 4 cores of an AMD EPYC-Milan at 2.0 GHz. The run generated 250 tokens (4.94 s of 32 kHz audio) from the prompt "calm lo-fi piano with soft drums". The model loaded in 3.5 s from cache (13.6 s on first download).

variant wall RTF RMS spectral flatness
fp32 (as shipped) 26.9 s 5.45 0.095 0.0001
int8 dynamic quantization of Linear layers 8.4 s 1.70 0.353 0.0001

Neither variant is real time. Int8 is three times faster and still 70% slower than playback. The flatness of both is far from noise (1.0), so the speed is not coming from broken output. A stream from either would fall further behind the listener every second, so neither is wired into the realtime route, which stays fail-closed.

Real time needs faster hardware: a GPU for MusicGen, or Apple silicon for MRT2. Neither has been measured here. Rerun the benchmark on the target host before wiring either in.