# Euterpe ↔ Magenta RealTime 2 sidecar

Bridges the **official MRT2 streaming engine** to Euterpe's realtime route. The
engine is the MLX `magenta_rt` Python package — this repo does **not**
reimplement the model; the sidecar calls it.

```
GET /v1/generation/realtime (BFF)
  → SidecarOnDeviceRuntime (@euterpe/realtime-gen)  spawns: python3 magenta_rt_sidecar.py
      ↕ newline-JSON over stdio
    magenta_rt_sidecar.py → MagentaRT2Mlx.generate(style, notes, drums, frames, state)
      → 48 kHz stereo PCM frames back (one `step` per inferFrame ≈ 200 ms)
```

The model is loaded **once** (warm) at handshake and `state` is threaded across
steps, so live control (WebMIDI notes → 128-pianoroll, text → MusicCoCa style,
audio-ref, drum toggle) changes the stream in real time. It never fabricates
audio: engine/weights absent → `not_ready`; any generation failure → `error`.

## Protocol (stdio, newline-delimited JSON)

| → sidecar                                                                           | sidecar →                                                                                                  |
| ----------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| `{"op":"handshake","tier":"230m"\|"2.4b"}`                                          | `{"type":"ready","sampleRate":48000,"channels":2,"frameSeconds":0.04}` / `{"type":"not_ready","reason":…}` |
| `{"op":"step","text":str\|null,"notes":[int…]\|null,"drums":-1\|0\|1,"frames":int}` | `{"type":"frame","pcm":"<base64 f32le interleaved stereo>"}` / `{"type":"error","reason":…}`               |
| `{"op":"audioref","uri":str}`                                                       | (applied to the next frame)                                                                                |
| `{"op":"stop"}`                                                                     | (resets streaming state)                                                                                   |

## The entire remaining runbook (this is all that's left)

```bash
# 1. Install the real engine on an Apple Silicon Mac.
uv pip install "magenta-rt[mlx]"        # provides `mrt` + the magenta_rt package

# 2. Download the weights (mrt2_small ≈ runs on a MacBook Air; mrt2_base needs M3 Pro/M2 Max).
mrt models download mrt2_small          # → ~/Documents/Magenta/magenta-rt-v2/models/…

# 3. Point the BFF at the sidecar.
export OSHUN_MRT2_ENABLED=true
export OSHUN_MRT2_SIDECAR="$PWD/tools/euterpe-mrt2-sidecar/magenta_rt_sidecar.py"
export OSHUN_MRT2_TIER=230m             # 230m | 2.4b
# optional: OSHUN_MRT2_PYTHON=python3   (interpreter that has magenta_rt)
```

That's it. With those done, `GET /v1/generation/realtime` streams **real** live
audio (WebMIDI / prompt / drums / style-ref → ~200 ms 48 kHz stereo). Without
them, every layer fails closed (`not_ready` → 503 → panel "engine not
configured"). Confirm the exact download name against `mrt models download`
(interactive picker if the name is omitted).

## Verify it (deploy smoke)

```bash
# engine-gated e2e (skips automatically unless magenta_rt + weights are present):
npx vitest run libs/euterpe/realtime-gen/src/runtime/sidecar-e2e.test.ts
```

## Architecture note

The Rust `@euterpe/realtime-engine` crate is the **discovery /
device-feasibility / codec-math helper** (it parses a downloaded bundle, counts
params, gates tiers). **Generation runs in this sidecar**, not in the Rust
crate. Weights live in the `magenta_rt` checkpoints dir (above), not
`OSHUN_MRT2_WEIGHTS_DIR` (which only fed the Rust discovery helper).

## A CPU backend was tried, and cannot keep up in real time (2026-09-21)

The realtime route streams audio as it is generated, so a backend has to make a
second of audio in under a second of compute (real-time factor, RTF, below 1).
MRT2's engine is MLX, which runs only on Apple silicon, so on the Linux dev
server the candidate was MusicGen-small through Hugging Face transformers, on
CPU. `cpu_rtf_benchmark.py` measures it:

```bash
python3 -m venv --system-site-packages .venv-euterpe-musicgen   # system torch 2.11.0
.venv-euterpe-musicgen/bin/pip install "transformers==5.17.0" "scipy==1.16.2"
.venv-euterpe-musicgen/bin/python tools/euterpe-mrt2-sidecar/cpu_rtf_benchmark.py
```

Measured on 4 cores of an AMD EPYC-Milan at 2.0 GHz. The run generated 250
tokens (4.94 s of 32 kHz audio) from the prompt "calm lo-fi piano with soft
drums". The model loaded in 3.5 s from cache (13.6 s on first download).

| variant                                    | wall   | RTF      | RMS   | spectral flatness |
| ------------------------------------------ | ------ | -------- | ----- | ----------------- |
| fp32 (as shipped)                          | 26.9 s | **5.45** | 0.095 | 0.0001            |
| int8 dynamic quantization of Linear layers | 8.4 s  | **1.70** | 0.353 | 0.0001            |

Neither variant is real time. Int8 is three times faster and still 70% slower
than playback. The flatness of both is far from noise (1.0), so the speed is not
coming from broken output. A stream from either would fall further behind the
listener every second, so neither is wired into the realtime route, which stays
fail-closed.

Real time needs faster hardware: a GPU for MusicGen, or Apple silicon for MRT2.
Neither has been measured here. Rerun the benchmark on the target host before
wiring either in.
