On-device Rust engine for Magenta RealTime 2 (MRT2) — the SpectroStream
codec language model that drives frame-level, ~200 ms-latency music generation in
the Euterpe toolbox. Bridged to TypeScript (@euterpe/realtime-gen) via napi-rs,
mirroring the @kalika/cas-engine workspace.
Crates#
| Crate | Type | Responsibility |
|---|---|---|
mrt2-core |
rlib (pure) | Frame/token budget math, on-device feasibility (numerically identical to @euterpe/genesis frontier), tier selection (2.4B/230M), the live conditioning model (MIDI/text/audio-ref + sliding-window context), and the SpectroStream codec descriptor. No I/O, no model. |
mrt2-engine |
rlib | Real .safetensors header parsing + bundle discovery/validation, feasibility-gated load, and the fail-closed inference seam. |
mrt2-native |
cdylib + rlib | napi-rs bridge exposing the above to Node. |
Fail-closed by design#
mrt2-engine does every weight-dependent thing it can without the model's
forward graph — it validates a downloaded bundle, counts real parameters, and
runs the device feasibility gate — then reports not_configured honestly
instead of fabricating audio. No code path ever returns invented PCM.
Enabling generation is NOT "download weights → go" — it bridges the official engine#
MRT2's published runtime is a C++/MLX inference engine (models compiled to an
.mlxfn weights+graph container) and the magenta-rt Python package
(pip install magenta-rt); weights live at
google/magenta-realtime-2,
code at github.com/magenta/magenta-realtime.
The model is a dual transformer — Temporal_θ (compresses frame history) +
Depth_φ (depth-wise RVQ token decode) — with causal sliding-window attention
over the SpectroStream codec (48 kHz stereo, 25 Hz / 40 ms frames, 12 RVQ
levels × 1024 vocab, 3 kbps), conditioned by quantized MusicCoCa embeddings
(style), a 128-channel 25 Hz pianoroll (notes) and OaF binary drums.
This crate therefore does not reimplement that forward pass (the layer dims
aren't even public). Enabling generation means bridging the official engine
(an MLX/magenta-rt sidecar over local IPC, or FFI to the C++ engine) and
flipping INFERENCE_BACKEND_LINKED — a deploy-time integration that requires the
engine + weights on an Apple Silicon host. Until then everything fails closed.
Commands#
# From libs/euterpe/realtime-engine (bypasses Nx per the worktree rules):
cargo test --workspace # unit tests (budget, feasibility, codec, safetensors, discovery)
cargo check --workspace # typecheck
cargo fmt --all --check # lint
# Stage the native .node addon (fail-closed engine):
scripts/build-native.sh # → dist/libs/euterpe/realtime-engine/native/<platform>-<arch>/mrt2_native.node
This crate does discovery / feasibility / codec math — it does NOT generate.
Generation runs in the magenta_rt MLX sidecar (tools/euterpe-mrt2-sidecar/,
wired via OSHUN_MRT2_SIDECAR); see that README for the runbook. The Rust napi
path stays fail-closed.
Environment#
The Rust discovery helper (this crate, via the napi NativeOnDeviceModelRuntime):
| Var | Meaning |
|---|---|
OSHUN_MRT2_RUNTIME |
auto | native | off (default auto) |
OSHUN_MRT2_WEIGHTS_DIR |
Path to an unpacked bundle for the Rust discovery helper (NOT where generation weights live) |
OSHUN_MRT2_TIER |
230m | 2.4b (also selects the sidecar model size) |
OSHUN_MRT2_ENGINE_PATH |
Explicit .node addon path for the discovery helper |
The generation sidecar (the real engine; tools/euterpe-mrt2-sidecar/):
| Var | Meaning |
|---|---|
OSHUN_MRT2_ENABLED |
true to enable the realtime route |
OSHUN_MRT2_SIDECAR |
Absolute path to magenta_rt_sidecar.py (selects the sidecar over the Rust helper) |
OSHUN_MRT2_PYTHON |
Interpreter that has magenta_rt (default python3) |
Generation weights live in the magenta_rt checkpoints dir (mrt models download),
not OSHUN_MRT2_WEIGHTS_DIR.