Eve library

@euterpe/realtime-engine

Other

Revision 1

On this page

On-device Rust engine for Magenta RealTime 2 (MRT2) — the SpectroStream codec language model that drives frame-level, ~200 ms-latency music generation in the Euterpe toolbox. Bridged to TypeScript (@euterpe/realtime-gen) via napi-rs, mirroring the @kalika/cas-engine workspace.

Crates#

Crate Type Responsibility
mrt2-core rlib (pure) Frame/token budget math, on-device feasibility (numerically identical to @euterpe/genesis frontier), tier selection (2.4B/230M), the live conditioning model (MIDI/text/audio-ref + sliding-window context), and the SpectroStream codec descriptor. No I/O, no model.
mrt2-engine rlib Real .safetensors header parsing + bundle discovery/validation, feasibility-gated load, and the fail-closed inference seam.
mrt2-native cdylib + rlib napi-rs bridge exposing the above to Node.

Fail-closed by design#

mrt2-engine does every weight-dependent thing it can without the model's forward graph — it validates a downloaded bundle, counts real parameters, and runs the device feasibility gate — then reports not_configured honestly instead of fabricating audio. No code path ever returns invented PCM.

Enabling generation is NOT "download weights → go" — it bridges the official engine#

MRT2's published runtime is a C++/MLX inference engine (models compiled to an .mlxfn weights+graph container) and the magenta-rt Python package (pip install magenta-rt); weights live at google/magenta-realtime-2, code at github.com/magenta/magenta-realtime. The model is a dual transformer — Temporal_θ (compresses frame history) + Depth_φ (depth-wise RVQ token decode) — with causal sliding-window attention over the SpectroStream codec (48 kHz stereo, 25 Hz / 40 ms frames, 12 RVQ levels × 1024 vocab, 3 kbps), conditioned by quantized MusicCoCa embeddings (style), a 128-channel 25 Hz pianoroll (notes) and OaF binary drums.

This crate therefore does not reimplement that forward pass (the layer dims aren't even public). Enabling generation means bridging the official engine (an MLX/magenta-rt sidecar over local IPC, or FFI to the C++ engine) and flipping INFERENCE_BACKEND_LINKED — a deploy-time integration that requires the engine + weights on an Apple Silicon host. Until then everything fails closed.

Commands#

bash
# From libs/euterpe/realtime-engine (bypasses Nx per the worktree rules):
cargo test --workspace          # unit tests (budget, feasibility, codec, safetensors, discovery)
cargo check --workspace         # typecheck
cargo fmt --all --check         # lint

# Stage the native .node addon (fail-closed engine):
scripts/build-native.sh         # → dist/libs/euterpe/realtime-engine/native/<platform>-<arch>/mrt2_native.node

This crate does discovery / feasibility / codec math — it does NOT generate. Generation runs in the magenta_rt MLX sidecar (tools/euterpe-mrt2-sidecar/, wired via OSHUN_MRT2_SIDECAR); see that README for the runbook. The Rust napi path stays fail-closed.

Environment#

The Rust discovery helper (this crate, via the napi NativeOnDeviceModelRuntime):

Var Meaning
OSHUN_MRT2_RUNTIME auto | native | off (default auto)
OSHUN_MRT2_WEIGHTS_DIR Path to an unpacked bundle for the Rust discovery helper (NOT where generation weights live)
OSHUN_MRT2_TIER 230m | 2.4b (also selects the sidecar model size)
OSHUN_MRT2_ENGINE_PATH Explicit .node addon path for the discovery helper

The generation sidecar (the real engine; tools/euterpe-mrt2-sidecar/):

Var Meaning
OSHUN_MRT2_ENABLED true to enable the realtime route
OSHUN_MRT2_SIDECAR Absolute path to magenta_rt_sidecar.py (selects the sidecar over the Rust helper)
OSHUN_MRT2_PYTHON Interpreter that has magenta_rt (default python3)

Generation weights live in the magenta_rt checkpoints dir (mrt models download), not OSHUN_MRT2_WEIGHTS_DIR.