MuZero-class planning with learned models (Phase 176.4), PyTorch, CPU-runnable with overfit-verified training.
Components#
- Networks (
networks.py,support.py) — representationh, dynamicsg, predictionfwith min-max-normalised hidden states and categorical value/reward heads over the MuZero scaling support. (176.4.1.1) - Training (
muzero.py) — joint unrolled training with categorical value/reward cross-entropy, ½-scaled recurrent gradients, n-step value targets, and aReanalyserthat recomputes value/policy targets with the current network via MCTS. (176.4.1.1) - MCTS (
mcts.py) — PUCT selection (c1=1.25, c2=19652), Dirichlet root noise (α=0.3), min-max Q normalisation, and batched tree search that stacks the per-simulation network calls into single forward passes. (176.4.1.2) - Planner kernel (
planner.py) —LookaheadPlannerdrives MCTS over anyPlanningModel(a learnedMuZeroNetworkor aSymbolicModelbuilt from explicit tool/code-action transition/reward callbacks), ready to ship to@iris/reasoning-thinkingfor tool-use look-ahead. (176.4.1.4)
Not included (external — left unchecked in TODOS)#
The local efficientzero.py implementation adds EfficientZero stopped-target
latent consistency and value-prefix heads plus Stochastic MuZero
afterstate/chance-node dynamics. 176.4.1.3 remains open because training these
models on Atari-100k and the Hathor simulator to the published 1.9× human score
needs full-scale accelerated training on Atari frames.
Tests#
cd libs/nous/muzero && python3 -m pytest tests/ -q
Verifies the support transform inverse + twohot reconstruction, hidden-state normalisation, MCTS visit concentration on a solvable model, the n-step target closed form, an overfit of a deterministic MDP (all three loss heads drop), and the planner selecting the optimal tool sequence in a decision tree.