# nous-dreamer

DreamerV3-class model-based reinforcement learning (Phase 176.3), implemented in
PyTorch with CPU-runnable forward **and** backward passes verified by overfit
tests.

## Components

- **RSSM world model** (`rssm.py`, `world_model.py`) — deterministic GRU branch
  - 32×32 categorical stochastic branch with a straight-through sampler, prior /
    posterior heads, KL balancing (α=0.8), free bits (1.0 nat), unimix 1 %,
    symlog observation reconstruction, and twohot symexp reward + Bernoulli
    continue heads. (176.3.1.1)
- **Actor-critic in imagination** (`actor_critic.py`) — backward λ-returns
  (λ=0.95), percentile return normalisation, annealed entropy schedule (3e-4 →
  1e-4), twohot value head with a slow EMA target, and gradient clipping at
  norm 100. Discrete and tanh-Normal continuous actors. (176.3.1.2)
- **Offline mode** (`offline.py`) — trains the world model and imagination
  actor-critic from a fixed dataset of logged flywheel trajectories with a
  behaviour-regularisation term, no live environment. (176.3.2.2)
- **Hierarchical Director** (`hierarchical.py`) — a goal autoencoder (discrete
  goal codebook), a manager policy acting every K steps, and a goal-conditioned
  worker with cosine-gain intrinsic reward. (176.3.2.3)
- **DayDreamer loop** (`daydreamer.py`) — bounded episode-safe robot replay and
  interleaved environment/training cadence with action repeat. This implements
  the on-robot algorithm seam; the one-hour quadruped result remains external.
- **VLM conditional prior** (`vlm_prior.py`) — injected PaliGemma goal encoder
  and gated language/vision-conditioned RSSM categorical prior for multi-task
  Dreamer. The production checkpoint remains an external asset.

## Not included (require GPU + full datasets — left unchecked in TODOS)

`176.3.1.3` (DreamerV3-XL 200M on Crafter/Minecraft/Atari-200M), the physical
quadruped training/terrain-transfer outcome in `176.3.1.4`, and the trained
PaliGemma checkpoint evaluation in `176.3.2.1` need accelerated training,
hardware, and external assets.

## Tests

```
cd libs/nous/dreamer && python3 -m pytest tests/ -q
```

The suite verifies the symlog/twohot algebra against closed forms, the RSSM
KL/free-bits/straight-through behaviour, and — crucially — that the world model
overfits a fixed sequence, the actor-critic solves an imagined bandit, the
offline trainer reduces world loss, and the goal autoencoder reconstructs
compressible features.