## Phase 176: Neural World Models & Latent Dynamics — Self-Supervised Predictive Representations, Model-Based RL, Action-Conditioned Video Worlds & Interactive Neural Simulators

Sovereign replication and extension of the frontier world-model stack: JEPA /
V-JEPA / I-JEPA (Meta FAIR), DreamerV3 / DayDreamer (Hafner et al.), MuZero /
EfficientZero / Stochastic MuZero (DeepMind), Genie 1/2/3 (DeepMind), DIAMOND
(ICLR 2024), GameNGen (Google), Oasis (Decart), and Sora / Cosmos-class video
world models. Powers Maya's infinite interactive universes, Galatea's
sample-efficient robotic policies, Hathor's living NPC cognition, and
@iris/reasoning look-ahead search. Replaces hand-authored simulators with
learned ones wherever the data volume justifies it.

Tech split: Python (~55%) for model training (PyTorch + JAX); Rust (~25%) for
real-time inference runtime, frame schedulers, and action-token codecs;
TypeScript (~20%) for agent orchestration, world APIs, and Maya/Galatea
integration.

Library layout: `libs/nous/world-model-core/`, `libs/nous/jepa/`,
`libs/nous/dreamer/`, `libs/nous/muzero/`, `libs/nous/genie/`,
`libs/nous/diffusion-world/`, `libs/nous/wm-memory/`, `libs/nous/wm-serving/`.

---

### 176.1 Shared Foundations — `@nous/world-model-core`

#### 176.1.1 Core Abstractions

- [x] **176.1.1.1** Create `libs/nous/world-model-core/` with `project.json`,
      `package.json`, `tsconfig.json`. Define `WorldModel<S, A, O>` interface:
      `encode(observation: O) → LatentState`,
      `step(state: LatentState,     action: A) → { next: LatentState; reward: number; continue: number;     pred: O? }`,
      `imagine(state, policy, horizon) → Trajectory`,
      `getUncertainty(state, action) → EpistemicUncertainty`
- [x] **176.1.1.2** Define `LatentState` as typed tagged-union of
      `DeterministicLatent` (vector), `StochasticLatent` (mean + logvar +
      sample), `CategoricalLatent` (k×c one-hot) and `TokenLatent` (codebook
      indices) — matching Dreamer, JEPA, and Genie state families
- [x] **176.1.1.3** Define `ActionSpace` algebra — `ContinuousBox`,
      `DiscreteFinite`, `MultiDiscrete`,
      `MultiModal (mouse + keyboard +     text)`, with serialisation to/from
      Genie-style latent action tokens
- [x] **176.1.1.4** Build unified replay-buffer + trajectory-dataset library
      consuming Phase 85 flywheel outputs (Maya sessions, Galatea teleop, Psyche
      computer-use) with prioritised sampling, n-step returns, and
      variable-horizon slicing
- [x] **176.1.1.5** Build imagination-rollout engine reused across Dreamer /
      MuZero / latent-policy gradient training — produces synthetic trajectories
      in latent space without decoder calls

---

### 176.2 Self-Supervised Predictive Representations (JEPA Family)

#### 176.2.1 I-JEPA & Masked Latent Prediction

- [ ] **176.2.1.1** Train I-JEPA base (ViT-H/14, 1600 epochs on ImageNet-1k +
      LVD-142M) predicting latent target blocks from a context block, to a
      published linear-probe target. Needs an owner decision on a compute and
      data budget before any run — this is frontier-scale neural-net training,
      not a coding task (stated 2026-09-18). `blocked:governance`
- [ ] **176.2.1.2** Integrate a published I-JEPA encoder as the
      `@nous/world-model-core` vision backbone for Psyche and Galatea
      perception, replacing DINOv2 where throughput allows. Use the publicly
      released pretrained I-JEPA weights (no training) exported to ONNX and run
      on CPU. Verify: a spec loads the encoder, embeds a fixture image set,
      asserts the embedding dimension and determinism, and a throughput
      benchmark records CPU latency vs the current DINOv2 backbone so the "where
      throughput permits" swap is a measured decision.
- [ ] **176.2.1.3** Train I-JEPA-3D on ShapeNet + Objaverse-XL for the 3D
      representation priors `@maya/environments` uses for world seeding. Needs
      an owner decision on a compute and data budget before any run — this is
      frontier-scale neural-net training, not a coding task (stated 2026-09-18).
      The dataset does not exist in a usable, licensed form here (Objaverse-XL
      requires bulk licensed 3D-asset download and curation).
      `blocked:governance` `blocked:corpus`

#### 176.2.2 V-JEPA 2 Video Prediction

- [ ] **176.2.2.1** Train V-JEPA 2 on Something-Something-v2 + Ego4D + HowTo100M
      (1.2B clips) with latent tube masking and an EMA target, as the Maya
      world-model visual backbone. Needs an owner decision on a compute and data
      budget before any run — this is frontier-scale neural-net training, not a
      coding task (stated 2026-09-18). The dataset does not exist in a usable,
      licensed form here (Ego4D and HowTo100M are access-restricted /
      license-gated video corpora). `blocked:governance` `blocked:corpus`
- [ ] **176.2.2.2** Build and train the V-JEPA 2 action-conditioned variant —
      latent prediction conditioned on past latents and an action sequence — for
      zero-shot action-plan rollouts. Needs an owner decision on a compute and
      data budget before any run — this is frontier-scale neural-net training,
      not a coding task (stated 2026-09-18). `blocked:governance`
- [ ] **176.2.2.3** Fine-tune V-JEPA 2 for Galatea egocentric manipulation on
      DROID + Open-X-Embodiment (2.4M episodes), quantifying the VLA-downstream
      success-rate lift from frozen features. Needs an owner decision on a
      compute and data budget before any run — this is frontier-scale neural-net
      training, not a coding task (stated 2026-09-18). The dataset does not
      exist in a usable, licensed form here (DROID and Open-X-Embodiment are
      large robotics corpora that must be fetched and stored).
      `blocked:governance` `blocked:corpus`

#### 176.2.3 Audio-JEPA & Multimodal JEPA

- [ ] **176.2.3.1** Train AudioJEPA on AudioSet-2M + Common Voice + AudioCaps
      with spectrogram tube masking, as a shared encoder for Iris prosody,
      Euterpe music embedding and more. Needs an owner decision on a compute and
      data budget before any run — this is frontier-scale neural-net training,
      not a coding task (stated 2026-09-18). `blocked:governance`
- [ ] **176.2.3.2** Train the MM-JEPA joint encoder aligning V-JEPA +
      AudioJEPA + Gemini-Embeddings-2 latents via a dual-tower contrastive head.
      Needs an owner decision on a compute and data budget before any run — this
      is frontier-scale neural-net training, not a coding task (stated
      2026-09-18). Also note: it aligns to Gemini-Embeddings-2, a proprietary
      frontier embedding — a sovereign model should not bake in that dependency;
      resolve that in the owner decision. `blocked:governance`

---

### 176.3 DreamerV3-Class Model-Based RL

#### 176.3.1 RSSM World Model

- [x] **176.3.1.1** Implement Recurrent State-Space Model (RSSM) — deterministic
      GRU branch + 32×32 categorical stochastic branch, KL balancing α=0.8,
      free-bits 1.0 nats per latent, symlog observation predictor, twohot reward
      / continue heads — byte-for-byte parity with DreamerV3 reference
- [x] **176.3.1.2** Implement actor-critic in imagination — percentile-based
      reward normalisation, lambda-returns (λ=0.95), entropy schedule 3e-4 →
      1e-4, gradient clip 100 — reproduce published DreamerV3 Atari-100k mean
      HNS
- [ ] **176.3.1.3** Train DreamerV3-XL (200M params, 256-dim latent) on Crafter,
      Minecraft diamond-from-scratch, DMLab-30, Atari-200M and the Hathor
      economy sim, to the published diamond-rate target. Needs an owner decision
      on a compute and data budget before any run — this is frontier-scale
      neural-net training, not a coding task (stated 2026-09-18).
      `blocked:governance`
- [ ] **176.3.1.4** Implement DayDreamer-style on-robot world-model training for
      Galatea (quadruped locomotion, ~1 h on-hardware training) and validate
      zero-shot terrain transfer. Needs a physical quadruped robot to train on,
      which the fleet does not have, and a compute/data-budget decision (stated
      2026-09-18). `blocked:hardware` `blocked:governance`

#### 176.3.2 DreamerV3 Specialisations

- [x] **176.3.2.1** Integrate PaliGemma-class VLM as conditional prior for
      multi-task DreamerV3, enabling natural-language goal specification in Maya
      quests and Galatea tasks
- [x] **176.3.2.2** Build offline DreamerV3 mode consuming Phase 85
      passive-flywheel trajectories (no live env) for sample-efficient policy
      pre-training before on-env fine-tuning
- [x] **176.3.2.3** Build hierarchical DreamerV3 (Director-style options)
      decomposing long-horizon Maya quests into sub-goals proposed by a
      high-level latent policy

---

### 176.4 MuZero-Class Planning with Learned Models

#### 176.4.1 MuZero Core

- [x] **176.4.1.1** Implement MuZero — representation net (CNN/ViT → hidden
      state), dynamics net (hidden + action → next hidden + reward), prediction
      net (hidden → policy + value) — joint training on replay with n-step
      returns and reanalysed targets
- [x] **176.4.1.2** Implement MuZero MCTS (PUCT with c1=1.25, c2=19652,
      Dirichlet noise α=0.3, 50–800 simulations per move) with batched GPU tree
      search — reproduce AlphaZero-class Go / Chess / Shogi parity on curated
      positions
- [ ] **176.4.1.3** Train EfficientZero / Stochastic MuZero on Atari-100k and a
      custom Hathor political-intrigue simulator, to the published Atari-100k
      1.9x-human target. Needs an owner decision on a compute and data budget
      before any run — this is frontier-scale neural-net training, not a coding
      task (stated 2026-09-18). `blocked:governance`
- [x] **176.4.1.4** Ship MuZero inference kernel to `@iris/reasoning-thinking`
      as a general-purpose look-ahead planner for tool-use and code-action
      decision trees

---

### 176.5 Genie-Class Action-Conditioned Video World Models

#### 176.5.1 Latent Action + Spatio-Temporal Transformer

- [ ] **176.5.1.1** Train a VQ-VAE video tokenizer (8-frame temporal, 16x16
      spatial, codebook 16384, EMA) on a 500k-hour mixed corpus. Needs an owner
      decision on a compute and data budget before any run — this is
      frontier-scale neural-net training, not a coding task (stated 2026-09-18).
      The dataset does not exist in a usable, licensed form here (the 500k-hour
      Maya-playthrough corpus does not exist and bulk YouTube gameplay is not
      licensed for training). `blocked:governance` `blocked:corpus`
- [ ] **176.5.1.2** Train the latent action model (LAM) — a causal
      ST-transformer predicting next-frame tokens and jointly learning a
      discrete latent-action codebook via a VQ bottleneck. Needs an owner
      decision on a compute and data budget before any run — this is
      frontier-scale neural-net training, not a coding task (stated 2026-09-18).
      `blocked:governance`
- [ ] **176.5.1.3** Train Genie-Sovereign (11B params, ST-transformer, 16-frame
      context, autoregressive next-frame-token generation conditioned on latent
      action) to a Genie-1-class interactivity target. Needs an owner decision
      on a compute and data budget before any run — this is frontier-scale
      neural-net training, not a coding task (stated 2026-09-18).
      `blocked:governance`
- [ ] **176.5.1.4** Train Genie-Sovereign-3 (real-time, memory-persistent,
      re-promptable) with cross-attention to a persistent memory bank and
      mid-world text prompting. Needs an owner decision on a compute and data
      budget before any run — this is frontier-scale neural-net training, not a
      coding task (stated 2026-09-18). `blocked:governance`
- [x] **176.5.1.5** Build Maya integration — `/maya/genie/create` endpoint
      accepting text prompt → interactive world session (WebRTC video + input
      stream); world persists to a durable state tree indexed by session

#### 176.5.2 Controllability & Memory

- [x] **176.5.2.1** Build long-horizon memory module — vector memory of visited
      regions keyed by camera pose, retrieved into dynamics model context so
      that "go back to the start" reproduces the original scene (Genie 3
      persistence requirement)
- [x] **176.5.2.2** Build adversarial-prompt interface enabling a second agent
      to rewrite the live world (Hadsell's "adversarially prompting your
      experience") — wire to `@maya/multiplayer` as a new PvP mode
- [x] **176.5.2.3** Build education-mode wrapper for `@metis/lessons` —
      teacher-specified prompt yields an interactive exploratory scene (e.g.
      "origami lizard in origami world", "inside the mitochondrion") with
      embedded learning objectives and assessment hooks

---

### 176.6 Diffusion-Based World Models (DIAMOND / GameNGen / Oasis)

#### 176.6.1 Diffusion Dynamics

- [x] **176.6.1.1** Implement DIAMOND — EDM diffusion over next-frame images
      conditioned on (past frames, action) with 3-step Heun sampler; train on
      Atari-100k and evaluate mean HNS versus DreamerV3 and IRIS
- [x] **176.6.1.2** Implement GameNGen-class Doom simulator — train diffusion
      dynamics on agent-collected rollouts (1B frames) reaching 20 FPS
      interactive generation; use as stress-test for the memory module
      (176.5.2.1) in long dungeon traversal
- [ ] **176.6.1.3** Implement and train an Oasis-style open-domain diffusion
      world on Minecraft-YouTube + Maya-alpha for free-form text-conditioned
      exploration. Needs an owner decision on a compute and data budget before
      any run — this is frontier-scale neural-net training, not a coding task
      (stated 2026-09-18). The dataset does not exist in a usable, licensed form
      here (bulk Minecraft-YouTube is not licensed for training and Maya-alpha
      playthrough data does not exist yet). `blocked:governance`
      `blocked:corpus`
- [x] **176.6.1.4** Build hybrid pipeline: Genie latent dynamics for long-range
      structure + diffusion decoder for photorealistic frames, balancing
      throughput and visual fidelity at runtime

---

### 176.7 Robotic World Models & Integration

#### 176.7.1 Galatea VLA World-Model Fusion

- [ ] **176.7.1.1** Train the Galatea world model on Open-X-Embodiment + DROID +
      internal teleop using the V-JEPA backbone and RSSM dynamics, for
      imagine-rollout policy generation. Needs an owner decision on a compute
      and data budget before any run — this is frontier-scale neural-net
      training, not a coding task (stated 2026-09-18). The dataset does not
      exist in a usable, licensed form here (the internal teleop dataset (Phase
      85.20.1.1) does not exist and Open-X/DROID need bulk fetch/storage).
      `blocked:governance` `blocked:corpus`
- [x] **176.7.1.2** Implement sim-to-real world-model adaptation via domain
      randomisation inside imagination (texture / lighting / physics noise) to
      halve the real-world fine-tune budget
- [x] **176.7.1.3** Build safety-shield imagination — before executing any
      VLA-proposed action, roll out N imagined trajectories; if any crosses a
      safety-violation classifier threshold, reject and resample — wired into
      `@galatea/safety` and `@kuanyin/precognition`

#### 176.7.2 NPC & Cognitive Integration

- [x] **176.7.2.1** Integrate world model with `@hathor/llm-npc` — NPCs plan via
      DreamerV3 imagination over a Hathor-simulation latent, giving
      goal-directed behaviour that respects economy / politics / culture
      dynamics without hand-written rules
- [x] **176.7.2.2** Integrate world model with `@iris/reasoning-thinking`
      tool-action planner — MuZero look-ahead over simulated tool responses
      before committing to a tool-call plan; latency-gated behind agent
      difficulty classifier
- [x] **176.7.2.3** Build Phase 81 NPC-cognitive-core upgrade path wiring the
      on-device SLM through the Phase 176.3 RSSM so NPCs can dream-plan on
      consumer hardware at ≤ 30 ms per decision

---

### 176.8 Serving, Memory & Evaluation

#### 176.8.1 Real-Time World-Model Runtime

- [x] **176.8.1.1** Build `@nous/wm-serving` Rust inference runtime — batched
      action-conditioned token generation, KV cache, speculative decoding for
      future-frame preview, WebRTC out; target 24 FPS 720p on single H100
- [x] **176.8.1.2** Build world-session persistence service — durable
      Zstd-compressed memory-bank + seed + prompt log in MinIO, signed
      `session_id → resumable URL`
- [x] **176.8.1.3** Build multi-player world joins — two Maya users enter the
      same Genie world with deterministic seed + shared memory bank and
      conflict-resolution on simultaneous prompts
- [x] **176.8.1.4** Build GPU autoscaler with world-session stickiness (session
      ↔ GPU affinity) and graceful migration on node drain

#### 176.8.2 Benchmarks & Data Flywheel

- [x] **176.8.2.1** Build world-model benchmark harness — Procgen, Atari-100k,
      Crafter, Minecraft diamond, DMLab-30, MineRL BASALT, RT-Bench, and
      internal Hathor / Maya suites — publish per-checkpoint scorecards into
      Phase 91 evaluation hub
- [x] **176.8.2.2** Build open-ended evaluation via POET-class automatic
      environment generation feeding Phase 176.5 Genie + Phase 176.3 Dreamer in
      a co-evolutionary loop, scoring emergent complexity
- [x] **176.8.2.3** Extend Phase 85 flywheel with `world-model-trajectories`
      collector — every Maya / Hathor / Galatea session emits
      `(obs_0, a_0, o_1, r_1, …)` tuples with quality labels for continual
      world-model retraining
- [x] **176.8.2.4** Build human-factors evaluation loop (consistency, memory
      fidelity, prompt-adherence, enjoyability) with paid raters and automatic
      LLM-judge backstop, wired into promotion gating for new world-model
      checkpoints

---
