Historical task source. The Eve task board holds current task state.
Phase 176: Neural World Models & Latent Dynamics — Self-Supervised Predictive Representations, Model-Based RL, Action-Conditioned Video Worlds & Interactive Neural Simulators#
Sovereign replication and extension of the frontier world-model stack: JEPA / V-JEPA / I-JEPA (Meta FAIR), DreamerV3 / DayDreamer (Hafner et al.), MuZero / EfficientZero / Stochastic MuZero (DeepMind), Genie 1/2/3 (DeepMind), DIAMOND (ICLR 2024), GameNGen (Google), Oasis (Decart), and Sora / Cosmos-class video world models. Powers Maya's infinite interactive universes, Galatea's sample-efficient robotic policies, Hathor's living NPC cognition, and @iris/reasoning look-ahead search. Replaces hand-authored simulators with learned ones wherever the data volume justifies it.
Tech split: Python (~55%) for model training (PyTorch + JAX); Rust (~25%) for real-time inference runtime, frame schedulers, and action-token codecs; TypeScript (~20%) for agent orchestration, world APIs, and Maya/Galatea integration.
Library layout: libs/nous/world-model-core/, libs/nous/jepa/,
libs/nous/dreamer/, libs/nous/muzero/, libs/nous/genie/,
libs/nous/diffusion-world/, libs/nous/wm-memory/, libs/nous/wm-serving/.
176.1 Shared Foundations — @nous/world-model-core#
176.1.1 Core Abstractions#
- 176.1.1.1 Create
libs/nous/world-model-core/withproject.json,package.json,tsconfig.json. DefineWorldModel<S, A, O>interface:encode(observation: O) → LatentState,step(state: LatentState, action: A) → { next: LatentState; reward: number; continue: number; pred: O? },imagine(state, policy, horizon) → Trajectory,getUncertainty(state, action) → EpistemicUncertainty - 176.1.1.2 Define
LatentStateas typed tagged-union ofDeterministicLatent(vector),StochasticLatent(mean + logvar + sample),CategoricalLatent(k×c one-hot) andTokenLatent(codebook indices) — matching Dreamer, JEPA, and Genie state families - 176.1.1.3 Define
ActionSpacealgebra —ContinuousBox,DiscreteFinite,MultiDiscrete,MultiModal (mouse + keyboard + text), with serialisation to/from Genie-style latent action tokens - 176.1.1.4 Build unified replay-buffer + trajectory-dataset library consuming Phase 85 flywheel outputs (Maya sessions, Galatea teleop, Psyche computer-use) with prioritised sampling, n-step returns, and variable-horizon slicing
- 176.1.1.5 Build imagination-rollout engine reused across Dreamer / MuZero / latent-policy gradient training — produces synthetic trajectories in latent space without decoder calls
176.2 Self-Supervised Predictive Representations (JEPA Family)#
176.2.1 I-JEPA & Masked Latent Prediction#
- 176.2.1.1 Train I-JEPA base (ViT-H/14, 1600 epochs on ImageNet-1k +
LVD-142M) predicting latent target blocks from a context block, to a
published linear-probe target. Needs an owner decision on a compute and
data budget before any run — this is frontier-scale neural-net training,
not a coding task (stated 2026-09-18).
blocked:governance - 176.2.1.2 Integrate a published I-JEPA encoder as the
@nous/world-model-corevision backbone for Psyche and Galatea perception, replacing DINOv2 where throughput allows. Use the publicly released pretrained I-JEPA weights (no training) exported to ONNX and run on CPU. Verify: a spec loads the encoder, embeds a fixture image set, asserts the embedding dimension and determinism, and a throughput benchmark records CPU latency vs the current DINOv2 backbone so the "where throughput permits" swap is a measured decision. - 176.2.1.3 Train I-JEPA-3D on ShapeNet + Objaverse-XL for the 3D
representation priors
@maya/environmentsuses for world seeding. Needs an owner decision on a compute and data budget before any run — this is frontier-scale neural-net training, not a coding task (stated 2026-09-18). The dataset does not exist in a usable, licensed form here (Objaverse-XL requires bulk licensed 3D-asset download and curation).blocked:governanceblocked:corpus
176.2.2 V-JEPA 2 Video Prediction#
- 176.2.2.1 Train V-JEPA 2 on Something-Something-v2 + Ego4D + HowTo100M
(1.2B clips) with latent tube masking and an EMA target, as the Maya
world-model visual backbone. Needs an owner decision on a compute and data
budget before any run — this is frontier-scale neural-net training, not a
coding task (stated 2026-09-18). The dataset does not exist in a usable,
licensed form here (Ego4D and HowTo100M are access-restricted /
license-gated video corpora).
blocked:governanceblocked:corpus - 176.2.2.2 Build and train the V-JEPA 2 action-conditioned variant —
latent prediction conditioned on past latents and an action sequence — for
zero-shot action-plan rollouts. Needs an owner decision on a compute and
data budget before any run — this is frontier-scale neural-net training,
not a coding task (stated 2026-09-18).
blocked:governance - 176.2.2.3 Fine-tune V-JEPA 2 for Galatea egocentric manipulation on
DROID + Open-X-Embodiment (2.4M episodes), quantifying the VLA-downstream
success-rate lift from frozen features. Needs an owner decision on a
compute and data budget before any run — this is frontier-scale neural-net
training, not a coding task (stated 2026-09-18). The dataset does not
exist in a usable, licensed form here (DROID and Open-X-Embodiment are
large robotics corpora that must be fetched and stored).
blocked:governanceblocked:corpus
176.2.3 Audio-JEPA & Multimodal JEPA#
- 176.2.3.1 Train AudioJEPA on AudioSet-2M + Common Voice + AudioCaps
with spectrogram tube masking, as a shared encoder for Iris prosody,
Euterpe music embedding and more. Needs an owner decision on a compute and
data budget before any run — this is frontier-scale neural-net training,
not a coding task (stated 2026-09-18).
blocked:governance - 176.2.3.2 Train the MM-JEPA joint encoder aligning V-JEPA +
AudioJEPA + Gemini-Embeddings-2 latents via a dual-tower contrastive head.
Needs an owner decision on a compute and data budget before any run — this
is frontier-scale neural-net training, not a coding task (stated
2026-09-18). Also note: it aligns to Gemini-Embeddings-2, a proprietary
frontier embedding — a sovereign model should not bake in that dependency;
resolve that in the owner decision.
blocked:governance
176.3 DreamerV3-Class Model-Based RL#
176.3.1 RSSM World Model#
- 176.3.1.1 Implement Recurrent State-Space Model (RSSM) — deterministic GRU branch + 32×32 categorical stochastic branch, KL balancing α=0.8, free-bits 1.0 nats per latent, symlog observation predictor, twohot reward / continue heads — byte-for-byte parity with DreamerV3 reference
- 176.3.1.2 Implement actor-critic in imagination — percentile-based reward normalisation, lambda-returns (λ=0.95), entropy schedule 3e-4 → 1e-4, gradient clip 100 — reproduce published DreamerV3 Atari-100k mean HNS
- 176.3.1.3 Train DreamerV3-XL (200M params, 256-dim latent) on Crafter,
Minecraft diamond-from-scratch, DMLab-30, Atari-200M and the Hathor
economy sim, to the published diamond-rate target. Needs an owner decision
on a compute and data budget before any run — this is frontier-scale
neural-net training, not a coding task (stated 2026-09-18).
blocked:governance - 176.3.1.4 Implement DayDreamer-style on-robot world-model training for
Galatea (quadruped locomotion, ~1 h on-hardware training) and validate
zero-shot terrain transfer. Needs a physical quadruped robot to train on,
which the fleet does not have, and a compute/data-budget decision (stated
2026-09-18).
blocked:hardwareblocked:governance
176.3.2 DreamerV3 Specialisations#
- 176.3.2.1 Integrate PaliGemma-class VLM as conditional prior for multi-task DreamerV3, enabling natural-language goal specification in Maya quests and Galatea tasks
- 176.3.2.2 Build offline DreamerV3 mode consuming Phase 85 passive-flywheel trajectories (no live env) for sample-efficient policy pre-training before on-env fine-tuning
- 176.3.2.3 Build hierarchical DreamerV3 (Director-style options) decomposing long-horizon Maya quests into sub-goals proposed by a high-level latent policy
176.4 MuZero-Class Planning with Learned Models#
176.4.1 MuZero Core#
- 176.4.1.1 Implement MuZero — representation net (CNN/ViT → hidden state), dynamics net (hidden + action → next hidden + reward), prediction net (hidden → policy + value) — joint training on replay with n-step returns and reanalysed targets
- 176.4.1.2 Implement MuZero MCTS (PUCT with c1=1.25, c2=19652, Dirichlet noise α=0.3, 50–800 simulations per move) with batched GPU tree search — reproduce AlphaZero-class Go / Chess / Shogi parity on curated positions
- 176.4.1.3 Train EfficientZero / Stochastic MuZero on Atari-100k and a
custom Hathor political-intrigue simulator, to the published Atari-100k
1.9x-human target. Needs an owner decision on a compute and data budget
before any run — this is frontier-scale neural-net training, not a coding
task (stated 2026-09-18).
blocked:governance - 176.4.1.4 Ship MuZero inference kernel to
@iris/reasoning-thinkingas a general-purpose look-ahead planner for tool-use and code-action decision trees
176.5 Genie-Class Action-Conditioned Video World Models#
176.5.1 Latent Action + Spatio-Temporal Transformer#
- 176.5.1.1 Train a VQ-VAE video tokenizer (8-frame temporal, 16x16
spatial, codebook 16384, EMA) on a 500k-hour mixed corpus. Needs an owner
decision on a compute and data budget before any run — this is
frontier-scale neural-net training, not a coding task (stated 2026-09-18).
The dataset does not exist in a usable, licensed form here (the 500k-hour
Maya-playthrough corpus does not exist and bulk YouTube gameplay is not
licensed for training).
blocked:governanceblocked:corpus - 176.5.1.2 Train the latent action model (LAM) — a causal
ST-transformer predicting next-frame tokens and jointly learning a
discrete latent-action codebook via a VQ bottleneck. Needs an owner
decision on a compute and data budget before any run — this is
frontier-scale neural-net training, not a coding task (stated 2026-09-18).
blocked:governance - 176.5.1.3 Train Genie-Sovereign (11B params, ST-transformer, 16-frame
context, autoregressive next-frame-token generation conditioned on latent
action) to a Genie-1-class interactivity target. Needs an owner decision
on a compute and data budget before any run — this is frontier-scale
neural-net training, not a coding task (stated 2026-09-18).
blocked:governance - 176.5.1.4 Train Genie-Sovereign-3 (real-time, memory-persistent,
re-promptable) with cross-attention to a persistent memory bank and
mid-world text prompting. Needs an owner decision on a compute and data
budget before any run — this is frontier-scale neural-net training, not a
coding task (stated 2026-09-18).
blocked:governance - 176.5.1.5 Build Maya integration —
/maya/genie/createendpoint accepting text prompt → interactive world session (WebRTC video + input stream); world persists to a durable state tree indexed by session
176.5.2 Controllability & Memory#
- 176.5.2.1 Build long-horizon memory module — vector memory of visited regions keyed by camera pose, retrieved into dynamics model context so that "go back to the start" reproduces the original scene (Genie 3 persistence requirement)
- 176.5.2.2 Build adversarial-prompt interface enabling a second agent
to rewrite the live world (Hadsell's "adversarially prompting your
experience") — wire to
@maya/multiplayeras a new PvP mode - 176.5.2.3 Build education-mode wrapper for
@metis/lessons— teacher-specified prompt yields an interactive exploratory scene (e.g. "origami lizard in origami world", "inside the mitochondrion") with embedded learning objectives and assessment hooks
176.6 Diffusion-Based World Models (DIAMOND / GameNGen / Oasis)#
176.6.1 Diffusion Dynamics#
- 176.6.1.1 Implement DIAMOND — EDM diffusion over next-frame images conditioned on (past frames, action) with 3-step Heun sampler; train on Atari-100k and evaluate mean HNS versus DreamerV3 and IRIS
- 176.6.1.2 Implement GameNGen-class Doom simulator — train diffusion dynamics on agent-collected rollouts (1B frames) reaching 20 FPS interactive generation; use as stress-test for the memory module (176.5.2.1) in long dungeon traversal
- 176.6.1.3 Implement and train an Oasis-style open-domain diffusion
world on Minecraft-YouTube + Maya-alpha for free-form text-conditioned
exploration. Needs an owner decision on a compute and data budget before
any run — this is frontier-scale neural-net training, not a coding task
(stated 2026-09-18). The dataset does not exist in a usable, licensed form
here (bulk Minecraft-YouTube is not licensed for training and Maya-alpha
playthrough data does not exist yet).
blocked:governanceblocked:corpus - 176.6.1.4 Build hybrid pipeline: Genie latent dynamics for long-range structure + diffusion decoder for photorealistic frames, balancing throughput and visual fidelity at runtime
176.7 Robotic World Models & Integration#
176.7.1 Galatea VLA World-Model Fusion#
- 176.7.1.1 Train the Galatea world model on Open-X-Embodiment + DROID +
internal teleop using the V-JEPA backbone and RSSM dynamics, for
imagine-rollout policy generation. Needs an owner decision on a compute
and data budget before any run — this is frontier-scale neural-net
training, not a coding task (stated 2026-09-18). The dataset does not
exist in a usable, licensed form here (the internal teleop dataset (Phase
85.20.1.1) does not exist and Open-X/DROID need bulk fetch/storage).
blocked:governanceblocked:corpus - 176.7.1.2 Implement sim-to-real world-model adaptation via domain randomisation inside imagination (texture / lighting / physics noise) to halve the real-world fine-tune budget
- 176.7.1.3 Build safety-shield imagination — before executing any
VLA-proposed action, roll out N imagined trajectories; if any crosses a
safety-violation classifier threshold, reject and resample — wired into
@galatea/safetyand@kuanyin/precognition
176.7.2 NPC & Cognitive Integration#
- 176.7.2.1 Integrate world model with
@hathor/llm-npc— NPCs plan via DreamerV3 imagination over a Hathor-simulation latent, giving goal-directed behaviour that respects economy / politics / culture dynamics without hand-written rules - 176.7.2.2 Integrate world model with
@iris/reasoning-thinkingtool-action planner — MuZero look-ahead over simulated tool responses before committing to a tool-call plan; latency-gated behind agent difficulty classifier - 176.7.2.3 Build Phase 81 NPC-cognitive-core upgrade path wiring the on-device SLM through the Phase 176.3 RSSM so NPCs can dream-plan on consumer hardware at ≤ 30 ms per decision
176.8 Serving, Memory & Evaluation#
176.8.1 Real-Time World-Model Runtime#
- 176.8.1.1 Build
@nous/wm-servingRust inference runtime — batched action-conditioned token generation, KV cache, speculative decoding for future-frame preview, WebRTC out; target 24 FPS 720p on single H100 - 176.8.1.2 Build world-session persistence service — durable
Zstd-compressed memory-bank + seed + prompt log in MinIO, signed
session_id → resumable URL - 176.8.1.3 Build multi-player world joins — two Maya users enter the same Genie world with deterministic seed + shared memory bank and conflict-resolution on simultaneous prompts
- 176.8.1.4 Build GPU autoscaler with world-session stickiness (session ↔ GPU affinity) and graceful migration on node drain
176.8.2 Benchmarks & Data Flywheel#
- 176.8.2.1 Build world-model benchmark harness — Procgen, Atari-100k, Crafter, Minecraft diamond, DMLab-30, MineRL BASALT, RT-Bench, and internal Hathor / Maya suites — publish per-checkpoint scorecards into Phase 91 evaluation hub
- 176.8.2.2 Build open-ended evaluation via POET-class automatic environment generation feeding Phase 176.5 Genie + Phase 176.3 Dreamer in a co-evolutionary loop, scoring emergent complexity
- 176.8.2.3 Extend Phase 85 flywheel with
world-model-trajectoriescollector — every Maya / Hathor / Galatea session emits(obs_0, a_0, o_1, r_1, …)tuples with quality labels for continual world-model retraining - 176.8.2.4 Build human-factors evaluation loop (consistency, memory fidelity, prompt-adherence, enjoyability) with paid raters and automatic LLM-judge backstop, wired into promotion gating for new world-model checkpoints