Eve library

phase-176.md

Tracker

Revision 1

On this page

Historical task source. The Eve task board holds current task state.

Phase 176: Neural World Models & Latent Dynamics — Self-Supervised Predictive Representations, Model-Based RL, Action-Conditioned Video Worlds & Interactive Neural Simulators#

Sovereign replication and extension of the frontier world-model stack: JEPA / V-JEPA / I-JEPA (Meta FAIR), DreamerV3 / DayDreamer (Hafner et al.), MuZero / EfficientZero / Stochastic MuZero (DeepMind), Genie 1/2/3 (DeepMind), DIAMOND (ICLR 2024), GameNGen (Google), Oasis (Decart), and Sora / Cosmos-class video world models. Powers Maya's infinite interactive universes, Galatea's sample-efficient robotic policies, Hathor's living NPC cognition, and @iris/reasoning look-ahead search. Replaces hand-authored simulators with learned ones wherever the data volume justifies it.

Tech split: Python (~55%) for model training (PyTorch + JAX); Rust (~25%) for real-time inference runtime, frame schedulers, and action-token codecs; TypeScript (~20%) for agent orchestration, world APIs, and Maya/Galatea integration.

Library layout: libs/nous/world-model-core/, libs/nous/jepa/, libs/nous/dreamer/, libs/nous/muzero/, libs/nous/genie/, libs/nous/diffusion-world/, libs/nous/wm-memory/, libs/nous/wm-serving/.


176.1 Shared Foundations — @nous/world-model-core#

176.1.1 Core Abstractions#

  • 176.1.1.1 Create libs/nous/world-model-core/ with project.json, package.json, tsconfig.json. Define WorldModel<S, A, O> interface: encode(observation: O) → LatentState, step(state: LatentState, action: A) → { next: LatentState; reward: number; continue: number; pred: O? }, imagine(state, policy, horizon) → Trajectory, getUncertainty(state, action) → EpistemicUncertainty
  • 176.1.1.2 Define LatentState as typed tagged-union of DeterministicLatent (vector), StochasticLatent (mean + logvar + sample), CategoricalLatent (k×c one-hot) and TokenLatent (codebook indices) — matching Dreamer, JEPA, and Genie state families
  • 176.1.1.3 Define ActionSpace algebra — ContinuousBox, DiscreteFinite, MultiDiscrete, MultiModal (mouse + keyboard + text), with serialisation to/from Genie-style latent action tokens
  • 176.1.1.4 Build unified replay-buffer + trajectory-dataset library consuming Phase 85 flywheel outputs (Maya sessions, Galatea teleop, Psyche computer-use) with prioritised sampling, n-step returns, and variable-horizon slicing
  • 176.1.1.5 Build imagination-rollout engine reused across Dreamer / MuZero / latent-policy gradient training — produces synthetic trajectories in latent space without decoder calls

176.2 Self-Supervised Predictive Representations (JEPA Family)#

176.2.1 I-JEPA & Masked Latent Prediction#

  • 176.2.1.1 Train I-JEPA base (ViT-H/14, 1600 epochs on ImageNet-1k + LVD-142M) predicting latent target blocks from a context block, to a published linear-probe target. Needs an owner decision on a compute and data budget before any run — this is frontier-scale neural-net training, not a coding task (stated 2026-09-18). blocked:governance
  • 176.2.1.2 Integrate a published I-JEPA encoder as the @nous/world-model-core vision backbone for Psyche and Galatea perception, replacing DINOv2 where throughput allows. Use the publicly released pretrained I-JEPA weights (no training) exported to ONNX and run on CPU. Verify: a spec loads the encoder, embeds a fixture image set, asserts the embedding dimension and determinism, and a throughput benchmark records CPU latency vs the current DINOv2 backbone so the "where throughput permits" swap is a measured decision.
  • 176.2.1.3 Train I-JEPA-3D on ShapeNet + Objaverse-XL for the 3D representation priors @maya/environments uses for world seeding. Needs an owner decision on a compute and data budget before any run — this is frontier-scale neural-net training, not a coding task (stated 2026-09-18). The dataset does not exist in a usable, licensed form here (Objaverse-XL requires bulk licensed 3D-asset download and curation). blocked:governance blocked:corpus

176.2.2 V-JEPA 2 Video Prediction#

  • 176.2.2.1 Train V-JEPA 2 on Something-Something-v2 + Ego4D + HowTo100M (1.2B clips) with latent tube masking and an EMA target, as the Maya world-model visual backbone. Needs an owner decision on a compute and data budget before any run — this is frontier-scale neural-net training, not a coding task (stated 2026-09-18). The dataset does not exist in a usable, licensed form here (Ego4D and HowTo100M are access-restricted / license-gated video corpora). blocked:governance blocked:corpus
  • 176.2.2.2 Build and train the V-JEPA 2 action-conditioned variant — latent prediction conditioned on past latents and an action sequence — for zero-shot action-plan rollouts. Needs an owner decision on a compute and data budget before any run — this is frontier-scale neural-net training, not a coding task (stated 2026-09-18). blocked:governance
  • 176.2.2.3 Fine-tune V-JEPA 2 for Galatea egocentric manipulation on DROID + Open-X-Embodiment (2.4M episodes), quantifying the VLA-downstream success-rate lift from frozen features. Needs an owner decision on a compute and data budget before any run — this is frontier-scale neural-net training, not a coding task (stated 2026-09-18). The dataset does not exist in a usable, licensed form here (DROID and Open-X-Embodiment are large robotics corpora that must be fetched and stored). blocked:governance blocked:corpus

176.2.3 Audio-JEPA & Multimodal JEPA#

  • 176.2.3.1 Train AudioJEPA on AudioSet-2M + Common Voice + AudioCaps with spectrogram tube masking, as a shared encoder for Iris prosody, Euterpe music embedding and more. Needs an owner decision on a compute and data budget before any run — this is frontier-scale neural-net training, not a coding task (stated 2026-09-18). blocked:governance
  • 176.2.3.2 Train the MM-JEPA joint encoder aligning V-JEPA + AudioJEPA + Gemini-Embeddings-2 latents via a dual-tower contrastive head. Needs an owner decision on a compute and data budget before any run — this is frontier-scale neural-net training, not a coding task (stated 2026-09-18). Also note: it aligns to Gemini-Embeddings-2, a proprietary frontier embedding — a sovereign model should not bake in that dependency; resolve that in the owner decision. blocked:governance

176.3 DreamerV3-Class Model-Based RL#

176.3.1 RSSM World Model#

  • 176.3.1.1 Implement Recurrent State-Space Model (RSSM) — deterministic GRU branch + 32×32 categorical stochastic branch, KL balancing α=0.8, free-bits 1.0 nats per latent, symlog observation predictor, twohot reward / continue heads — byte-for-byte parity with DreamerV3 reference
  • 176.3.1.2 Implement actor-critic in imagination — percentile-based reward normalisation, lambda-returns (λ=0.95), entropy schedule 3e-4 → 1e-4, gradient clip 100 — reproduce published DreamerV3 Atari-100k mean HNS
  • 176.3.1.3 Train DreamerV3-XL (200M params, 256-dim latent) on Crafter, Minecraft diamond-from-scratch, DMLab-30, Atari-200M and the Hathor economy sim, to the published diamond-rate target. Needs an owner decision on a compute and data budget before any run — this is frontier-scale neural-net training, not a coding task (stated 2026-09-18). blocked:governance
  • 176.3.1.4 Implement DayDreamer-style on-robot world-model training for Galatea (quadruped locomotion, ~1 h on-hardware training) and validate zero-shot terrain transfer. Needs a physical quadruped robot to train on, which the fleet does not have, and a compute/data-budget decision (stated 2026-09-18). blocked:hardware blocked:governance

176.3.2 DreamerV3 Specialisations#

  • 176.3.2.1 Integrate PaliGemma-class VLM as conditional prior for multi-task DreamerV3, enabling natural-language goal specification in Maya quests and Galatea tasks
  • 176.3.2.2 Build offline DreamerV3 mode consuming Phase 85 passive-flywheel trajectories (no live env) for sample-efficient policy pre-training before on-env fine-tuning
  • 176.3.2.3 Build hierarchical DreamerV3 (Director-style options) decomposing long-horizon Maya quests into sub-goals proposed by a high-level latent policy

176.4 MuZero-Class Planning with Learned Models#

176.4.1 MuZero Core#

  • 176.4.1.1 Implement MuZero — representation net (CNN/ViT → hidden state), dynamics net (hidden + action → next hidden + reward), prediction net (hidden → policy + value) — joint training on replay with n-step returns and reanalysed targets
  • 176.4.1.2 Implement MuZero MCTS (PUCT with c1=1.25, c2=19652, Dirichlet noise α=0.3, 50–800 simulations per move) with batched GPU tree search — reproduce AlphaZero-class Go / Chess / Shogi parity on curated positions
  • 176.4.1.3 Train EfficientZero / Stochastic MuZero on Atari-100k and a custom Hathor political-intrigue simulator, to the published Atari-100k 1.9x-human target. Needs an owner decision on a compute and data budget before any run — this is frontier-scale neural-net training, not a coding task (stated 2026-09-18). blocked:governance
  • 176.4.1.4 Ship MuZero inference kernel to @iris/reasoning-thinking as a general-purpose look-ahead planner for tool-use and code-action decision trees

176.5 Genie-Class Action-Conditioned Video World Models#

176.5.1 Latent Action + Spatio-Temporal Transformer#

  • 176.5.1.1 Train a VQ-VAE video tokenizer (8-frame temporal, 16x16 spatial, codebook 16384, EMA) on a 500k-hour mixed corpus. Needs an owner decision on a compute and data budget before any run — this is frontier-scale neural-net training, not a coding task (stated 2026-09-18). The dataset does not exist in a usable, licensed form here (the 500k-hour Maya-playthrough corpus does not exist and bulk YouTube gameplay is not licensed for training). blocked:governance blocked:corpus
  • 176.5.1.2 Train the latent action model (LAM) — a causal ST-transformer predicting next-frame tokens and jointly learning a discrete latent-action codebook via a VQ bottleneck. Needs an owner decision on a compute and data budget before any run — this is frontier-scale neural-net training, not a coding task (stated 2026-09-18). blocked:governance
  • 176.5.1.3 Train Genie-Sovereign (11B params, ST-transformer, 16-frame context, autoregressive next-frame-token generation conditioned on latent action) to a Genie-1-class interactivity target. Needs an owner decision on a compute and data budget before any run — this is frontier-scale neural-net training, not a coding task (stated 2026-09-18). blocked:governance
  • 176.5.1.4 Train Genie-Sovereign-3 (real-time, memory-persistent, re-promptable) with cross-attention to a persistent memory bank and mid-world text prompting. Needs an owner decision on a compute and data budget before any run — this is frontier-scale neural-net training, not a coding task (stated 2026-09-18). blocked:governance
  • 176.5.1.5 Build Maya integration — /maya/genie/create endpoint accepting text prompt → interactive world session (WebRTC video + input stream); world persists to a durable state tree indexed by session

176.5.2 Controllability & Memory#

  • 176.5.2.1 Build long-horizon memory module — vector memory of visited regions keyed by camera pose, retrieved into dynamics model context so that "go back to the start" reproduces the original scene (Genie 3 persistence requirement)
  • 176.5.2.2 Build adversarial-prompt interface enabling a second agent to rewrite the live world (Hadsell's "adversarially prompting your experience") — wire to @maya/multiplayer as a new PvP mode
  • 176.5.2.3 Build education-mode wrapper for @metis/lessons — teacher-specified prompt yields an interactive exploratory scene (e.g. "origami lizard in origami world", "inside the mitochondrion") with embedded learning objectives and assessment hooks

176.6 Diffusion-Based World Models (DIAMOND / GameNGen / Oasis)#

176.6.1 Diffusion Dynamics#

  • 176.6.1.1 Implement DIAMOND — EDM diffusion over next-frame images conditioned on (past frames, action) with 3-step Heun sampler; train on Atari-100k and evaluate mean HNS versus DreamerV3 and IRIS
  • 176.6.1.2 Implement GameNGen-class Doom simulator — train diffusion dynamics on agent-collected rollouts (1B frames) reaching 20 FPS interactive generation; use as stress-test for the memory module (176.5.2.1) in long dungeon traversal
  • 176.6.1.3 Implement and train an Oasis-style open-domain diffusion world on Minecraft-YouTube + Maya-alpha for free-form text-conditioned exploration. Needs an owner decision on a compute and data budget before any run — this is frontier-scale neural-net training, not a coding task (stated 2026-09-18). The dataset does not exist in a usable, licensed form here (bulk Minecraft-YouTube is not licensed for training and Maya-alpha playthrough data does not exist yet). blocked:governance blocked:corpus
  • 176.6.1.4 Build hybrid pipeline: Genie latent dynamics for long-range structure + diffusion decoder for photorealistic frames, balancing throughput and visual fidelity at runtime

176.7 Robotic World Models & Integration#

176.7.1 Galatea VLA World-Model Fusion#

  • 176.7.1.1 Train the Galatea world model on Open-X-Embodiment + DROID + internal teleop using the V-JEPA backbone and RSSM dynamics, for imagine-rollout policy generation. Needs an owner decision on a compute and data budget before any run — this is frontier-scale neural-net training, not a coding task (stated 2026-09-18). The dataset does not exist in a usable, licensed form here (the internal teleop dataset (Phase 85.20.1.1) does not exist and Open-X/DROID need bulk fetch/storage). blocked:governance blocked:corpus
  • 176.7.1.2 Implement sim-to-real world-model adaptation via domain randomisation inside imagination (texture / lighting / physics noise) to halve the real-world fine-tune budget
  • 176.7.1.3 Build safety-shield imagination — before executing any VLA-proposed action, roll out N imagined trajectories; if any crosses a safety-violation classifier threshold, reject and resample — wired into @galatea/safety and @kuanyin/precognition

176.7.2 NPC & Cognitive Integration#

  • 176.7.2.1 Integrate world model with @hathor/llm-npc — NPCs plan via DreamerV3 imagination over a Hathor-simulation latent, giving goal-directed behaviour that respects economy / politics / culture dynamics without hand-written rules
  • 176.7.2.2 Integrate world model with @iris/reasoning-thinking tool-action planner — MuZero look-ahead over simulated tool responses before committing to a tool-call plan; latency-gated behind agent difficulty classifier
  • 176.7.2.3 Build Phase 81 NPC-cognitive-core upgrade path wiring the on-device SLM through the Phase 176.3 RSSM so NPCs can dream-plan on consumer hardware at ≤ 30 ms per decision

176.8 Serving, Memory & Evaluation#

176.8.1 Real-Time World-Model Runtime#

  • 176.8.1.1 Build @nous/wm-serving Rust inference runtime — batched action-conditioned token generation, KV cache, speculative decoding for future-frame preview, WebRTC out; target 24 FPS 720p on single H100
  • 176.8.1.2 Build world-session persistence service — durable Zstd-compressed memory-bank + seed + prompt log in MinIO, signed session_id → resumable URL
  • 176.8.1.3 Build multi-player world joins — two Maya users enter the same Genie world with deterministic seed + shared memory bank and conflict-resolution on simultaneous prompts
  • 176.8.1.4 Build GPU autoscaler with world-session stickiness (session ↔ GPU affinity) and graceful migration on node drain

176.8.2 Benchmarks & Data Flywheel#

  • 176.8.2.1 Build world-model benchmark harness — Procgen, Atari-100k, Crafter, Minecraft diamond, DMLab-30, MineRL BASALT, RT-Bench, and internal Hathor / Maya suites — publish per-checkpoint scorecards into Phase 91 evaluation hub
  • 176.8.2.2 Build open-ended evaluation via POET-class automatic environment generation feeding Phase 176.5 Genie + Phase 176.3 Dreamer in a co-evolutionary loop, scoring emergent complexity
  • 176.8.2.3 Extend Phase 85 flywheel with world-model-trajectories collector — every Maya / Hathor / Galatea session emits (obs_0, a_0, o_1, r_1, …) tuples with quality labels for continual world-model retraining
  • 176.8.2.4 Build human-factors evaluation loop (consistency, memory fidelity, prompt-adherence, enjoyability) with paid raters and automatic LLM-judge backstop, wired into promotion gating for new world-model checkpoints