Eve library

phase-177.md

Tracker

Revision 1

On this page

Historical task source. The Eve task board holds current task state.

Phase 177: Interpretability & Continual-Learning Research Track — Sparse Autoencoders, Circuit Discovery, Progressive Networks, EWC, CLEAR Replay, PackNet & Policy Distillation#

Sovereign research track covering the two frontier safety-and-robustness disciplines that the current ML Sovereignty phases (85–96) do not address in depth: mechanistic interpretability (Anthropic, DeepMind, OpenAI — sparse autoencoders, circuit discovery, attribution patching, transcoders) and continual learning (Hadsell et al. — progressive networks, EWC, online EWC, synaptic intelligence, MAS, CLEAR, PackNet, policy distillation, Kickstarting). Powers Kuan Yin safety audits, Galatea fleet-wide lifelong learning, and alignment research defensibility in front of UK AISI / US AISI.

Tech split: Python (~70%) for research-grade experimentation (PyTorch + TransformerLens + sae-lens); Rust (~10%) for high-throughput SAE inference kernels and streaming activation capture; TypeScript (~20%) for dashboards (interp explorer, continual-learning fleet monitor) and serving integration.

Library layout: libs/nous/interp-core/, libs/nous/sae/, libs/nous/circuits/, libs/nous/probes/, libs/nous/continual-core/, libs/nous/progressive-nets/, libs/nous/replay/, libs/nous/distillation/, libs/nous/interp-dashboard/.


177.1 Interpretability Foundations — @nous/interp-core#

177.1.1 Activation Capture & Tooling#

  • 177.1.1.1 Create libs/nous/interp-core/ with project.json, package.json, tsconfig.json. Define HookedModel<M> wrapper over any @nous/llm or @nous/vision model exposing streaming hooks at every residual-stream layer, MLP pre/post, attention Q/K/V/O, and logits — TransformerLens-compatible semantics
  • 177.1.1.2 Build ActivationCapture recorder writing Zarr-chunked activation tensors keyed by (model_rev, dataset, sample_id, layer, point) with per-sample metadata and deterministic replay
  • 177.1.1.3 Build ActivationStats online estimator (mean, cov, diagonal Fisher, percentile-100 activation magnitudes) used by SAE initialisation and Fisher-based EWC (177.7)
  • 177.1.1.4 Ship a cross-model adapter covering Llama-4 MoE, Qwen3 dense + MoE, Gemma-4, Mixtral / DeepSeek V3 MoE routers, and Nous's own fine-tunes — guarantees every Oshun-served model is mechanically inspectable out of the box

177.1.2 Attribution Primitives#

  • 177.1.2.1 Implement activation patching (causal intervention: replace activation at (layer, point, token) from donor → recipient, measure logit shift) with batched single-backward implementation
  • 177.1.2.2 Implement Integrated Gradients, SmoothGrad, and LIME attribution backends for feature-attribution dashboards and safety-relevant model cards
  • 177.1.2.3 Implement attribution-patching (gradient-based approximation to activation patching) enabling per-token per-layer attribution at whole-dataset scale with O(1) forward/backward per prompt

177.2 Sparse Autoencoders — @nous/sae#

177.2.1 SAE Training#

  • 177.2.1.1 Implement JumpReLU SAE (DeepMind 2024) with per-feature threshold + straight-through estimator — primary SAE family, target L0 ≈ 60 and reconstruction MSE parity with published Gemma Scope
  • 177.2.1.2 Implement TopK SAE (OpenAI 2024) with aux_k dead-feature revival loss and 4× expansion factor — complementary family for controlled comparisons
  • 177.2.1.3 Implement GatedSAE baseline for architecture ablations
  • 177.2.1.4 Build SAE training pipeline — stream activations from 177.1.1.2, shuffle via reservoir sampler, batch 4096, Adam β₂=0.999, learning-rate warmup, λ annealing, feature-resampling on dead features every 25k steps, EMA checkpointing every 10k
  • 177.2.1.5 Train a full SAE suite on the Gemma-4-9B / Qwen3-8B residual stream at every layer (expansion x16/x32/x64) and MLP-post, publishing feature dashboards. Needs an owner decision on a compute budget — this is large-scale SAE/interpretability training or a circuit investigation over frontier checkpoints, not a coding task (stated 2026-09-18). blocked:governance
  • 177.2.1.6 Train SAEs on the fine-tuned @iris/code and @kuanyin/safety heads and publish per-domain feature dashboards. Needs an owner decision on a compute budget — this is large-scale SAE/interpretability training or a circuit investigation over frontier checkpoints, not a coding task (stated 2026-09-18). It also needs those fine-tuned checkpoints, which are themselves frontier-trained. blocked:governance

177.2.2 Feature Analysis#

  • 177.2.2.1 Build autointerp pipeline (Bills et al. style) — for each SAE feature, sample top-activating contexts, ask a frontier LLM for a natural-language label, score label fidelity via held-out-example prediction — run at publish-scale over every SAE
  • 177.2.2.2 Build feature-splitting detector tracking how features divide under scale-ups (×8 → ×32 → ×64) to identify canonical vs derivative features
  • 177.2.2.3 Build feature-ablation steering — zero or scale specific SAE features at inference, measure downstream behavioural shift (refusal rate, sentiment, tool-use style) on eval suites — expose as @nous/interp-core steering API for Kuan Yin safety researchers
  • 177.2.2.4 Build concept-dictionary export — feature → label → safety category (toxicity, deception, CSAM, self-harm, jailbreak-adjacent) used by Kuan Yin classifiers as auxiliary supervision (Phase 85.5 integration)

177.3 Circuits & Feature Graphs — @nous/circuits#

177.3.1 Circuit Discovery#

  • 177.3.1.1 Implement ACDC (Automatic Circuit DisCovery) — iterative edge ablation over a computational graph constructed from SAE features + attention edges, converging to minimal circuits explaining a task
  • 177.3.1.2 Implement EAP-IG (Edge Attribution Patching with Integrated Gradients) as the scalable alternative to ACDC for large models and large datasets
  • 177.3.1.3 Implement transcoder training (per-MLP, predicting MLP output from residual) for clean feature-graph edges between residual and MLP modules — Anthropic transcoder pattern
  • 177.3.1.4 Build feature-circuit visualiser — interactive Reagent dashboard showing feature nodes (SAE), attention edges, MLP edges, per-prompt activation traces, and downstream token effects

177.3.2 Targeted Interpretability Campaigns#

  • 177.3.2.1 Run a refusal-circuit investigation on the safety-aligned @nous/llm checkpoint — identify the causal features and attention heads. Needs an owner decision on a compute budget — this is large-scale SAE/interpretability training or a circuit investigation over frontier checkpoints, not a coding task (stated 2026-09-18). The investigation needs a trained @nous/llm checkpoint to run against. blocked:governance
  • 177.3.2.2 Run a factual-recall circuit investigation (Ghanaian history, Yorùbá deities, African philosophy) to characterise failure modes. Needs an owner decision on a compute budget — this is large-scale SAE/interpretability training or a circuit investigation over frontier checkpoints, not a coding task (stated 2026-09-18). blocked:governance
  • 177.3.2.3 Run a code-generation circuit investigation on the @iris/code model to find the features behind common bug-prone patterns. Needs an owner decision on a compute budget — this is large-scale SAE/interpretability training or a circuit investigation over frontier checkpoints, not a coding task (stated 2026-09-18). blocked:governance
  • 177.3.2.4 Run a deception / sycophancy circuit investigation on pre- and post-RLHF checkpoints to quantify steering targets. Needs an owner decision on a compute budget — this is large-scale SAE/interpretability training or a circuit investigation over frontier checkpoints, not a coding task (stated 2026-09-18). It needs paired pre/post-RLHF checkpoints. blocked:governance

177.4 Probing & Representation Analysis — @nous/probes#

177.4.1 Classical Probing#

  • 177.4.1.1 Implement linear probing harness (L2-regularised logistic regression with bootstrap CIs) over activation tensors, supporting per-layer and per-position sweeps
  • 177.4.1.2 Implement Representational Similarity Analysis (RSA) and Centered Kernel Alignment (CKA) between checkpoints, model families, and between model and brain activations (NSD, Algonauts datasets) for neuroscience-bridge research
  • 177.4.1.3 Implement concept activation vectors (TCAV) + causal concept effect, surfaced in the model-card for every published Nous checkpoint
  • 177.4.1.4 Build emergent-capability probes — monitor downstream probes (truthfulness, multilingual, arithmetic) across training checkpoints so scaling-law experiments (Phase 89.8) can track capability emergence, not just loss

177.5 Continual-Learning Foundations — @nous/continual-core#

177.5.1 Task Abstractions#

  • 177.5.1.1 Create libs/nous/continual-core/ with the ContinualTaskStream<T> abstraction — an ordered stream of tasks emitting (task_id, dataset, eval_fn) supporting both disjoint task-incremental and blurry class-incremental regimes
  • 177.5.1.2 Define ForgettingMetric — backward transfer (BWT), forward transfer (FWT), average accuracy (ACC), and per-task learning curve — computed automatically after every task in the stream
  • 177.5.1.3 Build a standard CL benchmark suite (Permuted MNIST, Split CIFAR-100, CLEAR-100, Continual World 20, LORA-CL, and an Oshun synthetic stream combining Iris intent + Isis style + Kuan Yin harms)
  • 177.5.1.4 Build a per-domain Oshun continual stream — Iris conversations by week, Kuan Yin emerging harms by month, Galatea teleop by fleet cohort, Veritas news by cycle — wired to Phase 85 flywheel via retrograde-stream sampling

177.6 Progressive Networks & Parameter Expansion#

177.6.1 Progressive Networks (Rusu & Hadsell 2016)#

  • 177.6.1.1 Implement Progressive Network column architecture — for each new task, instantiate a new column, freeze all prior columns, add lateral adapters from prior columns' activations at each layer; forward pass concatenates lateral contributions; backward flows only through the new column
  • 177.6.1.2 Integrate progressive columns with Galatea VLA (Phase 176.7.1) — each new deployment fleet / new task class gets its own column, preserving prior proficiency exactly while reusing lateral features — no forgetting by construction
  • 177.6.1.3 Build column-pruning pass that distils rarely-used progressive columns into a shared base (177.9) when parameter budget tightens, preserving measured skill retention

177.6.2 Dynamically Expandable Networks & Supermasks#

  • 177.6.2.1 Implement DEN (Dynamically Expandable Networks) — grow the hidden width on demand guided by loss-threshold heuristics, with task-selective retraining
  • 177.6.2.2 Implement PackNet — iterative magnitude pruning and task-mask packing, allowing many tasks to share a single dense backbone with per-task bitmasks
  • 177.6.2.3 Implement supermask subnetwork discovery (Zhou et al.) for inference-time routing between continual tasks without weight mutation

177.7 Regularisation-Based Continual Learning#

177.7.1 Fisher / Path-Integral / Importance-Based#

  • 177.7.1.1 Implement Elastic Weight Consolidation (EWC) — per-task diagonal Fisher estimate F_θ, quadratic penalty Σ λ_i (θ_i − θ*_i)² F_i on subsequent task losses; validated on Permuted MNIST (target BWT ≥ −2%) and Atari-continual
  • 177.7.1.2 Implement Online EWC with running Fisher accumulation, avoiding per-task Fisher storage, used in long streams (177.5.1.4)
  • 177.7.1.3 Implement Synaptic Intelligence (Zenke et al.) — path integral of parameter movement weighted by loss contribution — cheaper than Fisher, validated against EWC on Oshun synthetic stream
  • 177.7.1.4 Implement Memory-Aware Synapses (MAS) — importance from output-gradient magnitude, supports unsupervised continual streams
  • 177.7.1.5 Build automatic λ-tuning — Bayesian optimisation over held-out BWT/ACC frontier to avoid hand-tuned EWC coefficients per task

177.8 Replay-Based Continual Learning — @nous/replay#

177.8.1 Experience & Generative Replay#

  • 177.8.1.1 Implement CLEAR (Continual Learning with Experience And Replay) — off-policy behavioural cloning + on-policy RL mixed replay with V-trace corrections, per Rolnick et al. (Hadsell group)
  • 177.8.1.2 Implement Gradient Episodic Memory (GEM) and A-GEM — project gradients to avoid increasing prior-task loss (GEM quadprog, A-GEM projection); evaluate on Permuted MNIST + Split CIFAR-100
  • 177.8.1.3 Implement Dark Experience Replay (DER++) — logit distillation across the episodic memory, SOTA baseline for many class-incremental benchmarks
  • 177.8.1.4 Implement Generative Replay with a small diffusion / VQ-GAN model producing prior-task pseudo-samples when storage of real replay violates consent or retention policy (Phase 85.15)
  • 177.8.1.5 Build privacy-preserving replay — encrypted, k-anonymised, differentially-private replay buffer integrated with Phase 85.15.1.1

177.9 Policy Distillation & Kickstarting — @nous/distillation#

177.9.1 Distillation#

  • 177.9.1.1 Implement policy distillation (Rusu et al.) — teacher policy → student policy KL distillation over shared env rollouts, compressing multiple progressive columns into a single deployable model
  • 177.9.1.2 Implement Kickstarting (Schmitt et al.) — mixture-of- experts teacher auxiliary loss added to student RL objective, smoothly decayed — used to bootstrap new Galatea robots from fleet teachers
  • 177.9.1.3 Implement DistBelief-style data-free distillation producing a compact @nous/llm-edge continual student consuming teacher logits from the progressive stack
  • 177.9.1.4 Build distillation-quality scorecard — per-task retention after distillation vs pre-distillation progressive stack, promotion gated at ≥ 97% mean retention

177.10 Galatea Fleet & Safety Integration#

177.10.1 Fleet-Wide Continual Learning#

  • 177.10.1.1 Wire Phase 85.20.1.3 federated-learning rounds through Phase 177.8.1.5 privacy-preserving replay and Phase 177.7.1.1 EWC so that fleet rollouts strictly monotonically improve (BWT ≥ 0) on previously certified skills
  • 177.10.1.2 Build fleet dashboard (@nous/interp-dashboard) showing per-robot per-skill retention, forgetting alerts, and roll-back recommendations — integrated with Phase 92 MLOps governance
  • 177.10.1.3 Build continual-evaluation watchdog — every fleet continual update re-runs a regression suite on 177.5.1.4 streams; any BWT < −1% on safety-critical skills freezes the update

177.10.2 Interpretability-Driven Safety#

  • 177.10.2.1 Publish an "SAE safety report" per major Nous checkpoint (feature inventory, refusal circuit, steering vulnerabilities). Needs an owner decision on a compute budget — this is large-scale SAE/interpretability training or a circuit investigation over frontier checkpoints, not a coding task (stated 2026-09-18). It rolls up the SAE runs of 177.2/177.3, which are themselves budget-gated. blocked:governance
  • 177.10.2.2 Build interpretability-integrated eval (Phase 91) — in addition to behavioural scores, publish circuit-stability and feature-drift metrics per checkpoint so regressions in internal structure, not just outputs, gate promotion
  • 177.10.2.3 Build red-team interp tooling — given an elicited failure, automatically surface top contributing SAE features and circuits for root-cause analysis within minutes, not weeks
  • 177.10.2.4 Wire SAE feature activations as side-channel signals into Kuan Yin precognition (Phase 85.5.2.4) — early-warning features (deception / coercion / CSAM-adjacent) fire detectors before behavioural manifestation