Historical task source. The Eve task board holds current task state.
Phase 177: Interpretability & Continual-Learning Research Track — Sparse Autoencoders, Circuit Discovery, Progressive Networks, EWC, CLEAR Replay, PackNet & Policy Distillation#
Sovereign research track covering the two frontier safety-and-robustness disciplines that the current ML Sovereignty phases (85–96) do not address in depth: mechanistic interpretability (Anthropic, DeepMind, OpenAI — sparse autoencoders, circuit discovery, attribution patching, transcoders) and continual learning (Hadsell et al. — progressive networks, EWC, online EWC, synaptic intelligence, MAS, CLEAR, PackNet, policy distillation, Kickstarting). Powers Kuan Yin safety audits, Galatea fleet-wide lifelong learning, and alignment research defensibility in front of UK AISI / US AISI.
Tech split: Python (~70%) for research-grade experimentation (PyTorch + TransformerLens + sae-lens); Rust (~10%) for high-throughput SAE inference kernels and streaming activation capture; TypeScript (~20%) for dashboards (interp explorer, continual-learning fleet monitor) and serving integration.
Library layout: libs/nous/interp-core/, libs/nous/sae/,
libs/nous/circuits/, libs/nous/probes/, libs/nous/continual-core/,
libs/nous/progressive-nets/, libs/nous/replay/, libs/nous/distillation/,
libs/nous/interp-dashboard/.
177.1 Interpretability Foundations — @nous/interp-core#
177.1.1 Activation Capture & Tooling#
- 177.1.1.1 Create
libs/nous/interp-core/withproject.json,package.json,tsconfig.json. DefineHookedModel<M>wrapper over any@nous/llmor@nous/visionmodel exposing streaming hooks at every residual-stream layer, MLP pre/post, attention Q/K/V/O, and logits — TransformerLens-compatible semantics - 177.1.1.2 Build
ActivationCapturerecorder writing Zarr-chunked activation tensors keyed by(model_rev, dataset, sample_id, layer, point)with per-sample metadata and deterministic replay - 177.1.1.3 Build
ActivationStatsonline estimator (mean, cov, diagonal Fisher, percentile-100 activation magnitudes) used by SAE initialisation and Fisher-based EWC (177.7) - 177.1.1.4 Ship a cross-model adapter covering Llama-4 MoE, Qwen3 dense + MoE, Gemma-4, Mixtral / DeepSeek V3 MoE routers, and Nous's own fine-tunes — guarantees every Oshun-served model is mechanically inspectable out of the box
177.1.2 Attribution Primitives#
- 177.1.2.1 Implement activation patching (causal intervention: replace activation at (layer, point, token) from donor → recipient, measure logit shift) with batched single-backward implementation
- 177.1.2.2 Implement Integrated Gradients, SmoothGrad, and LIME attribution backends for feature-attribution dashboards and safety-relevant model cards
- 177.1.2.3 Implement attribution-patching (gradient-based approximation to activation patching) enabling per-token per-layer attribution at whole-dataset scale with O(1) forward/backward per prompt
177.2 Sparse Autoencoders — @nous/sae#
177.2.1 SAE Training#
- 177.2.1.1 Implement JumpReLU SAE (DeepMind 2024) with per-feature threshold + straight-through estimator — primary SAE family, target L0 ≈ 60 and reconstruction MSE parity with published Gemma Scope
- 177.2.1.2 Implement TopK SAE (OpenAI 2024) with aux_k dead-feature revival loss and 4× expansion factor — complementary family for controlled comparisons
- 177.2.1.3 Implement GatedSAE baseline for architecture ablations
- 177.2.1.4 Build SAE training pipeline — stream activations from 177.1.1.2, shuffle via reservoir sampler, batch 4096, Adam β₂=0.999, learning-rate warmup, λ annealing, feature-resampling on dead features every 25k steps, EMA checkpointing every 10k
- 177.2.1.5 Train a full SAE suite on the Gemma-4-9B / Qwen3-8B residual
stream at every layer (expansion x16/x32/x64) and MLP-post, publishing
feature dashboards. Needs an owner decision on a compute budget — this is
large-scale SAE/interpretability training or a circuit investigation over
frontier checkpoints, not a coding task (stated 2026-09-18).
blocked:governance - 177.2.1.6 Train SAEs on the fine-tuned
@iris/codeand@kuanyin/safetyheads and publish per-domain feature dashboards. Needs an owner decision on a compute budget — this is large-scale SAE/interpretability training or a circuit investigation over frontier checkpoints, not a coding task (stated 2026-09-18). It also needs those fine-tuned checkpoints, which are themselves frontier-trained.blocked:governance
177.2.2 Feature Analysis#
- 177.2.2.1 Build autointerp pipeline (Bills et al. style) — for each SAE feature, sample top-activating contexts, ask a frontier LLM for a natural-language label, score label fidelity via held-out-example prediction — run at publish-scale over every SAE
- 177.2.2.2 Build feature-splitting detector tracking how features divide under scale-ups (×8 → ×32 → ×64) to identify canonical vs derivative features
- 177.2.2.3 Build feature-ablation steering — zero or scale specific SAE
features at inference, measure downstream behavioural shift (refusal rate,
sentiment, tool-use style) on eval suites — expose as
@nous/interp-coresteering API for Kuan Yin safety researchers - 177.2.2.4 Build concept-dictionary export — feature → label → safety category (toxicity, deception, CSAM, self-harm, jailbreak-adjacent) used by Kuan Yin classifiers as auxiliary supervision (Phase 85.5 integration)
177.3 Circuits & Feature Graphs — @nous/circuits#
177.3.1 Circuit Discovery#
- 177.3.1.1 Implement ACDC (Automatic Circuit DisCovery) — iterative edge ablation over a computational graph constructed from SAE features + attention edges, converging to minimal circuits explaining a task
- 177.3.1.2 Implement EAP-IG (Edge Attribution Patching with Integrated Gradients) as the scalable alternative to ACDC for large models and large datasets
- 177.3.1.3 Implement transcoder training (per-MLP, predicting MLP output from residual) for clean feature-graph edges between residual and MLP modules — Anthropic transcoder pattern
- 177.3.1.4 Build feature-circuit visualiser — interactive Reagent dashboard showing feature nodes (SAE), attention edges, MLP edges, per-prompt activation traces, and downstream token effects
177.3.2 Targeted Interpretability Campaigns#
- 177.3.2.1 Run a refusal-circuit investigation on the safety-aligned
@nous/llmcheckpoint — identify the causal features and attention heads. Needs an owner decision on a compute budget — this is large-scale SAE/interpretability training or a circuit investigation over frontier checkpoints, not a coding task (stated 2026-09-18). The investigation needs a trained@nous/llmcheckpoint to run against.blocked:governance - 177.3.2.2 Run a factual-recall circuit investigation (Ghanaian
history, Yorùbá deities, African philosophy) to characterise failure
modes. Needs an owner decision on a compute budget — this is large-scale
SAE/interpretability training or a circuit investigation over frontier
checkpoints, not a coding task (stated 2026-09-18).
blocked:governance - 177.3.2.3 Run a code-generation circuit investigation on the
@iris/codemodel to find the features behind common bug-prone patterns. Needs an owner decision on a compute budget — this is large-scale SAE/interpretability training or a circuit investigation over frontier checkpoints, not a coding task (stated 2026-09-18).blocked:governance - 177.3.2.4 Run a deception / sycophancy circuit investigation on pre-
and post-RLHF checkpoints to quantify steering targets. Needs an owner
decision on a compute budget — this is large-scale SAE/interpretability
training or a circuit investigation over frontier checkpoints, not a
coding task (stated 2026-09-18). It needs paired pre/post-RLHF
checkpoints.
blocked:governance
177.4 Probing & Representation Analysis — @nous/probes#
177.4.1 Classical Probing#
- 177.4.1.1 Implement linear probing harness (L2-regularised logistic regression with bootstrap CIs) over activation tensors, supporting per-layer and per-position sweeps
- 177.4.1.2 Implement Representational Similarity Analysis (RSA) and Centered Kernel Alignment (CKA) between checkpoints, model families, and between model and brain activations (NSD, Algonauts datasets) for neuroscience-bridge research
- 177.4.1.3 Implement concept activation vectors (TCAV) + causal concept effect, surfaced in the model-card for every published Nous checkpoint
- 177.4.1.4 Build emergent-capability probes — monitor downstream probes (truthfulness, multilingual, arithmetic) across training checkpoints so scaling-law experiments (Phase 89.8) can track capability emergence, not just loss
177.5 Continual-Learning Foundations — @nous/continual-core#
177.5.1 Task Abstractions#
- 177.5.1.1 Create
libs/nous/continual-core/with theContinualTaskStream<T>abstraction — an ordered stream of tasks emitting(task_id, dataset, eval_fn)supporting both disjoint task-incremental and blurry class-incremental regimes - 177.5.1.2 Define
ForgettingMetric— backward transfer (BWT), forward transfer (FWT), average accuracy (ACC), and per-task learning curve — computed automatically after every task in the stream - 177.5.1.3 Build a standard CL benchmark suite (Permuted MNIST, Split CIFAR-100, CLEAR-100, Continual World 20, LORA-CL, and an Oshun synthetic stream combining Iris intent + Isis style + Kuan Yin harms)
- 177.5.1.4 Build a per-domain Oshun continual stream — Iris conversations by week, Kuan Yin emerging harms by month, Galatea teleop by fleet cohort, Veritas news by cycle — wired to Phase 85 flywheel via retrograde-stream sampling
177.6 Progressive Networks & Parameter Expansion#
177.6.1 Progressive Networks (Rusu & Hadsell 2016)#
- 177.6.1.1 Implement Progressive Network column architecture — for each new task, instantiate a new column, freeze all prior columns, add lateral adapters from prior columns' activations at each layer; forward pass concatenates lateral contributions; backward flows only through the new column
- 177.6.1.2 Integrate progressive columns with Galatea VLA (Phase 176.7.1) — each new deployment fleet / new task class gets its own column, preserving prior proficiency exactly while reusing lateral features — no forgetting by construction
- 177.6.1.3 Build column-pruning pass that distils rarely-used progressive columns into a shared base (177.9) when parameter budget tightens, preserving measured skill retention
177.6.2 Dynamically Expandable Networks & Supermasks#
- 177.6.2.1 Implement DEN (Dynamically Expandable Networks) — grow the hidden width on demand guided by loss-threshold heuristics, with task-selective retraining
- 177.6.2.2 Implement PackNet — iterative magnitude pruning and task-mask packing, allowing many tasks to share a single dense backbone with per-task bitmasks
- 177.6.2.3 Implement supermask subnetwork discovery (Zhou et al.) for inference-time routing between continual tasks without weight mutation
177.7 Regularisation-Based Continual Learning#
177.7.1 Fisher / Path-Integral / Importance-Based#
- 177.7.1.1 Implement Elastic Weight Consolidation (EWC) — per-task diagonal Fisher estimate F_θ, quadratic penalty Σ λ_i (θ_i − θ*_i)² F_i on subsequent task losses; validated on Permuted MNIST (target BWT ≥ −2%) and Atari-continual
- 177.7.1.2 Implement Online EWC with running Fisher accumulation, avoiding per-task Fisher storage, used in long streams (177.5.1.4)
- 177.7.1.3 Implement Synaptic Intelligence (Zenke et al.) — path integral of parameter movement weighted by loss contribution — cheaper than Fisher, validated against EWC on Oshun synthetic stream
- 177.7.1.4 Implement Memory-Aware Synapses (MAS) — importance from output-gradient magnitude, supports unsupervised continual streams
- 177.7.1.5 Build automatic λ-tuning — Bayesian optimisation over held-out BWT/ACC frontier to avoid hand-tuned EWC coefficients per task
177.8 Replay-Based Continual Learning — @nous/replay#
177.8.1 Experience & Generative Replay#
- 177.8.1.1 Implement CLEAR (Continual Learning with Experience And Replay) — off-policy behavioural cloning + on-policy RL mixed replay with V-trace corrections, per Rolnick et al. (Hadsell group)
- 177.8.1.2 Implement Gradient Episodic Memory (GEM) and A-GEM — project gradients to avoid increasing prior-task loss (GEM quadprog, A-GEM projection); evaluate on Permuted MNIST + Split CIFAR-100
- 177.8.1.3 Implement Dark Experience Replay (DER++) — logit distillation across the episodic memory, SOTA baseline for many class-incremental benchmarks
- 177.8.1.4 Implement Generative Replay with a small diffusion / VQ-GAN model producing prior-task pseudo-samples when storage of real replay violates consent or retention policy (Phase 85.15)
- 177.8.1.5 Build privacy-preserving replay — encrypted, k-anonymised, differentially-private replay buffer integrated with Phase 85.15.1.1
177.9 Policy Distillation & Kickstarting — @nous/distillation#
177.9.1 Distillation#
- 177.9.1.1 Implement policy distillation (Rusu et al.) — teacher policy → student policy KL distillation over shared env rollouts, compressing multiple progressive columns into a single deployable model
- 177.9.1.2 Implement Kickstarting (Schmitt et al.) — mixture-of- experts teacher auxiliary loss added to student RL objective, smoothly decayed — used to bootstrap new Galatea robots from fleet teachers
- 177.9.1.3 Implement DistBelief-style data-free distillation producing
a compact
@nous/llm-edgecontinual student consuming teacher logits from the progressive stack - 177.9.1.4 Build distillation-quality scorecard — per-task retention after distillation vs pre-distillation progressive stack, promotion gated at ≥ 97% mean retention
177.10 Galatea Fleet & Safety Integration#
177.10.1 Fleet-Wide Continual Learning#
- 177.10.1.1 Wire Phase 85.20.1.3 federated-learning rounds through Phase 177.8.1.5 privacy-preserving replay and Phase 177.7.1.1 EWC so that fleet rollouts strictly monotonically improve (BWT ≥ 0) on previously certified skills
- 177.10.1.2 Build fleet dashboard (
@nous/interp-dashboard) showing per-robot per-skill retention, forgetting alerts, and roll-back recommendations — integrated with Phase 92 MLOps governance - 177.10.1.3 Build continual-evaluation watchdog — every fleet continual update re-runs a regression suite on 177.5.1.4 streams; any BWT < −1% on safety-critical skills freezes the update
177.10.2 Interpretability-Driven Safety#
- 177.10.2.1 Publish an "SAE safety report" per major Nous checkpoint
(feature inventory, refusal circuit, steering vulnerabilities). Needs an
owner decision on a compute budget — this is large-scale
SAE/interpretability training or a circuit investigation over frontier
checkpoints, not a coding task (stated 2026-09-18). It rolls up the SAE
runs of 177.2/177.3, which are themselves budget-gated.
blocked:governance - 177.10.2.2 Build interpretability-integrated eval (Phase 91) — in addition to behavioural scores, publish circuit-stability and feature-drift metrics per checkpoint so regressions in internal structure, not just outputs, gate promotion
- 177.10.2.3 Build red-team interp tooling — given an elicited failure, automatically surface top contributing SAE features and circuits for root-cause analysis within minutes, not weeks
- 177.10.2.4 Wire SAE feature activations as side-channel signals into Kuan Yin precognition (Phase 85.5.2.4) — early-warning features (deception / coercion / CSAM-adjacent) fire detectors before behavioural manifestation