Eve library

@nous/tweak-loop

Other

Revision 1

On this page

Karpathy-style single-GPU overnight autoresearch loop and its async researcher-pool successor (TODOS/phase-178 §178.5).

Components#

Module Summary
ledger.py Hash-chained experiment ledger — append-only {trial_id, parent, diff, command, metric_before/after, delta, accepted, wall_clock, gpu_hours, model_rev}; each entry carries the SHA-256 of the previous, so tampering is detectable; tree/lineage reconstruction. (§178.5.1.4)
target.py Three editable-train.py tweak targets (nano/small/mid) with a program.md, frozen prep seed, and a deterministic held-out surrogate val_bpb surface (coupled-hyperparameter bowl) so the search is CPU-testable. (§178.5.1.2)
editor.py Multi-backend code-editor driver — a real CoordinateEditor (guided coordinate search) + fail-loud Claude/GPT/Gemini seams, with per-backend cost/latency/acceptance stats. (§178.5.1.3)
loop.py The overnight hill-climb: edit train.py → run training → read val_bpb → keep/discard on delta → log the decision (with diff). Stops on trial or GPU-hour budget. (§178.5.1.1)
launcher.py run_overnight(target, model_backend, budget_gpu_hours, max_trials) + a morning summary (best delta, top-5 changes, failed-hypothesis histogram, Markdown). (§178.5.1.5)
pool.py Async researcher-pool — N workers share the ledger, branch from ancestors, hold advisory subtree locks, gossip improvements; SubtreeAssigner (UCB over subtrees by recent-delta + idle-time); MergeResolver composes orthogonal wins and accepts only if the composition improves. (§178.5.2.1–.3)
dashboard.py Researcher-pool dashboard aggregation — trial tree, per-agent throughput, cost-per-accepted-delta, hypothesis histogram, cross-seed divergence. (§178.5.2.4)

The loop shape and ledger are exact; the "5-minute training" is a swappable target.evaluate — the tests show the loop and pool drive val_bpb down toward the surrogate optimum (1.05 → ~0.88 on tweak-small) and that the ledger hash-chain verifies and detects tampering. Point evaluate at a real nanochat trainer and nothing else changes.

Testing#

bash
python3 -m pytest tests/ -q     # 15 tests
python3 -m ruff check .