Karpathy-style single-GPU overnight autoresearch loop and its async researcher-pool successor (TODOS/phase-178 §178.5).
Components#
| Module | Summary |
|---|---|
ledger.py |
Hash-chained experiment ledger — append-only {trial_id, parent, diff, command, metric_before/after, delta, accepted, wall_clock, gpu_hours, model_rev}; each entry carries the SHA-256 of the previous, so tampering is detectable; tree/lineage reconstruction. (§178.5.1.4) |
target.py |
Three editable-train.py tweak targets (nano/small/mid) with a program.md, frozen prep seed, and a deterministic held-out surrogate val_bpb surface (coupled-hyperparameter bowl) so the search is CPU-testable. (§178.5.1.2) |
editor.py |
Multi-backend code-editor driver — a real CoordinateEditor (guided coordinate search) + fail-loud Claude/GPT/Gemini seams, with per-backend cost/latency/acceptance stats. (§178.5.1.3) |
loop.py |
The overnight hill-climb: edit train.py → run training → read val_bpb → keep/discard on delta → log the decision (with diff). Stops on trial or GPU-hour budget. (§178.5.1.1) |
launcher.py |
run_overnight(target, model_backend, budget_gpu_hours, max_trials) + a morning summary (best delta, top-5 changes, failed-hypothesis histogram, Markdown). (§178.5.1.5) |
pool.py |
Async researcher-pool — N workers share the ledger, branch from ancestors, hold advisory subtree locks, gossip improvements; SubtreeAssigner (UCB over subtrees by recent-delta + idle-time); MergeResolver composes orthogonal wins and accepts only if the composition improves. (§178.5.2.1–.3) |
dashboard.py |
Researcher-pool dashboard aggregation — trial tree, per-agent throughput, cost-per-accepted-delta, hypothesis histogram, cross-seed divergence. (§178.5.2.4) |
The loop shape and ledger are exact; the "5-minute training" is a swappable
target.evaluate — the tests show the loop and pool drive val_bpb down toward
the surrogate optimum (1.05 → ~0.88 on tweak-small) and that the ledger
hash-chain verifies and detects tampering. Point evaluate at a real nanochat
trainer and nothing else changes.
Testing#
bash
python3 -m pytest tests/ -q # 15 tests
python3 -m ruff check .