Autoresearch benchmarks, observability & experiment ledger for the Phase-178 stack (TODOS/phase-178 §178.9).
Components#
| Module | Summary |
|---|---|
harness.py |
An inspect_ai-shaped Task / Sample / Solver / Scorer harness + EvalRunner, a named-suite registry (MLE-Bench, SWE-Bench Verified, RE-Bench, PaperBench, ScienceAgentBench, DiscoveryBench, GAIA, Cybench, CTF, HLE wired by reference), and an inspect_ai_available() probe to delegate when installed. (§178.9.1.1) |
suites.py |
KarpathyLoopBench (delta reproduction), AlphaEvolveMathBench (50-problem deterministic sweep), OshunDomainBench (internal multi-domain Q&A). (§178.9.1.2) |
leaderboard.py |
Continuous-leaderboard publisher — per-checkpoint submissions to a Phase-91 hub sink with regression alarms and promotion gating. (§178.9.1.3) |
observability.py |
Dual Langfuse + W&B Weave exporters (real payload mappers) over a fail-loud transport. (§178.9.2.1) |
experiments.py |
Experiments-as-first-class diffing — per-prompt output/score/cost/latency/token deltas + side-by-side report. (§178.9.2.2) |
ledger.py |
Canonical hash-chained + HMAC-signed experiment ledger; tampering breaks the chain, forging breaks the signature. (§178.9.2.3) |
sandbox_pool.py |
SWE-ReX-class parallel sandbox pool — bounded workers, per-job resource envelope (fs/network/cpu/gpu/wall), isolated failures. (§178.9.2.4) |
flywheel.py |
Autoresearch-outcomes collector — (prompt, edit, delta, accepted) tuples → weighted preference training records. (§178.9.2.5) |
External benchmark data (Kaggle competitions, SWE-Bench repos) and the real Langfuse/Weave/inspect_ai backends are plug-in seams; the harness, suites, leaderboard gating, payload mapping, diffing, signed ledger, pool and flywheel logic are fully local and tested.
Testing#
bash
python3 -m pytest tests/ -q # 19 tests
python3 -m ruff check .