Eve library

@nous/autoresearch-evals

Other

Revision 1

On this page

Autoresearch benchmarks, observability & experiment ledger for the Phase-178 stack (TODOS/phase-178 §178.9).

Components#

Module Summary
harness.py An inspect_ai-shaped Task / Sample / Solver / Scorer harness + EvalRunner, a named-suite registry (MLE-Bench, SWE-Bench Verified, RE-Bench, PaperBench, ScienceAgentBench, DiscoveryBench, GAIA, Cybench, CTF, HLE wired by reference), and an inspect_ai_available() probe to delegate when installed. (§178.9.1.1)
suites.py KarpathyLoopBench (delta reproduction), AlphaEvolveMathBench (50-problem deterministic sweep), OshunDomainBench (internal multi-domain Q&A). (§178.9.1.2)
leaderboard.py Continuous-leaderboard publisher — per-checkpoint submissions to a Phase-91 hub sink with regression alarms and promotion gating. (§178.9.1.3)
observability.py Dual Langfuse + W&B Weave exporters (real payload mappers) over a fail-loud transport. (§178.9.2.1)
experiments.py Experiments-as-first-class diffing — per-prompt output/score/cost/latency/token deltas + side-by-side report. (§178.9.2.2)
ledger.py Canonical hash-chained + HMAC-signed experiment ledger; tampering breaks the chain, forging breaks the signature. (§178.9.2.3)
sandbox_pool.py SWE-ReX-class parallel sandbox pool — bounded workers, per-job resource envelope (fs/network/cpu/gpu/wall), isolated failures. (§178.9.2.4)
flywheel.py Autoresearch-outcomes collector — (prompt, edit, delta, accepted) tuples → weighted preference training records. (§178.9.2.5)

External benchmark data (Kaggle competitions, SWE-Bench repos) and the real Langfuse/Weave/inspect_ai backends are plug-in seams; the harness, suites, leaderboard gating, payload mapping, diffing, signed ledger, pool and flywheel logic are fully local and tested.

Testing#

bash
python3 -m pytest tests/ -q     # 19 tests
python3 -m ruff check .