This folder contains optional competitor baselines used by the competitive benchmark comparison report.
When present, python -m computer_use.benchmark.ci loads all *.json files in
this directory and includes them in the leaderboard written to:
benchmark_results_ci/competitive.jsonbenchmark_results_ci/competitive.md
Supported JSON Formats#
1) Explicit competitor + metrics (recommended)#
json
{
"competitor": {
"competitor_id": "model_x",
"name": "Model X",
"link": "https://example.com"
},
"metrics": {
"success_rate": 0.83,
"average_steps": 12.4,
"average_recovery_score": 0.76,
"category_metrics": {
"FILE_MANAGEMENT": { "rate": 0.92, "recovery_avg": 0.8 },
"WEB_BROWSING": { "rate": 0.78, "recovery_avg": 0.7 }
}
}
}
2) A saved benchmark run results file#
You can also drop in a full *_results.json produced by the benchmark runner
(top-level run.metrics is used).
3) Raw metrics dict (fallback)#
If the JSON object looks like a metrics payload (e.g. contains success_rate /
average_steps), it will be treated as metrics directly.