Eve library

Competitor Baselines

Other

Revision 1

On this page

This folder contains optional competitor baselines used by the competitive benchmark comparison report.

When present, python -m computer_use.benchmark.ci loads all *.json files in this directory and includes them in the leaderboard written to:

  • benchmark_results_ci/competitive.json
  • benchmark_results_ci/competitive.md

Supported JSON Formats#

json
{
  "competitor": {
    "competitor_id": "model_x",
    "name": "Model X",
    "link": "https://example.com"
  },
  "metrics": {
    "success_rate": 0.83,
    "average_steps": 12.4,
    "average_recovery_score": 0.76,
    "category_metrics": {
      "FILE_MANAGEMENT": { "rate": 0.92, "recovery_avg": 0.8 },
      "WEB_BROWSING": { "rate": 0.78, "recovery_avg": 0.7 }
    }
  }
}

2) A saved benchmark run results file#

You can also drop in a full *_results.json produced by the benchmark runner (top-level run.metrics is used).

3) Raw metrics dict (fallback)#

If the JSON object looks like a metrics payload (e.g. contains success_rate / average_steps), it will be treated as metrics directly.