Benchmarks
Benchmarks that come from real runs.
Most benchmarks are written by hand from imagined tasks. Ours are derived from execution traces — millions of real agent runs — so the evaluations measure what production work actually looks like.
What we measure.
Every benchmark is built from normalized, anonymized execution traces with published methodology.
Success rate
Task completion measured on real production workloads with defined outcome criteria.
Latency
Step-level and end-to-end timing distributions across tool calls, reasoning and recovery.
Cost efficiency
Cost per successful task, normalized across models, tools and configurations.
Recovery quality
How quickly and how well agents recover from errors, retries and partial failures.
Tool-selection accuracy
Whether the right tool is chosen at the right moment, judged against human corrections.
Multi-agent coordination
Handoffs, contention and delegation quality across multi-agent systems.
The benchmark suite.
| Benchmark | Domain | Primary metric | Status |
|---|---|---|---|
| AgentBench-Real | General agent work | Success rate | Stable |
| ToolTrail | Tool orchestration | Tool-selection accuracy | Stable |
| FailureForge | Error & edge cases | Recovery quality | Stable |
| RecoveryIQ | Retry & fallback | Recovery quality | Beta |
| CostCompass | Cost optimization | Cost per task | Beta |
| SyncBench | Multi-agent systems | Coordination score | Beta |
How benchmarks are built.
Full methodology, published with every release.
Collect real executions
Traces flow in from registered agents across thousands of production workloads.
Normalize & anonymize
Raw traces are structured into a canonical schema with personal data stripped.
Define tasks & metrics
Tasks are extracted from real runs; metrics are defined against observable outcomes.
Publish with methodology
Benchmarks ship with full methodology, seed documentation and version history.
What a scored run looks like
{
"benchmark": "ToolTrail v2",
"run_id": "9f3a1c",
"task": "invoice_reconciliation",
"score": 0.87,
"latency_p95_ms": 4102,
"cost_usd": 0.021,
"recovery": "2 retries, 0 escalations",
"verdict": "pass"
}
Measure agents on real work.
Run your agents against benchmarks derived from millions of anonymized real-world executions — and get evaluations on your own stack.