THETOOLDECATHLON: BENCHMARKINGLANGUAGE AGENTS FORDIVERSE, REALISTIC,ANDLONG- HORIZONTASKEXECUTION

Shared by Gary on Oct 31, 2025 • 03:49 AM UTC

Abstract

Real-world language agents must handle complex, multi-step workflows across diverse applications. For instance, an agent may manage emails by coordinating with calendars and file systems, or monitor a production database like BigQuery to detect anomalies and generate reports following a standard ope...

Subject

TOOLATHLON

language agents

MCP

Highlights

-TOOLATHLONspans 32 real-world applications and 604 tools across 108 tasks.

-The majority of tools are sourced from MCP servers, with several open-source implementations refined by the authors.

-The benchmark uses realistic initial environment states drawn from real software, such as Canvas courses and financial spreadsheets.

-The best-performing model Claude-4.5-Sonnet achieves 38.6% accuracy with 20.2 tool calling turns on average; among open weights, DeepSeek-V3.2-Exp reaches 20.1%.

Subject

Highlights

Strengths & Weaknesses

Commercial Applications

Societal Benefits & Applications

Comments