AA-AnalystAgent ranks Claude Opus 5 highest in new benchmark

Source published 40d 22h agoTerminal published 40d 22h ago

Summary

AA-AnalystAgent has been launched as a benchmark tool for evaluating the performance of agent-based models in quantitative analysis across various domains, with Claude Opus 5 topping the leaderboard at 54%. This benchmark emphasizes the importance of consistent reliability, as it measures how well models can repeatedly produce correct answers across multiple attempts, demonstrating that reliability is more critical than raw capability for real-world applications. Notably, professional judgment and expertise are also integral to interpreting complex documents, with models required to assess exceptions and methodologies accurately.

Tokens

$GOOG$Xiaomi$OPENAI

Analysis

OpenAI: OpenAI creates frontier AI models such as the GPT series. Its GPT-5.5 model placed second on the AA-AnalystAgent leaderboard in the reported results. Xiaomi: Xiaomi produces AI models such as the MiMo series. Its MiMo-V2.5-Pro achieved a notable score on AA-AnalystAgent while demonstrating significant cost efficiency compared to peers. GPT-5.5: GPT-5.5 is OpenAI's model variant that achieved strong average pass rates on AA-AnalystAgent while ranking second overall on the reliability-focused metric. Kimi K3: Kimi K3 is Kimi Moonshot's top-performing open-weights model on the AA-AnalystAgent benchmark at launch. Grok 4.5: Grok 4.5 is a model variant from the xAI family evaluated in the AA-AnalystAgent failure analysis for its handling of source material and assumptions. Anthropic: Anthropic develops advanced AI models including the Claude family. In the news, Anthropic's Claude Opus 5 leads the AA-AnalystAgent benchmark, with the company securing three of the top five positions overall. deepseek_ai: deepseek_ai builds open-weights AI models including the DeepSeek series. Its DeepSeek V4 Flash model was evaluated on AA-AnalystAgent as one of the open-weights entries. Claude Opus 5: Claude Opus 5 is Anthropic's leading model variant that topped the AA-AnalystAgent benchmark for consistent performance across repeated task attempts. Kimi Moonshot: Kimi Moonshot develops large language models including the Kimi series. Its Kimi K3 model ranked as the top open-weights performer on the new AA-AnalystAgent benchmark. MiMo-V2.5-Pro: MiMo-V2.5-Pro is Xiaomi's model variant tested on AA-AnalystAgent, matching another model's score at a substantially lower per-task cost. Claude Fable 5: Claude Fable 5 is an Anthropic model variant that placed third on the AA-AnalystAgent leaderboard among the evaluated systems. AA-AnalystAgent: AA-AnalystAgent is a benchmark for evaluating AI agents on quantitative analysis tasks using real-world spreadsheets and documents across business and scientific domains. It employs an agentic harness with tools for code execution and web access, scoring models on pass^5 reliability across five runs per task. The benchmark was launched as a standalone leaderboard to test consistent performance in analyst-like workflows. Google DeepMind: Google DeepMind develops AI systems including the Gemini family. Its Gemini 3.1 Pro Preview model participated in the AA-AnalystAgent evaluation, showing strengths in initial task solving but lower reliability across repeated runs. Claude Sonnet 4.6: Claude Sonnet 4.6 is an Anthropic model variant evaluated on AA-AnalystAgent, demonstrating mid-tier performance in the benchmark results. DeepSeek V4 Flash: DeepSeek V4 Flash is deepseek_ai's model variant that ranked among the open-weights entries on the AA-AnalystAgent leaderboard. Gemini 3.1 Pro Preview: Gemini 3.1 Pro Preview is Google DeepMind's model variant that showed high single-run success but lower pass^5 reliability on AA-AnalystAgent tasks. Benchmark Focus: Agentic AI evaluations increasingly prioritize consistent reliability across repeated runs over single-attempt accuracy for real-world analyst applications. Model Categories: Closed frontier models continue to lead dedicated benchmarks testing complex document interpretation and workflow consistency compared to open-weights alternatives.