Artificial Analysis launches Coding Agent Index v2.0 with new benchmarks

DeepSWE: DeepSWE v1.1 was the prior software engineering benchmark used in earlier versions of the Coding Agent Index before being replaced by FrontierCode v1.1. It previously contributed to evaluations of AI coding agent performance across repository tasks. Cognition: Cognition develops AI tools and benchmarks focused on software engineering tasks. It created the FrontierCode benchmark, which features a private test set of real open-source repository tasks and serves as the replacement for DeepSWE in the Coding Agent Index v2.0. The benchmark uses maintainer-merge criteria and rule-based scanning for evaluation. GPT-6 Luna: GPT-6 Luna is an AI model configured as the judge for the revised SWE-Atlas-QnA-AA grading in the Coding Agent Index. It processes each rubric criterion with five votes to determine verdicts on agent responses. Claude Opus: Claude Opus 5.5 is an advanced AI model designed for complex reasoning and coding applications. In the Coding Agent Index v2.0, the Claude Code configuration powered by Claude Opus 5.5 achieves the highest overall score and leads particularly on the FrontierCode benchmark component. Hugging Face: Hugging Face is a platform for hosting and sharing machine learning datasets, models, and evaluation resources. The updated SWE-Atlas-QnA-AA dataset and its associated judge configuration have been published there for public access. SWE-Atlas-QnA: SWE-Atlas-QnA comprises 124 repository Q&A tasks originally developed by Scale AI for inclusion in the Coding Agent Index. These tasks assess AI agent capabilities in understanding and answering questions about code repositories. SWE-Atlas-QnA-AA: SWE-Atlas-QnA-AA is the revised version of the Scale AI repository Q&A tasks used in the Coding Agent Index v2.0. Updates include reworded rubric criteria for consistency and adoption of GPT-6 Luna as the multi-vote judge, with the full dataset and configuration released on Hugging Face. FrontierCode v1.1: FrontierCode v1.1 is a benchmark consisting of 100 software engineering tasks on real open-source repositories. Developed by Cognition, it evaluates whether proposed changes would be merged by maintainers and incorporates scanning for solution material access. It forms one of the three equally weighted components of the updated Coding Agent Index. Model Integration: Leading AI models are increasingly paired with specialized agent harnesses to optimize performance across diverse coding and repository interaction scenarios. AI Coding Benchmarks: The field of AI agent evaluation is seeing increased focus on real-world software engineering tasks that mirror actual developer workflows and code review processes. Evaluation Refinements: Updates to AI benchmarks often involve stricter grading rubrics and independent judging models to improve consistency and reduce variability in results.