Artificial Analysis Cyber Index launches with new evaluation standards

IBM: IBM participates as a launch partner in the Artificial Analysis Cyber Index Alliance to help set standards for evaluating AI models on enterprise cyber defense tasks. Its involvement supports partner contributions to benchmark design and related datasets. NVIDIA: NVIDIA joins as a launch partner in the Artificial Analysis Cyber Index Alliance to advance standardized evaluation of AI performance in cyber defense. The company contributes to the collaborative effort on benchmark development alongside other industry participants. Vercel: Vercel develops benchmarks focused on AI model capabilities in vulnerability discovery within application code. It contributed the DeepsecBench-AA evaluation, which isolates the process of finding vulnerabilities and scoring them against expert-verified sets. Vercel is one of the launch partners in the Artificial Analysis Cyber Index Alliance. Collinear AI: Collinear AI develops cybersecurity benchmarks for AI coding agents, with a focus on auditing open-source codebases for security weaknesses and patching them. It contributed the CWE-Bench-AA evaluation, which covers auditing and patching tasks across multiple programming languages and all ten OWASP Top 10 (2025) categories. This benchmark forms one component of the Artificial Analysis Cyber Index. Artificial Analysis Cyber Index: The Artificial Analysis Cyber Index is a composite benchmark designed to evaluate how AI models perform on enterprise cyber defense tasks, specifically the defensive loop of discovering vulnerabilities in source code, reproducing them, and patching them without breaking existing functionality. It combines three partner-contributed evaluations and reports safety refusals separately from capability scores to provide a clear comparison for decision-makers. The Index was announced as a new independent standard focused solely on defense, excluding exploit realization or tasks without source access. Artificial Analysis Cyber Index Alliance: The Artificial Analysis Cyber Index Alliance is a collaborative group of industry partners working to establish standards for evaluating AI models on enterprise cyber defense tasks. It launches alongside the Cyber Index and enables partners to contribute expert input on benchmark design, implementation, and datasets. Launch partners include Collinear AI, IBM, NVIDIA, and Vercel, with the Alliance open to additional organizations. Benchmark Scope: The Artificial Analysis Cyber Index measures only defensive cyber work from source code access, covering discovery, reproduction, and patching of vulnerabilities while explicitly excluding exploit creation. Industry Partnership: The Cyber Index Alliance enables ongoing collaboration among partners to refine evaluations and incorporate new datasets as AI cyber capabilities evolve. Safety Considerations: Several frontier models decline a substantial portion of tasks on safety grounds, limiting their scores on the Index even when they demonstrate strong agentic coding abilities elsewhere.