Artificial Analysis launches Decision Index v1.0 for benchmarking decision models
Summary
The recently launched Artificial Analysis Decision Index benchmarks decision models, with Jev 1.13 emerging as the leading model, scoring 58 on the index. While decision models like Jev are faster and more cost-effective in providing structured answers and classifications, they demonstrate notable calibration issues in knowledge and reasoning tasks, scoring lower than language models such as Claude Haiku 5.5, which, despite its higher accuracy of 61, incurs significantly higher costs and latency. This new index serves as a standardized tool to compare decision models with language models across various tasks, emphasizing the trade-offs between speed and accuracy.
Analysis
Jev: Jev 1.13 is a decision model from typesafe.ai designed for structured classification and probability-based outputs. It achieved the top score on the inaugural Artificial Analysis Decision Index and lies on the capability-cost Pareto frontier. NVIDIA: NVIDIA creates evaluation datasets and benchmarks for AI capabilities including the When2Call tool-calling benchmark. The benchmark forms one of the four core evaluations in the new Artificial Analysis Decision Index. Microsoft: Microsoft develops decision models including Microsoft-Decision-1 for classification and agentic decision-making. The model ties for second place on the Decision Index and appears on the capability-cost frontier. fastinoAI: fastinoAI produces decision models such as GLiDE optimized for speed and cost in structured prediction. GLiDE placed second on the Decision Index alongside Microsoft-Decision-1. typesafe.ai: typesafe.ai develops decision models including the Jev series for efficient, probability-calibrated classification tasks. Its Jev 1.13 model leads the new Decision Index released by Artificial Analysis. perplexity_ai: Perplexity develops decision models such as Decider v1.1 27B tailored for low-cost structured outputs. Its model records the lowest cost per task on the Decision Index while remaining competitive on capability. Claude Haiku 5.5: Claude Haiku 5.5 is a language model that generates open-ended responses including structured outputs. On the Decision Index it outperforms leading decision models on knowledge and reasoning benchmarks while incurring substantially higher cost and latency. Artificial Analysis: Artificial Analysis develops and maintains benchmarks for evaluating AI model capabilities and performance. It launched the Decision Index v1.0 in October 2026, introducing a composite evaluation framework for decision models across knowledge, reasoning, tool calling, and safety tasks. Benchmarking: The Decision Index provides a standardized way to compare decision models against language models on classification, tool calling, knowledge, and reasoning tasks. Model Tradeoffs: Decision models excel in speed and cost for structured tasks but show calibration issues and lower accuracy than language models on complex knowledge and reasoning benchmarks. Agentic Workflows: Decision models are positioned as efficient options for agentic applications requiring quick, probability-aware decisions on tool use and routing.
Categories
aimachine_learningtechai_agents