Benchmark Reviews launches to audit AI benchmarks with 15 assessments

Epoch: Epoch AI is a research institute that investigates trends in machine learning and the economic impacts of AI, with a focus on capabilities measurement. It maintains a public benchmarking hub, develops original evaluations such as FrontierMath, and tracks model performance across tasks. In this announcement, Epoch is launching Benchmark Reviews as an independent auditing service while explicitly excluding its own benchmarks from review to prevent conflicts of interest. Rubric v1: Rubric v1 is Epoch AI's initial evaluation framework for assessing AI benchmarks, establishing minimum standards for verification and criteria for identifying substantive flaws. It includes sampling protocols for error rate analysis in benchmark tasks and guidelines for citing external reviews where available. The rubric supports consistent, transparent judgments on benchmark quality and is expected to evolve with new flaw types. Benchmark Reviews: Benchmark Reviews is Epoch AI's new initiative dedicated to systematically auditing the quality of AI benchmarks. It evaluates benchmarks using a structured rubric to assign verdicts of Verified, Flawed, or Not Enough Info. The project aims to help users better interpret AI capability results by highlighting issues that could affect reliability. Scope: Reviews cover a range of benchmark types including math problem collections, coding tasks, game-based learning tests, and factoid question sets. Launch: Epoch AI introduced its Benchmark Reviews initiative on September 17, 2026, publishing initial assessments alongside detailed methodology documentation. Objective: The effort provides a consistent, independent source of information on benchmark quality to improve accurate interpretation of AI model performance results.