UC Berkeley’s Sky Computing Lab explores AI model cost factors

Summary

In a recent analysis, a study led by Melissa Pan, PhD candidate at UC Berkeley’s Sky Computing Lab, compared the performance of three coding agents: Claude Code, Codex, and Pi, focusing on the significant impact of the "harness tax" on their cost and efficiency. The research revealed that the choice of harness affects cost more than accuracy, emphasizing the importance of evaluating overall cost per successful task rather than solely accuracy. For instance, while Claude Code achieved a high success rate of 97.8% at a cost of $1.33 per rollout, Pi performed slightly lower at 96.7% but at just $0.67, illustrating the cost benefits that can be gained while maintaining comparable quality.

Analysis