The Leaderboard Left Out the Price

Key Takeaways

• On the HumanEval coding benchmark, the tree-search agent LATS scored 88.0% for $134.50 a run. Calling GPT-4 directly, with no agent at all, scored 89.6% for $1.93. A simple retry baseline scored 92.0% for $2.51.
• Agent leaderboards rank accuracy and leave cost off the page. Kapoor and Narayanan found that for substantially similar accuracy the cost “can differ by almost two orders of magnitude”, and that cost was a top-line metric in none of the papers they reviewed.
• The reason is who the benchmark is for. For a lab studying a model, accuracy at fixed compute is the scientific question. For anyone deploying, dollars per task is the construct they actually pay on.
• 2026 makes this worse, not better. The newest gains come from spending more at inference. ARC-AGI is the rare benchmark that reports cost, and it showed a high-compute run costing thousands of dollars per task against roughly $5 for a human.
• Read a leaderboard as a frontier, not a ranking. Ask what the top row cost, and pick the cheapest system on the accuracy plateau you need.

I have argued here that an agent benchmark can be gamed: a perfect score on SWE-bench without solving a task, earned by an agent that reached the grader and rewrote its verdict. That is a story about a broken instrument, one whose reading you can no longer use.

This is a subtler failure, and it does not require anyone to cheat. Take a benchmark nobody has tampered with, reporting an honest accuracy number. It is still, for almost everyone who reads it, reporting half of the measurement they need.

What does an agent leaderboard leave out?

A leaderboard sorts agents by accuracy. The person reading it is usually trying to choose one, which means they are going to run it thousands or millions of times and pay for every token it spends on the way to each answer.

Those are two different quantities. One is how often the agent is right. The other is what being right costs. An agent leaderboard reports the first and, almost always, is silent about the second. Accuracy without cost is a price tag with the price torn off: it tells you the thing is good without telling you whether you can afford to use it.

For a while this did not matter much, because agents were cheap relative to how wrong they were. That has inverted. The gap between the cheapest way to reach a given accuracy and the most expensive way is now enormous, and the leaderboard points you at the wrong end of it.

How much did the top-ranked agent actually cost?

The cleanest measurement of this comes from AI Agents That Matter, by Sayash Kapoor, Benedikt Stroebl, Zachary Siegel, Nitya Nadgir and Arvind Narayanan, published in July 2024. They took the HumanEval coding benchmark and ran several published agent architectures against a set of deliberately simple baselines, recording accuracy and the dollar cost of getting there.

Read the bottom-heavy rows first. LATS is a tree-search agent, one of the more elaborate architectures in the field: it explores many reasoning paths, backtracks, and calls the model again and again. It scored 88.0%, at a total cost of $134.50. Calling GPT-4 directly, with no agent scaffolding of any kind, scored higher, 89.6%, for $1.93. A plain retry loop scored 92.0% for $2.51. LATS cost, in the authors’ words, “over 50 times more” than the warming baseline, and lost to it on accuracy.

The authors state the general finding flatly: “For substantially similar accuracy, the cost can differ by almost two orders of magnitude.” And the reason it goes unnoticed: “the cost of running these agents isn’t a top-line metric reported in any of these papers.” There is a leaderboard for the accuracy and no column for the bill.

The row at the very bottom is the counterweight to any nihilism. “Escalation” starts every task on a cheap model and moves up to a more expensive one only when a test fails. It scored 85.0% for twenty-seven cents. It gives up a handful of accuracy points and takes two orders of magnitude off the price. That trade is invisible on a leaderboard sorted by accuracy alone, where escalation sits near the bottom looking like the worst option in the table.

Why would a more complex agent score worse and cost more?

Because the complexity buys search, and search is paid for in tokens. A tree-search agent does not think harder in some free interior way. It calls the model many times, keeps several lines of attack alive, and discards most of them. Every discarded path was still billed. When the underlying model is already good at the task, most of that search is spent confirming an answer it would have reached on the first try.

That is why the simple baselines hold up. Retrying the same prompt, warming up with a few worked examples, or escalating from a cheap model on failure each capture most of the available accuracy with a small, bounded number of calls. Kapoor and Narayanan put the comparison plainly: “there is no significant accuracy difference between our warming strategy and the best-performing agent architecture.” The elaborate agent’s “state of the art” label attached to its novelty, not to any accuracy the baselines could not reach far more cheaply.

Who is the leaderboard for?

The deepest point in the paper is not a number. It is a question about audience.

“For model evaluation,” the authors write, “controlling for compute is reasonable.” A lab studying whether a model is getting smarter wants to hold resources fixed and vary the model, the same way you would not judge two runners by letting one start early. Accuracy at fixed compute is a clean scientific question, and for that question cost is noise to be controlled away.

“For downstream evaluation, cost is the actual construct of interest.” Someone building on an API “doesn’t care about the number of active parameters,” as the authors put it. “They simply care about the dollar cost relative to accuracy.” That is not a refinement of the accuracy question. It is a different question, whose answer is a ratio.

A single leaderboard tries to serve both readers and ends up serving the second one badly. It reports the scientist’s number to the engineer, who then makes a purchasing decision with the price hidden. The benchmark is not wrong. It is answering a question the reader was not asking, in a voice confident enough that the reader does not notice the swap.

Has this gotten better or worse since 2024?

Worse, because the newest capability gains are bought precisely where the leaderboard is blind: at inference time.

Reasoning models improve by spending more compute per question, and agents built on them inherit that. The clearest illustration is one of the few benchmarks that does report cost. ARC Prize treats efficiency as a required metric and plots score against cost per task. When OpenAI’s o3 posted its breakthrough ARC-AGI numbers in December 2024, the high-compute configuration reached its top score at a cost ARC Prize described as “thousands of dollars per task”, against “roughly $5 per task” for a human doing the same puzzles. The accuracy headline and the cost headline told opposite stories, and only one of them usually makes the leaderboard.

The scale of evaluation itself now says the same thing. A 2025 enterprise benchmarking effort by JV Roig ran “over 5.5 billion tokens across 35 model configurations” and lists cost-performance tradeoffs among its headline findings, calling them “critical for enterprises making deployment decisions.” When a single pass of the measuring instrument burns billions of tokens, cost has stopped being a footnote to the result. It is part of the result.

What does an accuracy-only score quietly assume?

Every measurement carries an assumption about when it is valid. An accuracy-only leaderboard assumes that cost is either free or fixed, so that the ranking which maximizes accuracy is also the ranking that maximizes value.

Hold compute constant, as a lab does, and the assumption holds. Pay per token, as a deployer does, and it breaks, because the quantity you care about is now accuracy divided by cost, and a ranking on the numerator alone can float the worst deal in the table to the top. The leaderboard does not lie about the accuracy. LATS really did score 88.0%. It simply presents that number in a way that implies a verdict it has not earned: that this is the one to use.

This is a cousin of the instrument problem I wrote about before, where a benchmark could no longer tell a solved task from a gamed grader. There the score meant less than it claimed because the test could be subverted. Here the score is entirely honest and still misleads, because it is only one coordinate of a two-coordinate answer, and the missing coordinate is the one with a dollar sign.

How should you actually read an agent benchmark?

Four habits recover most of what the ranking hides.

Read it as a frontier, not a list. Plot accuracy against cost per task. The agents worth your attention are the ones on the frontier, where nothing beats them on both axes at once. Everything inside the frontier is dominated: something is cheaper at the same accuracy, or better at the same price.

Ask what the top row cost. If the paper or leaderboard does not report cost, treat the accuracy as unpriced rather than free. An unpriced score is a claim you cannot act on, only admire.

Buy the accuracy you need, not the most there is. A few points of accuracy that cost fifty times more are rarely worth it outside a competition. Decide the accuracy your application actually requires, then find the cheapest system that clears it.

Build the cheap baseline first. Retry, warm with examples, escalate from a small model. Then make the elaborate agent beat that, on accuracy per dollar, before you adopt it. Measured against nothing, every agent looks worth its price. Measured against a retry loop, most of them do not.

Closing Thought

The models in that HumanEval table were not misbehaving. Neither were the benchmarks. Everyone optimized exactly what they were told to. We told the leaderboard to sort by accuracy, and it did, faithfully, all the way to a $134 answer that a $2 one beat.

A score is a claim about the conditions that produced it, and one of those conditions is the bill. When we quote the accuracy and drop the price, we are not measuring the agent. We are measuring the half of it that flatters the number, and then buying on the strength of what we chose not to look at.

Further Reading

• Sayash Kapoor, Benedikt Stroebl, Zachary Siegel, Nitya Nadgir, Arvind Narayanan: AI Agents That Matter, arXiv, July 2024
• ARC Prize: OpenAI o3 Breakthrough High Score on ARC-AGI-Pub, December 2024
• JV Roig: Towards a Standard, Enterprise-Relevant Agentic AI Benchmark, arXiv, November 2025
• Daniel Kang: AI Agent Benchmarks Are Broken, the accuracy-side companion to this cost-side argument

More essays at Call to Think, or start with the AI Evaluation collection.