Evaluating the Unpredictable: Why and How Evals Should Work in Agentic AI
Last updated on

Evaluating the Unpredictable: Why and How Evals Should Work in Agentic AI

By Aditya Karnam

Key Takeaways

  • Agentic AI changes what we’re evaluating: from static outputs to dynamic processes.
  • Traditional evals measure correctness at a point in time. Agentic evals must measure reliability across trajectories.
  • The gap is measurable: on WebArena, the best GPT-4 agent finished 14.41% of tasks against 78.24% for humans.
  • Reliability collapses under repetition. τ-bench’s pass^k metric — succeeding on all k attempts, not any one — drops below 25% at k=8 for a leading agent.
  • The right eval system distinguishes between arriving at the right answer through good judgment versus accidental luck.

There’s a fundamental mismatch between how we evaluate AI and what AI systems are becoming — and it’s becoming increasingly costly to ignore.

You’ve probably seen the familiar pattern: a company benchmarks their model, publishes throughput numbers, maybe a few hand-picked examples of impressive outputs. The eval dashboard shows high accuracy scores across multiple-choice benchmarks. And then, in production, an agent loops three times into a hallucinated database query before wasting $40 and missing a deadline.

The model wasn’t broken. The eval was.

This gap is measurable, and it is not small. When researchers built WebArena, a realistic web environment rather than a multiple-choice test, their best GPT-4-based agent completed 14.41% of tasks end-to-end against a human rate of 78.24%. GAIA, a benchmark of real-world assistant questions, found humans scoring 92% where GPT-4 with plugins managed 15% — an inversion of the usual story, where models beat humans on bar exams and chemistry problems. The tasks that models find hardest are the ones humans find routine, and conventional benchmarks are largely blind to them.

This is the gap I want to examine: why traditional evaluation strategies fail for agentic systems, and what a proper eval framework should look like instead. Not as theory — but as something that could actually be built.

What Assumption Are Traditional Evals Built On?

Traditional LLM evals rest on a simple assumption: input in, output out. You give the model a prompt, you collect the response, you compare it against an answer or score its quality. It works fine for classification, summarization, single-turn QA. The evaluation is static because the task is static.

An agent doesn’t work that way. An agent plans. It makes tool calls. It reads results. It re-evaluates its strategy mid-stream. It can loop back. It can recover from early mistakes — or compound them. It navigates ambiguity autonomously, making decisions at every fork in the road.

Evaluating agents against the same rubric as static completions is like evaluating a chess player by how good their first move is. The opening matters, but the game happens across fifty moves.

The evaluation needs to shift from points to trajectories. But here’s the uncomfortable part: that’s much harder to do well.

Why Is Outcome-Only Evaluation Misleading?

Let me be specific about what goes wrong when you only measure whether an agent “got the right answer.”

An agent can reach a correct outcome through terrible reasoning. It might randomly guess, hit a lucky database result, or reverse-engineer the answer through brute force. If your eval only checks the final output, this looks indistinguishable from competent problem-solving. That’s dangerous — because you’ve built confidence around something that isn’t reproducible.

This one has been quantified, and the result is stark. The τ-bench authors introduced a metric called pass^k — the probability an agent solves the same task on all k independent attempts, rather than the usual pass@k, which credits it for succeeding on any one of them. The distinction matters enormously: as k grows, pass@k climbs toward 1 while pass^k collapses. Running the same tasks eight times, a leading function-calling agent’s pass^8 fell below 25% in the retail domain. Same agent, same tasks, most of the apparent competence gone the moment you asked it to be right repeatedly rather than once.

That is the whole problem in a single number. A single-trial benchmark cannot distinguish a reliable agent from a lucky one, because it only ever gives the agent one chance to get lucky.

An agent can fail gracefully and look fine on a coarse metric. Consider an agent faced with ambiguous instructions. A sophisticated one might acknowledge the ambiguity, propose assumptions, and ask for clarification. A naive one might just hallucinate confidently. Both could score identically if evals only measure whether the user eventually got something back.

An agent can succeed accidentally through overpowered capabilities. If your agent has a massive context window or access to an exhaustive knowledge base, it might produce reasonable outputs without actually demonstrating planning, reasoning discipline, or tool-use strategy. The eval system credits “intelligence.” What existed was just scale.

These are not edge cases. They’re the baseline behavior of any system whose success you can’t observe from the end result alone.

What Should an Agentic Eval Actually Measure?

A meaningful eval framework for agents should measure three things, none of which a traditional benchmark covers:

1. Outcome Quality — Did it solve the problem?

This is the baseline and still necessary. Did the agent produce the right answer? Was the tool output correct? Were the recommendations actionable? This feels like the “obvious” thing to measure — but in an agentic system, it’s only one signal, not the whole story.

2. Process Discipline — Did it reason well along the way?

This is where most eval systems break down entirely. An agent that plans poorly, misuses tools, follows dead-end paths, or fails to revise course when evidence says “stop” might still land on a correct answer. The process was bad. The outcome was lucky.

Process evaluation means assessing things like:

  • Planning quality: Did the agent develop a coherent strategy before acting? Or did it react step-by-step from the first signal?
  • Tool-use discipline: Did it use tools intentionally and interpret results honestly — or does it treat them as prompts to be “worked around”?
  • Recovery behavior: When things went wrong, did the agent recover or escalate the error?

These aren’t nice-to-haves. They’re distinguishing factors between an agent that’s reliable and one that’s occasionally lucky.

This is a live research direction, not just a wish. The Agent-as-a-Judge work opens with essentially the same complaint this section makes — that existing evaluation methods “focus exclusively on final outcomes — ignoring the step-by-step nature of agentic systems, or require excessive manual labour” — and responds by giving the evaluator agentic capabilities of its own, so it can inspect the reasoning trajectory rather than just the answer at the end. Their reported result is the encouraging part: judging the process this way substantially outperformed LLM-as-judge and approached their human evaluation baseline.

3. Meta-Cognition — Did it know what it didn’t know?

This is the hardest axis to operationalize but arguably the most important for agents deployed in real systems. An agent should be able to:

  • Recognize when evidence is insufficient
  • Know when a tool output contradicts its assumptions
  • Stop and signal uncertainty rather than fabricate confidence
  • Escalate when it’s out of its domain

If an agent can’t distinguish between “I think I know” and “I actually know,” it will overreach — and the cost of that overreach grows with its capabilities.

How Do You Actually Build an Agentic Eval System?

This is the part most blogs skip. Frameworks are easy. Implementation isn’t. Here’s a rough sketch of what this looks like in practice.

Designing Eval Traces

For agents, you can’t just compare input → output. You need to capture and score the trajectory: what the agent planned, which tools it called with what parameters, how it interpreted each result, when it updated (or failed to update) its strategy.

A basic eval trace might look like:

{
   "task": "Find competitor pricing for X product in region Y",
   "plan": "Search docs → scrape site → compare",
   "tool_calls": [
     {"tool": "search", "params": {"query": "X competitor pricing"}, ...},
     {"tool": "scrape", "url": "..."}
   ],
   "reasoning_steps": ["Initial search returned incomplete...", ...],
   "final_answer": "...",
   "is_correct": true,
   "process_score": 0.72
}

Each step and the final trajectory get scored independently. The process score might differ from the outcome score — and that difference is where you learn something.

Scoring Methodologies

There are a few approaches, each with trade-offs:

  • Human scoring: Gold standard but expensive and slow. Best for building gold datasets and calibrating automated scores.
  • LLM-as-judge: Efficient at scale, vulnerable to the same biases as the rest of the model. The foundational study here is worth reading in both directions: strong judges like GPT-4 reached over 80% agreement with human preferences — about the rate at which humans agree with each other, which is genuinely good. The same paper names the failure modes: position bias (sensitivity to which response is shown first), verbosity bias (longer answers score better regardless of quality), and self-enhancement bias (models favour their own outputs). For process evaluation the last one should worry you most, since the judge and the agent are often the same model family. The meta-problem underneath: asking an LLM to evaluate another LLM’s reasoning can be gamed by any agent that learned to produce “correct-feeling” justifications.
  • Deterministic rules: Structured checks (tool args validated against schema, response format enforced, step count bounded). High precision, low recall — it catches the obvious violations but misses subtle failures.
  • Hybrid scoring: Combine deterministic guardrails with LLM-as-judge for process quality and human review for edge cases. This is where things are actually heading because no single approach covers the space well.

The Feedback Loop

An eval system that doesn’t feed back into the model’s improvement loop is just a scoreboard — and scoreboards don’t make anyone better.

For agents specifically, I think the feedback should target:

  • Planning strategies: Which approaches to solving problem X actually worked? Use this to adjust future plans.
  • Tool reliability: Which tools consistently return useful output vs. noise? The agent needs to weight them differently.
  • Failure patterns: What kinds of mistakes recur? This informs safety guardrails and escalation thresholds.

The eval system isn’t a report card. It’s part of the training signal — just in a delayed, post-hoc form.

Who Evaluates the Evaluator?

Here’s what keeps me up, and it’s worth being honest about: evaluating an eval system is itself hard. We’ve seen this pattern before with LLM benchmarks themselves — the benchmarks become proxies for capability rather than measurements of it. Agents will face the same pressure: once there are scores to optimize for, someone will game them.

What separates robust evals from fragile ones is whether they can detect when they’re being gamed. If an agent learns to produce evaluations that look like high-quality reasoning without actually demonstrating it, and your eval system doesn’t catch the difference, you’ve built a paper metric — one that sounds impressive but tells you nothing about actual capability.

We should design eval systems with this vulnerability in mind from the start, not retroactively after someone discovers how to inflate scores without improving the underlying behavior. The shape of the failure is predictable: any easily-countable proxy — tool calls completed, steps executed, tokens consumed — will get optimized because it is easy to count, not because it measures effectiveness. An agent that makes more tool calls is not thereby better at its job; it may simply be flailing more efficiently. The single-trial-versus-pass^k gap above is the same lesson in miniature, and it took someone deliberately re-running the identical task eight times to expose it.

Closing Thought

The shift from static models to agentic systems demands a rethinking of evaluation itself. We need frameworks that can meaningfully distinguish between:

  • An agent that solves problems reliably versus one that sometimes does
  • An agent that reasons through ambiguity versus one that ignores it
  • An agent that knows its limits versus one that overreaches

None of this is easy. But the cost of not doing it properly — deploying agents on metrics that don’t actually measure what we think they do — will be higher.

We built systems to reason for us. The least we can do is reason carefully about whether they’re actually reasoning.

Further Reading

More essays at Call to Think · About this project