Blog
#0145 min read

How to evaluate an AI agent before you trust it with your customers

An agent that's right 90% of the time fails far more than it seems. How to build evaluations for AI agents from real tasks, what to grade, and why to read the transcripts.

“We tested it and it works” is the most dangerous sentence in an agent project. It almost always means someone sent it ten messages, the answers looked fine and the team moved on. An agent doesn’t behave the same way twice, and what matters isn’t whether it can do the job, but whether it always does.

This is the fourth note in the series on how to build AI agents. Most of it comes from Anthropic’s guide to evaluating agents (Demystifying evals for AI agents, 2026), adapted to how I apply it on projects for companies.

The parts of an evaluation

It’s worth using the same words:

  • Task: one test case, with an input and a success criterion. “The customer wants to move their Tuesday appointment to Thursday afternoon.”
  • Trial: one run of the agent on that task. Since the agent doesn’t answer the same way every time, each task runs several times.
  • Transcript: the full record of a trial: messages, tool calls, reasoning and intermediate results.
  • Outcome: how the world ended up. Not what the agent said, but what happened.
  • Grader: the logic that checks one aspect of a trial and scores it.

The difference between what the agent says and the outcome is the most important one. An agent can reply “done, your appointment is now on Thursday” without having moved it. The evaluation checks the calendar, not the message.

The 90% that isn’t 90%

Say your agent handles a task correctly 90% of the time. Sounds good. Now think of a customer who writes to it ten times a year. The chance that all ten go well is 0.9 multiplied by itself ten times: under 35%.

Anthropic separates two metrics:

  • pass@k: the chance that at least one of k trials succeeds. It fits when you can retry and keep the best, like generating ideas.
  • pass^k: the chance that all k trials succeed. It’s the one that matters for a customer-facing agent, because every trial is a real conversation.
Fig.An agent that's right 90% of the time, across several trials

Source: My calculation at a 90% success rate per trial, following Anthropic's definitions

See the data
ItemEvery trial succeeds (pass^k)At least one succeeds (pass@k)
190%90%
281%99%
372.9%99.9%
465.6%100%
559%100%
653.1%100%
747.8%100%
843%100%
938.7%100%
1034.9%100%

That’s why each task runs several times. One run that goes well doesn’t tell you whether the agent is dependable.

Start with 20 to 50 real tasks

You don’t need hundreds of cases to start. Early on, changes to the agent produce big differences, and a few tasks are enough to see them. Anthropic recommends starting with 20 to 50, and for their research system they began with about 20 representative queries (How we built our multi-agent research system).

What matters is where they come from:

  • real conversations and cases from your business,
  • failures you already know about or that users reported,
  • the hard cases: confusing messages, missing data, angry customers, attempts to trick the agent.

Every time the agent fails in production, that conversation becomes a new task.

Grade the outcome, not the path

It’s tempting to check that the agent called the exact tools in the order you expected. But an agent can reach the right outcome by another valid path, and a rigid test punishes it for that. Mostly check what it produced: does the appointment exist in the calendar with the right date? Was the refund recorded for the right amount?

There are exceptions you do check on the path: the rules that must never break. That it didn’t call approve_refund without confirmation, or that it didn’t look up another customer’s data.

Three kinds of grader

GraderHow it worksGood forWatch out for
CodeCompares data, checks a record exists, validates a formatOutcomes you can verify exactlyIt’s brittle with open-ended answers
ModelAnother model scores against a written rubricTone, clarity, whether it answered what was askedIt needs calibrating against people
PeopleSomeone who knows the business reviews a sampleThe ground truth, and calibrating the other twoIt’s slow and expensive

Use code wherever you can, a model where code falls short, and people to check that the other two grade well.

Two test suites

  • Capability tests: what the agent doesn’t do well yet. They should start with a low pass rate, because they’re what you’re improving.
  • Regression tests: what the agent already does well. They should sit near 100%, and they run on every change to the prompt, the tools or the model.

When a capability task becomes stable, it moves to the regression suite.

The A/B bench: two versions of the agent, the same tasks, one scoreboard. The numbers pick the winner, not the demo.

Read the transcripts

No number replaces reading what happened. A failing task can be a real agent mistake or a grader that rejected a valid answer, and you only find out by reading. As Anthropic puts it, you won’t know whether your graders work unless you read the transcripts and grades from many trials.

And after launch

Evaluations don’t end when the agent goes live. Models change, customers write differently and cases appear that nobody foresaw. Log every conversation, review a sample every week and turn failures into new tasks. That way the test set grows with the business.

If the agent is going to talk to your customers, these are the other things to check: what to require before putting an AI agent on WhatsApp.