An AI assistant says it has moved a customer’s delivery to Friday.
That sounds helpful. The evaluation begins when you open the order and see what actually changed.
For an agent that can use tools and modify business records, a good final message is only one part of success. The requested action needs to be correct, authorized and complete, with relevant side effects checked. Otherwise, you have built a very agreeable narrator.
Anthropic’s January 2026 guide to agent evaluations makes a useful distinction between the agent’s conversation and the final state of its environment. Here is how I would apply that distinction to a hypothetical delivery-support agent.
Write one case with a clear answer
The customer requests a date change for one eligible order. The test environment contains that order, another order on the same account and a defined delivery cutoff. The customer is authorized to change the first order.
Success means that the correct order moves to an available Friday slot, the unrelated order stays unchanged and the customer receives an accurate explanation of the result. The test also specifies which notification or scheduling records should exist.
That sounds a little fussy until you consider the alternatives. “It mentioned Friday” is a wonderfully generous pass condition for an agent operating a delivery schedule.
The expected outcome needs agreement from someone who understands the business rules. If the team cannot decide whether a late request should be accepted, the evaluation has found an unresolved product decision before the agent has done anything.
Use realistic but appropriately protected data. Reset the environment between trials so an earlier attempt does not reserve the slot or change the order for the next one.
Curious what an AI agent could do for your team? See our AI agent services.
Inspect the action and its consequences
Run the case through the same relevant tools and permissions that the agent would use in the intended workflow. Then inspect the resulting records independently.
Did it select the correct order? Did it use an available date? Did it preserve the address and other fields? If the scheduling service rejected the change, did the agent leave the order alone and explain the problem?
The task can fail in different ways. One attempt may misunderstand the request. Another may choose the wrong tool argument. A third may make the correct change but tell the customer something misleading.
Keep those failure categories separate. They point toward different fixes: clearer product instructions, better tool definitions, stronger permission enforcement or improved result checking.
For objective records, use direct checks where practical. A model-based judge can help assess qualities such as whether the explanation is understandable, but it needs a clear rubric and calibration against human review. A second AI nodding thoughtfully is not independent proof that the truck will arrive on Friday.
Check out this blog: The AI feature is popular. Is it making the business better?
Expand the cases before trusting the score
The initial case establishes a useful baseline. Add variations that reflect the actual workflow: no available slot, an expired cutoff, an ambiguous order reference and a request from someone without permission.
Include a service interruption as well. A good result may be a clear escalation with no unauthorized change, rather than a completed reschedule. The evaluation should reward the behavior the business actually wants.
Repeat representative cases because an agent’s behavior can vary. Keep a record of the model, instructions, tool versions and environment so comparisons have meaning. When something changes, rerun the relevant cases and review both the outcomes and the traces of problematic attempts.
Avoid reducing all of that to one attractive percentage. A strong average can conceal a serious permission failure. Report important failure types and the conditions in which they occur.
The finish line is a verified outcome that matches the customer’s request and the business’s rules. A friendly explanation belongs there too. Ideally, “Done” should be the least surprising part of the result.