Evals for agents: why your AI agent breaks without tests

Building a working demonstration of an AI agent is nowadays a matter of a few hours. You connect a language model to a handful of external functions, give the system a well-thought-out system instruction, and watch the model make decisions on its own. The real challenge, however, begins once that agent is deployed to production. Where a simple prompt application at worst returns a poorly formulated answer, an autonomous agent has the ability to perform actions in the outside world. Without a solid testing infrastructure, this inevitably leads to stuck loops, corrupted database records, unnecessarily high API costs, and unpredictable errors on unexpected user input.

This article falls within Module 6 — Building Responsibly of the AI curriculum at leren.llmnet.nl. In this module, we focus on the architectural and operational safeguards needed to bring complex language model applications to production in a safe and reliable manner.

Why traditional unit tests don't work for AI agents

In classic software development, a unit test is deterministic. If you call a function with argument A and B, you expect exactly outcome C. If the code doesn't change, the test passes today, tomorrow, and a year from now. AI agents break this foundation in two ways: the underlying LLM is stochastic and the environment in which the agent operates changes dynamically.

An AI agent works according to a continuous loop of observing, reasoning, selecting tools, and executing. During each step in this loop, the model generates a response based on the current context. Because language models use probability distributions over tokens, the same prompt with an identical temperature setting can still produce subtly different reasoning paths. A classic assertion that checks for an exact string will therefore fail regularly, even when the agent has completed the task perfectly in substance.

When an agent makes a mistake in step two of a process consisting of six steps, that error propagates cumulatively through the remaining steps. The agent can get stuck on a wrong assumption, leading to hallucinations further along in the task. Take a look at the guide on understanding hallucinations in language models if you want to understand how flawed assumptions arise in the reasoning step and send the agent down the wrong path. For this reason, standard software tests fall short, and we need specialized evaluation frameworks: so-called agent evals.

The three levels of agent evaluation

An effective test architecture for an agent checks the system at three different levels. Each level highlights a different part of the decision-making chain and helps locate specific defects.

1. Step-by-step / Component evaluation

At the lowest level, you isolate the model's individual decisions. You don't test the entire process, but validate one specific step. Can the model select the correct tool based on a specific user question? Does the model generate valid JSON that exactly matches the expected schema?

Component evaluations are fast, cheap, and largely deterministic. If the agent fails here, the problem is almost always unclear system instructions or a poorly defined tool interface. Consult this guide if you want to learn how clarified function descriptions drastically reduce the number of incorrect tool calls, via writing effective tool descriptions for your agent.

2. Trajectory evaluation (Reasoning Trajectory)

At the middle level, you assess the route the agent takes to reach a final result. An agent might find the right solution but make twelve unnecessary API calls and search in the wrong direction three times along the way. That is undesirable in a production environment due to the accumulated latency and API costs.

With trajectory evaluation, you analyze the sequence of thoughts, actions, and observations. You measure things like the number of steps, re-executing already-successful actions, and recovering from error messages. Read this in-depth article if you're looking for quantitative protocols to compare agent trajectories across thousands of runs, via methodologies for benchmark-level agent evaluation.

3. Final result & effect evaluation

At the highest level, you assess the ultimate effect of the agent run. Was the database update actually applied correctly? Does the generated email draft contain all the required customer information? Was the resolved ticket closed with the correct status code?

Final result evaluation does not focus on *how* the agent completed the task, but exclusively on the quality and correctness of the final state change. It often combines physical inspection of the environment (for example a database query) with qualitative review via an evaluation model.

Trajectory evaluation: analyzing the agent loop

Analyzing the reasoning trajectory is the most unique challenge in testing agents. A trajectory can be represented as a sequence of states. When a user asks: "Cancel my most recent order and send a confirmation by email", the ideal trajectory looks like this:

[START] -> User Query
  │
  ├──> Thought: Zoek de klant-ID en meest recente bestelling op.
  ├──> Action: get_customer_orders(customer_id="123", limit=1)
  ├──> Observation: Bestelling #98765, status: "in behandeling", bedrag: € 45,00.
  │
  ├──> Thought: De bestelling kan worden geannuleerd. Roep de annulering-API aan.
  ├──> Action: cancel_order(order_id="98765")
  ├──> Observation: Bestelling #98765 succesvol geannuleerd.
  │
  ├──> Thought: Stuur nu een bevestigingsmail naar de klant.
  ├──> Action: send_email(customer_id="123", template="order_cancelled", order_id="98765")
  ├──> Observation: E-mail verzonden met id #abc12.
  │
  └──> Final Answer: Uw bestelling #98765 is geannuleerd en er is een bevestiging gemaild.
[EIND]

A broken agent can derail in this loop in several ways:

• Infinite Loop: The agent keeps calling `get_customer_orders` because the response isn't properly processed into the context.
• Tool hallucination: The agent tries to call a non-existent function such as `delete_order_permanently`.
• Premature give-up: The agent stops after the first step and tells the user the order has been canceled, while only the status was retrieved.
• Context pollution: The agent adds unnecessarily large amounts of raw JSON to its reasoning context, causing the maximum token limit to be exceeded quickly.

Visit the analysis of context engineering for strategies to keep the prompt history of your test cases compact and representative, via designing an effective context architecture.

Enabling test datasets and mocks for agent evals

Having live API calls execute against external systems during automated test runs is dangerous and slow. A test set of 100 scenarios that actually sends live emails, processes payments, or modifies CRM records pollutes your production data and costs a fortune in API credits.

That's why professional testing infrastructures work with two layers of isolation:

1. Mocking the external environment

All tools the agent uses must be replaced in a test environment with so-called mock functions. When the agent calls the function `get_customer_orders`, the mock returns a fixed JSON response without consulting a real database. This guarantees the test stays consistent, independent of changes in live data.

2. Recording and replaying trajectories (Recorded Trajectories)

When building a regression test set, you record a successful run. As soon as a change is made to the system instruction or the agent's code, you run exactly the same prompt again. The test framework compares the new trajectory with the recorded trajectory. Deviations in tool selection or reasoning steps are immediately flagged for inspection.

LLM-as-a-Judge vs. Deterministic Assertions

To determine whether a test passes or fails, you use a combination of deterministic checks and evaluation by a second-opinion model (LLM-as-a-judge).

When you deploy an LLM as a judge, it is essential to draw up a strict rubric. Never simply ask a judge model: "Is this a good answer?". Instead, give the model a clear scale with concrete criteria:

Beoordeel het onderstaande antwoord van de agent op een schaal van 1 tot 5 op basis van de volgende criteria:
- 1 punt: De agent heeft de actie niet uitgevoerd en geeft foutieve informatie.
- 3 punten: De agent heeft de actie uitgevoerd, maar mist verplichte details in het antwoord.
- 5 punten: De agent heeft de actie correct uitgevoerd en geeft een volledig, correct antwoord.

Retourneer uitsluitend een JSON-object: {"score": integer, "reasoning": "string"}

Browse this overview of test frameworks if you're looking for ready-made libraries for automated LLM evaluations, in an overview of evaluation and testing tools for LLMs.

Pitfalls when setting up agent evals

Setting up a test network for agents brings specific pitfalls that teams often only discover once their test suite becomes unreliable:

• Overfitting to specific phrasing: If your test set only contains questions phrased in exactly the same way, your agent will fail in production with the first user who uses a synonym or a typo. Include variations in your test dataset.
• Flaky Tests: Due to the stochastic nature of models, a test can pass one run and fail the next. Use metrics such as pass@k (how often the test passes over k attempts) instead of relying on a single successful run.
• Ignoring cost and latency: An agent that performs a task perfectly but needs 45 seconds and 8,000 tokens for it is unusable in practice. Include hard thresholds for duration and token consumption in your test criteria.
• Unmonitored changes to tool definitions: A small change to the docstring of a Python function that serves as a tool can completely change the model's behavior. Treat tool definitions as part of your prompt architecture and include them in version control.

A practical framework for regression testing in production

To prevent changes to prompts or model versions from breaking your agent, you integrate evals directly into your CI/CD pipeline. Every pull request that touches the code, the prompts, or the tool sets must automatically run through the test suite.

1. Fast pre-commit checks (Deterministic): Validate that all prompts and JSON schemas are syntactically correct and that all tool functions have a valid docstring. This takes a few seconds.
2. Small regression suite (Component & Trajectory): Run a subset of 20 critical scenarios with mocked APIs. Here you check whether the agent still calls the expected tools and doesn't cause infinite loops.
3. Nightly full evaluation (Final result & LLM-as-a-judge): Run hundreds of complex scenarios through the full evaluation chain with qualitative review by a heavyweight LLM model. This produces a detailed dashboard of performance changes over time.

Conclusion & Next Steps

Building reliable AI agents is not primarily a prompt-engineering challenge, but a software-engineering and evaluation challenge. Without systematic agent evals, it is impossible to confidently roll out updates to your system. By testing at the component level, analyzing trajectories, and checking final results with a combination of hard assertions and qualitative judges, you transform an unpredictable demo into a stable, production-ready application.