# Evals for agents: why AI agents fail without testing

[Skip to content](#lm-inhoud)Network/[NL](/en/evals-voor-agents-waarom-tests-essentieel-zijn)EN[Hubhub.llmnet.nlCompare models on task, language, cost and license.](https://hub.llmnet.nl/en/)[Communitycommunity.llmnet.nlPrompt techniques, patterns and system prompts.](https://community.llmnet.nl/en/)[APIapi.llmnet.nlLLMs in production: rate limits, routing, structured output.](https://api.llmnet.nl/en/)[Consultancyconsultancy.llmnet.nlRolling out AI in an organization, pilot to production.](https://consultancy.llmnet.nl/en/)[Newsnieuws.llmnet.nlAI developments, explained for the Netherlands.](https://nieuws.llmnet.nl/en/)[Benchmarkbenchmark.llmnet.nlMeasure AI quality yourself, on your own tasks.](https://benchmark.llmnet.nl/en/)[Careersvacatures.llmnet.nlAI roles, salaries and career paths in the Netherlands.](https://vacatures.llmnet.nl/en/)[Learnleren.llmnet.nlAI concepts in plain language, beginner to builder.](https://leren.llmnet.nl/en/)[Guidegids.llmnet.nlRun AI privately on your own Mac, PC, NAS or home server.](https://gids.llmnet.nl/en/)[Directorydirectory.llmnet.nlMapping the AI ecosystem: tools, models, companies.](https://directory.llmnet.nl/en/)[Radarradar.llmnet.nlSignals from X, research and communities for indie developers.](https://radar.llmnet.nl/en/)[Appsapps.llmnet.nlReviews of AI apps and open-source repos, with tips for builders.](https://apps.llmnet.nl/en/)[llmnet.nl — main site](https://llmnet.nl/en/)[](https://x.com/intent/post?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fevals-voor-agents-waarom-tests-essentieel-zijn&text=Evals%20for%20agents%3A%20why%20AI%20agents%20fail%20without%20testing)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fevals-voor-agents-waarom-tests-essentieel-zijn)[](https://www.reddit.com/submit?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fevals-voor-agents-waarom-tests-essentieel-zijn&title=Evals%20for%20agents%3A%20why%20AI%20agents%20fail%20without%20testing)[](#)[](https://x.com/intent/post?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fevals-voor-agents-waarom-tests-essentieel-zijn&text=Evals%20for%20agents%3A%20why%20AI%20agents%20fail%20without%20testing)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fevals-voor-agents-waarom-tests-essentieel-zijn)[](https://www.reddit.com/submit?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fevals-voor-agents-waarom-tests-essentieel-zijn&title=Evals%20for%20agents%3A%20why%20AI%20agents%20fail%20without%20testing)[](#)

 
 
 [leren.llmnet.nl](https://leren.llmnet.nl/en/)
 
 

 
 
 
# Evals for agents: why your AI agent breaks without tests

 By Ivo Donker — compiled with AI assistance (Claude & Gemini)

 Building a working demonstration of an AI agent is nowadays a matter of a few hours. You connect a language model to a handful of external functions, give the system a well-thought-out system instruction, and watch the model make decisions on its own. The real challenge, however, begins once that agent is deployed to production. Where a simple prompt application at worst returns a poorly formulated answer, an autonomous agent has the ability to perform actions in the outside world. Without a solid testing infrastructure, this inevitably leads to stuck loops, corrupted database records, unnecessarily high API costs, and unpredictable errors on unexpected user input.

 This article falls within Module 6 — Building Responsibly of the AI curriculum at leren.llmnet.nl. In this module, we focus on the architectural and operational safeguards needed to bring complex language model applications to production in a safe and reliable manner.

 
 
### What you need to know beforehand

 To fully understand the concepts in this article, it helps to be familiar with the basic principles of autonomous AI systems and steering:

 
 
- First read the explanation of [the difference between an agent and a chatbot](https://leren.llmnet.nl/en/wat-is-een-agent-vs-chatbot) if you want to understand exactly how an agent's autonomous action loop differs from a linear conversation interface.
 
- Take a look at the overview of [how guardrails work in AI applications](https://leren.llmnet.nl/en/guardrails-uitgelegd) for insight into how to constrain input and output at runtime.
 
- Consult the guide on [getting models to reason with chain-of-thought](https://leren.llmnet.nl/en/chain-of-thought) to see how intermediate reasoning steps influence a model's decision-making.
 
 

 
## Why traditional unit tests don't work for AI agents

 In classic software development, a unit test is deterministic. If you call a function with argument A and B, you expect exactly outcome C. If the code doesn't change, the test passes today, tomorrow, and a year from now. AI agents break this foundation in two ways: the underlying LLM is stochastic and the environment in which the agent operates changes dynamically.

 An AI agent works according to a continuous loop of observing, reasoning, selecting tools, and executing. During each step in this loop, the model generates a response based on the current context. Because language models use probability distributions over tokens, the same prompt with an identical temperature setting can still produce subtly different reasoning paths. A classic assertion that checks for an exact string will therefore fail regularly, even when the agent has completed the task perfectly in substance.

 When an agent makes a mistake in step two of a process consisting of six steps, that error propagates cumulatively through the remaining steps. The agent can get stuck on a wrong assumption, leading to hallucinations further along in the task. Take a look at the guide on [understanding hallucinations in language models](https://leren.llmnet.nl/en/hallucinaties-begrijpen) if you want to understand how flawed assumptions arise in the reasoning step and send the agent down the wrong path. For this reason, standard software tests fall short, and we need specialized evaluation frameworks: so-called agent evals.

 
## The three levels of agent evaluation

 An effective test architecture for an agent checks the system at three different levels. Each level highlights a different part of the decision-making chain and helps locate specific defects.

 
### 1. Step-by-step / Component evaluation

 At the lowest level, you isolate the model's individual decisions. You don't test the entire process, but validate one specific step. Can the model select the correct tool based on a specific user question? Does the model generate valid JSON that exactly matches the expected schema?

 Component evaluations are fast, cheap, and largely deterministic. If the agent fails here, the problem is almost always unclear system instructions or a poorly defined tool interface. Consult this guide if you want to learn how clarified function descriptions drastically reduce the number of incorrect tool calls, via [writing effective tool descriptions for your agent](https://community.llmnet.nl/en/tool-descriptions-schrijven-de-prompt-die-je-agent-niet-leest).

 
### 2. Trajectory evaluation (Reasoning Trajectory)

 At the middle level, you assess the route the agent takes to reach a final result. An agent might find the right solution but make twelve unnecessary API calls and search in the wrong direction three times along the way. That is undesirable in a production environment due to the accumulated latency and API costs.

 With trajectory evaluation, you analyze the sequence of thoughts, actions, and observations. You measure things like the number of steps, re-executing already-successful actions, and recovering from error messages. Read this in-depth article if you're looking for quantitative protocols to compare agent trajectories across thousands of runs, via [methodologies for benchmark-level agent evaluation](https://benchmark.llmnet.nl/en/agent-evaluatie).

 
### 3. Final result & effect evaluation

 At the highest level, you assess the ultimate effect of the agent run. Was the database update actually applied correctly? Does the generated email draft contain all the required customer information? Was the resolved ticket closed with the correct status code?

 Final result evaluation does not focus on *how* the agent completed the task, but exclusively on the quality and correctness of the final state change. It often combines physical inspection of the environment (for example a database query) with qualitative review via an evaluation model.

 
## Trajectory evaluation: analyzing the agent loop

 Analyzing the reasoning trajectory is the most unique challenge in testing agents. A trajectory can be represented as a sequence of states. When a user asks: "Cancel my most recent order and send a confirmation by email", the ideal trajectory looks like this:

 [START] -> User Query
 │
 ├──> Thought: Zoek de klant-ID en meest recente bestelling op.
 ├──> Action: get_customer_orders(customer_id="123", limit=1)
 ├──> Observation: Bestelling #98765, status: "in behandeling", bedrag: € 45,00.
 │
 ├──> Thought: De bestelling kan worden geannuleerd. Roep de annulering-API aan.
 ├──> Action: cancel_order(order_id="98765")
 ├──> Observation: Bestelling #98765 succesvol geannuleerd.
 │
 ├──> Thought: Stuur nu een bevestigingsmail naar de klant.
 ├──> Action: send_email(customer_id="123", template="order_cancelled", order_id="98765")
 ├──> Observation: E-mail verzonden met id #abc12.
 │
 └──> Final Answer: Uw bestelling #98765 is geannuleerd en er is een bevestiging gemaild.
[EIND]

 A broken agent can derail in this loop in several ways:

 
 
- Infinite Loop: The agent keeps calling `get_customer_orders` because the response isn't properly processed into the context.
 
- Tool hallucination: The agent tries to call a non-existent function such as `delete_order_permanently`.
 
- Premature give-up: The agent stops after the first step and tells the user the order has been canceled, while only the status was retrieved.
 
- Context pollution: The agent adds unnecessarily large amounts of raw JSON to its reasoning context, causing the maximum token limit to be exceeded quickly.
 

 Visit the analysis of context engineering for strategies to keep the prompt history of your test cases compact and representative, via [designing an effective context architecture](https://leren.llmnet.nl/en/context-engineering-uitgelegd).

 
## Enabling test datasets and mocks for agent evals

 Having live API calls execute against external systems during automated test runs is dangerous and slow. A test set of 100 scenarios that actually sends live emails, processes payments, or modifies CRM records pollutes your production data and costs a fortune in API credits.

 That's why professional testing infrastructures work with two layers of isolation:

 
### 1. Mocking the external environment

 All tools the agent uses must be replaced in a test environment with so-called mock functions. When the agent calls the function `get_customer_orders`, the mock returns a fixed JSON response without consulting a real database. This guarantees the test stays consistent, independent of changes in live data.

 
### 2. Recording and replaying trajectories (Recorded Trajectories)

 When building a regression test set, you record a successful run. As soon as a change is made to the system instruction or the agent's code, you run exactly the same prompt again. The test framework compares the new trajectory with the recorded trajectory. Deviations in tool selection or reasoning steps are immediately flagged for inspection.

 
## LLM-as-a-Judge vs. Deterministic Assertions

 To determine whether a test passes or fails, you use a combination of deterministic checks and evaluation by a second-opinion model (LLM-as-a-judge).

 
 
 
 Evaluation type | 
 When to use | 
 Advantages | 
 Disadvantages / Risks | 
 

 
 
 
 Deterministic Assertions | 
 Checking JSON schemas, status codes, SQL syntax, and presence of required fields. | 
 Very fast, free, 100% reproducible, and no margin of error in the check. | 
 Cannot judge the substantive intent, style, or nuance of human language. | 
 

 
 LLM-as-a-Judge | 
 Evaluating customer-friendliness, relevance, correctness of summaries, and logic in reasoning. | 
 Understands complex context and language nuances, and can judge according to a qualitative rubric. | 
 High cost, extra latency, risk of judgment errors (bias), and not 100% reproducible. | 
 

 
 

 When you deploy an LLM as a judge, it is essential to draw up a strict rubric. Never simply ask a judge model: "Is this a good answer?". Instead, give the model a clear scale with concrete criteria:

 Beoordeel het onderstaande antwoord van de agent op een schaal van 1 tot 5 op basis van de volgende criteria:
- 1 punt: De agent heeft de actie niet uitgevoerd en geeft foutieve informatie.
- 3 punten: De agent heeft de actie uitgevoerd, maar mist verplichte details in het antwoord.
- 5 punten: De agent heeft de actie correct uitgevoerd en geeft een volledig, correct antwoord.

Retourneer uitsluitend een JSON-object: {"score": integer, "reasoning": "string"}

 Browse this overview of test frameworks if you're looking for ready-made libraries for automated LLM evaluations, in [an overview of evaluation and testing tools for LLMs](https://directory.llmnet.nl/en/evaluatie-en-testgereedschap-voor-llm-toepassingen-evals).

 
## Pitfalls when setting up agent evals

 Setting up a test network for agents brings specific pitfalls that teams often only discover once their test suite becomes unreliable:

 
 
- Overfitting to specific phrasing: If your test set only contains questions phrased in exactly the same way, your agent will fail in production with the first user who uses a synonym or a typo. Include variations in your test dataset.
 
- Flaky Tests: Due to the stochastic nature of models, a test can pass one run and fail the next. Use metrics such as pass@k (how often the test passes over k attempts) instead of relying on a single successful run.
 
- Ignoring cost and latency: An agent that performs a task perfectly but needs 45 seconds and 8,000 tokens for it is unusable in practice. Include hard thresholds for duration and token consumption in your test criteria.
 
- Unmonitored changes to tool definitions: A small change to the docstring of a Python function that serves as a tool can completely change the model's behavior. Treat tool definitions as part of your prompt architecture and include them in version control.
 

 
## A practical framework for regression testing in production

 To prevent changes to prompts or model versions from breaking your agent, you integrate evals directly into your CI/CD pipeline. Every pull request that touches the code, the prompts, or the tool sets must automatically run through the test suite.

 
 
- Fast pre-commit checks (Deterministic): Validate that all prompts and JSON schemas are syntactically correct and that all tool functions have a valid docstring. This takes a few seconds.
 
- Small regression suite (Component & Trajectory): Run a subset of 20 critical scenarios with mocked APIs. Here you check whether the agent still calls the expected tools and doesn't cause infinite loops.
 
- Nightly full evaluation (Final result & LLM-as-a-judge): Run hundreds of complex scenarios through the full evaluation chain with qualitative review by a heavyweight LLM model. This produces a detailed dashboard of performance changes over time.
 

 
## Conclusion & Next Steps

 Building reliable AI agents is not primarily a prompt-engineering challenge, but a software-engineering and evaluation challenge. Without systematic agent evals, it is impossible to confidently roll out updates to your system. By testing at the component level, analyzing trajectories, and checking final results with a combination of hard assertions and qualitative judges, you transform an unpredictable demo into a stable, production-ready application.

 
 
### Continue with

 Deepen your knowledge of building and maintaining AI models with these follow-up steps from the curriculum:

 
 
- Consult the [complete AI and LLM glossary](https://leren.llmnet.nl/en/ai-begrippenlijst) to quickly look up all technical definitions around agentic workflows and evaluation metrics.
 
- Read the article on [implementing chain-of-thought reasoning](https://leren.llmnet.nl/en/chain-of-thought) to discover how to better structure and observe your agent's internal step-by-step logic.
 
 

 
 

 
 © 2026 llmnet.nl — Knowledge network about AI and LLMs