Evaluation and Testing
Measure, validate, and continuously improve AI system quality with structured evaluation pipelines.
Last updated: July 2026
You collect the AI system’s outputs on a set of inputs you care about.
Evaluation and testing for Generative AI (GenAI) and AI agents is the structured process of measuring how well an AI system performs. It covers accuracy, relevance, safety, reasoning, and behavior, before, during, and after deployment. In traditional software testing, outputs are deterministic. GenAI outputs are probabilistic, open-ended, and often subjective, which makes them harder to judge. For AI agents, evaluation goes beyond checking a final answer. It includes verifying the path the agent took: which tools it called, whether it reasoned correctly at each step, and whether its final response logically follows from its actions.
“A GenAI system that works in demos but fails in production is worse than no system at all. It erodes user trust and creates hidden liabilities. Evaluation is the only structured way to tell the difference between a system that appears to work and one that reliably works.”
| Component | What it does | Example |
|---|---|---|
Evaluation Dataset (Test Set) | A curated collection of inputs (prompts, queries, tasks) paired with expected outputs or evaluation criteria. Think of it as the exam your AI takes. It can be reference-based (with a known correct answer) or reference-free (judging quality alone). | For a legal document summarizer, the test set might include 50 contracts with expert-written reference summaries. Each prompt is a contract, and each reference is the gold-standard answer. |
Evaluator / Judge | The mechanism that scores outputs. It can be a human reviewer, an automated metric, or another LLM acting as a judge. Types include deterministic (regex, unit tests), statistical (BLEU, ROUGE), LLM-as-Judge, and human review. | After a medical FAQ bot answers a question, an LLM judge scores the response on accuracy, completeness, and safety using a predefined rubric. This removes the need for a doctor to review every response by hand. |
Metrics | Specific numeric signals used to measure quality dimensions of the output. For RAG systems: faithfulness (does the answer stay true to retrieved docs?), answer relevancy (does it address what the user asked?), contextual recall (were all needed facts retrieved?). | A RAG-based knowledge assistant is evaluated on all three metrics together, which reveals whether failures come from the model, the retriever, or the prompt. |
Trajectory Evaluation (Agent-Specific) | For AI agents, this evaluates the sequence of steps (tool calls, reasoning steps, decisions) taken to reach an answer, not just the final output. It measures tool selection accuracy, tool call order, parameter correctness, and efficiency. | A travel booking agent asked to find and book a flight should follow: search_flights → check_seat_availability → confirm_price → book_ticket. If it skips check_seat_availability, trajectory evaluation catches this gap even if the booking succeeds. |
Evaluation Pipeline | The automated system that runs evaluations continuously: on new builds, production samples, or scheduled batches, integrated into CI/CD with pass/fail thresholds. | A team connects their evaluation suite to GitHub Actions. Every pull request triggers a run of 200 test cases. If hallucination rate exceeds 5% or faithfulness drops below 0.85, the pipeline blocks the merge automatically. |
Full assembled example
A customer support chatbot evaluation pipeline: (1) Curate a test set of 300 real user queries with expert-written reference answers. (2) Run the bot on all queries. (3) Score each response with an LLM judge on faithfulness, relevancy, and empathy. (4) Flag any response with faithfulness < 0.8. (5) Block deployment if overall hallucination rate > 5%. (6) Store results in a dashboard for trend tracking. (7) Re-run on every prompt update.
Core Principles
- 1Evaluate multiple dimensions simultaneously - No single metric tells the whole story. A response can be relevant but unfaithful, or fluent but unsafe. A news summarizer scoring 0.9 ROUGE (a metric for evaluating text summaries) but 0.4 faithfulness produces readable summaries that fabricate facts. Evaluating only ROUGE would have missed a critical flaw.
- 2Use a layered evaluation approach - Combine automated metrics (first layer) with human reviewers (second layer). An automated metric might flag no issues with a customer service response, but a human reviewer catches that the tone is condescending, something automated metrics cannot detect reliably.
- 3Test for your specific use case, not just benchmarks - Standard benchmarks (MMLU, HellaSwag) measure general model capability, not production fitness for your task. A coding assistant may score well on MMLU but fail consistently on your internal codebase patterns. Only a custom eval set surfaces this gap.
- 4Evaluate continuously, not just at launch - Prompt changes, model updates, and new data can degrade performance silently. This is called regression. Say a team updates their system prompt to reduce verbosity. The AI gets more concise, but its faithfulness drops because it now omits key context. Only a regression test suite catches this before deploy.
- 5Evaluate the judge too - If you use an LLM-as-a-Judge, check that the judge itself is calibrated and consistent. A team using GPT-5.5 as their eval judge never validates its scores against human ratings. The judge consistently over-scores long responses (length bias), inflating quality metrics.
- 6For agents, evaluate the path, not only the destination - An agent that reaches the right answer through a flawed process is a liability in production, because it may fail unpredictably on novel inputs. A financial research agent gives the correct final recommendation but skips the 'verify source' tool call. Trajectory evaluation catches this, while response-only evaluation does not.
Reference-Based Evaluation
Compare the model's output against a known correct answer (gold standard) using overlap or similarity metrics. Uses BLEU (a metric for evaluating machine translation), ROUGE, METEOR, Exact Match, BERTScore.
Prompt
Task: Summarize this 500-word article about renewable energy. Reference: [expert-written summary] Model output: [generated summary] Metric: ROUGE-L score comparing overlap of key sentences
Output
A numeric score (e.g., ROUGE-L: 0.72) indicating how much the generated summary overlaps with the expert reference
LLM-as-a-Judge (Model-Graded Evaluation)
Use a capable LLM to evaluate another model's output against a rubric, assigning scores and providing reasoning. Subtypes: pointwise scoring, pairwise comparison, and reference-guided evaluation.
Prompt
You are evaluating an AI assistant's response.
User Question: {user_question}
AI Response: {ai_response}
Score the response on:
- Correctness (1–5): Is the information accurate?
- Relevance (1–5): Does it address the user's question?
- Safety (1–5): Is it free of harmful content?
Provide a score and one sentence of reasoning per criterion.Output
Structured scores with rationale: Correctness: 4/5 - Mostly accurate, minor omission. Relevance: 5/5 - Directly addresses the question. Safety: 5/5 - No harmful content.
Trajectory / Process Evaluation (Agent-Specific)
Evaluate the sequence of tool calls, reasoning steps, and decisions an agent makes, not only the final answer. It measures tool selection accuracy, sequence accuracy, parameter correctness, and task completion.
Prompt
Expected trajectory for a research agent:
web_search("climate policy 2025") → filter_results(relevance > 0.8) → summarize_top_3 → generate_report
Actual trajectory: agent calls generate_report directly without searching.
Trajectory metric: Tool selection accuracy = 0.5, sequence score = 0.25Output
Trajectory score: 0.38 → FAIL. Root cause: Agent skipped web_search and summarization steps.
Human Evaluation
Human raters assess model outputs on defined criteria, either as the primary method or to calibrate automated metrics. Methods: Likert scale (1–5 ratings), side-by-side A/B comparison, and expert annotation.
Prompt
A medical Q&A system's responses are reviewed by licensed nurses using a structured rubric. Each response is rated on medical accuracy, completeness, and safety. The nurse ratings then serve as the ground truth to validate the team's LLM-as-a-Judge scores.
Output
Human annotation labels that serve as ground truth for calibrating automated judges and catching subtle quality issues
Benchmark Evaluation
Test a model against standardized public datasets designed to measure specific capabilities. Key benchmarks: MMLU (general knowledge), TruthfulQA (hallucination), AgentBench (agent performance), GLUE/SuperGLUE (language understanding).
Prompt
A team evaluates a new model version on MMLU and TruthfulQA before deciding whether to upgrade their production system. The new model scores 5% higher on MMLU but 3% lower on TruthfulQA, so they decide the hallucination risk outweighs the knowledge gain.
Output
Standardized benchmark scores that enable direct comparison between model versions or vendor models
A/B Testing & Regression Testing
A/B testing compares two system versions side by side on real user queries. Regression testing re-runs a fixed test suite after every update to make sure nothing breaks.
Prompt
Team A updates their RAG pipeline's retrieval chunk size. They run A/B tests: old version vs. new on 500 real user queries. Faithfulness improves but response latency doubles. Regression tests confirm previously passing test cases still pass.
Output
A/B test result showing tradeoffs (faithfulness +8%, latency +120%), plus regression pass/fail status for all existing test cases
Common Mistakes
| Mistake | Real-World Scenario | Fix |
|---|---|---|
| Evaluating only the final output, not the trajectory | An agent gives the correct booking confirmation but skipped the 'check flight availability' step, so it will silently fail on sold-out flights | Add trajectory evaluation; verify each tool call's correctness and sequence |
| Using a single metric as the sole quality signal | A summarization bot scores 0.9 ROUGE but fabricates statistics; the team celebrates the ROUGE score and ships it | Use a metric portfolio: include at minimum relevancy, faithfulness, and safety |
| Trusting an LLM judge without validating it | The eval judge consistently favors longer responses (length bias), inflating quality scores; a prompt regression goes undetected | Benchmark the judge against human labels; test for known biases (length, position, self-preference) |
| Testing only happy-path inputs | Chatbot tested on well-formed questions only; in production it fails on typos, multilingual inputs, and ambiguous requests | Include adversarial, edge-case, and out-of-distribution test cases in your test set |
| Static evaluation set that never updates | Test set built in Q1 remains unchanged by Q4; new product features and user intents are never covered | Regularly refresh the test set with new production samples and emerging query patterns |
| Evaluating at launch only, not continuously | Team runs eval before v1 release; a silent regression after a prompt update 2 months later degrades accuracy by 30% and no one notices until users complain | Implement regression testing in CI/CD; run eval on every model/prompt update |
| Ignoring cost and latency in evaluation | Team optimizes faithfulness to 0.95 but the new pipeline costs 4x more per query and has 8-second latency, making it unusable in production | Include latency (p50/p95) and cost-per-query as first-class evaluation metrics alongside quality metrics |
Use Cases by Task / Industry
| Use Case | Real-World Example | Best Approach |
|---|---|---|
| RAG / Knowledge Assistant | Internal HR policy chatbot retrieves from policy documents; must answer accurately without hallucinating benefits rules | Faithfulness, contextual recall, answer relevancy metrics + LLM judge |
| Customer Support Chatbot | Retail bot handles returns and billing disputes; must be accurate, empathetic, and never fabricate policies | LLM judge (empathy, accuracy, resolution) + faithfulness against policy KB |
| Code Generation | Coding assistant suggests Python functions; must be syntactically correct and functionally accurate | Unit test pass/fail, static analysis, exact match on small functions |
| Multi-step Autonomous Agent | Research agent browses web, summarizes sources, drafts report; must use correct tools in correct order | Trajectory evaluation: tool selection accuracy, step sequence, parameter correctness |
| Document Summarization | Legal contract summarizer; must preserve key clauses without omission or distortion | ROUGE-L, BERTScore, faithfulness, expert human review |
| Medical / High-Stakes Q&A | Patient-facing symptom checker; must be accurate and safe, never recommend harmful actions | Expert human annotation + LLM judge calibrated against clinician ratings + safety-specific metrics |
Evaluating a RAG Chatbot Response (Faithfulness + Relevancy)
Prompt
User Question: "What is our company's parental leave policy for primary caregivers?" Retrieved Context (from HR policy doc): "Primary caregivers are entitled to 16 weeks of fully paid parental leave, starting from the date of birth or adoption. Leave must be taken within 12 months of the child's arrival." AI Response: "Primary caregivers receive 16 weeks of fully paid parental leave, which must begin within 12 months of the child's birth or adoption."
Output
Faithfulness check: - Claim 1: "16 weeks of fully paid leave" → Present in context ✅ - Claim 2: "must begin within 12 months" → Supported ✅ - Any unsupported claims? → None ✅ Faithfulness score: 1.0 (perfect) Answer Relevancy: 1.0 - directly addresses the primary caregiver question Result: PASS ✅ - The response is fully grounded in the retrieved document.
Evaluating an AI Agent's Trajectory
Prompt
Task: "Find the current price of AAPL stock and summarize the latest news about Apple." Expected trajectory: Step 1: search_stock_price(ticker="AAPL") Step 2: search_news(query="Apple Inc latest news", n=3) Step 3: summarize_results(sources=[step1_result, step2_result]) Step 4: generate_response() Actual trajectory: Step 1: search_stock_price(ticker="AAPL") ✅ Step 2: generate_response() ❌ (skipped news search and summarize)
Output
Tool selection accuracy: 2/4 steps correct = 0.50 Sequence accuracy: Steps out of order after step 1 = 0.25 Task completion: Partial - stock price included, news missing ❌ Trajectory score: 0.38 → FAIL ❌ Root cause: Agent skipped web_search for news (Step 2) and summarization (Step 3). Recommendation: Review agent planning prompt; add explicit instruction to complete all sub-tasks before generating final response.
LLM-as-a-Judge: Customer Support Response Evaluation
Prompt
User Message: "I ordered a laptop 2 weeks ago and it still hasn't arrived. This is unacceptable." AI Response: "I'm sorry to hear your order hasn't arrived yet. I can look into this for you right away. Could you please share your order number so I can check the status and escalate this to our shipping team if needed?" Judge Prompt: Evaluate on Empathy (1–5), Accuracy (1–5), Actionability (1–5). Provide score and one-sentence rationale per criterion.
Output
Empathy: 5/5 - Opens with a direct apology and validates the customer's frustration. Accuracy: 5/5 - Makes no false promises about delivery; asks for information before committing to a resolution. Actionability: 5/5 - Provides a clear, immediate next step (sharing order number) to unblock resolution. Overall: 15/15 - PASS ✅
Tools & Platforms
| Tool | Best For | Link |
|---|---|---|
| DeepEval (Confident AI) | LLM-as-a-Judge metrics, CI/CD integration, unit testing for LLM apps | deepeval.com |
| Braintrust | CI/CD-integrated eval workflows, dataset management, prompt comparison | braintrust.dev |
| Arize Phoenix | Open-source production observability, tracing, evaluation for agents and RAG | phoenix.arize.com |
| Ragas | RAG pipeline evaluation: faithfulness, relevancy, recall | ragas.io |
| LangSmith (LangChain) | Evaluating and tracing LangChain-based agents, prompt debugging | smith.langchain.com |
| Galileo | Hallucination detection at scale without ground truth, enterprise GenAI eval | rungalileo.io |
| Promptfoo | Red-teaming, security testing, adversarial prompt evaluation | promptfoo.dev |
| MLflow 3.0 | LLM experiment tracking, hallucination detection, production monitoring | mlflow.org |
Frequently asked questions
What is AI evaluation and testing?
AI evaluation and testing is the process of measuring an AI system's performance by assessing its accuracy, relevance, safety, reasoning, and behavior. Unlike traditional software, AI outputs are probabilistic and subjective, making evaluation crucial to ensure reliable performance before, during, and after deployment.
How do you evaluate AI agents?
Evaluating AI agents involves checking both the final output and the decision-making path. This includes verifying tool selection, reasoning steps, and the order of actions. Ensuring that each step logically leads to the final answer helps identify potential issues in the agent's process.
What are the key components of an AI evaluation pipeline?
An AI evaluation pipeline includes a test set, evaluation metrics, evaluators (which can be human or automated), and a system to continuously run evaluations. This setup helps identify performance issues by providing structured and ongoing assessments of AI outputs.
Why is continuous evaluation important for AI systems?
Continuous evaluation is vital to detect regressions that can occur with prompt changes, model updates, or new data. This ongoing process ensures that AI systems maintain their performance standards and do not degrade over time, which can happen silently without regular testing.
What are common mistakes in AI evaluation?
Common mistakes include evaluating only the final output without considering the process, relying on a single metric for quality assessment, and trusting an LLM judge without validation. These oversights can lead to unnoticed issues in AI performance and decision-making processes.