1. Introduction
Building an AI application does not end when the application produces a response successfully. Developers also need to determine whether the application produces useful, accurate, safe, reliable, and appropriate results.
AI application evaluation is the systematic process of measuring how well an AI-powered application performs against defined requirements.
This is especially important because an application can be technically functional while still producing poor results. An API request may succeed, the model may return a response, and the application may display it correctly while the information itself is incomplete, irrelevant, or unsafe.
2. What Is AI Application Evaluation?
Evaluation means collecting evidence about application performance and comparing that performance against defined expectations.
Depending on the application, evaluation may examine:
- Accuracy
- Relevance
- Completeness
- Consistency
- Grounding
- Safety
- Security
- Latency
- Reliability
- Cost
- User satisfaction
The appropriate criteria depend on the actual task performed by the application.
3. Model Evaluation vs Application Evaluation
Model evaluation focuses on the capabilities and behavior of an AI model.
Application evaluation examines the complete system surrounding the model.
An AI application may include:
- User interface
- Application logic
- Prompts
- AI model
- Retrieval system
- Database
- Tools
- Authentication
- Authorization
- Output processing
A strong model does not automatically make every application using that model effective. Application evaluation must therefore consider the complete workflow.
4. Define the Task First
Evaluation should begin by clearly defining what the AI application is supposed to accomplish.
For example, an AI customer support assistant might be expected to:
- Identify the customer question
- Retrieve relevant information
- Provide an accurate response
- Avoid inventing policy information
- Protect private customer information
- Escalate appropriate cases to human staff
Without a clearly defined task, it is difficult to determine whether an output is good or bad.
5. Evaluation Criteria
Evaluation criteria describe the characteristics that a successful output should have.
Criteria should be specific enough that different evaluators can apply them consistently.
For example, instead of simply asking whether a response is good, an evaluation can separately measure factual accuracy, relevance, completeness, and safety.
6. Accuracy
Accuracy measures whether the application produces correct information or correct results.
For an information extraction application, accuracy may involve comparing extracted fields with verified source information.
For a classification application, accuracy may involve comparing predicted categories with known labels.
For a question-answering system, accuracy may involve checking whether important factual claims are correct.
7. Relevance
Relevance measures whether the response addresses the actual task or question.
An answer can contain correct information while still being poor if it does not address what the user asked.
Evaluation should therefore distinguish correctness from relevance.
8. Completeness
Completeness measures whether the application provides the information required for the task.
An answer can be accurate but incomplete. For example, an extraction system may correctly identify several invoice fields while failing to return another required field.
Required fields and expected results should be defined before evaluation.
9. Consistency
Consistency measures whether the application behaves appropriately across similar inputs.
AI systems can sometimes produce different outputs for similar requests. Evaluation can help determine whether that variation is acceptable for the application.
Consistency is particularly important when predictable application behavior is required.
10. Grounding
Grounding measures whether AI responses are supported by the information that the application is expected to use.
This is especially important for RAG applications and internal knowledge assistants.
A grounded answer should be supported by relevant retrieved information rather than unsupported model-generated claims.
11. Creating an Evaluation Dataset
An evaluation dataset is a collection of representative inputs used to test an AI application.
A useful dataset can contain:
- Common user requests
- Typical business cases
- Edge cases
- Ambiguous requests
- Invalid inputs
- Security-related scenarios
- Cases requiring human escalation
The dataset should reflect the actual situations that the application is expected to handle.
12. Expected Results
Evaluation requires some definition of what acceptable behavior looks like.
Expected results do not always need to be an exact sentence. Depending on the task, they may define:
- Required facts
- Required fields
- Allowed categories
- Required citations
- Safety requirements
- Actions that must not occur
13. Evaluation Rubrics
A rubric provides structured criteria for judging an output.
For example, a customer support response can be evaluated separately for factual accuracy, relevance, completeness, tone, safety, and policy compliance.
Using a consistent rubric makes evaluation more repeatable.
14. Automated Evaluation
Some evaluation tasks can be automated.
Examples include:
- Checking whether required fields exist
- Comparing classifications with known labels
- Checking structured output format
- Measuring response latency
- Counting API errors
- Measuring retrieval metrics
Automated evaluation is useful for repeatable tests and large datasets.
15. Human Evaluation
Some qualities are difficult to measure reliably using simple automated checks.
Human evaluation can help assess:
- Usefulness
- Clarity
- Relevance
- Subtle factual problems
- Safety
- Overall response quality
Human evaluation should use clear criteria so different reviewers can evaluate outputs consistently.
16. Combining Automated and Human Evaluation
Strong evaluation programs often combine automated tests with human review.
Automated tests provide scale and repeatability. Human evaluation can examine qualities that are difficult to capture through simple rules.
The two approaches can complement each other.
17. Testing Structured Outputs
Structured outputs can be evaluated using schema and business-rule checks.
Tests can verify:
- Required fields are present
- Data types are correct
- Values are within allowed ranges
- Unexpected fields are handled appropriately
- Business rules are satisfied
Valid structure does not automatically mean that the information inside the structure is correct.
18. Evaluating RAG Retrieval
RAG applications require evaluation of both retrieval and final response generation.
A response can be poor because the model generated an incorrect answer, but it can also be poor because the retrieval system supplied irrelevant or incomplete information.
Evaluation should therefore examine whether the correct source information was retrieved.
19. Retrieval Evaluation
Retrieval evaluation measures whether relevant information is returned for a query.
Important questions include:
- Was the relevant document retrieved?
- Was the relevant section retrieved?
- Were important sources missing?
- Were irrelevant sources included?
- Did access controls filter the results correctly?
20. Evaluating Tool Calls
Applications that use tools should evaluate more than the final text response.
Evaluation can examine:
- Whether the correct tool was selected
- Whether arguments were correct
- Whether unauthorized actions were prevented
- Whether tool errors were handled correctly
- Whether the final response accurately reflected the tool result
21. Evaluating AI Agents
Agents require evaluation of multi-step behavior.
Evaluation can examine:
- Goal completion
- Tool selection
- Intermediate decisions
- Stopping behavior
- Error recovery
- Security boundaries
- Cost
An agent should not be judged only by whether it eventually produces a final answer.
22. Safety Evaluation
AI applications should be tested for unsafe behavior.
Safety evaluation can include scenarios involving:
- Harmful requests
- Sensitive information
- High-impact decisions
- Requests outside the intended application scope
- Attempts to bypass application controls
23. Security Evaluation
Security testing examines whether the application protects its data, tools, credentials, and capabilities.
Tests can include prompt injection scenarios, unauthorized access attempts, malicious inputs, unsafe tool requests, and attempts to expose protected information.
24. Evaluating Refusal Behavior
An AI application should not only answer appropriate requests. It should also handle inappropriate or unsupported requests correctly.
Evaluation should therefore include cases where the correct application behavior is to refuse, limit the response, request clarification, or escalate to a human.
25. Latency
Latency measures how long the application takes to respond.
AI applications may involve multiple steps such as retrieval, model requests, tool calls, and output processing.
High latency can reduce usability even when response quality is good.
26. Reliability
Reliability measures whether the application continues to function correctly when services fail or conditions change.
Evaluation can include:
- API failures
- Timeouts
- Unavailable models
- Tool failures
- Invalid responses
- Database failures
27. Cost Evaluation
AI applications should also be evaluated for resource usage and cost.
Useful measurements can include:
- Tokens per request
- AI requests per task
- Tool calls per task
- Cost per user
- Cost per completed task
28. User Satisfaction
User feedback can provide important evidence about application quality.
Useful signals may include ratings, task completion, repeated usage, abandonment, support requests, and user comments.
User satisfaction should complement technical evaluation rather than replace it.
29. Regression Testing
Regression testing checks whether previously working behavior continues to work after changes.
AI applications can change when developers modify:
- Prompts
- Models
- Retrieval systems
- Tool definitions
- Application logic
- Data sources
A change that improves one scenario can unintentionally make another scenario worse.
30. Evaluation Before Deployment
Important evaluation should happen before a new AI feature is released to users.
A pre-deployment evaluation can identify major quality, security, reliability, and cost problems before they affect production users.
31. Evaluation After Deployment
Evaluation should continue after deployment.
Real-world inputs can differ from development test cases. Production monitoring and periodic evaluation can reveal new failure patterns.
32. Monitoring Production Quality
Production monitoring can track:
- Error rates
- Latency
- Validation failures
- Tool failures
- Retrieval quality signals
- AI usage
- Cost
- User feedback
Monitoring provides evidence about how the application behaves under real workloads.
33. Evaluation and Model Changes
Changing the AI model can change application behavior.
A new model may improve some tasks while producing different results on others.
Existing evaluation datasets should therefore be used when important model changes are introduced.
34. Evaluation and Prompt Changes
Prompt changes can also affect application behavior.
Even a small instruction change can alter response format, content, tool selection, or refusal behavior.
Important prompt changes should therefore be evaluated before production deployment.
35. Example: Customer Support Assistant
Consider an AI customer support assistant connected to a product knowledge base.
An evaluation framework could test:
- Question classification
- Retrieval relevance
- Answer accuracy
- Source grounding
- Privacy protection
- Escalation behavior
- Response latency
- Cost per interaction
36. Example: Invoice Extraction
An invoice extraction application can be evaluated against verified invoices.
Tests can measure whether required fields are extracted correctly, whether values have the correct types, whether important fields are missing, and whether invalid information is detected.
37. Example: RAG Knowledge Assistant
A knowledge assistant can be evaluated in two stages.
First, evaluate whether relevant documents and sections are retrieved. Second, evaluate whether the generated response correctly uses the retrieved information.
38. Common Beginner Mistakes
- Testing only successful examples
- Evaluating only the final text
- Ignoring retrieval quality
- Ignoring security scenarios
- Ignoring cost
- Changing prompts without regression tests
- Assuming a strong model guarantees a strong application
- Stopping evaluation after deployment
39. A Practical Evaluation Framework
A practical evaluation process can follow these steps:
- Define the application task.
- Identify important quality and safety criteria.
- Create representative evaluation cases.
- Define expected behavior.
- Run automated checks where appropriate.
- Perform human review where necessary.
- Record results.
- Fix identified problems.
- Repeat evaluation after important changes.
- Continue monitoring after deployment.
40. Developer Mental Model
A useful mental model is:
Build → Test → Evaluate → Improve → Deploy → Monitor → Re-evaluate.
AI application development is an iterative process. Evaluation provides the evidence needed to understand whether changes actually improve the application.
41. Where This Module Goes Next
The next lesson focuses on deploying an AI feature. After understanding evaluation, the next step is learning how to move a tested AI feature into production while managing configuration, security, reliability, monitoring, and operational risks.
Conclusion
Evaluating an AI application means measuring the complete system against clearly defined requirements. Good evaluation considers accuracy, relevance, completeness, grounding, retrieval, tool behavior, safety, security, latency, reliability, cost, and user experience.
The most important principle is simple: an AI application should be tested and evaluated as a complete software system, not judged only by whether an AI model can generate convincing text.