Evaluating RAG with Python and RAGAS: A Practical Introduction
A RAG application can return an answer for almost any question. That does not mean the answer is correct.
To understand how well a RAG pipeline performs, we need a repeatable evaluation process.
In this article, we'll explore how to evaluate RAG applications using Python and RAGAS, from preparing a dataset to interpreting evaluation results.
1. Install the Required Packages
Start by installing the evaluation library:
pip install ragas datasets
The exact integration and metric APIs may vary by RAGAS version, so pin your dependency versions in a real project.
2. Prepare an Evaluation Dataset
An evaluation dataset should represent the questions your application is expected to answer.
For example:
evaluation_data = [
{
"question": "What is the refund policy?",
"answer": "Customers can request a refund within 30 days.",
"contexts": [
"Customers can request a refund "
"within 30 days of purchase."
],
"ground_truth": (
"Customers can request a refund "
"within 30 days of purchase."
),
}
]
This example illustrates the required information conceptually. For a real evaluation, your answer and contexts should come from the RAG application being tested, while ground_truth should be independently verified.
Never use the model's own generated answer as ground truth.
3. Understand the Evaluation Metrics
Before running an evaluation, understand what each metric measures.
Faithfulness
Does the generated answer stay consistent with the retrieved context?
If the context says the refund period is 30 days, an answer claiming 90 days should be flagged as unsupported.
Answer Relevancy
Does the answer address the question?
An answer can contain correct information but still fail to answer what the user asked.
Context Precision
Are relevant chunks ranked highly among the retrieved results?
Irrelevant chunks can introduce noise and make generation less reliable.
Context Recall
Does the retrieved context contain the information needed to answer the question?
A system can fail even with a good LLM if its retriever misses important information.
Answer Correctness
Does the generated answer agree with a verified reference answer?
This is a different question from faithfulness: a model can faithfully repeat incorrect source material.
Not every metric requires the same dataset fields. Some metrics need reference answers, while others can operate with questions, answers, and retrieved contexts.
4. Connect Evaluation to Your RAG Pipeline
A typical evaluation workflow looks like this:
def evaluate_rag_case(question, rag_pipeline):
result = rag_pipeline.ask(question)
return {
"question": question,
"answer": result["answer"],
"contexts": result["contexts"],
"sources": result.get("sources", []),
}
This is a simplified example. It assumes your RAG pipeline exposes an ask() method that returns an answer and the retrieved contexts.
For each question in your evaluation dataset:
- Run the RAG pipeline.
- Capture the generated answer.
- Record the actual retrieved contexts.
- Compare the output with verified references.
- Store the evaluation results.
Capturing the actual contexts is important. Evaluating only the final answer hides information about retrieval failures.
5. Run RAGAS Evaluation
RAGAS provides metrics and evaluation utilities for RAG systems. The exact code depends on the installed version, metric selection, and configured model providers.
Conceptually, the workflow is:
# Conceptual evaluation workflow
dataset = prepare_evaluation_dataset(
questions=questions,
answers=generated_answers,
contexts=retrieved_contexts,
references=ground_truth_answers,
)
results = run_ragas_evaluation(
dataset=dataset,
metrics=[
"faithfulness",
"answer_relevancy",
"context_precision",
"context_recall",
],
)
The functions above are illustrative pseudocode, not built-in RAGAS APIs. Use the documentation for your pinned version to configure the dataset, metrics, and evaluator models.
6. How Should You Interpret the Results?
Avoid looking at one aggregate score and immediately declaring the system successful.
Instead, analyze failure patterns.
| Observed issue | Possible investigation |
|---|---|
| Low context recall | Chunking, embeddings, retrieval settings, or missing documents |
| Low context precision | Search ranking, metadata filters, or reranking |
| Low faithfulness | Prompt design, conflicting context, or generation behavior |
| Low answer correctness | Source quality, retrieval, generation, or reference-data problems |
| High latency | Retrieval cost, model latency, reranking, or unnecessary processing |
These are diagnostic clues, not guaranteed explanations. Inspect individual failed examples before changing the system.
7. Turn Evaluation Into an Engineering Workflow
A useful next step is to run evaluations whenever you change:
- Embedding models
- Chunking strategies
- Retrieval parameters
- Rerankers
- Prompts
- LLM versions
Store the configuration and results for each experiment.
Over time, your evaluation dataset becomes a regression suite that helps you detect quality changes before deployment.
Conclusion
RAGAS can help structure RAG evaluation, but the quality of your evaluation depends on the quality of your test data and the way you interpret the metrics.
Start with a small, verified dataset. Measure retrieval and generation separately. Review failed examples. Then expand your evaluation as your application grows.
A RAG system is easier to improve when you can identify exactly where it fails.
Useful next step: build a small evaluation pipeline that records each question, retrieved chunk, generated answer, reference answer, and metric result.
Connect with me
- YouTube: https://www.youtube.com/@Tahahussein-Ai
- LinkedIn: https://www.linkedin.com/in/taha-hussein-b0a583425/
- GitHub: https://github.com/Taha2hussein
#Python #RAG #RAGAS #LLM #MachineLearning
