Evaluating RAG end to end without fooling yourself

A RAG system can fail in its index, retriever, reranker, packing, generator or citations, and one “answer accuracy” number blends all of them. This note is a method: evaluate each layer, connect the layers to the end-to-end result, and report numbers that would survive a statistician.

Evaluate the layers, then the system

nDCG rewards ranking highly relevant passages near the top, with a logarithmic discount by position [1]. For RAG, recall at the depth you actually pass to the generator usually predicts answer quality better, because the generator reads everything you pack. Citation precision and recall come from ALCE [2], which builds on the AIS framework for judging whether a statement is attributable to a source [3].

Your relevance labels are incomplete, and that is a bias

Ground truth is usually built by judging a pool of top results from a few systems. Documents that no pooled system retrieved are never judged and get treated as irrelevant, a bias studied since the TREC era [4]. A new retriever that finds good passages your old one missed is penalised for being better. Before trusting a comparison, judge a sample of the new system’s unjudged results.

Synthetic test sets have their own bias. A question generated from a single chunk tends to reuse that chunk’s wording, which makes lexical retrieval look better than it will on real users, who ask in their own words. Generate from several chunks, paraphrase aggressively, and keep a slice of real queries from logs as the set that decides releases.

Judge retrieval by what it does to the answer

Relevance labels measure whether a passage is about the question. The generator cares whether the passage helps it answer. eRAG turns that into the metric: feed each retrieved document to the generator on its own, score the output against the reference, and use that score as the document’s label. Salemi and Zamani found this correlates with end-to-end RAG performance better than evaluations built on query-document relevance labels, while using up to 50 times less GPU memory than full end-to-end runs [5].

It also exposes the near-miss problem: a passage can be relevant by any human standard and still lower answer quality [6]. A label that only says “relevant” can’t see that.

Faithfulness, one claim at a time

“Is this answer grounded?” is too coarse to judge reliably. FActScore splits a long answer into atomic facts and checks each against a knowledge source [7]. RAGChecker applies the idea to RAG: claims from the answer and the reference are checked for entailment against each other and against the retrieved context, which yields diagnostics such as context utilisation, noise sensitivity and hallucination rate rather than one score [8]. RAGAS popularised reference-free versions of these metrics [9].

Those are the base rates to beat [10, 11]. CRAG’s headline is worth repeating: adding retrieval in a straightforward way lifted accuracy only from about a third to 44%.

Validate the judge before you trust it

LLM judges are the only practical way to score claim-level metrics at scale, and their biases are measurable. Zheng et al. documented position, verbosity and self-enhancement bias [12]; Wang et al. showed that swapping the order of two answers can flip a judge’s verdict [13]; Panickssery et al. found judges recognise and favour their own generations [14]. Shankar et al. add a subtler problem, criteria drift: people refine what “good” means while grading, so a rubric fixed in advance quietly stops matching what humans want [15].

So: label 100 to 200 cases by hand, measure agreement with Cohen’s kappa [16], swap positions in every pairwise test, prefer binary claim-level checks over 1 to 10 scales, and re-label a sample whenever the rubric changes.

Valid intervals from a biased judge

This is the most useful technique in the note. You have a small human-labelled set, say 150 questions, and a large judge-labelled set, say 5,000. The judge is cheap and biased; the humans are unbiased and few. Prediction-powered inference combines them: take the judge’s mean on the large set and correct it by the judge’s average error on the human set [17]. ARES applies it to RAG evaluation [18].

The estimate stays unbiased even when the judge is systematically optimistic. When the judge agrees with humans most of the time, the interval is far tighter than the 150 human labels could give you alone.

Do the statistics, including the part people skip

Miller’s “Adding Error Bars to Evals” lays out the method: report standard errors, use paired differences when two systems answer the same questions, and cluster standard errors when questions are not independent [19]. The last point bites RAG specifically. Eval sets are usually built several questions per document, and questions about the same passage succeed or fail together.

That is Kish’s design effect from survey sampling [20]: 200 clustered questions carry about as much information as 54 independent ones. For comparing two systems, count discordant pairs. If 9 questions flipped from fail to pass and 6 from pass to fail, McNemar’s test gives (9 − 6)² / (9 + 6) = 0.6, p ≈ 0.44 [21]: a +3 indistinguishable from chance. The 15 flipped questions are where to read.

Validate online, not only offline

• Runtime groundedness. Run a cheap claim-support check before an answer ships, and abstain or regenerate when it fails.
• Citation verification. Reject any answer that cites a passage ID that was not in the retrieved set.
• Canaries. Keep a fixed set of questions with known answers and run it on every index rebuild. Re-embedding, chunking changes and deletes break retrieval silently.
• Abstention as a metric. Track correct refusals on unanswerable questions next to accuracy. A system that never says “I don’t know” looks accurate until it isn’t.
• Adversarial slices. Conflicting documents, stale versions, near-miss distractors and planted injections [22], each as its own slice with its own threshold.

A RAG eval is a measurement instrument. Calibrate it against people, put error bars on it, and know which layer every number belongs to.

References

1. 01Järvelin & Kekäläinen, 2002. Cumulated Gain-Based Evaluation of IR Techniques. ACM TOIS.
2. 02Gao et al., 2023. Enabling Large Language Models to Generate Text with Citations. EMNLP 2023.
3. 03Rashkin et al., 2023. Measuring Attribution in Natural Language Generation Models. Computational Linguistics.
4. 04Buckley & Voorhees, 2004. Retrieval Evaluation with Incomplete Information. SIGIR 2004.
5. 05Salemi & Zamani, 2024. Evaluating Retrieval Quality in Retrieval-Augmented Generation. SIGIR 2024.
6. 06Cuconasu et al., 2024. The Power of Noise: Redefining Retrieval for RAG Systems. SIGIR 2024.
7. 07Min et al., 2023. FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. EMNLP 2023.
8. 08Ru et al., 2024. RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation. NeurIPS 2024 Datasets and Benchmarks.
9. 09Es et al., 2024. RAGAS: Automated Evaluation of Retrieval Augmented Generation. EACL 2024 Demos.
10. 10Liu, Zhang & Liang, 2023. Evaluating Verifiability in Generative Search Engines. Findings of EMNLP 2023.
11. 11Yang et al., 2024. CRAG: Comprehensive RAG Benchmark. NeurIPS 2024 Datasets and Benchmarks.
12. 12Zheng et al., 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks.
13. 13Wang et al., 2024. Large Language Models are not Fair Evaluators. ACL 2024.
14. 14Panickssery et al., 2024. LLM Evaluators Recognize and Favor Their Own Generations. NeurIPS 2024.
15. 15Shankar et al., 2024. Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences. UIST 2024.
16. 16Cohen, 1960. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement.
17. 17Angelopoulos et al., 2023. Prediction-Powered Inference. Science.
18. 18Saad-Falcon et al., 2024. ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems. NAACL 2024.
19. 19Miller, 2024. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. arXiv.
20. 20Kish, 1965. Survey Sampling. Wiley.
21. 21McNemar, 1947. Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages. Psychometrika.
22. 22Zou et al., 2025. PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models. USENIX Security 2025.