RAG past the tutorial: where retrieval quietly breaks

Most writing about retrieval-augmented generation stops at the pipeline diagram: chunk, embed, take the top k, generate [1]. This note starts where that one ends. Recall has plateaued, the reranker no longer moves the numbers, and the remaining failures are subtle. Each section is a ceiling in the standard setup, the research that explains it, and what I would change.

A single vector has a capacity limit

Dense retrieval compresses a passage into one vector and scores it against the query with a dot product [2]. It is fast and it generalises, and it has a geometric ceiling. Weller et al. show that the number of distinct top-k result sets a d-dimensional embedding can return is bounded by d, and the bound holds even when embeddings are optimised directly on the test set [3]. Their LIMIT dataset uses deliberately simple queries and still defeats state-of-the-art embedding models.

In production this shows up on combinatorial queries: “contracts with a termination clause and a foreign counterparty”, “papers that cite A but not B”. The relevant set is an intersection, and one vector per document cannot represent every intersection a user might ask for. More training data does not lift a dimensional bound.

The alternatives each cost something. Late-interaction models such as ColBERT keep a vector per token and score with MaxSim [4]; ColBERTv2’s residual compression is what makes that index affordable [5]. Learned sparse models such as SPLADE expand documents into weighted vocabulary terms and keep the exactness of an inverted index [6]. Cross-encoders read query and passage together and have no capacity bottleneck at all, which is exactly why they belong at the end of the cascade, on a short list [7].

Hybrid search is coverage, and RRF is mostly a vote

BEIR evaluated retrievers zero-shot across 18 datasets. BM25 held up as a strong baseline out of domain, dense retrievers trained on one domain often underperformed elsewhere, and reranking and late-interaction models were best on average at a much higher cost [8]. Keyword plus vector search is not a hedge. It is two different failure surfaces covering each other.

Reciprocal Rank Fusion is the usual way to merge them [9]. Its constant k changes the behaviour more than most people expect.

At k = 60, a document a list ranks 1st gets only 1.15 times the weight of one it ranks 10th. RRF behaves like a vote counter: appearing in both lists matters far more than where you appear in either.

That is what you want when the two lists are complementary, and the wrong behaviour when one retriever is clearly better for a query class. For identifier-shaped queries (error codes, SKUs, function names) you want BM25’s first hit to dominate. Lower k for that class, or route those queries to the lexical leg before fusion. And fuse at the depth you rerank: a document missing from a list contributes zero, so fusing two lists of depth 10 silently drops everything one retriever ranked 11th.

Chunking is a modelling decision

Chunk boundaries decide what can be retrieved together. There are four serious answers, and they fit different question shapes.

Contextual retrieval prepends a short, model-written description of where each chunk sits in its document before embedding and indexing it. Anthropic reported it roughly halved failed retrievals in their tests, with a further gain when combined with reranking [10]. Late chunking reaches a similar goal without an LLM call per chunk: embed the whole document with a long-context encoder, then pool the token embeddings per chunk, so every chunk vector is conditioned on the full document [11]. RAPTOR clusters and recursively summarises chunks into a tree, so a question can match a summary node when no leaf holds the answer [12]. GraphRAG extracts an entity graph, summarises its communities, and answers global questions by map-reducing over those summaries [13]. It is the right tool for “what are the themes” questions, and an expensive one: every chunk passes through an extraction call at indexing time.

The reranker cannot fix what the first stage never fetched

A cascade is bounded by first-stage recall at the depth you rerank. If the gold passage sits at position 140 and you rerank the top 100, the best cross-encoder available returns a confident wrong answer. The number to watch is first-stage recall@100, not final recall@5.

LLM rerankers are now competitive with trained cross-encoders; RankGPT showed listwise reranking with a sliding window over the candidates [14]. Two details matter in production. Listwise rerankers are sensitive to the order candidates arrive in, and aggregating over several shuffled orders measurably improves them [15], which is worth the cost on high-stakes queries. And reranker scores are not calibrated across queries: 0.7 means something different for a short factoid than for a long analytical question. A single global “abstain below 0.5” threshold will be wrong for some query class. Calibrate thresholds per query type, or use the margin between the first and second candidate as the confidence signal.

The dangerous passage is the near miss

Long context windows do not remove the ordering problem. Liu et al. found models use information at the start and end of the input more reliably than in the middle [16], and Shi et al. showed irrelevant context measurably distracts models even on simple problems [17]. Cuconasu et al. then found something more uncomfortable: passages that rank highly but do not contain the answer hurt accuracy, while adding random documents raised accuracy by up to 35% in their setup [18].

The practical reading is that your worst passages are your reranker’s confident false positives, because they look exactly like evidence. Track a distractor rate on the eval set: the share of packed passages that score as relevant but do not support the answer. Then pack fewer passages, put the strongest first, and restate the question at the end of the prompt.

Decide whether to retrieve, then check what came back

Not every query needs retrieval, and not every retrieval should be trusted. Self-RAG trains a model to emit reflection tokens that decide when to retrieve and whether the retrieved passages support the output [19]. Corrective RAG adds a lightweight evaluator that grades retrieved documents and triggers a fallback, such as web search or refinement, when confidence is low [20]. You don’t need their training recipes to use the idea: a cheap router in front of retrieval and a sufficiency check after it catch a large class of failures before generation.

On the query side, HyDE embeds a generated hypothetical answer instead of the question [21], and Ma et al. trained a small rewriter with reinforcement learning against downstream answer quality [22]. The trick I’d add: rewriters paraphrase away exact tokens. Send the original query to the lexical leg and the rewritten one to the dense leg, then fuse. You keep the identifier match and gain the semantic expansion.

The index is an approximation too

HNSW, the default graph index in most vector databases, is approximate by design [23]. Its recall against exact search is below 1 and moves with parameters like ef_search, and few teams measure it. Sample a few hundred real queries, run brute-force search on them, and compare. If ANN recall@10 is 0.92, no amount of embedding or reranker tuning recovers the missing 8%.

Filters make this worse. Under a restrictive metadata filter, say one tenant and one month, post-filtering an HNSW result can leave almost nothing, and pre-filtering can cut the graph into pieces so the search stalls in a region with no valid nodes. Predicate-aware indexes such as ACORN were built for this case [24]. If you can’t change the index, partition by the most selective filter, which is usually the tenant.

Compression has the same shape. Product quantisation [25] and Matryoshka embeddings, which can be truncated to fewer dimensions [26], cut memory dramatically at some recall cost. The standard recovery is two passes: search with compressed vectors, then rescore the top few hundred with full-precision ones before reranking.

Your corpus is an attack surface

Anything retrievable can be injected. Greshake et al. demonstrated indirect prompt injection through retrieved content in real LLM-integrated applications [27]. PoisonedRAG measured how little it takes: five malicious texts per target question, in a knowledge base of millions of texts, reached a 90% attack success rate [28].

• Keep provenance on every chunk (source, owner, ingestion time, trust tier) and filter on it at retrieval time.
• Present passages as quoted data, never as instructions, and never let retrieved text change which tools the model may call.
• Watch for new documents that suddenly rank first for many unrelated queries. An optimised poison looks exactly like that.

The checklist I actually use

• Measure first-stage recall at rerank depth, ANN recall against brute force, and distractor rate, as three separate numbers.
• Route identifier-shaped queries to lexical search, and tune RRF’s k per query class.
• Choose the chunking strategy from question shape, not from a default size.
• Calibrate abstention thresholds per query type.
• Version the embedding model together with the index, and migrate through a shadow index rather than in place.
• Keep provenance on every chunk, and filter on it.

Every ceiling in RAG is a retrieval ceiling first. The model can only reason over what you hand it.

References

1. 01Lewis et al., 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS 2020.
2. 02Karpukhin et al., 2020. Dense Passage Retrieval for Open-Domain Question Answering. EMNLP 2020.
3. 03Weller et al., 2025. On the Theoretical Limitations of Embedding-Based Retrieval. ICLR 2026.
4. 04Khattab & Zaharia, 2020. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. SIGIR 2020.
5. 05Santhanam et al., 2022. ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction. NAACL 2022.
6. 06Formal et al., 2021. SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking. SIGIR 2021.
7. 07Nogueira & Cho, 2019. Passage Re-ranking with BERT. arXiv.
8. 08Thakur et al., 2021. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. NeurIPS 2021 Datasets and Benchmarks.
9. 09Cormack, Clarke & Büttcher, 2009. Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods. SIGIR 2009.
10. 10Anthropic, 2024. Introducing Contextual Retrieval. Engineering blog.
11. 11Günther et al., 2024. Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models. arXiv.
12. 12Sarthi et al., 2024. RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval. ICLR 2024.
13. 13Edge et al., 2024. From Local to Global: A Graph RAG Approach to Query-Focused Summarization. arXiv.
14. 14Sun et al., 2023. Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents. EMNLP 2023.
15. 15Tang et al., 2024. Found in the Middle: Permutation Self-Consistency Improves Listwise Ranking in Large Language Models. NAACL 2024.
16. 16Liu et al., 2024. Lost in the Middle: How Language Models Use Long Contexts. TACL 2024.
17. 17Shi et al., 2023. Large Language Models Can Be Easily Distracted by Irrelevant Context. ICML 2023.
18. 18Cuconasu et al., 2024. The Power of Noise: Redefining Retrieval for RAG Systems. SIGIR 2024.
19. 19Asai et al., 2024. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. ICLR 2024.
20. 20Yan et al., 2024. Corrective Retrieval Augmented Generation. arXiv.
21. 21Gao et al., 2023. Precise Zero-Shot Dense Retrieval without Relevance Labels. ACL 2023.
22. 22Ma et al., 2023. Query Rewriting for Retrieval-Augmented Large Language Models. EMNLP 2023.
23. 23Malkov & Yashunin, 2018. Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs. IEEE TPAMI.
24. 24Patel et al., 2024. ACORN: Performant and Predicate-Agnostic Search Over Vector Embeddings and Structured Data. SIGMOD 2024.
25. 25Jégou, Douze & Schmid, 2011. Product Quantization for Nearest Neighbor Search. IEEE TPAMI.
26. 26Kusupati et al., 2022. Matryoshka Representation Learning. NeurIPS 2022.
27. 27Greshake et al., 2023. Not What You’ve Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. AISec 2023.
28. 28Zou et al., 2025. PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models. USENIX Security 2025.