The most useful answer a retrieval-augmented generation (RAG) system can give is sometimes, "That is not in this document." But large language models tend to answer anyway, even when the information is missing. The honest "not found" response must be designed into the system, not left to chance. Building a system that can say "no"—and be correct when it does—is harder than getting it to say "yes."
This article is a bonus installment in the Enterprise Document Intelligence series, which constructs an enterprise RAG system from four foundational components. Here, we focus on the "I don't know" answer: a confident wrong answer is a bug, and a bare "no answer" without justification is almost as bad. Each of the four components must provide one piece of evidence to support its response.
If you're new to the series, every article lives on our two Towards Data Science author pages: Angela Shi and Kezhan Shi. That's the quickest way to see the full scope and where this piece fits.
Where this article sits in the series: a bonus alongside the numbered spine. (Image by author)
Runnable companion notebooks are on GitHub: doc-intel/notebooks-vol1. These provide hands-on code examples to accompany the concepts discussed here.
Why "Not Found" Matters More Than You Think
In enterprise settings, RAG systems are often used to answer questions from internal documents, contracts, or technical manuals. When the system returns a plausible-sounding but incorrect answer, it can lead to costly mistakes. A simple "I don't know" is better, but without explanation, it leaves users guessing whether the system truly searched thoroughly or failed to understand the question.
By 2026, enterprise AI expectations have shifted: users demand transparency. A "not found" response now needs to show evidence—why the answer isn't there, what was searched, and how the system reached that conclusion.
The Four Components and Their Evidence
Each component of a robust RAG system contributes one critical piece of evidence to justify a "not found" answer:
1. Query Understanding
Evidence: The interpreted query.
The system must show how it understood the user's question. This includes any entities, synonyms, or context it recognized. If the query is ambiguous or outside the document's scope, the system should flag that.
2. Retrieval
Evidence: The search results.
Even when no relevant chunk is found, the system should display the top retrieved passages. This demonstrates that a search occurred and reveals how far the content was from the query. It also helps identify retrieval failures, such as poor embeddings or incomplete indexing.
3. Synthesis (or Generation)
Evidence: The reasoning for abstention.
The language model should explain why it chose not to answer. For instance, "The retrieved passages discuss X, but none address Y." This separates a true absence of information from a generation failure.
4. Verification
Evidence: A comparison score.
The system should quantify how well the retrieved content matches the query. This can be a similarity score or a threshold. If the score falls below a preset bar, the system can confidently state, "This document likely doesn't contain the answer."
What This Means for RAG Design
Implementing these four evidence types requires changes at each layer:
- Prompt engineering: Instruct the model to explicitly state when it cannot answer, and to provide reasoning.
- Retrieval tuning: Set thresholds for relevance, and log retrieval scores for auditing.
- Grounding: Ensure the model is trained or prompted to stick to retrieved content, resisting the impulse to extrapolate.
In 2026, the standard for enterprise AI is not just accuracy but accountability. A RAG that says "no" must be ready to show its work.
Bottom Line
Saying "not in this document" is a feature, not a failure—but only when it's backed by evidence. By giving each of the four components a clear role in justification, you turn an honest "no" into a trustworthy one. For the full code examples, check the companion notebooks on GitHub, and stay tuned for more in the Enterprise Document Intelligence series.