Ask an AI engineer how to take a RAG system beyond a single document, and the answer has been the same for three years: put every page of every file into a vector store, embed the question, and let similarity search pick the passages.
On a demo folder, that works. On an enterprise shelf, it fails—not because a parameter is set wrong, but because similarity search returns passages, never whole documents. The answer gets assembled from three files that were never meant to be read together.
The step that gets skipped is the one before that. A collection of documents is not one kind of thing. A shared drive of unrelated reports, five thousand copies of the same contract, and a stack of claim folders are three different problems. An architecture that suits one of them suits the other two poorly.
This article maps that decision:
- Five ways a flat pile fails once the shelf grows—and why a re-ranker fixes none of them.
- Three questions that reveal which of the three corpus shapes you have.
- What each shape requires you to prepare, and what building the wrong shape costs.
- The four bricks of the single-document pipeline, viewed one level up.
- A baseline you can run today, and the waste it exposes on five real PDFs.
This article opens Part IV of Enterprise Document Intelligence, a series that builds an enterprise RAG system from four foundational components. In Part IV, the input shifts from a single file to a folder, and the first task is to identify which kind of folder you're dealing with.
🧭 New to the series? Start with the map: Prompt, Context, Loop outlines the three engineering layers every RAG system relies on—the prompt (the call itself), the context (what fills the model's window), and the loop (when the next call fires and when it stops)—and walks the entire series through that lens, article by article. It's the quickest way to see what's covered and where this piece sits.
📓 Run the five-PDF baseline in the companion notebook: one question, five model calls, and a per-document table showing one real answer and four blanks. Then point the same loop at a folder of your own and watch the same ratio appear. Repo → doc-intel/notebooks-vol1.
The worked example in Section 4 uses five public NIST publications: the Cybersecurity Framework v1.1 (NIST CSWP 04162018), FIPS 199, SP 800-207, AI 100-1, and CSWP 29. All are US Government works, in the public domain in the US (see the NIST copyright statement).
1. A Quarter of a Million Documents and One Question
Everything in the series so far assumed the system knew which document to look at—because there was only one. This section covers what happens when that assumption disappears.
1.1 What the Expert Does, and What the Pipeline Does
A mid-size insurance broker keeps about a quarter of a million documents in its claims and policy systems. A claims adjuster with ten years of experience can pull the right file in under a minute. The RAG pipeline, fed the same corpus, produces an answer that mixes three unrelated claims into a coherent narrative. The difference isn't the model—it's the decision about what to retrieve. The expert selects a document; the pipeline selects passages. Until that mismatch is fixed, no amount of prompt tuning will help.