Multi-Document RAG: A Folder of Unrelated PDFs Is One Long Document with a Nested Outline

When scaling a Retrieval-Augmented Generation (RAG) system beyond a single document, the standard advice is to build an index: one row per document, one column per field for filtering, then apply a filter before the search. However, this approach assumes the documents share common fields—a situation that many folders do not satisfy.

Consider a research folder containing a 492-page catalogue of security controls, a zero-trust architecture specification, an AI risk framework, thirty-one machine-learning papers, and seven commodity market reports. These documents share no common column. There is no client, amount, or effective date that holds consistent meaning across all of them, nor any other field a business user could name.

When no shared field exists, no table can be built. This might seem like a gap, but it is the opposite: it collapses the preparation step into two artefacts, one of which the parser already returns for free. This article explores that case in depth, covering:

The two questions that distinguish this scenario from one that requires an index.
The minimal preparation: one summary line per file plus each file's own table of contents, and nothing else.
Why the summary line must be written for a router, not for a reader.
The two-level routing that answers a question, and why the nested outline never enters a prompt in its entirety.
Four ways this approach breaks, including the point where a flat file list stops scaling.

This article is part of Part IV of Enterprise Document Intelligence, a series that builds an enterprise RAG system from four fundamental bricks. Part IV addresses what changes when the input is a folder instead of a file, and the answer depends on the folder type. This one focuses on the type that requires the least new machinery.

If you are new to the series, start with the map: 'Prompt, Context, Loop' outlines the three engineering layers every RAG system relies on—the prompt (the call itself), the context (what fills the model's window), and the loop (when the next call fires and when it stops)—and walks through the entire series through that lens, article by article. It is the quickest way to see what is covered and where this piece fits.

!Where this article sits: Part IV, the folder that needs no index – Image by author

You can build the two-level index over your own folder in the companion notebook: write one summary line per file, print the level-0 list, then run the routing call and watch it keep one file out of sixty-three before a single page is read. The repository is available at doc-intel/notebooks-vol1.

!The public companion-code repo at doc-intel/notebooks-vol1 – Image by author

The folder used throughout this article consists of 63 public PDFs, totaling 4,211 pages: 25 NIST files (24 publications plus a three-page scan of one, all US Government work in the public domain, per the NIST copyright statement), 31 arXiv papers (each under the arXiv non-exclusive distribution license declared on its abstract page), and 7 issues of the World Bank Commodity Markets Outlook (CC BY 3.0 IGO, as declared on the OKR publication page for April 2026).

1. Which Kind of Folder Do You Have?

The first decision is to identify whether your folder fits this scenario. Two questions help:

Do all documents share at least one field with consistent semantics (e.g., client ID, date, or amount)? If yes, a traditional index may be appropriate. If no, proceed to the next question.
Is the goal to answer questions that span across the entire corpus, rather than querying specific fields? If yes, the two-level routing approach described here is likely the right fit.

If your folder lacks shared fields and requires cross-document reasoning, you are working with a heterogeneous corpus. In this case, the standard index-based advice breaks down, and a different strategy is needed.

2. The Preparation: Two Artefacts Only

The preparation step simplifies to two artefacts:

A summary line per file: A one-sentence description of each document's content, purpose, and scope, written to support routing (see Section 3).
The file's own table of contents (ToC): Often extracted automatically during PDF parsing, so this comes without extra effort.

No additional metadata extraction or schema design is required. This minimal setup is sufficient for effective routing.

3. Write the Summary Line for a Router, Not a Reader

The summary line is not for human readers; it is for an LLM-based router that decides which documents are relevant to a query. Therefore, it should be:

Specific: Mention the document type, topic, and key scope (e.g., 'A catalogue of security controls for federal IT systems').
Consistent: Use a uniform structure across all files to make comparisons easier for the router.
Routing-oriented: Highlight aspects that differentiate it from other documents, such as domain, perspective, or time period.

The goal is to give the router enough signal to narrow down a 63-file list to a handful of candidates without reading the full content.

4. Two-Level Routing: Answering Questions with a Nested Outline

Instead of a single flat search, the system uses two levels of routing:

Level-0 routing: The router reads the summary lines (the 'flat list') and selects the top candidate files. For example, given a query about AI risk management, it picks 5 out of 63 files.
Level-1 routing: Within each selected file, the router uses the file's own ToC to choose the most relevant sections or chapters.

The nested outline—the ToCs of all files—never enters a prompt in its entirety. Only the summaries are used at level 0, and only the ToC of the selected files is used at level 1. This keeps token usage low and improves accuracy by focusing the model's attention.

5. Four Ways It Breaks (and Where It Stops Scaling)

While this approach reduces machinery, it has failure points:

Ambiguous summaries: If a summary line is vague or misleading, the router may select the wrong files, leading to missed information.
Heading inconsistency: If a file's ToC uses cryptic or non-descriptive headings, level-1 routing becomes unreliable.
Overlapping scope: If two documents have very similar summaries, the router may not distinguish them, but this is often acceptable if both are relevant.
Scaling limit: The flat file list stops scaling when the number of files exceeds the router's context window. For very large folders (e.g., thousands of files), you need a hierarchical clustering or a database to manage summaries.

In practice, this method works well for folders up to a few hundred files, but beyond that, a different approach is required.

Conclusion

A folder of unrelated PDFs is best treated as a single long document with a nested outline, rather than a collection of records. With just a summary line per file and each file's own ToC, you can build an effective two-level routing system that answers questions across a heterogeneous corpus with minimal setup. By writing summaries for a router, you optimize for decision-making, not human readability, and you keep the prompt lean. As the folder grows, watch for the scaling limit and consider moving to more complex indexing if needed.

This approach is part of the broader Enterprise Document Intelligence series, where you can explore how the prompt, context, and loop layers adapt to different folder types.

via Towards Data Science