A car-insurance contract prints its guarantees as a 40-row table: one row per covered event, with columns for the event, its cap, its deductible, and the eligibility condition. A user opens the search bar and asks, "What is the cap for vehicle theft?" The naive retrieval approach indexes the whole table as one chunk. When the question matches, the generation model receives 39 rows of unrelated events on top of the one that answers, forcing it to guess which line the reader meant. Sending the entire table makes the model do the filtering that the retriever should have done—the unit that actually matches the question is one row, not the whole rectangle.
This article is part of Part II of Enterprise Document Intelligence, a series that builds an enterprise RAG system from four foundational bricks. The retrieval brick treats retrieval as filtering rather than vector search, runs keyword and embedding signals in parallel, lets an LLM arbiter rank the finalists, and routes long documents through their table of contents. This companion addresses a case they leave open: a document whose answer resides inside a table, where the unit that fits the question is one row.
🧭 New to the series? Start with the map: Prompt, Context, Loop outlines the three engineering layers every RAG system relies on—the prompt (the call itself), the context (what fills the model's window), and the loop (when the next call fires and when it stops)—and walks through the series article by article. It's the quickest way to see the full coverage and where this piece fits.
!Article 7sexies sits in Part II as a retrieval companion. – Image by author
📓 The runnable companion is on GitHub: Load Table 1 of the Attention paper, watch serializetablerows turn it into four row-level chunks, then run a targeted keyword query to see it return just the relevant row instead of the whole table. Access it at doc-intel/notebooks-vol1.
!The public companion-code repo at doc-intel/notebooks-vol1 – Image by author
We demonstrate this using Attention Is All You Need (Vaswani et al. 2017; arXiv non-exclusive distribution license, as declared on the arXiv abstract page). Its Table 1 (page 6) is a compact real-world case: four rows, four columns, each row representing a distinct answer. Runnable code paths call OpenAI services governed by OpenAI's Terms of Use. Parsing uses Docling (MIT license).
1. Table vs. row: The unit of retrieval
A table on paper is a bounded region: a rectangle of cells with a header row on top and body rows below. A retrieval system that treats the whole rectangle as one chunk collapses that structure. The chunk either matches—and the generation model receives every row, most of them irrelevant to the question—or it doesn't, leaving the generation model with nothing because the one relevant row is buried among all others that the scorer had to average over.
The mismatch lies between the document's unit of information (one rectangle) and the reader's unit of question (one row). A question like "cap for vehicle theft?" targets one row. A question like "which events are covered?" targets the whole rectangle. Both are legitimate. A retrieval brick that only offers the rectangle answers the second question but mishandles the first.
The solution isn't to pick one scale over the other, but to make the unit of retrieval align with the query's granularity. In practice, this means generating row-level chunks for table data, enabling the retriever to match the precise row that answers the user's question. By 2026, leading RAG systems have shifted toward hierarchical chunking strategies—breaking tables into rows, sections into paragraphs, and documents into sections—so that retrieval can operate at the most relevant level. This approach reduces noise in the generation context, improves answer precision, and lowers token costs, as the model receives only the information it needs.
Implementing row-level chunks involves a few key steps:
- Parse the table structure: Use a tool like Docling to extract the table and identify the header row and body rows.
- Serialize each row: Convert each row into a self-contained text chunk, including column names and values, e.g., "Event: vehicle theft; Cap: $10,000; Deductible: $500; Condition: comprehensive coverage."
- Index row-level chunks: Store these chunks in the retrieval index, alongside or instead of the whole-table chunk, depending on the use case.
- Query accordingly: For row-specific questions, the retriever matches the relevant row; for table-wide questions, it can retrieve the whole table or a summary.
This method is especially valuable in domains like insurance, finance, and legal documents, where tables encode critical data points. By giving the retriever fine-grained access, you minimize hallucination risks and ensure that the generation model focuses on the exact answer.
The companion notebook demonstrates this workflow end-to-end: parsing Table 1 from the Attention paper, creating four row-level chunks, and running a keyword query that returns only the matching row. This approach is a practical addition to any enterprise RAG system, bridging the gap between structured data and unstructured queries.