In my RAG-ING Ahead series, I worked through a complete cloud-native retrieval stack: speech and document processing, chunking, embeddings, Azure AI Search, and an assistant layer on top. That series answered the question I had at the time: how do I get a language model to answer questions about documents it was never trained on?
The stack still works. Retrieval-Augmented Generation (RAG) remains the most practical way to ground a model in private, domain-specific, or recently changed information without retraining anything [1]. If you have a corpus and need answers from it, RAG is still where you start.
But after running that pattern on projects that lasted longer than a demo, a different question began to bother me: The system retrieves the same paragraph, reasons over it, produces a good answer—and then throws all that reasoning away. Tomorrow, someone asks a related question, and it does the identical work again, from scratch, at the same cost, with no guarantee of reaching the same conclusion.
My first instinct was to tune the machinery rather than question it. I experimented with embedding-based semantic caching: recognizing when an incoming question was semantically close to one already answered, and serving the earlier response instead of paying for a full retrieval-and-generation pass. Semantic caching genuinely helps with cost and latency, and I would still recommend it. But it took me a while to admit what it actually is: it caches answers, not understanding. The cached response is exactly as disposable as the original one. Nothing about the system's model of the domain has improved, and the moment a question falls outside the similarity threshold, the work starts from zero again. Whatever I tweaked, the main RAG architectural concept underneath remained the same.
That is not a retrieval problem. Retrieval is doing exactly what it was designed to do. It is an architecture problem. There is nowhere in a standard RAG system for understanding to accumulate. No amount of caching, re-ranking, or chunking strategy fixes that, because they all optimize the lookup; none of them gives the system a memory.
This article is about building that missing place. It is the result of my latest work and experimentation around RAG, GraphRAG, and agentic reasoning over a corpus of documents—the point where incremental tweaks stopped being enough and the design itself had to change. I will present it in three parts.
Part I is vendor-neutral. It describes the architecture as a design pattern: the layers, the object model, the failure modes it exists to survive, and the governance it demands. None of it depends on Azure, or on any particular database or model provider. If you are on AWS, GCP, or running Postgres with pgvector and a local model, the design still holds, and I hope it is useful to you.
Part II is the Azure implementation. Service by service, with the reasoning for each choice, real infrastructure-as-code, and a running FastAPI application you can clone and deploy.
Part III is the demonstration. A synthetic property insurer called Ostermere Mutual, twenty-one interconnected documents, and three walkthroughs that show the pattern doing something a retrieval system genuinely cannot.
Everything in the dataset is synthetic. Ostermere Mutual does not exist. Neither does the regulator, the policy, the claims, the people, the wind zones, or the figures. Nothing here is insurance, legal, underwriting, or claims advice, and no page in the demo represents a real interpretation of any real policy.