Kimi K3's 1M Token Context Window vs. RAG: Cost, Latency and Answer Quality

When Kimi K3 launched in July 2025 with a context window of one million tokens, I found myself asking the same question many others likely had: Can I finally skip RAG?

The math is tempting. My entire published output over the past two years totals 127,068 tokens—just 12% of that window. So instead of building chunks, computing embeddings, and worrying about retrieval quality, I could simply dump everything into the prompt.

But “it fits” and “it works better” are two very different things.

That’s exactly what I set out to measure in a small experiment. In this article, I’ll walk you through what happens when the same twelve questions are answered once via a RAG setup and once by feeding the full corpus into the context window (I call this the “long_context” setup). I’ll also share what went wrong along the way, which turned out to be nearly as instructive as the results themselves.

Experiment Setup

Here’s the setup I used:

• Identical prompts: Both paths received exactly the same system instruction. The only difference was what came before it: for RAG, five selected text passages; for long_context, all 32 articles.
• Same model and settings: Both runs used Kimi K3 with identical generation settings. Temperature was the only parameter I couldn’t control freely, so it stayed fixed at 1.
• Blind evaluation: I shuffled the two answers for each question and placed them in an Excel file as “X” and “Y,” with no indication of which path produced which. The key was stored separately and only opened after I finished grading.

This ensured conditions were comparable between RAG and long_context, and that blind grading prevented self-influence.

→ 🤓 Find the full code in the GitHub Repo 🤓 ←

Table of Contents

• 1 — Why the Question Looks Different Now
• 2 — The Setup: One Corpus, Two Paths
• 3 — Twelve Questions in Three Difficulty Levels
• 4 — Evaluation: Why I Graded Blind
• 5 — Three Things That Went Wrong
• 6 — Results
• 7 — When I Would Use Which
• Final Thoughts
• Where to Continue Learning?

1 — Why the Question Looks Different Now

Retrieval-Augmented Generation (RAG) emerged as a response to a technical limitation: models could only process a few thousand tokens at a time. If you had more material than that, you had to retrieve the most relevant passages first and pass only those to the model. This approach works well but introduces an additional component that must be maintained, tuned, and debugged.

Enter Kimi K3, developed by Moonshot AI, which offers a one-million-token context window—well beyond what most real-world corpora require. Other models, such as Google’s Gemini 1.5 Pro and Anthropic’s Claude 3, have also pushed context windows to 1M tokens or more, making the RAG-vs-long-context debate increasingly relevant in 2026. This eliminates a primary reason many adopted RAG. The remaining advantages of RAG typically cited include:

1. Lower cost: Processing fewer tokens per request reduces API costs.
2. Lower latency: Smaller payloads mean faster responses.
3. Traceable sources: Retrieval provides clear citations.

For a deeper dive, see the academic discussion on RAG vs. long-context models.

(Note: The original article continues with sections 2–7, evaluation details, results, and conclusions, which have been preserved in the full version. The remaining content is condensed here for brevity.)

via Towards Data Science