AI decisions Architecture and technology
Long context windows vs RAG: do you still need a retrieval pipeline if the model can read the whole dataset?
Use long context windows when you need to synthesise, summarise, or compare information across a single massive document in a single turn. Use RAG when you need to answer questions from a large, constantly updating enterprise repository where permissions, query costs, and latency matter. Most enterprise systems use RAG to find the right needle, and long context windows to understand the haystack that needle was found in.
The options
Long context windows
You pass entire documents, books or codebases directly into the model's prompt, relying on its ability to read massive amounts of text at once.
RAG
The system searches a database for the specific paragraphs relevant to the user's query, and only sends those fragments to the model.
Side by side
| Criterion | Long context windows | RAG |
|---|---|---|
| Upfront pipeline effort | Low. You drop the files into the prompt without chunking, embedding or indexing. | High. Requires data pipelines, chunking strategies, embedding models and a vector database. |
| Cost per query | High. You pay for millions of input tokens on every single question asked by the user. | Low. You only pay for the prompt and the small, highly relevant fragments retrieved. |
| Latency | Slower. Reading and processing millions of tokens takes seconds to minutes before answering. | Faster. A quick database search followed by a short prompt generates answers almost instantly. |
| Data scale | Limited by the model's maximum window (e.g., a few million tokens or a few thousand pages). | Unlimited. The vector database can hold terabytes of company data across thousands of systems. |
| Access control | Hard to scale. You must dynamically build the prompt with only the files the user is allowed to see. | Native. The database filters search results by the user's permissions before sending anything to the model. |
| Best use case | Synthesising trends across a long report, or finding contradictions across a few massive contracts. | Fact-finding across an entire corporate intranet, customer support tickets or product wikis. |
| Traceability | The model can cite the text, but you rely on its attention mechanism to not miss details inside the prompt. | Explicit. You know exactly which document fragments were retrieved and fed to the model. |
Choose Long context windows when…
- The user needs to summarise, translate or format a specific, large document in its entirety.
- The query requires comparing complex reasoning spread across a single long text.
- You are building a quick prototype and want to avoid the engineering overhead of a retrieval pipeline.
- The total corpus size is well within the model's context limit and query volume is extremely low.
Choose RAG when…
- The knowledge base is larger than any available context window.
- The application serves many users, making the cost of massive prompts financially unsustainable.
- Fast response times are critical for the user experience, such as in customer-facing chatbots.
- Different users have different access rights, and the system must strictly respect document permissions.
- The underlying data changes constantly, and you need to add or remove facts without rewriting a massive prompt.
When to combine them
They work best together. As context windows grow, RAG chunking strategies can become less granular. Instead of retrieving isolated paragraphs, RAG can retrieve entire chapters or multiple full documents, and the long context window can synthesise them perfectly. Use RAG to filter out the noise and enforce permissions, and use long context models to reason over the retrieved, highly relevant documents.
Common mistakes
- Assuming a 2-million token window means you can just paste your entire corporate Google Drive into the prompt for every user question.
- Ignoring the 'lost in the middle' effect, where models sometimes fail to extract facts buried deep inside a massive prompt.
- Comparing API costs based on output tokens while ignoring the massive input token cost of sending a whole book on every query.
- Building a complex RAG pipeline for a use case that only ever involves chatting with single, standalone PDF files.
How Thinkia approaches it
We design enterprise architectures based on cost predictability, latency and strict access controls. While long context models are powerful, dumping unindexed data into a prompt does not scale for thousands of employees. Our approach with Enterprise Knowledge AI is to build a governed retrieval layer that respects permissions, finds the right evidence, and keeps the query cost sustainable.
We leverage expanding context windows to make RAG better, not to replace it. With Synapse, we can route queries dynamically: using fast, cheap models for standard retrieval tasks, and passing larger retrieved chunks—like entire policy documents—to models with long context windows only when deep synthesis is required. This gives you the reasoning power of long context without the massive token tax on every simple question.
Thinkia products involved
- Enterprise Knowledge AIGoverned knowledge layer: source-grounded answers with citations and audit trail.
- SynapseGoverned agentic platform: agents, models, costs and data in one place.
Related AI solutions
Frequently asked questions
Do long context windows make vector databases obsolete?
No. Vector databases are essential for searching across terabytes of data, enforcing user permissions at the document level, and keeping query costs and latency low. They act as the filter before the prompt.
Why is latency worse with long context windows?
Processing millions of tokens requires heavy computation. Even if the model reads it fast, the time to first token (when the model starts answering) is significantly longer than processing a short RAG prompt.
Are models perfectly accurate with long context?
Not always. Models can suffer from the 'lost in the middle' phenomenon, where they recall information at the beginning and end of a massive prompt well, but sometimes miss specific facts buried in the middle.
How do access permissions work with long context models?
The model itself cannot enforce permissions. You must write software to check what the user is allowed to see before passing those files into the prompt. RAG architectures handle this naturally at the database search level.
Does RAG still require chunking if context windows are huge?
Yes, but the chunks can be much larger. Instead of splitting a document into single paragraphs, you can index and retrieve whole pages or chapters, preserving the surrounding context and improving the model's understanding.
Keep exploring
Related decisions
- RAG vs fine-tuning: which one does your enterprise use case need?
- RAG vs enterprise search: do your people need answers or documents?
- GraphRAG vs vector RAG: when does your knowledge assistant need a graph?
Key terms
Thinkia articles
- Long-Context LLM Serving: Why Infrastructure Optimization Is the New Moat
- Prompt Engineering Limits: Why More Context Isn't Always Better for AI
- Vector Databases and RAG: Revolutionizing Data Management for Generative AI
Whitepapers
- The company with memory. Storing is not remembering.Companies do not have operational memory; they have amnesia governed by folders. A repository is not memory: real recall answers in the workflow.
- From the core to the agentic layer.Modernise the core or put agents on top? What decides is not the system but how legible its rules are. A four-quadrant framework for the 2027 budget.