Skip to main content

AI glossary Models & Architecture

What is a context window in an LLM?

← All glossary terms
In short

The maximum amount of text, measured in tokens, that a language model can take into account in a single request, including its own reply.

Explained

The context window is the maximum amount of input and output, measured in tokens, that a large language model can process in one request. Everything the model sees counts against it: system instructions, conversation history, retrieved documents, tool results and the answer it writes. Larger windows allow longer documents and conversations, but they raise cost and latency, and models do not always use information in the middle of a long input as well as information at the start or end. Choosing what goes into the window is therefore as important as its size.

How the context window works

Models read text as tokens, pieces of words whose length depends on the language and the tokeniser. Each request sends the model a sequence of tokens, and the window is the ceiling on how long that sequence, plus the generated reply, can be.

The model has no memory between requests beyond what is sent in the window. In a chat, the history is resent each turn; in an agent, tool results pile up step by step. When the limit is reached, something must be dropped, summarised or moved out to retrieval.

Bigger is not always better

  • Cost and latency usually grow with the number of tokens sent.
  • Research has shown that models can use information at the start and end of a long input better than information in the middle.
  • Irrelevant text in the window can distract the model and lower answer quality.
  • A large window does not replace access control: anything placed in it can appear in the answer.

Working within the limits

The usual techniques are retrieval-augmented generation (RAG), which brings in only the relevant passages; summarising older conversation turns; trimming tool outputs to what matters; and keeping stable instructions short. Taken together, deciding what the model sees at each step is what is now called context engineering.

Frequently asked questions

What is a token in a context window?

A token is the unit of text a model reads: often a word fragment, a whole short word or a punctuation mark. The number of tokens for the same text varies by language and tokeniser. Context window sizes and API prices are both expressed in tokens.

Does the context window include the model's answer?

Yes, in most models the window covers both the input and the generated output. Providers also set a separate maximum for output tokens. If the input is close to the limit, there is less room for the reply.

What happens when you exceed the context window?

The API returns an error, or the application drops or summarises earlier content to make room. In chat products this can look as if the assistant has forgotten the start of the conversation. Well-designed systems manage this explicitly rather than letting text fall off silently.

Does a large context window replace RAG?

Not in most enterprise cases. Sending everything is costly and slow, permissions still need to be enforced per user, and quality can drop with very long inputs. Long windows and RAG work well together: retrieval selects, the window holds what was selected.

How big are context windows today?

It depends on the model and changes with each release. Current window sizes are published in each provider's documentation, which is the reference to check before designing a system around a limit.

Decision guides that use this term

Where it shows up at Thinkia

Further reading

Sources

Updated: 2026-10-21

Want to see how this applies to your organisation?

Talk to an AI Expert