A context window is the fixed-size input buffer of a language model — the total number of tokens (prompt plus response) the model can process in a single interaction. It determines how much text, code, conversation history, or data the model can reference when generating its next output. Everything the model knows about the current conversation — the system prompt, the user’s question, any retrieved documents, previous messages, and the response it is generating — must fit within this window. Once the window is full, information must be dropped or summarized to make room.
Context windows are to language models what working memory is to humans. A person with a short working memory can hold a phone number; a person with a long working memory can hold an entire conversation in their head. The analogy is imperfect — models do not “forget” in the human sense — but the practical consequence is the same: the context window defines the boundary of what the model can consider at any given moment.
The context window expansion timeline
Context windows have expanded by three orders of magnitude in five years, fundamentally changing what language models can do:
| Model | Release | Context window | Approximate word equivalent |
|---|---|---|---|
| GPT-2 | 2019 | 1,024 tokens | ~750 words |
| GPT-3 | 2020 | 4,096 tokens | ~3,000 words |
| GPT-3.5 Turbo | 2023 | 16,384 tokens | ~12,000 words |
| Claude 2 | 2023 | 100,000 tokens | ~75,000 words |
| GPT-4 Turbo | 2023 | 128,000 tokens | ~96,000 words |
| Claude 3 Opus | 2024 | 200,000 tokens | ~150,000 words |
| Gemini 1.5 Pro | 2024 | 1,000,000 tokens | ~750,000 words |
| Gemini 2.5 Pro | 2025 | 1,000,000 tokens | ~750,000 words |
| Claude Opus 4 | 2025 | 200,000 tokens | ~150,000 words |
| Claude Sonnet 4 | 2025 | 200,000 tokens | ~150,000 words |
At 200K tokens, Claude can process a 500-page book in a single prompt. At 1M tokens, Gemini can process roughly 4-5 books, an entire codebase, or hours of meeting transcripts. These are not theoretical limits — they are production capabilities available through standard APIs.
How context windows work technically
The context window size is determined by the model’s architecture, specifically its positional encoding scheme and attention mechanism.
Positional encoding
Transformers process all tokens in parallel, so they need a way to encode the position of each token in the sequence. The original transformer used fixed sinusoidal positional encodings with a predetermined maximum length. Modern models use Rotary Position Embeddings (RoPE), which encode relative positions through rotation matrices applied to the attention computation.
RoPE’s key advantage is extensibility — models trained with RoPE at one context length can be extended to longer contexts through techniques like YaRN (Yet another RoPE extensioN) and NTK-aware scaling, which adjust the frequency components of the rotary encodings. This has enabled models originally trained at 8K or 16K context to be extended to 128K or beyond with relatively modest additional training, typically 1-5% of the original training compute.
Attention mechanism and the quadratic problem
Standard self-attention computes relationships between every pair of tokens in the context, creating an attention matrix of size N x N where N is the sequence length. This means memory usage scales quadratically with context length: doubling the context quadruples the memory required for attention computation.
For a model processing 200K tokens, the attention matrix alone (in FP16) would require approximately 75 GB of memory per layer — clearly impractical with standard attention. Several architectural innovations address this:
Flash Attention (Tri Dao, 2022) does not reduce the computational complexity of attention but reorganizes the computation to be dramatically more memory-efficient by tiling the attention computation and keeping data in fast GPU SRAM rather than slower HBM. Flash Attention 2 and 3 further optimize this, enabling practical attention computation at 200K+ context lengths. This is the single most important engineering advance behind long-context models.
Ring Attention distributes the attention computation across multiple GPUs, with each GPU handling a segment of the sequence and passing key-value pairs in a ring topology. This enables context lengths beyond what a single GPU can handle. Google used ring attention to achieve 1M-token context for Gemini.
Sliding window attention restricts each token to attend only to nearby tokens (a window of 4K-32K), reducing attention from O(N^2) to O(N * W) where W is the window size. Mistral’s models use sliding window attention in some layers while maintaining full attention in others, balancing local detail with global context.
Multi-Query Attention (MQA) and Grouped-Query Attention (GQA) reduce memory by sharing key and value heads across multiple query heads. This does not change context length directly but reduces the KV cache memory by 4-8x, making longer contexts practical. Llama 3 uses GQA, and most modern architectures have adopted it.
The KV cache: the real bottleneck
While attention computation is often cited as the context length bottleneck, the key-value (KV) cache is the more pressing practical constraint. During auto-regressive generation, the model stores the key and value tensors for every token already processed, so it does not need to recompute them for each new token.
For a model like Llama 3.1 70B (80 layers, 8 KV heads, 128 dimensions per head), the KV cache per token is approximately 40 KB in FP16. At 128K context, the KV cache alone requires approximately 5 GB — per request. A server handling 32 concurrent requests at 128K context needs 160 GB just for KV caches, before accounting for the model weights themselves.
This explains several practical phenomena:
-
Why long-context inference is more expensive. Providers charge more for long contexts not just because of attention computation but because of KV cache memory. Each long-context request monopolizes more GPU memory, reducing the number of concurrent requests the server can handle.
-
Why context caching exists. Anthropic’s prompt caching and Google’s context caching features store the KV cache for a given prefix across requests. If multiple queries share the same system prompt and document context (common in RAG and agentic applications), the KV cache is computed once and reused, reducing both latency and cost. Anthropic’s prompt caching reduces input token costs by up to 90% for cached portions.
-
Why KV cache quantization matters. Quantizing the KV cache from FP16 to INT8 or INT4 halves or quarters the memory requirement, enabling longer effective context on the same hardware. This is an active area of optimization in inference frameworks like vLLM and TensorRT-LLM.
The “lost in the middle” problem
Longer context windows do not automatically mean better utilization of the information within them. A landmark 2023 paper by Stanford and UC Berkeley researchers demonstrated that language models show a U-shaped attention pattern: they attend strongly to information at the beginning and end of the context but are significantly worse at using information in the middle.
In their experiments, models achieved 90%+ accuracy when relevant information appeared in the first or last 10% of the context, but accuracy dropped to 50-70% when the same information was placed in the middle third. This “lost in the middle” effect has significant practical implications:
For RAG systems: Place the most relevant retrieved documents at the top of the context, not sorted by relevance score descending. Some systems duplicate the most important passages at both the beginning and end of the context.
For long document analysis: When asking questions about specific sections of a long document, explicitly reference the section or page rather than expecting the model to find the relevant passage on its own.
For conversation history: In long conversations, critical context from early messages may be effectively invisible if buried under dozens of subsequent exchanges. Periodic summarization of key points helps maintain coherence.
The severity of this effect varies by model and has improved with newer architectures. Models trained specifically on long-context tasks (Gemini 1.5/2.5 Pro, Claude 3/Opus 4) show less pronounced middle-context degradation than models extended to long contexts after initial training.
Context window pricing economics
Context length directly affects the cost of using language model APIs, and the pricing structures vary significantly:
Linear pricing. Most providers charge per token regardless of context length. Claude Sonnet 4 costs $3 per million input tokens whether the context is 1K or 200K tokens. A 200K-token prompt costs $0.60 — affordable for high-value tasks but expensive at scale.
Tiered pricing. Some providers charge more for longer contexts. Google charges a higher per-token rate for Gemini prompts exceeding 128K tokens, reflecting the higher compute cost of very long contexts.
Context caching discounts. Anthropic’s prompt caching charges $3.75/million tokens to write to cache (a 25% premium) but only $0.30/million tokens for cache reads — a 90% discount. For applications where the same document context is used across hundreds of queries, caching transforms the economics of long context from expensive to cheap.
A practical example illustrates the cost dynamics. Consider an application that processes a 100-page contract (approximately 50,000 tokens) with 200 user questions per day:
| Approach | Daily token cost | Monthly cost |
|---|---|---|
| No caching (50K context per query) | 200 x 50K x $3/M = $30 | $900 |
| With prompt caching (50K cached) | Cache write + 200 cache reads = $16.88 + $3 = ~$20 | $600 |
| RAG (retrieve 5K per query) | 200 x 5K x $3/M = $3 + retrieval costs | $90-150 |
When context window size matters
Context window size is most impactful in specific use cases:
Codebase analysis. A typical software project has 10,000-100,000 lines of code across dozens of files. At 200K tokens, Claude can process approximately 50,000-70,000 lines of code in a single prompt — enough for most individual repositories. Claude Code leverages this by reading entire codebases into context when planning changes, enabling it to understand cross-file dependencies and maintain consistency.
Document analysis. Legal contracts, research papers, financial filings, and policy documents often span 50-500 pages. A 200K-token context handles documents up to roughly 500 pages. A 1M-token context handles approximately 2,500 pages — sufficient for most individual documents but still requiring chunking or RAG for large document collections.
Long conversations. Each message in a conversation (user and assistant) accumulates in the context. A typical conversational exchange uses 500-2,000 tokens per turn. At 200K tokens, a conversation can sustain 100-400 turns before hitting the context limit. For extended sessions (debugging, tutoring, collaborative writing), this is often sufficient but can be exceeded, requiring message compaction or summarization.
Multi-file editing. Agentic coding tasks that modify multiple files simultaneously benefit from seeing all relevant files in context. Without sufficient context, the agent must make changes file-by-file, risking inconsistencies. Larger context windows reduce the need for sequential processing and improve the coherence of multi-file changes.
Context window vs. RAG: the design decision
A persistent question in AI architecture is whether to use larger context windows or RAG to handle large information sets. The answer depends on the specific constraints:
Use large context windows when:
- The entire corpus fits within the window
- Every part of the content might be relevant (e.g., analyzing a single document)
- Simplicity is important (no retrieval infrastructure to maintain)
- The content changes frequently (no index to update)
- You need the model to synthesize across the entire corpus
Use RAG when:
- The corpus is larger than the context window
- Only a small fraction of the corpus is relevant to each query
- Cost per query matters (retrieving 5K tokens is cheaper than ingesting 500K)
- You need citations and source attribution
- The corpus is growing continuously
Use both when:
- The corpus is large but you want multi-document reasoning
- You need the precision of retrieval combined with the reasoning depth of full context
- Your application has varying query types (some need broad context, others need specific passages)
In practice, the “both” approach is increasingly common. RAG retrieves the most relevant documents, and a large context window allows including more retrieved results plus conversation history, system prompts, and instructions.
Context window management techniques
When applications approach or exceed context limits, several techniques extend effective capacity:
Conversation compaction. Instead of dropping old messages, summarize them into a condensed context block that preserves key information. Claude Code uses this approach — when a coding session approaches the context limit, earlier tool calls and their results are summarized while preserving the overall task state and decisions made.
Sliding window with summary. Maintain the most recent N messages in full detail, with a rolling summary of older messages prepended. This preserves recency and important context while staying within limits.
Hierarchical context. Organize context into layers: a persistent system prompt, a task-level context block, and a turn-level working memory. When space is tight, compress the task-level context while preserving the system prompt and recent turns.
Dynamic context selection. For agentic applications, intelligently select which information to include in each model call. A coding agent does not need to include every file in every call — it can include only the files relevant to the current subtask, using the model’s previous observations to determine relevance.
The effective context frontier
Context window size on paper and useful context length in practice are different things. Several factors reduce the effective context:
Attention degradation. Even without the “lost in the middle” effect, model performance on needle-in-a-haystack tasks degrades as context length increases. A model might achieve 99% accuracy at 10K context, 95% at 100K, and 85% at 500K. The degradation curve varies by model and task.
Inference latency. Processing a 200K-token prompt takes 10-30 seconds for initial encoding before the first output token appears. At 1M tokens, this can exceed 60 seconds. For interactive applications, this latency limits practical context use even when the model supports it.
Cost scaling. At $3/million input tokens, a 200K prompt costs $0.60. At 100 queries per day, that is $60/day or $1,800/month — just for input tokens. The economic ceiling on context usage is often lower than the technical ceiling.
Output quality. Empirically, models produce higher-quality outputs when given focused, relevant context rather than maximum context. Including irrelevant information can actually degrade performance by diluting the model’s attention across non-relevant passages. This is why RAG (selecting relevant context) often outperforms brute-force context stuffing even when the full corpus fits in the window.
Frequently asked questions
What happens when you exceed the context window? The behavior depends on the implementation. API calls that exceed the context limit return an error. Chat interfaces typically truncate or remove older messages to fit within the limit. Some applications implement automatic summarization of older context. The model itself does not process tokens beyond its context limit — any content that exceeds the window is simply not seen by the model.
Is a bigger context window always better? Not necessarily. Bigger context windows provide more capacity, but using that capacity has costs: higher latency, higher price, and potential attention degradation. A 200K-token model processing a 5K-token prompt performs identically to a 1M-token model processing the same 5K-token prompt. The value of larger windows lies in the ability to process more information when needed, not in a general quality improvement.
How do context windows affect conversation memory? Language models have no memory between separate conversations. Within a single conversation, the context window determines how much history the model can reference. A 200K-token window can hold approximately 100-400 conversational turns. Once the window is exceeded, the application must drop or summarize earlier messages. Some applications (like Claude) use external memory systems to persist information across conversations, but this is separate from the context window.
Why do different models have different context window sizes? Context window size is an architectural and economic choice. Larger windows require more GPU memory (for KV cache), more compute (for attention), and more training data at long context lengths. Each model maker balances these costs against the value of longer context for their target use cases. Google prioritizes very long context (1M tokens) for document processing. Anthropic’s 200K context balances capability with cost-effectiveness for coding and analysis tasks.
Does context caching reduce the context window? No. Context caching is a performance and cost optimization, not a context size change. When you cache a 50K-token prefix, the model still processes and attends to those 50K tokens — it just does not recompute the KV cache for them on subsequent requests. The context window remains the same size, and the cached content counts toward it. The benefits are reduced latency (cached tokens are processed faster) and reduced cost (cached input tokens are priced at a discount).