Elicit: Context Length Impact on LLM Accuracy
How Task Accuracy Varies with Context Length in LLMs: From Older Models to GPT-5+, Claude Opus 4+, and Gemini 3+
Executive summary
Across a wide range of benchmarks, a consistent empirical pattern emerges: as context length increases, task accuracy tends to decrease, often substantially, and frequently well before reaching a model’s advertised maximum window. This degradation is observed in retrieval-style tasks (needle-in-a-haystack variants), realistic document-grounded QA, long-output generation, multi-task in-context learning, tool calling, and long-horizon multi-step execution.
Newer frontier models have improved robustness at very long contexts on some standardized retrieval/reasoning evaluations (notably MRCR/MRCRv2) relative to earlier generations, but the improvement is uneven across task types and settings. For example, GPT-5.2 Thinking shows markedly higher MRCRv2 accuracy than GPT-5.1 Thinking across multiple length ranges (4k–8k through 128k–256k), even though its accuracy still declines as length increases. Similarly, Claude Opus 4.6 reports strong multi-needle retrieval at 1M tokens in MRCR v2, while Gemini 3.1 Pro is reported much lower on the same 1M, 8-needle condition, illustrating that “million-token support” does not imply comparable effective long-context accuracy across providers.
A key shift since the pre-2025 generation is that several models can sustain high accuracy further into the 128k–256k+ regime on certain retrieval-style benchmarks, but difficult “deep” long-context uses—multi-round agentic workflows, multi-needle reasoning, latent-association retrieval, and tasks where length alone impairs reasoning—still produce large accuracy drops.
Foundational phenomenon
A recurring positional effect is “lost in the middle,” where models perform best when relevant information appears near the beginning or end of the context and worst when it appears in the middle, producing a U-shaped accuracy curve over position. In multilingual needle retrieval, the worst performance occurs when the needle is placed in the middle and is in a language outside the English family, and performance is better in English than non-English languages. In large-scale retrieval heatmaps, accuracy for most models decreases toward the middle of the context window, aligning with prior lost-in-the-middle observations.
A second foundational concept is the gap between nominal context length and “effective” context length: multiple sources conclude that accuracy degrades as the window grows and that many models’ reliable performance limits are significantly shorter than their supported maximums. One operational definition used in multilingual retrieval defines effective length as the maximum context length where performance does not decrease by more than 25% relative to baseline accuracy.
A third foundational result is that degradation is not purely a retrieval failure: controlled experiments show that even when models can “perfectly retrieve all the evidence” (including exact-match recitation), downstream performance on math, QA, and code tasks still degrades substantially as input length increases, including under minimal-distraction and masked-attention conditions. This supports the interpretation that input length itself can hurt performance independent of retrieval quality, not just distractor interference or missing evidence.
Benchmarks that reveal degradation
Different long-context benchmarks isolate different failure modes, and their results collectively show that “solving NIAH” is not the same as robust long-context task performance. The following table summarizes what each benchmark is designed to test and the core degradation signal it surfaces.
| Benchmark or study | What it stresses | Typical degradation signal as length increases |
|---|---|---|
| Vanilla NIAH variants and heatmaps | Single-fact retrieval across haystack lengths and positions | Recall often declines after certain lengths and varies by prompt content; prompt sensitivity can produce large differences at the same nominal length. |
| Sequential-NIAH | Retrieving multiple needles in a specified sequential order | Accuracy decreases across length groups and becomes harder with longer contexts and more needles; best model reported at 63.50%. |
| RULER | Retrieval plus multi-hop tracing, aggregation, and QA across 4k–128k | Many models are near-perfect on vanilla NIAH yet degrade substantially on RULER’s more complex tasks as sequence length increases; only about half of models claiming ≥32k maintain satisfactory performance at 32k. |
| NoLiMa | Retrieval requiring latent association (minimal lexical overlap) | Strong short-context performance (<1k) degrades significantly with length; at 32k many models drop below 50% of short-length baselines. |
| LongBench v2 | Multi-task long-context MCQ across length bins | Best models perform best on <32k and drop on 32k–128k and >128k ranges. |
| BABILong | Reasoning over facts embedded in long natural documents | Models use only 10–20% of context effectively and decline sharply with length and task complexity; many models only “efficiently use” up to about 4k except some stronger models reaching 16k–64k depending on model. |
| LOFT | Realistic retrieval, RAG, and compositional tasks up to 1M | Performance degrades when scaling corpora to millions of tokens, and performance drops when gold documents are moved toward the end of the corpus. |
| TLDM | Novel-length narrative understanding | Performance is comparable for <64k but begins to degrade when book lengths exceed 64k; summary/storyworld drop faster than time estimation. |
| Task Haystack | Multi-task Lifelong ICL with task-interleaved contexts | Pass rates decrease as context length increases; many configurations fall below 90% pass rate. |
| LongGenBench | Long output generation with many questions answered in one response | Performance degradation in long-context generation ranges from 1.2% to 47.1%, and accuracy declines as the question index increases within the long response. |
| LongFuncEval | Tool calling under large catalogs, long tool outputs, long conversations | Large degradation ranges (e.g., up to 85.58% with tool-catalog scaling; up to ~91% with long tool responses; 13–40% in long conversations). |
| MMNeedle and MM-NIAH | Multimodal retrieval in large visual/textual haystacks | Accuracy drops sharply as multimodal “context size” grows, and performance degrades with increasing multimodal context length across tasks. |
Older generation
Pre-2025 model evaluations repeatedly show that many systems can appear strong on simplified retrieval tasks, yet degrade quickly on more complex or realistic long-context tasks as length scales from a few thousand tokens into the 8k–128k range. In RULER, models benchmarked from 4k to 128k are described as “nearly perfect” on vanilla NIAH while “almost all models exhibit large degradation on more complex tasks in RULER as sequence length increases.” Even among models claiming ≥32k context, only about half maintain satisfactory performance at 32k, highlighting an early and persistent advertised-vs-effective gap.
More realistic document-grounded QA benchmarks also show sharp degradation at long lengths even for strong closed models. On the TAIL benchmark, the top four tested models can exceed 90% accuracy at 8k tokens but drop below 70% when context extends to 128k tokens; TAIL also reports GPT-4o cannot exceed 80% at 128k, despite achieving nearly 100% on a standard NIAH test.
NoLiMa demonstrates that removing literal lexical overlap can make long-context retrieval substantially harder and reveals strong length sensitivity even at 32k tokens. Reasoning-heavy long-context tasks show similar sensitivity. Long-output generation also degrades as the generation becomes longer and more question-dense.
Frontier models
Frontier 2025–2026 era systems show clear progress on some standardized long-context retrieval/reasoning evaluations, while still exhibiting material degradation as context length grows.
GPT-5+
OpenAI reports that GPT-5.2 Thinking “sets a new state of the art in long-context reasoning” on MRCRv2 and is the first model they have seen reaching “near 100% accuracy on the 4-needle MRCR variant (out to 256k tokens).” Other reported data points also suggest degradation at very long lengths for GPT-5 family variants on MRCR2 Needle and long-context reasoning tasks.
Claude Opus 4+
Anthropic frames “context rot” as a common complaint, defined as performance degrading as conversations exceed a certain number of tokens, and reports major gains for Opus 4.6 over earlier Claude variants on a 1M-token 8-needle MRCR v2 condition.
Gemini 3+
Direct benchmark evidence for “Gemini 3+” in the provided sources is concentrated in MRCR v2 at 1M tokens, reporting that Gemini 3.1 Pro scores 26.3% on this benchmark.
Methods and architectures
Recent work emphasizes that measuring long-context ability requires moving beyond simplistic NIAH tests toward broader downstream task suites, and that training and evaluation protocols can materially change observed long-context performance. Some approaches target million-token or “infinite” contexts via external memory or chunked retrieval that changes the underlying computation from “attend over everything” to “retrieve then generate.”
Persistent failure modes
Even as newer models improve on some long-context retrieval benchmarks, the evidence indicates multiple failure modes persist. First, complex reasoning and multi-needle settings remain harder than single-needle retrieval, and they tend to exhibit steep degradation as length or needle count increases.
Second, multi-round agentic depth can be more harmful than single-round noisy long-context “width.”
Third, “context rot” can appear even when retrieval is controlled, minimized, or perfect, implying that prompt length can directly reduce reasoning quality.
Fourth, real-world narrative and comprehension tasks show performance declines beyond tens of thousands of tokens even in 2025 frontier models.
Finally, long-context function/tool calling can cause large drops that appear distinct from traditional retrieval-only benchmarks.