Measuring the lost-in-the-middle effect in long context windows

When a language model supports a large context window, that doesn't automatically mean all information across the full span is processed equally. In practice, a phenomenon often occurs where facts buried deep in the middle of the prompt get overlooked, while information at the beginning and end is retrieved flawlessly. This is known as the lost in the middle effect. This article differs from general retrieval tests by zooming in specifically on the mathematical spread of attention loss across varying depths and window sizes.

The decision informed by this measurement is fundamental to any software architecture: do you rely on the inherent memory of a large context window for analyzing extensive documents, or is an additional retrieval layer with targeted reranking of document fragments necessary? Anyone who blindly trusts a large context window without validation risks having crucial clauses in legal files or variables in codebases silently ignored.

The mechanics of attention degradation

Attention mechanisms in transformer architectures compute pairwise interactions between all tokens in a sequence via the softmax operation. In theory, every token can attend to every other token. In practice, however, the attention distribution isn't spread homogeneously. As the number of tokens increases, dispersion occurs: the attention vectors have to be normalized across tens of thousands of input positions, causing the weights for individual tokens in the middle to become diluted.

A second cause lies in positional encoding. Many modern architectures use Rotary Position Embeddings (RoPE) to capture relative distances between tokens. Read the article on Rotary Position Embedding and long texts to understand how frequency scaling and base values affect the relative distance calculation across tens of thousands of tokens. When a context window is artificially stretched via interpolation techniques, the model often loses resolution at relatively large distances from the instruction prompt, which is usually located at the beginning or end of the window.

The result is a characteristic U-shaped performance curve: at the beginning (right after the system prompt) and at the end (just before the question), the model performs strongly, but in the central section accuracy drops significantly. To determine whether a specific model is suitable for a given use case, this performance curve needs to be mapped out precisely.

Scope: synthetic needles versus semantic synthesis

When evaluating context windows, there are two different testing approaches. The classic synthetic test places a random fact in a sea of irrelevant text and asks the model to reproduce that fact verbatim. See the guide on the needle-in-a-haystack test method for the basic principle behind such synthetic retrieval tests.

Measuring the lost in the middle effect, however, goes a step further than mere binary retrieval accuracy. Two specific dimensions are varied simultaneously here:

• Context depth (relative position): the exact percentage location of the information within the total input.
• Context length (absolute size): the total length of the context in tokens.

In addition, a distinction must be made between single-needle retrieval (retrieving one unique fact) and multi-needle reasoning (combining two related facts that are both hidden at different depths in the middle). While models often still achieve reasonable results on single-needle tests, accuracy on multi-needle tasks in the middle drops considerably more.

The systematic measurement protocol: the depth matrix

A robust measurement protocol requires a controlled grid evaluation. A matrix is defined that plots total context length against relative depth. To gather reliable statistics, each cell in the grid is repeated with different seed values and varying document fragments.

The table above shows the qualitative pattern of degradation: with shorter contexts, performance is maintained across the whole span, but at maximum lengths, accuracy plummets in the middle. To build such test files, the interactive needle-in-a-haystack generator tool can be used to assemble standardized input files with configurable depths.

Noise distribution and Dutch-language specificity

The composition of the background text greatly influences the measurement outcome. When the background text consists of monotonous repetitions of a single sentence, the contrast ratio between the needle and the noise is unrealistically high. Attention mechanisms can isolate an anomalous fact in synthetic noise far more easily than in realistic documents.

For a reliable evaluation, the noise must consist of semantically coherent material within the same domain as the question. When testing a legal search application, the context is filled with authentic court rulings or statutory articles. When testing code analysis, actual source code files are used.

This also involves a specific pitfall for Dutch-language texts. Dutch makes extensive use of long compound words and complex clause structures in which verbs are placed far apart. Tokenizers often split long Dutch compounds into multiple sub-tokens. As a result, the absolute token volume per unit of information increases compared to English. If a keyword in the middle of the text gets chopped up into rare sub-tokens, this further lowers attention activation, which can noticeably reinforce the lost-in-the-middle effect in Dutch-language texts.

Scoring mechanisms and evaluation errors

Determining whether an answer is correct seems trivial for factual questions, but it has subtle methodological pitfalls. There are three primary scoring methods:

• Exact string match (regex): Checks whether the exact key value is present in the output. This is fast and deterministic, but can produce false negatives if the model paraphrases the answer.
• F1 score over sub-tokens: Calculates the overlap between the generated tokens and the reference answer. Offers more nuance, but can be misleading with negated sentence openings.
• LLM-as-a-Judge: An external evaluation model judges whether the generated answer is semantically correct based on the source needle. This catches paraphrases, but introduces its own margin of error and additional token costs.

A common mistake is that a model starts hallucinating based on the surrounding background text instead of using the specific needle. If the scoring method only looks for terms that also appear elsewhere in the document, a hallucination gets incorrectly marked as correct. The needle should therefore always contain a unique identifier or key value that doesn't appear anywhere else in the test context.

Statistics, repetitions, and sample size

A single measurement per cell in the matrix is statistically worthless. LLMs exhibit slight non-deterministic variation due to floating-point rounding in parallel computations. Moreover, attention focus depends heavily on the exact token positions of surrounding punctuation.

Each cell in the test matrix requires multiple independent repetitions with randomized needle values and different segment orderings of the background text. See the guidelines on statistics for LLM evaluations for calculating how confidence intervals behave as sample size increases.

When a cell shows moderate accuracy based on a few runs, the actual confidence interval is wide. Only with enough repetitions per cell can sufficient resolution be obtained to determine with certainty whether a drop at the middle depth differs significantly from the score at the edges.

Practical measurement setup in Python

Below is a modular Python script that runs a systematic depth test. The script generates a context, injects a unique needle at a specified percentage of the total length, calls the model under test via a standardized API interface, and evaluates the result.

import math
import random
import time
from typing import List, Dict, Tuple

# Constanten voor de evaluatiematrix
DEPTH_PERCENTAGES = [0.0, 0.25, 0.50, 0.75, 1.0]
CONTEXT_LENGTHS = [4000, 8000, 16000, 32000]
RUNS_PER_CELL = 10

def generate_background_corpus(target_tokens: int) -> str:
  """Genereert achtergrondtekst op basis van alinea's."""
  base_paragraph = (
    "De inspectie van industriële installaties vereist periodieke controle "
    "van hydraulische systemen, sensordata en mechanische toleranties. "
    "Afwijkingen in drukprofielen moeten direct worden gelogd in het "
    "centrale onderhoudsregister conform de geldende veiligheidsnormen.\n\n"
  )
  words_needed = int(target_tokens / 1.3)
  paragraph_words = len(base_paragraph.split())
  repeats = math.ceil(words_needed / paragraph_words)
  return (base_paragraph * repeats)[:words_needed * 6]

def inject_needle(
  haystack: str,
  needle: str,
  depth_fraction: float
) -> str:
  """Plaatst de naald op de relatieve diepte van het document."""
  paragraphs = haystack.split("\n\n")
  total_p = len(paragraphs)
  insert_idx = int(total_p * depth_fraction)
  insert_idx = min(max(insert_idx, 0), total_p)
  
  paragraphs.insert(insert_idx, needle)
  return "\n\n".join(paragraphs)

def evaluate_response(response: str, secret_key: str) -> bool:
  """Controleert op deterministische aanwezigheid van de sleutel."""
  return secret_key.lower() in response.lower()

def run_depth_benchmark(
  model_name: str,
  api_client
) -> Dict[Tuple[int, float], float]:
  results = {}
  
  for length in CONTEXT_LENGTHS:
    for depth in DEPTH_PERCENTAGES:
      successes = 0
      for run_id in range(RUNS_PER_CELL):
        secret_id = f"NL-VAL-{random.randint(100000, 999999)}"
        needle = (
          f"BELANGRIJK DOSSIERKENMERK: Het unieke autorisatienummer "
          f"voor deze specifieke sessie is {secret_id}."
        )
        prompt_question = (
          "Wat is het unieke autorisatienummer voor deze sessie? "
          "Geef uitsluitend de code terug."
        )
        
        raw_haystack = generate_background_corpus(length)
        full_context = inject_needle(raw_haystack, needle, depth)
        
        messages = [
          {"role": "system", "content": "Je bent een feitelijke data-assistent."},
          {"role": "user", "content": f"{full_context}\n\nVraag: {prompt_question}"}
        ]
        
        response = api_client.chat.completions.create(
          model=model_name,
          messages=messages,
          temperature=0.0,
          max_tokens=50
        )
        
        output_text = response.choices[0].message.content
        if evaluate_response(output_text, secret_id):
          successes += 1
          
      accuracy = successes / RUNS_PER_CELL
      results[(length, depth)] = accuracy
      print(f"Lengte: {length} | Diepte: {depth*100}% | Score: {accuracy*100:.1f}%")
      
  return results

Token budgeting and evaluation costs

Benchmarking long contexts is financially and computationally expensive. A matrix of multiple lengths and depths quickly contains dozens of cells. With multiple repetitions per cell, this results in hundreds of separate API calls.

See the analysis on the cost of evaluation in LLM applications for strategies to manage the test budget without sacrificing statistical power, using staged sampling and local proxy models.

Validating architectural mitigations

Once measurement results show that a model suffers from the lost-in-the-middle effect, the model doesn't need to be discarded right away. The measurement serves as a baseline for validating architectural mitigations.

The most effective intervention is reordering documents before adding them to the prompt. Instead of placing documents linearly in order of relevance, they are sorted according to an alternating pattern. This puts the most relevant chunks at the very beginning and the very end of the context, exactly where the attention mechanisms perform best.

See the article on preventing lost-in-the-middle with smart context ordering for concrete implementations of such sorting algorithms in Python. After implementing such an algorithm, the exact same measurement protocol is run again. Only when the depth matrix shows a balanced pattern across all positions is the application ready for safe production deployment.

From test matrix to decision

Measuring the lost-in-the-middle effect turns vague assumptions about context windows into hard technical facts. A model that claims to support huge numbers of tokens may in reality only perform reliably up to a fraction of that when information is evenly distributed across the input.

By periodically quantifying the performance curve across different lengths and depths, you avoid costly production errors and make an informed choice between pure prompt input or a hybrid architecture with dynamic document reranking.