Bootstrapping confidence intervals for LLM tests

When evaluating large language models, decisions are often made on the basis of a single point score: model A scores 84.2% accuracy and model B reaches 86.1%. Without insight into the statistical uncertainty around these measurements, however, such a comparison is misleading. LLM evaluations show inherent variance arising from sampling fluctuations in the dataset, randomness in sampling parameters and scoring deviations. By bootstrapping non-parametric confidence intervals, we transform a fragile point estimate into a robust interval that enables well-founded choices about model promotion and prompt optimization.

This article deals exclusively with the mathematical technique of bootstrapping and calculating confidence intervals over empirical test results. For determining the required sample size ahead of a test, see the guide on determining sample size for reliable LLM evaluations. Here we focus on the analysis phase: what do the collected measurement points tell us about the true underlying population performance?

Why point estimates fail in language model evaluations

Traditional software tests are generally deterministic: a unit test passes or fails. LLM evaluations resemble clinical trials or econometric measurements more closely. When we fire a test set of two hundred questions at an LLM, that set is merely a random sample from an infinite collection of possible prompts within a specific domain. A reported average reflects performance on those two hundred specific examples and nothing more.

On top of that, outcomes in generative AI are rarely normally distributed. Error patterns in tasks such as classification or extraction result in binary vectors (right/wrong), while evaluations of text generation often produce heavily left-skewed scores in which most answers score high and a small tail fails catastrophically. Classical parametric methods such as Student's t-test assume normality of the underlying population or lean on the central limit theorem. With small samples or heavily asymmetric distributions, however, parametric assumptions lead to unrealistically narrow or even physically impossible confidence bounds (such as scores above 100%).

The general theoretical basis for evaluation statistics can be found in the overview article on statistics for LLM evaluations and sampling. Bootstrapping solves the parametric problem by making no assumption whatsoever about the shape of the data distribution: the observed sample serves as its own population model.

The bootstrap principle step by step

Bootstrapping is a non-parametric resampling method introduced in 1979 by Bradley Efron. In the context of LLM benchmarks, the technique works as follows: we repeatedly draw new samples of exactly the same size from our original dataset, with replacement. On each generated bootstrap sample we recalculate the desired metric (the mean score, the F1 score or the 95th percentile, for example).

By repeating this process thousands of times, we build an empirical distribution of the test statistic. From this distribution we can then read off confidence intervals directly. The process follows four fixed steps:

1. Fix the dataset: Collect the evaluation scores of N unique test cases. This results in a vector X = [x₁, x₂, ..., xₙ].
2. Resample with replacement: Draw N elements uniformly and randomly from X, where individual elements may occur several times or not at all. This yields a bootstrap sample X* .
3. Calculate the metric: Calculate the test statistic θ* = f(X*), for example the arithmetic mean of X*.
4. Repeat and aggregate: Carry out steps 2 and 3 a total of B times (where B typically lies between 2,000 and 10,000). Store all calculated statistics in a vector Θ* = [θ*₁, θ*₂, ..., θ*B].

The 95% confidence interval can then be established directly via the percentile method by taking the ordered vector Θ* and reading off the values at the 2.5th and 97.5th percentile.

Percentile versus BCa intervals: when do you use which?

Although the percentile method is extremely simple to implement, it shows systematic shortcomings when the sample is small or when the sampling distribution is skewed. In those situations the resulting interval covers the true population value less often than the nominal level (only 91% coverage instead of the intended 95%, for instance).

To correct these deviations, we prefer the Bias-Corrected and Accelerated (BCa) bootstrap interval. The BCa method adjusts the percentile bounds through two parameters:

• Bias correction (z₀): Measures the proportion of bootstrap replications falling below the original point estimate. If the distribution is symmetrical around the estimate, z₀ = 0.
• Acceleration (a): Corrects for the degree to which the standard error of the estimator changes with the true parameter value (skewness). This factor is calculated via a jackknife approach (leave-one-out resampling across the dataset).

Paired bootstrapping for A/B tests between prompts and models

A common mistake when comparing two prompts or models is calculating two confidence intervals separately and checking whether they overlap. If interval A and interval B overlap, analysts wrongly conclude that there is no statistically significant difference. This ignores the paired correlation between the measurements: after all, both models were evaluated on the same set of test questions.

To determine whether prompt 2 genuinely outperforms prompt 1, we bootstrap the difference per test item. Let dᵢ = score(M₂, qᵢ) - score(M₁, qᵢ) be the performance difference on question qᵢ. We then draw bootstrap samples from the difference vector D = [d₁, d₂, ..., dₙ].

If the 95% confidence interval of the difference vector D* does not contain the value 0 (an interval of [+0.021, +0.084], for example), we can state with 95% certainty that model 2 is superior to model 1, even when the individual absolute intervals of both models overlapped. This paired approach eliminates item-specific variance (some questions are fundamentally harder than others) and raises the statistical power of the test considerably.

When setting up software quality controls, this method is indispensable. Read more about how to set up acceptance tests for non-deterministic output to avoid mistaking random noise for a genuine improvement in quality.

A reproducible measurement setup and code implementation

To make a bootstrap evaluation reproducible, every source of randomness must be fixed. This concerns both the model's generation phase and the statistical resampling phase. On the generative side it is essential to understand how sampling parameters influence one another; for this, see the manual on calibrating temperature and top-p for deterministic output.

In the Python implementation below we use NumPy to run a fast, vector-based non-parametric bootstrap over evaluation results. The code calculates both the absolute percentile interval and the paired difference interval.

import numpy as np

def bootstrap_ci_paarsgewijs(scores_a, scores_b, n_bootstraps=5000, alpha=0.05, seed=42):
  rng = np.random.default_rng(seed)
  n = len(scores_a)
  
  if len(scores_b) != n:
    raise ValueError("Beide scorelijsten moeten exact even lang zijn.")
  
  # Bereken item-gewijze verschillen
  verschillen = np.array(scores_b) - np.array(scores_a)
  oorspronkelijk_verschil = np.mean(verschillen)
  
  # Genereer alle bootstrap indices in een matrix (B x N)
  boot_indices = rng.integers(0, n, size=(n_bootstraps, n))
  
  # Bereken de gemiddelden over de resamples
  boot_verschillen = np.mean(verschillen[boot_indices], axis=1)
  
  # Bepaal percentielgrenzen
  ondergrens_pct = 100 * (alpha / 2)
  bovengrens_pct = 100 * (1 - alpha / 2)
  
  ci_laag = np.percentile(boot_verschillen, ondergrens_pct)
  ci_hoog = np.percentile(boot_verschillen, bovengrens_pct)
  
  # Bereken de empirische tweezijdige p-waarde rond 0
  if oorspronkelijk_verschil > 0:
    p_waarde = 2 * np.mean(boot_verschillen <= 0)
  else:
    p_waarde = 2 * np.mean(boot_verschillen >= 0)
  p_waarde = min(p_waarde, 1.0)
  
  return {
    "delta_gemiddelde": float(oorspronkelijk_verschil),
    "ci_ondergrens": float(ci_laag),
    "ci_bovengrens": float(ci_hoog),
    "p_waarde": float(p_waarde),
    "significant": bool(ci_laag > 0 or ci_hoog < 0)
  }

# Voorbeeld met binaire evaluatiedata (0 = fout, 1 = correct)
data_oud = [1, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0, 1, 1, 0, 1, 1, 1, 0, 1, 0]
data_nieuw = [1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1]

resultaat = bootstrap_ci_paarsgewijs(data_oud, data_nieuw, seed=1337)
print(f"Delta: {resultaat['delta_gemiddelde']:.3f}")
print(f"95% CI: [{resultaat['ci_ondergrens']:.3f}, {resultaat['ci_bovengrens']:.3f}]")
print(f"Significant: {resultaat['significant']}")

The reproducibility of this outcome is guaranteed by the use of np.random.default_rng(seed). For a broader view on securing consistent test results within the team, we recommend the article on reproducibility of AI evaluations .

Dutch-language issues and scoring asymmetry

When evaluating Dutch-language model output, specific linguistic phenomena arise that strongly influence the variance of test scores. Where English sentences often have an unambiguous grammatical structure, Dutch has complex rules around compound words, inflections and loanwords.

A model may, for instance, consistently fail to write long compounds as a single word (such as kwaliteitsbeheersingssysteem versus the incorrect kwaliteits beheersings systeem) or struggle with the distinction between formal and informal forms of address (u versus je/jij). If a test set contains ten questions about compound nouns and ninety general questions, a so-called 'clustered error structure' arises.

When we bootstrap in the standard way across all one hundred items, we treat each item as statistically independent. If the test set contains thematic clusters, however, ordinary resampling leads to an underestimation of the variance. In such cases stratified bootstrapping is necessary: we then draw bootstrap samples within each language category separately (10 samples from the compounds group and 90 from the general group, for example), after which we combine the results. This prevents rare but crucial Dutch language pitfalls from being under- or over-represented in the confidence bounds.

Cost and turnaround analysis of resampling

Unlike generating LLM output via APIs, bootstrapping is purely a post-processing operation carried out locally on the CPU. The costs in terms of tokens and API budget are therefore zero euros. The only investment is the computing time of the evaluation machine.

Because bootstrapping is vectorizable in NumPy or C++, generating 10,000 bootstrap repetitions over a dataset of 1,000 test items takes less than 50 milliseconds on a modern laptop. Even a more complex BCa calculation with jackknife resampling takes only a few seconds at N = 500 . Bootstrapping therefore adds virtually no overhead to a CI/CD evaluation pipeline, while enormously increasing the interpretive value of the test.

Weaknesses and preconditions of bootstrapping

Although bootstrapping is a powerful mathematical instrument, the method has strict theoretical limits that must not be ignored:

• No remedy for an unrepresentative test set: Bootstrapping measures sampling variability and nothing else. If the original test set is systematically biased, poorly annotated or missing important production use cases, the bootstrap interval will frame the wrong value with great precision (garbage in, garbage out).
• Underestimation of the tail with extremely small samples: With fewer than 30 to 40 data points (N < 30), bootstrapping cannot accurately recover the true variance. The resamples then repeat the same rare outliers too often, which can make the interval artificially narrow.
• Non-independent observations: If the evaluation data contains sequential dependencies (such as successive turns in a multi-turn dialogue, or time series), standard bootstrapping violates the i.i.d. assumption (independent and identically distributed). In such scenarios, block bootstrapping must be deployed to keep the correlation structure intact.
• Right-bound truncation with perfect scores: When a model achieves a score of 100% on every item in the sample, each bootstrap resample will likewise yield 100%. The bootstrap interval then becomes [1.0, 1.0] with a variance of zero. This creates the false illusion of perfect certainty, while the true population error rate with a finite sample is simply not exactly zero (the so-called 'rule of three' states that with 0 errors across N tests, the 95% upper bound on the error rate lies around 3/N .

Conclusion: from scoreboard to scientific decision-making

Reporting bare percentages in LLM benchmarks belongs in marketing material, not in serious software engineering. By integrating non-parametric confidence intervals into evaluation reports, we make it visible whether a measured quality difference rests on chance or on an actual structural improvement.

The combination of paired bootstrapping and robust BCa intervals allows development teams to make firm statements about prompt changes, model migrations and quantization losses at minimal computational cost. Anchoring this statistical discipline prevents costly regressions and forms the foundation of a mature AI development process.