Determining sample size for reliable LLM evaluations

When building, fine-tuning, and deploying language models to production, one central question arises whenever a prompt or model version changes: how do we know for certain that the model is genuinely performing better, and that the observed difference isn't just random sampling error? Anyone curating an evaluation dataset must constantly strike a balance between statistical power on the one hand and practical feasibility on the other. A sample size that is too small leads to false certainty, where random noise is mistaken for performance gains. A sample size that is too large unnecessarily drives up token costs and compute time, slowing down rapid development cycles.

This article covers the mathematical and practical methodology for calculating the exact minimum sample size for both binary pass/fail tasks and continuous quality scores. We focus specifically on the statistical power analyses required to back hard production decisions. What sets this article apart from general test set guidelines is our deep dive into parametric and non-parametric sample size calculations, power functions, and variance reduction techniques, rather than focusing solely on data collection or drafting annotation guidelines.

The decision: when is a difference statistically significant?

Before applying formulas, we must clearly define which decision hinges on the evaluation outcome. In software development involving LLM components, two scenarios typically emerge. The first is a comparative A/B test between two prompt variations, model architectures, or quantization levels. The second is an absolute acceptance test: does a fine-tuned model meet a predetermined quality threshold, such as a minimum of 95 percent accuracy on an extraction task?

When a new model scores 88 percent on a random set of 50 questions and the old model scores 84 percent, it may appear at first glance to be an improvement. Statistically speaking, however, this difference of two extra correct answers in a sample of just 50 items is completely meaningless; the 95% confidence intervals of both measurements overlap heavily. Anyone jumping in without a foundational understanding of variance and reliability may want to first review the fundamentals of evaluations for beginners to understand how scoring and evaluation criteria are constructed at a fundamental level.

Determining the sample size fundamentally revolves around controlling two types of error risks. On the one hand, there is the risk of accidentally promoting an inferior model to production (Type I error or false positive, bounded by the significance level α). On the other hand, there is the risk of overlooking a real, valuable quality improvement (Type II error or false negative, bounded by β). Anyone who also wants to prevent test questions from having already been seen by the model during pre-training and distorting the calculated power should consult the guidelines on building a test set without data contamination to ensure that the calculated sample consists exclusively of clean, unseen data.

Binary evaluations: Cochran's formula and confidence intervals

Many automated LLM evaluations have a discrete, binary outcome: a generated response either meets or fails a strict validation schema, contains or does not contain a hallucination, or correctly follows a set of negative constraints. For such binary proportions, we base the sample size calculation on the binomial distribution and the central limit theorem.

When the goal is to estimate an absolute pass rate with a predefined absolute margin of error ( e) at a chosen confidence level (with z-score Z), we use Cochran's classical formula:

n = (Z^2 * p * (1 - p)) / e^2

Here, Z represents the critical value corresponding to the two-sided confidence interval (1.96 for 95% confidence; 2.576 for 99% confidence), p represents the expected proportion of successes in the population, and e represents the maximum allowable half-width of the interval. If we have no prior knowledge, we conservatively choose p = 0,5. This yields the maximum product p * (1 - p) = 0,25 , resulting in the largest possible required sample size.

If a model task is expected to have a high success rate of around 90 percent (p = 0,9), the binomial variance drops to 0,9 * 0,1 = 0,09. In that scenario, only 139 independent prompts are needed to demonstrate with 95 percent confidence that the actual score lies within the interval [85%, 95%]. For engineers who want to further study the formal proofs and mathematical background of confidence intervals, the in-depth guide on statistics for evaluations a thorough overview of hypothesis tests, z-scores, and distribution functions.

A/B comparisons and statistical power

When comparing two model checkpoints or prompt variants, estimating a single confidence interval is not enough. We need to formally test whether the difference between two proportions (p1 and p2) deviates significantly from zero. Four fundamental variables play a role here:

1. Significance level (α): the threshold for the risk of a false-positive conclusion (set by default to 0.05).
2. Statistical power (1 - β): the probability that a true existing performance difference is actually detected (typically 0.80 or 0.90).
3. Baseline quality (p1): the current accuracy level of the production model.
4. Minimum Detectable Effect (MDE, δ = |p1 - p2|): the smallest absolute performance improvement that is operationally relevant.

The required sample size per variant (n) for a two-tailed two-sample z-test for independent groups is:

n = [ (Z_alpha/2 * sqrt(2 * p_gem * (1 - p_gem)) + Z_beta * sqrt(p1*(1-p1) + p2*(1-p2)))^2 ] / (p1 - p2)^2

Where p_gem = (p1 + p2) / 2. Suppose we test a prompt change hoping to increase accuracy from 80% to 85% (α = 0.05, power = 0.80). According to this formula, approximately 1,077 unique test items per prompt variant are required, totaling 2,154 model calls. If one wants to detect a more granular difference of just 2 percentage points (from 80% to 82%), the required sample size explodes to over 6,800 items per group. To conduct such experiments in a controlled and reproducible manner, the article on A/B testing prompts explains how to evaluate variants in parallel without bias in test case selection.

Paired versus unpaired measurement setups (McNemar's test)

The classical formula for two independent samples assumes group A and group B are tested on different prompts. In the practice of software evaluation, that is exceptionally inefficient. We can use a paired measurement design : model A and model B are presented with the exact same set of test prompts.

By using the same input, we directly eliminate the external variance caused by differences in inherent difficulty across individual test prompts. The statistical test therefore shifts from a two-sample z-test to McNemar's test for paired nominal data. In a paired evaluation, we construct a 2x2 contingency table:

               | Model B: Correct | Model B: Fout |
---------------+------------------+---------------+
Model A: Goed  |        a         |       b       |
Model A: Fout  |        c         |       d       |

The concordant pairs (a and d) represent prompts where both models achieve the same result; they provide no distinguishing evidence. Only the discordant pairs (b and c) determine the test statistic. The chi-square (χ²) test statistic with continuity correction is:

chi^2 = (|b - c| - 1)^2 / (b + c)

Because models perform identically on many prompts (for instance, an 80% overlap in outcomes), a paired design often reduces the required sample size by a factor of 3 to 5 compared to an unpaired test. Where an unpaired A/B test requires 1,000 prompts per variant, a paired setup with 250 to 350 prompts can already achieve the same statistical power.

Continuous evaluations and Likert scales

Not every task evaluation can be captured in a binary 0 or 1. When measuring answer relevance, semantic embedding distance, stylistic alignment, or LLM-as-a-Judge ratings on a scale of 1 to 5, we work with continuous or ordinal metrics. For continuous variables, the minimum sample size directly depends on the population variance (σ²).

The formula for estimating a continuous population mean within an absolute margin of error E is given by:

n = (Z^2 * sigma^2) / E^2

The greatest challenge with continuous metrics is that the standard deviation (σ) is unknown in advance. An exploratory pilot measurement on 30 to 50 representative prompts is therefore a mandatory first step to empirically estimate σ. If the pilot reveals that quality scores fluctuate heavily (high standard deviation), the required sample size scales quadratically to maintain the desired precision.

For strictly structured extraction tasks, where the output must adhere to a rigid JSON schema, we can often reduce continuous quality assessments back to deterministic, binary parsing and validation rules. How to program and test such schema requirements is further detailed in the overview on reliable JSON and structured output.

Non-deterministic models and repetitions per prompt

A unique complication when quantitatively evaluating modern LLMs is non-determinism. Even when the temperature parameter is set to 0, floating-point rounding orders in parallel GPU decoding mini-batches can cause subtle variations in token probabilities. When sampling with a temperature above 0, the model introduces explicit stochasticity.

This creates a hierarchical data structure with two distinct variance components:

1. The variance between different prompts in the test set (σ²_prompt).
2. The variance between repeated generations on the same prompt (σ²_run).

The combined standard error (SE) of the estimated mean across N unique prompts, each with k repetitions, is given by:

SE = sqrt( (sigma^2_prompt / N) + (sigma^2_run / (N * k)) )

This equation exposes a crucial design principle: increasing the number of unique prompts (N) reduces both variance terms simultaneously, whereas increasing the number of repetitions per prompt (k) only dampens the run-level variance. In virtually all practical use cases, σ²_prompt is many times larger than σ²_run. It is therefore statistically superior to invest in a broad collection of, say, 300 unique prompts with k = 1 or k = 2, rather than querying 30 prompts 20 times each. To ensure that repeated runs are not skewed by changing system environments, it is advisable to follow the guidelines for reproducibility in AI evaluations when configuring hardware and parameters.

Dutch-language pitfalls and stratified sampling

A common mistake in evaluations for the Dutch language area is drawing a purely random sample from an unstructured collection of production logs. Dutch exhibits specific linguistic characteristics that are distributed very unevenly in language use, yet remain decisive for the user experience:

1. Long compound words: Models often tokenize Dutch compounds ('aansprakelijkheidsverzekering', 'klimaatadaptatiestrategie') into inefficient sub-word tokens, which can lead to hallucinations or spelling errors.
2. Forms of address and register: The choice between the formal 'u' form and the informal 'je/jij' form requires consistent context steering.
3. Regional language variation: Differences between Northern Dutch and Belgian Dutch (Flemish) lexicon and sentence structure.
4. Domain-specific jargon: The correct handling of English loanwords within a Dutch grammatical context.

When a test set of 200 questions is randomly drawn, it may happen that only two or three sentences with formal 'u' address or complex compound words end up in the sample. A sub-evaluation of that specific capability then carries a massive margin of error. The methodological solution is stratified sampling (stratified sampling). Here, we partition the population beforehand into homogeneous strata (subgroups) and calculate the minimum required sample size per stratum. This ensures that rare yet business-critical linguistic phenomena are also covered with sufficient statistical reliability.

Cost versus statistical power: a pragmatic step-by-step plan

Every additional evaluation question incurs direct API costs, latency, and potentially annotation expenses. To maintain a financially manageable evaluation strategy, we go through the following four phases:

Step 1: Determine the Minimum Detectable Effect (MDE). Establish in advance which performance difference justifies migrating to a larger model or revising the system prompt. A 1% gain rarely justifies doubling latency or costs, whereas a 5% difference often does.

Step 2: Run a pilot measurement (N = 30 to 50). Measure the baseline success rate (p) and standard deviation (σ). This replaces assumptions with hard empirical estimates.

Step 3: Calculate the definitive sample size. Apply the paired McNemar formula or Cochran's formula using the desired confidence level (95%) and target power (80%).

Step 4: Use sequential analysis (early stopping). For large-scale validation runs, interim testing can be performed using Wald's Sequential Probability Ratio Test (SPRT). If a new model performs significantly worse than the production baseline after just 80 of the planned 400 prompts, the run can be aborted immediately to conserve budget.

For an in-depth analysis of the trade-offs between token pricing, model selection, and evaluation costs, consult the overview on controlling evaluation costs.

Integration into the development and deployment pipeline

A sample size calculation is not a static, one-time exercise, but a dynamic tool within continuous integration (CI/CD). In modern engineering pipelines, we differentiate the sample size based on the validation stage:

How to integrate such automated regression tests with hard break criteria and failure thresholds into deployment workflows is explained step by step in the guide on setting up pipeline evaluations.

Conclusion and checklist for evaluation setup

Determining the right sample size transforms LLM evaluations from guesswork into reproducible science. By mathematically defining the desired confidence level, margin of error, and minimum detectable effect in advance, you prevent budget from being wasted on needlessly large test sets or decisions from being based on statistical noise.

When setting up a new test collection, always verify the following core points:

• Is the evaluation objective explicitly defined as an absolute proportion estimate or a comparative A/B test?
• Was a paired design on identical prompts chosen to eliminate noise caused by prompt variation?
• Has the variance been estimated beforehand with a representative pilot of 30 to 50 items?
• Is expanding unique prompts (N) structurally prioritized over repeated runs per prompt (k)?
• Has the dataset been checked via stratified sampling for representative coverage of specific Dutch language constructs?