Anyone working professionally with Large Language Models (LLMs) quickly runs into a fundamental problem: how do you know for sure if a new prompt, a new model, or a modified generation architecture is actually better than the previous one? In practice, we often see developers run ten or twenty test questions through a model. If the output looks better at first glance, the change is immediately pushed to production.
This approach is statistically highly risky. Language models are inherently non-deterministic; they produce variable output. A small sample size can therefore easily mislead you. A temporary 'lucky shot' on a handful of questions is then mistaken for a structural improvement. To build robust and reliable AI applications, we must approach LLM evaluations as a statistical process. In this article, we cover the essential statistics for evaluations in plain English: from determining the right sample size to calculating confidence intervals and significance.
Suppose you have a customer service bot and you decide to adjust the system prompt to make the bot friendlier and more factual. You come up with fifteen typical customer questions and test them with the old and the new prompt. The old prompt answers twelve correctly (80%). The new prompt answers fourteen correctly (93%). A clear winner, right?
Not necessarily. With a sample of only fifteen questions, it is statistically very possible that this 'improvement' is pure coincidence. If you repeat the test tomorrow with fifteen different questions, the old prompt might suddenly perform better. We call this phenomenon noise. The smaller your test set, the greater the impact of noise on your final result. Without enough data, you are essentially gambling with the quality of your application. Especially if you want to automate decisions about A/B testing prompts, a solid statistical foundation is indispensable.
The need for statistics stems from variance. In traditional software development, code is deterministic: if you provide the same input twice, you get the exact same output. With LLMs, depending on the 'temperature' setting and the API architecture, this is often not the case. Even at a temperature of 0 (which should result in the most likely output), we still sometimes see subtle differences in the generated text due to GPU-level optimizations.
Additionally, the input space is infinite. Users can ask the same question in a thousand different ways. A model that performs well on grammatically perfect questions might fail as soon as the user makes spelling mistakes. Your evaluation set is only a small window into an infinite reality. We use statistics to estimate to what extent performance within this small test window generalizes to the real world.
The most frequently asked question in LLM evaluation is: "How large should my test set be?" The honest answer is that it depends on the level of certainty you want and how small of a difference you want to be able to detect. Still, there is a mathematical way to calculate this.
If we assume a binary evaluation (an LLM answer is 'correct' or 'incorrect'), we can use the formula for the sample size of a proportion. The basic formula (without correction for finite populations) looks like this:
Let's break down these variables in plain language:
Since not everyone wants to constantly fill in formulas, we use the following rules of thumb in practice for the size of evaluation sets:
Suppose you have run a benchmark test on 200 questions and your new model achieves a score of 85% (170 out of 200 questions correct). Does this mean the actual intelligence of this model on your task is exactly 85%? No. That 85% is a so-called point estimate.
A fairer and safer way to communicate model performance is through confidence intervals (CI). A confidence interval indicates a range within which the 'true' performance of the model on the entire population of possible questions falls with a certain level of certainty.
For our example of 85% on 200 questions, the 95% confidence interval (calculated via the standard error of a proportion) lies roughly between 80% and 90%. This means we can be 95% confident that if we were to test the model on ten thousand or a million similar questions, the final success rate would end up somewhere between 80% and 90%.
This insight changes the way you look at benchmarks. If Model A scores 84% and Model B scores 86% on a test set of 200 questions, their confidence intervals overlap significantly. Based on this test, you simply cannot claim that Model B is superior. Statistically speaking, they are equivalent. For an in-depth guide on how to set up frameworks for robust evaluation, we recommend looking at the methodologies on build frameworks for evaluation.
When we compare the performance of two models, or two different prompts, we want to know if the difference in accuracy is statistically significant. Significance means that the probability is very small (usually less than 5%) that the observed difference occurred purely by chance (noise).
When evaluating LLMs, we almost always use paired testing. This means we run Model A and Model B on the exact same set of test questions. Because the test cases are not independent (a very difficult question will likely be answered incorrectly by both models), we cannot use a standard unpaired t-test. The correct statistical test for comparing binary outcomes on the same test set is the McNemar's test.
McNemar's test looks specifically at the questions where the models disagree. If Model A and Model B both have 150 questions correct and 20 questions incorrect, those 170 questions provide no information about which model is better. The test focuses on the remaining 30 questions: the cases where Model A was correct and B was incorrect, versus the cases where B was correct and A was incorrect.
You have a test set of 300 questions. You test a basic prompt (Model A) against an advanced Chain-of-Thought prompt (Model B). We sort the results in a matrix:
| Model B: Incorrect | Model B: Correct | |
|---|---|---|
| Model A: Incorrect | 40 (Both incorrect) | 45 (B won) |
| Model A: Correct | 15 (A won) | 200 (Both correct) |
At first glance, Model A went from 215/300 (71.6%) to Model B with 245/300 (81.6%). An increase of 10 percentage points. But is this significant? We look at the discrepancies. Model B won 45 times where A failed. Model A won 15 times where B failed.
Without going too deep into the formula: the resulting p-value (the probability that this difference is due to chance) is extremely low (p < 0.001). Because the p-value is much smaller than our threshold of 0.05 (5%), we can state with great certainty that the difference is statistically significant. The Chain-of-Thought prompt is indeed an improvement, and not a lucky shot.
An additional complication in modern evaluations is that we increasingly use another LLM (often GPT-4 or Claude 3.5 Opus) to evaluate the answers of our models. This is called LLM-as-a-judge. This solves the problem of manual review, but introduces a new layer of variance and potential bias.
The evaluator LLM itself is also not 100% accurate or consistent. Sometimes the 'judge' evaluates an answer as perfect today, and the same answer as mediocre tomorrow. This extra noise means your confidence intervals become even wider. If you work with automated evaluators, you must take into account that a small gain (for example, +2%) can be completely negated by the margin of error of the judge itself. Striving for absolute reproducibility in your evaluation pipeline (for example, by setting temperature to 0 and averaging scores over multiple evaluations) is crucial to keep this variance in check.
To avoid getting lost in the math, you can use these practical rules of thumb when building your evaluation infrastructure:
Statistics is not a luxury in the era of generative AI; it is your only line of defense against the chaos of non-deterministic models. By working with fixed confidence intervals, robust sample sizes (towards 400+ for key decisions), and significance tests like McNemar's, you transform LLM evaluation from a subjective "vibe check" into a hard, quantifiable science.