# Setting up an Elo rating system for model comparison

[Skip to content](#lm-inhoud)Network/[NL](/en/elo-ratingsysteem-opzetten-voor-paarsgewijze-modelvergelijking)EN[Hubhub.llmnet.nlCompare models on task, language, cost and licence.](https://hub.llmnet.nl/en/)[Communitycommunity.llmnet.nlPrompt techniques, patterns and system prompts.](https://community.llmnet.nl/en/)[APIapi.llmnet.nlLLMs in production: rate limits, routing, structured output.](https://api.llmnet.nl/en/)[Consultancyconsultancy.llmnet.nlRolling out AI in an organisation, pilot to production.](https://consultancy.llmnet.nl/en/)[Newsnieuws.llmnet.nlAI developments, explained for the Netherlands.](https://nieuws.llmnet.nl/en/)[Benchmarkbenchmark.llmnet.nlMeasure AI quality yourself, on your own tasks.](https://benchmark.llmnet.nl/en/)[Careersvacatures.llmnet.nlAI roles, salaries and career paths in the Netherlands.](https://vacatures.llmnet.nl/en/)[Learnleren.llmnet.nlAI concepts in plain language, beginner to builder.](https://leren.llmnet.nl/en/)[Guidegids.llmnet.nlRun AI privately on your own Mac, PC, NAS or home server.](https://gids.llmnet.nl/en/)[Directorydirectory.llmnet.nlMapping the AI ecosystem: tools, models, companies.](https://directory.llmnet.nl/en/)[Radarradar.llmnet.nlSignals from X, research and communities for indie developers.](https://radar.llmnet.nl/en/)[Appsapps.llmnet.nlReviews of AI apps and open-source repos, with tips for builders.](https://apps.llmnet.nl/en/)[llmnet.nl — main site](https://llmnet.nl/en/)[](https://x.com/intent/post?url=https%3A%2F%2Fbenchmark.llmnet.nl%2Fen%2Felo-ratingsysteem-opzetten-voor-paarsgewijze-modelvergelijking&text=Setting%20up%20an%20Elo%20rating%20system%20for%20model%20comparison)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fbenchmark.llmnet.nl%2Fen%2Felo-ratingsysteem-opzetten-voor-paarsgewijze-modelvergelijking)[](https://www.reddit.com/submit?url=https%3A%2F%2Fbenchmark.llmnet.nl%2Fen%2Felo-ratingsysteem-opzetten-voor-paarsgewijze-modelvergelijking&title=Setting%20up%20an%20Elo%20rating%20system%20for%20model%20comparison)[](#)[](https://x.com/intent/post?url=https%3A%2F%2Fbenchmark.llmnet.nl%2Fen%2Felo-ratingsysteem-opzetten-voor-paarsgewijze-modelvergelijking&text=Setting%20up%20an%20Elo%20rating%20system%20for%20model%20comparison)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fbenchmark.llmnet.nl%2Fen%2Felo-ratingsysteem-opzetten-voor-paarsgewijze-modelvergelijking)[](https://www.reddit.com/submit?url=https%3A%2F%2Fbenchmark.llmnet.nl%2Fen%2Felo-ratingsysteem-opzetten-voor-paarsgewijze-modelvergelijking&title=Setting%20up%20an%20Elo%20rating%20system%20for%20model%20comparison)[](#)

 
# Setting up an Elo rating system for pairwise model comparison

 By Ivo Donker — compiled with AI assistance (Claude & Gemini)

 When evaluating large language models for open-ended tasks — such as creative summaries, domain-specific customer service, or complex reasoning problems — absolute numeric scales (such as a grade from 1 to 10) structurally fall short. Raters scale inconsistent standards, scores are hard to calibrate, and a 7 from one annotator is a 9 from another. A pairwise comparison (A versus B) resolves this scale inconsistency by reducing the question to a binary or three-way choice: which response answers the question better, or is it a tie?

 To distill a coherent, one-dimensional ranking from hundreds or thousands of individual duels, an Elo or Bradley-Terry rating system is the industry standard. This article explains how to set up a robust pairwise measurement pipeline yourself, which mathematical formulas to use, how to neutralize systematic measurement errors, and how to calculate the required sample size. This article specifically distinguishes itself from absolute scoring methods and focuses purely on relative tournament structures between competing model versions.

 
## The foundation: from individual duels to a relative scale

 The classic Elo system, originally developed for chess by Arpad Elo, models the relative skill of entities based on the outcome of direct confrontations. When model A beats model B, model A's rating rises and model B's falls. The magnitude of the adjustment depends on the prior expectation: if a heavily favored model beats a weaker variant, the score barely changes; if the weaker model wins unexpectedly, a substantial correction occurs.

 The mathematical core consists of two steps: calculating the expected win probability and updating the scores after the observed outcome. The expected score \(E_A\) of model A against model B with current ratings \(R_A\) and \(R_B\) is calculated with a logistic function:

 E_A = 1 / (1 + 10^((R_B - R_A) / 400))
E_B = 1 / (1 + 10^((R_A - R_B) / 400))

 Once the duel is complete and the actual outcome \(S_A\) is known (where 1 stands for a win for A, 0.5 for a tie, and 0 for a win for B), the ratings are updated with a fixed factor \(K\):

 R'_A = R_A + K * (S_A - E_A)
R'_B = R_B + K * (S_B - E_B)

 The constant 400 determines the scale: a rating difference of 400 points implies that the stronger model has a theoretical win probability of about 91% against the opponent, while a difference of 0 points results in a win probability of exactly 50%.

 
## Classic online Elo versus Bradley-Terry with maximum likelihood

 In a static benchmark environment, iteratively updating classic Elo scores has an important drawback: the order in which duels are processed influences the final end score (order dependence). Moreover, the system reacts strongly to the chosen \(K\) factor. To solve this, modern LLM evaluations preferably use the Bradley-Terry model via Maximum Likelihood Estimation (MLE).

 The Bradley-Terry model postulates that each model \(i\) has a latent skill parameter \(\beta_i\), such that the probability that model \(i\) beats model \(j\) equals:

 P(i wint van j) = exp(\beta_i) / (exp(\beta_i) + exp(\beta_j)) = 1 / (1 + exp(-(\beta_i - \beta_j)))

 Instead of incremental updates, all collected duels are optimized in one batch by maximizing the combined log-likelihood of all observed outcomes via logistic regression. The resulting coefficients \(\beta\) are then linearly transformed to the familiar scale (often with an anchor value of 1000 or 1500 for the base model):

 Elo_i = 1000 + \beta_i * (400 / ln(10))

 The great advantage of Bradley-Terry via MLE is that the ranking is completely order-independent, makes optimal use of all historical data, and directly allows confidence intervals to be calculated. To determine how many duels are necessary before a measured difference is statistically significant, consult the overview on [statistics for LLM evaluations and sample sizes](https://benchmark.llmnet.nl/en/statistiek-voor-evaluaties), which details the theoretical foundation of confidence margins in model testing.

 
## Matchmaking strategies: efficient allocation of duels

 With \(N\) models, the number of unique model pairs grows quadratically according to \(N(N-1)/2\). If you want to compare 10 model variants, there are 45 unique pairs. With 50 models, this rises to 1225 pairs. Selecting pairs completely uniformly at random is inefficient: duels between an extremely strong model and a very weak model yield almost no informational value (the outcome is virtually certain), yet still cost inference tokens and reviewer hours.

 
 
 
 
 Strategy | 
 Mechanism | 
 Advantage | 
 Pitfall | 
 

 
 
 
 Uniform random | 
 Every pair has an equal chance of being drawn. | 
 Simple to implement, no selection bias. | 
 Wastes duels on uneven matchups; slow convergence. | 
 

 
 Swiss-system / K-nearest | 
 Pairs models with similar interim ratings. | 
 Maximum information gain per duel (win probability around 50%). | 
 Risk of isolated clusters; requires occasional cross-tier duels. | 
 

 
 Anchor-based (hub) | 
 New models primarily fight against calibrated anchor models. | 
 Fast positioning of new checkpoints with minimal runs. | 
 Sensitive to circular, non-transitive interactions with the anchor model. | 
 

 
 
 

 For a production evaluation, a hybrid setup works best: start with a short exploration phase (uniform random or anchor duels) to get a rough estimate, then switch to active sampling in which pairs are chosen whose absolute difference in estimated skill is as small as possible.

 
## Setting up the arbiter: human versus LLM-as-a-Judge

 The outcome of a duel (\(S_A\)) must be determined by an independent arbiter. This can be a human domain expert or a powerful automated language model. Both approaches require specific measures to ensure validity.

 When a human panel is deployed, strict annotation instructions must prevent subjective taste from taking over. For guidelines on calibrating human raters and measuring inter-rater reliability, see the article on [setting up human evaluation and annotation guidelines](https://benchmark.llmnet.nl/en/menselijke-evaluatie) which explains methods for building consensus.

 When a language model acts as referee, we must account for known systematic deviations. For an in-depth analysis of the error margins and validation methods of automated arbiters, the dossier on [LLM-as-a-Judge methodology](https://benchmark.llmnet.nl/en/llm-as-a-judge) offers indispensable background information on model choice and prompt design.

 
## Three persistent measurement errors and how to neutralize them

 Pairwise evaluations with models as judges suffer from three specific measurement errors that can heavily distort Elo scores if not explicitly corrected for:

 
### 1. Position bias

 Many language models have a strong preference for the first candidate (option A) or, conversely, the second (option B), purely because of the order in which they appear in the prompt. To neutralize this, a swap test is mandatory: each duel is run twice, with the answers swapping places on the second pass.

 If model X wins in both position A and position B, the outcome is unambiguously a win for X. If the candidate in position A wins regardless of which model is there, position bias is present and the result must be recorded as an invalid measurement or a tie.

 
### 2. Length bias (verbosity bias)

 Both human raters and LLM arbiters unconsciously associate longer, more elaborate answers with higher quality, even when the extra text is pure padding. To counter this, you can use length-normalized prompts (giving both models a strict token limit) or explicitly instruct the arbiter to reward conciseness and information density.

 
### 3. Self-preference (self-enhancement bias)

 When model family Z acts as the judge of a duel in which a model from the same family participates, the arbiter disproportionately often awards the win to its own lineage. You solve this by deploying a jury of multiple heterogeneous models (from different providers) or by choosing a model that does not itself participate in the tournament.

 
## Confidence intervals via bootstrapping

 An Elo score without a confidence interval suggests a false precision that does not actually exist. A difference of 15 Elo points between two models could be pure chance if only 100 duels have been played. To quantify the margin of uncertainty, we apply non-parametric bootstrapping .

 The bootstrapping algorithm proceeds as follows:

 
 
- Take the full dataset of \(M\) played duels.
 
- Draw a new sample of exactly size \(M\) with replacement (resampling with replacement).
 
- Calculate the Bradley-Terry Elo ratings over this resampled dataset.
 
- Repeat steps 2 and 3 at least 1000 times.
 
- Determine the 2.5th and 97.5th percentile of the distribution for each model; this forms the 95% confidence interval.
 

 Only when the confidence intervals of two models do not overlap can it be concluded with statistical certainty that one model is superior on the tested prompt set.

 
## Practical example: Python implementation with Bradley-Terry

 Below is a compact, self-contained implementation script in Python that converts a list of duel results into Bradley-Terry ratings via logistic regression (scikit-learn), including normalization to a recognizable Elo scale.

 import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression

def bereken_bradley_terry_elo(duels_df, basis_rating=1000.0, schaalfactor=400.0):
 """
 Berekent Elo-ratings via Bradley-Terry logistische regressie.
 duels_df bevat kolommen: 'model_a', 'model_b', 'winnaar' ('model_a', 'model_b', of 'tie')
 """
 # Filter gelijkspelen of splits ze in halve overwinningen
 duels = []
 for _, row in duels_df.iterrows():
 if row['winnaar'] == 'model_a':
 duels.append((row['model_a'], row['model_b'], 1.0))
 elif row['winnaar'] == 'model_b':
 duels.append((row['model_a'], row['model_b'], 0.0))
 elif row['winnaar'] == 'tie':
 # Gelijkspel telt als twee halve duels in spiegelbeeld
 duels.append((row['model_a'], row['model_b'], 0.5))

 modellen = sorted(list(set(duels_df['model_a']).union(set(duels_df['model_b']))))
 model_idx = {model: i for i, model in enumerate(modellen)}
 n_modellen = len(modellen)

 # Bouw featuresmatrix (1 voor model_a, -1 voor model_b)
 X = []
 y = []
 sample_weights = []

 for ma, mb, res in duels:
 row = np.zeros(n_modellen)
 row[model_idx[ma]] = 1.0
 row[model_idx[mb]] = -1.0
 
 if res == 0.5:
 # Gelijkspel gewogen verwerken
 X.append(row)
 y.append(1.0)
 sample_weights.append(0.5)
 X.append(row)
 y.append(0.0)
 sample_weights.append(0.5)
 else:
 X.append(row)
 y.append(res)
 sample_weights.append(1.0)

 X = np.array(X)
 y = np.array(y)
 weights = np.array(sample_weights)

 # Fit logistische regressie zonder intercept met L2-regularisatie
 lr = LogisticRegression(fit_intercept=False, C=1.0, solver='lbfgs')
 lr.fit(X, y, sample_weight=weights)

 coefs = lr.coef_[0]
 # Converteer log-odds coëfficiënten naar Elo-schaal
 elo_scores = coefs * (schaalfactor / np.log(10))
 # Centreer rond basis_rating
 elo_scores = elo_scores - np.mean(elo_scores) + basis_rating

 resultaten = pd.DataFrame({
 'Model': modellen,
 'Elo': np.round(elo_scores, 1)
 }).sort_values(by='Elo', ascending=False).reset_index(drop=True)

 return resultaten

 
## Cost and token calculation for an evaluation run

 Running a large-scale Elo tournament involves significant compute and API costs. Let's work through the token load for a tournament with 6 competing models on a dataset of 250 evaluation questions.

 
 Calculation example (indicative):
 With 6 models and a round-robin tournament, there are 15 unique pairs. With 250 questions and a mandatory swap test (2 positions per question), this results in \(15 \times 250 \times 2 = 7,500\) unique duels.
 Each duel requires:
 - Prompt input for the arbiter (question + answer A + answer B): 900 tokens on average.
 - Assessment reasoning + final verdict: 250 tokens on average.
 Total volume for the arbiter: 6.75M input tokens and 1.87M output tokens, excluding the initial generation costs of the 6 candidate models (each of which must generate 250 answers).
 

 To manage these costs, it's wise to draw up a tight budget plan in advance. In the article on [controlling the costs of evaluation in LLM applications](https://benchmark.llmnet.nl/en/kosten-van-evalueren) you'll find strategies to drastically limit the number of required API calls without sacrificing statistical power.

 When you call dozens of model variants in parallel across various providers, infrastructure complexity can quickly increase. Centrally routing these calls through an intermediate layer helps absorb rate limits and outages; see the guide on [the power of an LLM API aggregator](https://api.llmnet.nl/en/aggregator-uitleg) for technical integration patterns.

 
## Dutch-language pitfalls in pairwise model duels

 When benchmarking Dutch-language model output, specific linguistic phenomena occur that can distort pairwise ratings if the arbiter is not calibrated for them:

 
 
- Form-of-address inconsistency (je/u): Many models switch between formal and informal forms of address within a single answer. An arbiter trained on English data often misses this style error, while a human reader notices it immediately.
 
- Anglicisms and loan translations: Models regularly translate English idiomatic constructions literally into Dutch (such as "dat maakt zin" instead of "dat heeft zin"). If both models make the same mistake, the arbiter doesn't notice; if one model uses correct Dutch, the scoring prompt must explicitly instruct that natural language use weighs more heavily than technical terms.
 
- Compounds and spacing errors: The notorious English disease (writing separate words where a hyphen or single word is required, such as model evaluatie instead of modelevaluatie) occurs massively in smaller open-source checkpoints. The evaluation instruction must explicitly state whether spelling errors lead to point deductions.
 

 
## Quality control: when do you reject a tournament outcome?

 A calculated Elo ranking should never be adopted uncritically. There are four concrete rejection criteria based on which a tournament run must be declared invalid:

 
 
- High position inconsistency (> 15%): If in more than 15% of duels the outcome reverses purely by swapping the order of candidates A and B, the arbiter is insufficiently calibrated for the task.
 
- Non-transitive loops (rock-paper-scissors): If model A structurally beats B, B beats C, but C decisively beats A, the data violates the transitivity assumption of the Bradley-Terry model. This usually points to a dataset with widely diverging subdomains (for example, code versus poetry) that should not be lumped together.
 
- Excessive length correlation (\(r > 0.65\)): If the correlation between the number of generated tokens and the final Elo rating is extremely high, the tournament is probably measuring text length rather than substantive quality.
 
- Confidence interval too wide: If the 95% confidence interval of the top candidates is wider than 80 Elo points, simply too few duels have been played to base a responsible architecture choice on it.
 

 By structurally building in these checks, you create a measurement environment that enables reproducible, statistically robust, and business-critical decisions about model selection.