Calculating inter-annotator agreement for LLM juries

Deploying language models to judge the quality of other models saves hundreds of hours of manual review, but at the same time introduces a critical measurement problem: how do you validate whether an automated judgment is reliable? Without statistical testing of the inter-annotator agreement (IAA), an automated jury remains an opaque black box. We then run the real risk of mistaking random model variance, prompt sensitivities, or inherent biases for genuine quality differences in the application under test.

This article covers the mathematical and practical methods for quantifying the reliability of LLM juries. The focus here is not on designing prompts for evaluation, but purely on the statistical justification: how do you calculate whether two jury models, or a language model and a human expert, are sufficiently aligned to justify decisions about production deployments? Anyone wanting to review the conceptual foundations of automated judges can first study the background in Setting up LLM-as-a-Judge Below, we work through the formulas, worked examples, measurement protocols, code implementations, and language-specific pitfalls step by step.

Why percentage agreement is misleading

The most intuitive way to determine agreement between raters is the raw proportion: the number of times the jury members reach exactly the same verdict, divided by the total number of items evaluated. In software evaluations, however, this simple calculation almost always leads to a dangerous overestimation of actual reliability.

The core problem is the lack of a correction for chance agreement. As soon as a test set involves significant class imbalance — which is standard in practical AI evaluations — raw agreement inflates enormously. Suppose we test a RAG system for the occurrence of hallucinations, and 90 percent of the generated answers in the benchmark are factually correct. If we were to deploy two completely random jury models that blindly assign the label "Correct" 90 percent of the time, an agreement of over 80 percent would arise purely from probability:

P(beide toevallig 'Correct') = 0,90 * 0,90 = 0,81
P(beide toevallig 'Fout')    = 0,10 * 0,10 = 0,01
Verwachte toevalskans (Pe)   = 0,81 + 0,01 = 0,82

In this scenario, a naive measurement reports 82 percent agreement, creating the illusion of a solid evaluation pipeline. In reality, the jury panel adds no informative value whatsoever. To determine whether an automated judgment actually has discriminative power, we need to switch to chance-corrected coefficients, in which the observed agreement ($P_o$) is weighed against the agreement expected by chance ($P_e$).

Cohen's kappa for pairwise categorical scores

For situations where we compare exactly two raters across a fixed set of categories — for example an LLM jury versus a human domain expert, or two different LLM models against each other — Cohen's kappa ($\kappa$) is the classic standard. The coefficient expresses what proportion of the potential agreement above chance level has actually been achieved.

The mathematical definition is:

kappa = (Po - Pe) / (1 - Pe)

Here, $P_o$ is the observed proportion of agreement and $P_e$ is the expected agreement under the assumption of statistical independence between the raters. The possible values of $\kappa$ range from -1 to +1:

• $\kappa = 1.0$: Perfect, deterministic agreement between both annotators across all categories.
• $\kappa = 0.0$: The agreement is exactly equal to what would be expected based on chance alone, given the marginal distributions.
• $\kappa < 0.0$: Systematic disagreement; the raters contradict each other more often than would be expected by chance.

In this illustrative example, a seemingly acceptable raw score of 82 percent drops back to a modest $\kappa$ of approximately 0.46. This mercilessly exposes that a substantial share of the agreement was driven by the dominant class. To determine whether your sample size is large enough to draw reliable conclusions about these kappa values, the overview on statistics for evaluations provides the necessary mathematical frameworks around sample sizes and power calculations.

Weighted kappa for ordinal evaluation scales

Not all LLM evaluations are binary. Many benchmarks use a Likert scale (for example 1 to 5 stars for answer quality) or a rubric with quality gradations such as Poor, Fair, Good, and Excellent. With such ordered categories, a standard unweighted kappa is too strict: a disagreement between a score of 4 and a score of 5 is penalized just as heavily as a fundamental conflict between a score of 1 and a score of 5.

Weighted kappa ($\kappa_w$) solves this by introducing a weight matrix $w_{ij}$ that quantifies the distance between categories $i$ and $j$. The formula uses the disagreement matrix:

kappa_w = 1 - (som(w_ij * O_ij) / som(w_ij * E_ij))

Here, depending on the task, we choose between two commonly used weighting schemes:

• Linear weighting: The weight $w_{ij} = |i - j| / (k - 1)$. The penalty for disagreement grows linearly with the number of steps of difference. Suitable when each step on the scale represents a proportional increase in quality difference.
• Quadratic weighting: The weight $w_{ij} = (i - j)^2 / (k - 1)^2$. Small deviations of one point receive a relatively light penalty, while extreme contrasts are penalized disproportionately heavily. This scheme is mathematically equivalent to the intraclass correlation coefficient (ICC).

Krippendorff's alpha for larger panels and missing data

When we deploy a multi-member jury — for example an ensemble of three different LLM models combined with two human annotators — Cohen's kappa falls short. Fleiss' kappa can handle multiple raters, but requires that every item be evaluated by exactly the same number of raters, and it doesn't support missing data points or ordinal weights.

Krippendorff's alpha ($\alpha$) is the most robust and universal metric for advanced LLM evaluation architectures. The metric has crucial properties for modern benchmarks:

• Supports any number of raters ($\ge 2$).
• Handles missing annotations seamlessly (when an LLM call fails due to a rate limit or timeout, the entire test case doesn't need to be discarded).
• Supports different levels of measurement (nominal, ordinal, interval, ratio) via custom difference functions $\delta^2(c, k)$.
• Remains reliable for both very small and very large samples.

The general form of Krippendorff's alpha is:

alpha = 1 - (D_o / D_e)

Where $D_o$ is the observed disagreement within the units and $D_e$ is the disagreement expected by chance across all observations made. In the scientific literature, the general rule is: $\alpha \ge 0.80$ indicates a highly reliable measurement setup; values between $0.667$ and $0.800$ allow for cautious conclusions, and at $\alpha < 0.667$ the evaluation framework should be rejected and recalibrated.

Python implementation for automated validation

In a continuous evaluation pipeline, we calculate these statistics automatically over the logged jury scores. The script below demonstrates the calculation of nominal kappa, weighted ordinal kappa, and Krippendorff's alpha using scikit-learn and krippendorff.

import numpy as np
from sklearn.metrics import cohen_kappa_score
import krippendorff

# Voorbeelddata: 10 testcases beoordeeld op een schaal van 0 (Fout) tot 3 (Uitstekend)
# Beoordelaars: Expert (Mens), LLM_Rechter_A, LLM_Rechter_B
mens_expert   = [3, 2, 0, 1, 3, 0, 2, 3, 1, 2]
llm_rechter_a = [3, 2, 0, 2, 3, 0, 1, 3, 1, 2]
llm_rechter_b = [3, 1, 0, 1, 2, 0, 2, 3, 1, 2]

# 1. Cohen's Kappa (ongecorrigeerd vs lineair gewogen) Mens vs LLM A
kappa_ongecorrigeerd = cohen_kappa_score(mens_expert, llm_rechter_a)
kappa_lineair = cohen_kappa_score(mens_expert, llm_rechter_a, weights="linear")
kappa_kwadratisch = cohen_kappa_score(mens_expert, llm_rechter_a, weights="quadratic")

print(f"Mens vs LLM A - Nominale Kappa:     {kappa_ongecorrigeerd:.3f}")
print(f"Mens vs LLM A - Lineair Gewogen:    {kappa_lineair:.3f}")
print(f"Mens vs LLM A - Kwadratisch Gewogen:{kappa_kwadratisch:.3f}")

# 2. Krippendorff's Alpha over het voltallige panel (inclusief ontbrekende data)
# Matrix-formaat voor krippendorff-library: shape (aantal_beoordelaars, aantal_items)
# Stel voor dat LLM B een time-out had op item index 4 (aangeduid als np.nan)
llm_rechter_b_met_nan = [3, 1, 0, 1, np.nan, 0, 2, 3, 1, 2]

betrouwbaarheids_matrix = np.array([
    mens_expert,
    llm_rechter_a,
    llm_rechter_b_met_nan
])

alpha_ordinaal = krippendorff.alpha(
    reliability_data=betrouwbaarheids_matrix,
    level_of_measurement="ordinal"
)

print(f"Panel Krippendorff's Alpha (Ordinaal): {alpha_ordinaal:.3f}")

When the calculation shows that agreement between the LLM jury and the human raters is lagging, the cause is often not the model itself, but ambiguous instructions in the evaluation protocol. A thorough revision of the human evaluation guidelines ensures unambiguous rubrics, which directly increases both human inter-rater reliability and correlation with LLM juries.

Biases that artificially influence agreement

When analyzing inter-annotator agreement for LLMs, we need to watch out for systematic artifacts. Language models exhibit specific cognitive biases that can contaminate the calculated statistics in two ways: by artificially inflating agreement, or by unjustly deflating it.

1. Position bias

In pairwise comparisons (where a judge must choose between Answer A and Answer B), virtually every language model has a strong preference for the candidate in the first position, regardless of substantive quality. If we run two identical model instances on the same prompts without swapping the order, both models will choose option A more often. The calculated kappa rises significantly as a result, but it measures nothing more than shared positional preference.

To neutralize this effect, every comparison must be evaluated symmetrically: once as $(A, B)$ and once as $(B, A)$. Only when a model consistently chooses the same candidate regardless of order does the verdict count as a valid annotation. See the methods for measuring and neutralizing position bias for a full analysis of swap inconsistency.

2. Length bias (verbosity bias)

LLM juries structurally award higher scores to longer, verbose answers with complex formatting (bullet points, bold headings), even when a shorter answer is factually more accurate and concise. Two different models from the same model family often show exactly the same length sensitivity, which leads to high mutual agreement that does not correlate with human experts who prefer conciseness.

3. Self-preference (self-enhancement bias)

A model acting as a judge generally rates texts generated by its own model architecture or family more highly than texts from competing architectures. If a benchmark deploys LLM-X to compare LLM-X with LLM-Y, a structural deviation from neutral human annotators arises, resulting in a low external kappa.

Dutch-language quirks in jury agreement

When performing reliability measurements on Dutch-language evaluation sets, specific linguistic frictions arise that put pressure on inter-annotator agreement between models and native speakers:

• Forms of address and pragmatics: In Dutch, the distinction between formal "u" and informal "je/jij" is context-dependent. Many internationally trained LLMs incorrectly regard the informal "je" form in business or technical contexts as a stylistic error or a lack of professionalism, while a Dutch panel judges this as perfectly natural. This produces systematic scoring conflicts.
• Compound words and the 'English disease': Models primarily optimized on English-language corpora struggle with long Dutch compound words (for example kwaliteitsbeoordelingsprotocol) and regularly split them incorrectly with spaces. If an LLM jury fails to recognize such spacing errors, or wrongly approves them, discrepancies with human language experts arise.
• Translation interference and idiom: Literal translations of English expressions ("dat maakt geen zin" instead of the correct "dat heeft geen zin", both meaning "that doesn't make sense") are often overlooked by an LLM jury because the underlying semantics seem logical to the model, while a human reviewer rejects the answer immediately.

To verify that the factual claims in a Dutch-language text hold up before the style assessment takes place, it's advisable to integrate the procedures for fact-checking AI answers into the annotators' validation chain.

Costs, sample size, and computational overhead

Measuring inter-annotator agreement increases the computational cost of an evaluation campaign. Where a naive evaluation performs a single LLM call per test case, a scientifically sound measurement setup requires at least two independent model evaluations plus symmetric position swaps.

Token consumption scales according to the following parameters:

To keep these costs manageable without sacrificing methodological rigor, we apply a two-step approach: we perform the full agreement calculation (including human calibration) on a random sample of 15 to 20 percent of the test set. Only once the calculated Krippendorff's alpha on this sample exceeds the threshold of $\alpha \ge 0.75$ is the automated configuration rolled out across the remaining 80 percent of the evaluation volume.

Practical protocol for continuous quality monitoring

Establishing inter-annotator agreement is not a one-time exercise when setting up a benchmark. LLM providers regularly update their weights and API endpoints, which can cause the behavior and scoring standards of an LLM jury to shift silently (model drift).

A robust validation protocol consists of four fixed phases:

1. Establish a calibration set: Build a permanent gold standard of at least 150 representative test cases, manually annotated by at least two human experts with a mutual agreement of $\kappa \ge 0.80$.
2. Determine the baseline: Before the benchmark starts, calculate the weighted kappa between the LLM jury and the gold standard.
3. Enforce symmetric scoring: Run all automated comparisons twice with swapped input order to isolate positional variance.
4. Periodic regression test: Re-evaluate the calibration set monthly with the same LLM prompts. If the calculated $\alpha$ drops by more than 0.05 relative to the baseline, the jury prompt must be recalibrated.

By embedding inter-annotator agreement as a fixed quality threshold, you transform a subjective and shaky model evaluation into a reproducible, statistically grounded measurement instrument.