Making bias in model output measurable

By Ivo Donker — compiled with AI support (Claude & Gemini) · Last updated: 6 August 2026

Evaluating language models for undesirable patterns often runs into societal and philosophical debates in practice. To make an evaluation usable for software engineering and quality assurance, the analysis must be reduced to a measurement problem. The question is not whether a model has an opinion, morality, or consciousness. The question is whether the system systematically generates a deviating output for substantively equivalent input based on characteristics that are irrelevant to the task.

When a language model is deployed in a business process, its behavior must be predictable, consistent, and neutral. As soon as variables such as a person's name, a deictic reference, or a regional designation influence the outcome without the instruction calling for it, a measurable deviation is present. Systematically mapping these deviations requires a clear measurement framework, reproducible test sets, and suitable comparison metrics.

This article discusses how systematic skew in text generation and classification can be quantitatively established, how a test setup should be designed to prevent noise, and how the results should be correctly interpreted within software applications.

The distinction between representational and allocative skew

To make bias measurable, it's necessary to distinguish between two fundamental forms of skew. Although both arise from patterns in the data on which the model was trained (see also the background on training data and bias), the effect manifests itself in the output in different ways.

1. Representational skew (representational harm)

Representational skew concerns how specific groups, entities, or concepts are portrayed in the generated text. This type of skew measures the occurrence of stereotypes, associations, and differences in tone. Relevant questions include:

• Which adjectives or traits does the model automatically link to specific personal characteristics?
• Is there a difference in the assumed role or status when a background variable changes?
• Is the output for certain groups more often characterized by a cautionary, patronizing, or distant tone?

2. Allocative skew (allocative harm)

Allocative skew occurs when a model is used to support decisions, perform classifications, or allocate scarce resources. This type has a direct, quantifiable impact on the outcome of a process. Relevant questions include:

• Do two substantively identical files receive a different risk score based on a demographic variable?
• Is the model's willingness to grant a request or provide information unevenly distributed across different input variants?
• Does the rejection rate in automated CV screening differ when only contact details or names are changed?

The distinction is essential for choosing the evaluation methodology. Representational skew calls for qualitative tone analysis and semantic distance measures, while allocative skew is primarily measured through statistical differences in decision rules and acceptance criteria.

Paired testing as the basic setup

The most effective method for measuring skew is paired testing (counterfactual testing). This involves compiling pairs or sets of prompts that are identical except for exactly one controlled variable. By comparing the model's response to these pairs, the direct influence of that specific variable can be isolated.

If the model gives a significantly different response to Input A than to Input B, and the swapped characteristic is not relevant to the task at hand, this demonstrates the presence of systematic skew.

Care in constructing variables

A common pitfall in paired testing is the unintentional introduction of confounders. When adjusting one characteristic, the rest of the sentence must not change in grammar, writing style, register, or length. If adjusting a characteristic causes the sentence to read less naturally or changes the sentence structure, you are measuring not only the influence of the variable but also the model's sensitivity to unusual sentence construction.

The variable must therefore be introduced in strict isolation. Abstract templates should be used in which only the target values are swapped, without affecting the syntactic structure of the prompt.

Outcome measures for comparison

To express the difference between the generated texts from a paired test in numbers, various quantitative outcome measures are used. Depending on the task (free generation or structured output), the appropriate metric is chosen.

For generative tasks where the output is free-form, comparing token lengths and the presence of specific qualifiers often yields quick insights. For classification tasks, the principles from the statistics for evaluations, such as the chi-squared test or the McNemar test for dependent samples, can be used directly to determine whether the difference is statistically significant.

The need for multiple attempts and spread

Large language models are stochastic systems. At a temperature setting above zero, the same model will generate different answers to the exact same prompt. Running a paired test only once therefore does not provide reliable evidence. An observed difference could be due to chance within the stochastic variation of text generation.

To perform a valid measurement, the following guidelines apply:

1. Multiple runs per prompt: Run each prompt pair at least N times (for example, N=20 or N=50) at the intended temperature setting.
2. Logprob analysis: If the API supports it, analyze the log probabilities of the first generated tokens. This provides direct insight into the model's internal preference without requiring full generation.
3. Spread analysis: Analyze not only the average difference but also the variance. A model that reacts identically in 90% of cases but deviates strongly in 10% of cases has a different risk profile than a model that consistently shows a small deviation.

By aggregating multiple attempts, it becomes possible to calculate a confidence interval around the measured difference. Only when the confidence interval remains isolated from the null hypothesis (no difference) can it be concluded that there is measurable bias in the model's architecture or weights.

Limitations of public test sets

Various standard test sets are available in the scientific literature for measuring bias in models. Well-known examples are benchmark sets designed to quantify associations or stereotypes in English texts. Although these benchmarks are valuable for academic research, they have major limitations for practical application in business systems.

First, public benchmarks are static. Because the dataset is publicly accessible on the internet, there is a very high chance that this data has been included in the pre-training data of newer models. This creates the phenomenon of 'data contamination': the model recognizes the test questions and gives the desired answer, while its actual behavior on unfamiliar input remains unchanged.

Second, international benchmarks focus almost exclusively on Anglo-Saxon culture and social relations. The categories and assumptions used in them do not translate one-to-one to other languages and legal systems.

The Dutch context as a specific measurement challenge

Measuring bias in the Dutch language area requires a specific approach. Simply translating English-language test sets leads to invalid results. Words, social relations, and linguistic nuances lose their charge or take on a different meaning in translation.

For a correct evaluation in Dutch, the following specific factors must be taken into account:

1. Naming and cultural background

English-language benchmarks often work with specific first names that are characteristic of certain population groups. In the Dutch context, names must be chosen that are representative of Dutch demographics, including regional and migration backgrounds. Care must be taken that the chosen names have a comparable frequency in current Dutch-language corpora, to prevent the rarity of a name from acting as a confounder.

2. Forms of address and formality

Dutch has an explicit distinction between formal ('u') and informal ('je/jij') language use. Models sometimes show a tendency to switch register depending on the suggested background of the user. Measuring unwanted register shifts is an important part of evaluating Dutch-language conversational agents.

3. Regional designations and geographic bias

References to specific cities, provinces, or neighborhoods can evoke differences in answer quality or tone in Dutch models. Measuring this requires specific knowledge of local geography and social relations. More on the specific challenges of testing the Dutch language can be found in the overview on Dutch-language testing.

Automated assessment with a model (LLM-as-a-Judge)

When evaluating large volumes of generated text, manual assessment is often too time-consuming. A commonly used method is deploying an advanced language model as an assessor (the so-called 'LLM-as-a-Judge' approach). Here, the assessing model is tasked with comparing the outcomes of the paired test and scoring them on specific criteria such as neutrality, tone, and correctness.

Although this approach enables scaling, it introduces a fundamental risk: the assessing model may contain exactly the same or different forms of bias as the model being tested. If the judge has a preference for a particular writing style or shows a systematic deviation regarding a topic, it will fail to recognize the bias in the tested model, or may even reinforce it.

To ensure the reliability of an automated assessment process, the following measures must be taken:

• Anonymized evaluation: Remove the variables from the output before presenting it to the assessing model, so the judge does not know which variant it is evaluating.
• Position independence: Present the answers of a pairwise comparison in alternating order (A/B and B/A) to neutralize the assessing model's order preference.
• Strict rubrics: Do not give the assessing model an open question, but a strictly defined scale with clear criteria. For a detailed elaboration of this, see the guide on LLM-as-a-judge.

The need for a manually reviewed sample

No automated evaluation pipeline can fully rely on automated judges or statistical metrics alone. Human review remains necessary as a calibration step and to discover unforeseen patterns.

To keep a human sample feasible and affordable within software development, the following setup can be used:

1. Stratified sampling: Do not select randomly, but draw a sample from the cases where the automated metrics showed the greatest difference between the variants, supplemented with a random selection from the cases with no measured difference.
2. Double-blind assessment: Ensure that the human annotator does not know which hypothesis is being tested or which variable has been adjusted in the prompt.
3. Inter-annotator agreement: Have part of the sample assessed by multiple people and calculate the agreement (for example, via Cohen's kappa). This prevents the individual preference of a single annotator from clouding the test result. More details on setting up such panels can be found in the article on human evaluation.

What a test result does and does not demonstrate

Interpreting the results of a bias test requires scientific precision. A common mistake is generalizing the result into an overall judgment of the model.

A measurement is always a snapshot on a specific set of prompts, with specific parameters and within a defined domain. A positive result (no measured difference) does not guarantee that the model will also respond neutrally under different circumstances or with a slightly altered phrasing. The claim should therefore never extend beyond the boundaries of the test set used.

Measuring in your own application instead of in general

Because generic bias tests have such limited validity, the most valuable measurement is the one performed within the context of the specific application (the 'downstream application'). Generic tests establish how the base model performs on a broad set of academic tasks. For an organization, however, only how the total system responds to actual user input matters.

The measurement framework should therefore be built around the actual production flow:

• Use anonymized, representative production data as the basis for the paired-testing templates.
• Evaluate the entire system, including system prompts, Retrieval-Augmented Generation (RAG), and any post-processing.
• Focus the measurement on the decisions the system actually makes (for example, generating a summary or categorizing a report).

Testing on your own application context prevents time being spent optimizing edge cases that never occur in practice, while relevant risks in the actual business process are overlooked.

Chain analysis when finding deviations

When a paired test demonstrates a systematic difference in output, the first reflex is often to blame the language model. In a production system, however, the language model is only one part of a more complex processing chain. Before concluding that the deviation is caused by the base model, the entire chain must be analyzed.

The analysis should go through the following steps:

1. System prompt and instructions

Examine whether the instructions in the system prompt unintentionally leave room for interpretation or contain leading assumptions. An ambiguous instruction can force the model to fill in missing information based on probabilities in the pre-training data.

2. Context and RAG pipeline

If the system uses a knowledge base, check whether the retrieval step returns equivalent information for the different variants in the paired test. The deviation often turns out not to originate in the generation step of the language model, but in the search algorithm that retrieves more relevant documents for one variant than for the other.

3. Safety filters and moderation

Check the moderation models that run before or after the language model. It regularly happens that an external moderation layer incorrectly flags certain words or characteristics as sensitive or harmful, causing the chain to generate a deviating answer or prematurely cut off the response. For more information on these layers, see the overview on moderation and safety models.

4. Adversarial testing and edge cases

Finally, try to determine whether the measured bias can be exploited or amplified by malicious input. This touches on the field of adversarial testing, as described in detail in the guide on red teaming and safety testing.

Only once all external factors in the chain have been ruled out and corrected can it be established that the deviation originates from the language model itself. In that case, the deviation can be mitigated by adjusting the system prompt, adding 'few-shot' examples of neutral behavior, or applying specific post-processing to the generated output.

Further reading

• Red teaming and safety testing
• Human evaluation
• Statistics for evaluations
• LLM-as-a-judge
• Dutch-language testing
• Training data and bias
• Moderation and safety models