Benchmarks for Dutch-language model output
Quantifying the performance of large language models happens for the most part on the basis of English-language evaluation sets. When an organization wants to deploy a model for Dutch-language applications, these English figures fall short. A high score on an English-language reasoning test guarantees neither correct processing of Dutch grammar nor a proper understanding of the Dutch and Belgian legal, cultural or administrative context.
Measuring model quality specifically for Dutch requires insight into existing multilingual data sets, the limitations of translated tests and the design of your own evaluation methods. To get a broad picture of how language models score on Dutch in general, you can consult the overview of how to test the Dutch language proficiency of models.
Existing multilingual benchmarks with a Dutch segment
In the academic literature and the open-source community, various test collections are available that contain a Dutch component. These collections vary widely in their design, the type of source material and the ultimate goal of the measurement.
MMLU (Massive Multitask Language Understanding) translations
MMLU is one of the most widely used standards for testing the knowledge and reasoning capabilities of a model across dozens of domains. To measure performance in other languages, automated and semi-automated translations of MMLU into Dutch have been made. These tests mainly measure encyclopedic knowledge and multiple-choice reasoning. As a rule, the Dutch items came about through machine translation of the original English-language questions, sometimes supplemented with a manual review round.
Belebele
Belebele is a parallel reading comprehension benchmark developed by Meta. The dataset consists of short stories with accompanying multiple-choice questions, available in dozens of languages including Dutch. The dataset is designed to compare directly how a model processes the same text and questions in different languages. The questions were rendered by professional translators, which makes the grammatical quality higher than in purely machine-translated datasets.
Fleurs
Fleurs is a sub-dataset of the Massive Multilingual Speech project and focuses primarily on speech recognition and text-to-speech. For written text evaluation, the text component of Fleurs is sometimes used to measure how well a model recognizes the sentence structure and vocabulary of a language. The Dutch items consist of parallel sentences that have been checked by translators.
XGLUE and related multilingual sets
XGLUE is a collection of tasks put together to assess multilingual representational models (such as mBERT or XLM-RoBERTa) and generative models. The tasks range from news classification and question-answer pairs to recognizing grammatical errors. The Dutch components in XGLUE come partly from existing multilingual scraping corpora and partly from translations of English-language data sources.
| Benchmark | Type of measurement | Origin of Dutch items |
|---|---|---|
| MMLU (NL translation) | Knowledge and reasoning (multiple choice) | Machine translation of English-language source |
| Belebele | Text comprehension and reading comprehension | Professional human translation |
| Fleurs | Language and speech processing | Human translation and parallel corpora |
| XGLUE | Classic NLP tasks and classification | Combination of web scraping and translation |
Why English-language scores do not translate to Dutch
It is a misconception that a model that scores well on English-language evaluation sets automatically performs just as well in Dutch. During processing by the language model, shifts occur that affect the final output.
Translation loss and cultural noise
When a model reasons internally via English-language representations, or when a question has been translated directly from English, translation loss occurs. Concepts in English do not always have a direct one-to-one relationship with Dutch terms. A legal term such as indictment does not simply translate to Dutch or Belgian legal practice without losing context.
Forms of address and register
Apart from in specific dialects, English has no strict distinction between the informal you and the formal you. In Dutch, the use of je/jij versus u is essential for the tone and the intended audience. Models that are trained primarily on English-language data often switch randomly between the je and u forms within a single Dutch paragraph, which strongly reduces usability in business environments.
Compounds and morphology
Dutch makes frequent use of closed compounds, such as langetermijnplanning or klimaataanpassingsstrategie. In English these concepts are written as separate words (long term planning). Many tokenizers that are optimized for English texts cut Dutch compounds into a large number of separate subwords or tokens. This not only increases token consumption, but also degrades the model's ability to understand the semantic coherence within long words.
Regional differences: BE-NL versus NL-NL
Dutch has clear regional variations. Vocabulary, idiom and sentence structure in Flanders (Belgian Dutch) differ on specific points from the norm in the Netherlands (Netherlandic Dutch). English-language benchmarks ignore these nuances completely, which means a model can score high on a general test while coming across as unnatural to a specific regional audience. If the main goal of the evaluation is comparing written texts in specific styles, we refer to the analysis of Dutch-language content comparison.
Examples of task differences between English and Dutch
A task that is trivial in English, such as identifying anagrams or counting the number of letters in a word, becomes extra complex in Dutch because of the way the tokenizer handles compound words and diacritics. Summarizing official documents is also relatively straightforward in English thanks to a uniform structure, but in Dutch it requires a specific understanding of Dutch public administration terminology.
The gap in the existing collections: machine translations
Many publicly available Dutch benchmarks have been translated automatically from an English-language source set. This brings substantial quality problems that undermine the reliability of the evaluation.
Characteristics of contaminated item quality
Machine-translated test sets almost always show specific patterns by which the low quality can be recognized:
- English sentence structure and anglicisms: Sentences follow the English word order, such as placing verbs too early or translating idiomatic expressions literally (for example "dat maakt zin" instead of "dat heeft zin").
- Loss of question logic: With multiple-choice questions, the translation can turn answer options into synonyms of each other, so that several answers are correct or the right answer is no longer among the options.
- Incorrect tokenization of punctuation: Translation engines sometimes change quotation marks or the formatting, which leads to parser errors in automatic processing.
When specific translation errors have to be measured quantitatively with advanced metrics, you can use the guide to evaluating translation quality with COMET and BLEURT.
How to build a Dutch test collection yourself
To measure the actual performance of a model for your specific applications, building your own, authentically Dutch-language test collection is the most reliable route.
Writing your own items instead of translating
The starting point for a high-quality test set is that the items are drawn up directly in Dutch by native speakers. This prevents English language structures from creeping into the questions and makes sure the idiom being tested is natural.
Choosing three essential categories
A balanced Dutch test set should consist of at least the following three categories:
- Domain-specific knowledge: Questions that test knowledge of Dutch legislation, local regulations or specific organizational processes.
- Instruction following: Complex assignments in which the model has to meet specific formal requirements, such as drawing up a letter in a fixed format with specific sections.
- Language sensitivity and style: Assignments that test whether the model strikes the right tone, such as rewriting a legal text to B1 level or handling the u form correctly.
Using a fixed scoring rubric
Every item in the test set must come with a clear scoring rubric. Determine in advance what earns an answer 0, 1 or 2 points. Record explicitly which criteria are assessed, such as grammatical correctness, the presence of the required key facts and adherence to the requested style.
The pitfall of your own writing style
When drawing up test items by hand, there is a risk that the creator unconsciously treats his or her own writing style, word choice or sentence structure as the norm. If only one person writes the test set, the benchmark does not so much measure the general level of Dutch as the model's ability to imitate that specific author. So always involve several authors in the process.
When putting together your own evaluation set, it is also necessary that the questions do not end up in the training data of the models. You can read an extensive explanation of how to prevent this in the article on setting up a test set without data leaks.
Interpreting existing NL benchmarks
When you read reports or publications about Dutch benchmarks, it is important to analyze the results critically before you make strategic choices for a specific model.
Representativeness of items
Check which type of texts are included in the benchmark. A benchmark that consists mainly of summarizing Wikipedia articles says very little about how well a model performs when answering customer questions about insurance policies.
Coverage of the test
Analyze how many unique topics and language constructions the test covers. A set of 50 questions can give a quick indication, but it is not enough to determine with statistical certainty whether one model is better than another for business use.
When a score says nothing about Dutch
If a benchmark only checks whether the final answer (option A, B, C or D, for example) is correct, the test measures the internal knowledge of the model, but not the quality of the generated Dutch text. A model can choose the right answer "A" and still produce poor Dutch when it is asked to write out the reasoning.
Assessing NL output with LLM-as-a-judge
Automatically assessing generated texts by having a larger or stronger language model act as the assessor (the so-called LLM-as-a-judge method) is popular because of its scalability. In the Dutch language area this brings specific challenges.
Deviating error patterns in Dutch
An English-language assessor LLM judges Dutch texts through English glasses. This leads to specific deviations:
- Tolerance for anglicisms: An English-language assessment model does not recognize literal translations from English as errors, because the underlying semantics are logical to the model.
- Incorrect assessment of B1 level: Models have difficulty estimating whether a Dutch text actually meets B1 language level, because the definition of plain language differs per culture and language structure.
- Strictness on punctuation versus style errors: Assessors sometimes attach too much weight to small typos, while they completely overlook inconsistencies in the u/je form.
Calibrating the assessor
To deploy an LLM reliably as a Dutch assessor, the system must be calibrated against human assessments. Take a sample of at least 100 generated answers, have them assessed by human experts, and run exactly the same set through the assessor LLM. Adjust the instructions (prompts) of the assessor until the scores of the model match the human judgments sufficiently. For further discussion of the dynamics between multilingual instructions, consult the documentation on multilingual prompts on the community.
Costs and time investment of evaluation
Evaluating Dutch-language model output brings costs in the form of token consumption and human hours. It is useful to weigh this investment in advance against simply adopting an existing English-language benchmark score.
Calculation example of your own NL evaluation
Suppose an organization develops its own test set of 200 specific questions to compare three different models. The costs and hours break down as follows:
- Development of the test set: 20 to 30 person-hours for writing the questions and the reference answers and drawing up the assessment rubric.
- Execution and API costs: Generating answers for 200 questions by 3 models yields 600 results. With an average length of 500 tokens per answer and using a high-quality model as the assessor (LLM-as-a-judge), the total API costs are usually between €15 and €50, depending on the chosen assessment model.
- Human validation and sampling: 5 to 10 person-hours to check the assessments of the LLM judge on a sample of 10% of the data.
Compared with adopting English-language MMLU scores for free, your own evaluation requires a clear investment. In return, the outcome is directly applicable to your own operations and the reliability of the results for the Dutch language is many times higher.
Further reading
- Testing the Dutch language proficiency of models — An overview of how you can analyze the general language command of models.
- Evaluating translation quality with COMET and BLEURT — A deeper dive into measuring translations quantitatively via automated metrics.
- Setting up a test set without data leaks — Practical steps to prevent your test questions from ending up in training datasets.
- Dutch-language content comparison — Comparative methods for written Dutch texts and styles.
- Multilingual prompts on the community — Exchange of experiences and best practices on steering multilingual models.
- Guide to better Dutch — Pointers for optimizing the language structure in automated systems.
