When we evaluate the performance of a Large Language Model (LLM), we want to measure the model's reasoning ability and intelligence, not its photographic memory. Yet, that is exactly what often happens. Because public models are trained on unimaginably large amounts of internet data, there is a high probability that known test questions were already in the model's training set. This problem is known as data contamination or a data leak.
If a model is presented with a text it already knows during evaluation, this results in an artificially high score. This makes comparing models via public benchmarks increasingly difficult. In this article, we discuss how to build a robust, custom test set that allows you to evaluate models objectively, without the risk of data contamination.
What is data contamination in the context of LLMs?
Data contamination occurs when the data used to test a model is accidentally (or intentionally) included in the dataset on which the model was trained. In traditional machine learning, training data is strictly separated from test data. With LLMs, which are trained on massive collections of web pages, books, code repositories, and forums, this separation is particularly difficult to maintain.
Datasets like Common Crawl contain billions of web pages. If researchers publish a benchmark somewhere on the internet, it will sooner or later be picked up by web crawlers and included in the next generation of training datasets. The model "memorizes" the answers to the test. As a result, the model performs excellently on that specific benchmark, but fails in real, new situations.
The shortcomings of public benchmarks
Public benchmarks are widely used to rank models on leaderboards. While they are useful as a general indication, they have significant drawbacks for specific business applications. Models optimize implicitly (and sometimes explicitly) for these specific sets. A model that scores highly on a well-known math or programming benchmark is not guaranteed to perform proportionally well on your organization's codebase or specific customer service data.
In addition, it is almost impossible to prove that a closed, commercial model (whose training data is secret) has not had access to public test sets. The only foolproof solution to truly test a model is to use a privately held, unique test set that has never seen the public internet.
Strategies for building a private test set
Creating an uncontaminated test set requires a careful approach. There are several strategies to obtain qualitative data that has remained out of reach of public crawlers.
1. Using non-public company data
The most effective method is using your own, shielded company data. These are documents that live behind firewalls and in secure databases. Examples include:
- Internal wiki pages and technical documentation.
- Anonymized customer service tickets and email correspondence.
- Closed codebase repositories and internal API documentation.
- Meeting minutes or internal process descriptions.
By formulating test questions based on this internal data, you ensure that the model must distill and generate the answers based on reasoning ability or via your own Retrieval-Augmented Generation (RAG) pipeline, and not from the pre-training weights.
2. Air-gapping the test creation
When you create a test set, the process itself must not lead to a data leak. For example, if you ask a public, cloud-based LLM to "come up with 50 difficult questions for my new benchmark," and you enter those questions via an unsecured web interface, you run the risk of this interaction being stored and later used for model training.
To prevent this, the creation and storage of the test set must take place in a controlled environment. This means:
- Human effort: Have domain experts manually write out complex test cases in shielded systems (for example, offline or in on-premise applications).
- Local generation: Use locally run models (open-weights) on your own hardware to generate synthetic test cases, without data flowing to the internet.
3. Dynamic and procedural data generation
A static test set can still leak over time. A robust approach is to generate a dynamic test set. Instead of hardcoded questions and answers, you write a script (for example, in Python) that continuously adjusts the variables in the questions.
Suppose you are testing the mathematical capacity of a model. Instead of the static question "Calculate the compound interest on 1000 euros after 5 years at 4%", you use a template where the deposit, term, and percentage are generated procedurally. As long as the template is generic enough and allows for billions of combinations, it is impossible for the model to have memorized the specific calculations.
Risk management when evaluating models
Even if you have built the perfect, private test set, the evaluation itself can cause a leak. As soon as you send your carefully constructed test set via an API to an external model, you need to be sure what the provider does with that data.
Therefore, always use enterprise API endpoints with a strict zero-data-retention policy. Consumer interfaces (such as web chats) often use user input for further training by default. For an in-depth overview of secure connections and configuring API requests that respect your data, please refer to our documentation on secure API integrations.
Comparison: Public benchmarks versus Custom test sets
The choice between using existing benchmarks or developing your own set depends on your goals. Below is a brief overview:
| Feature | Public Benchmarks | Custom Private Test Set |
|---|---|---|
| Risk of Data Contamination | Very high | Very low (if correctly isolated) |
| Setup Costs & Time | Low (immediately available) | High (requires manual work or complex scripts) |
| Relevance to Business | General, often academic | Specific and directly applicable to the use case |
| Comparability | Easy to compare with the market | Only internal comparison possible |
Structure of a good test item
A professional test set is systematically structured, preferably in a machine-readable format such as JSON or JSONL. Each item in the test set must contain not only the input (the prompt) and the expected output (the answer), but also metadata about the task type, difficulty level, and the required evaluation metrics.
{
"id": "test-fin-0042",
"category": "financial_extraction",
"difficulty": "advanced",
"input": "Extract the total revenue from the following internal quarterly report: [TEXT]",
"expected_output_format": "integer",
"expected_value": 450000,
"evaluation_method": "exact_match",
"created_at": "2026-07-25"
}By consistently applying this structure, you can automate the evaluation. You send the input to the model and systematically compare the output with the expected_value based on the chosen evaluation_method. For tasks where an exact answer is not possible (such as summaries), the expected output can be replaced by reference texts, applying more advanced techniques for quality control, such as an LLM-as-a-Judge method.
Conclusion
Building your own test set without data contamination is essential for anyone seriously implementing language models in critical business processes. Although setting up an internal, shielded benchmark takes time and effort, it provides the only reliable predictor of how a model will perform in your specific production environment. By avoiding public data, using dynamic generation, and strictly managing API traffic, you safeguard the integrity of your evaluation process and protect your organization's intellectual property.