Filtering synthetic evaluation data by information density

When a language model generates its own test material, a subtle validity problem arises: synthetic datasets look impressive in terms of volume, but often contain negligible information density. Large language models have a strong tendency toward stylistic verbosity, repetitive syntactic patterns, and trivial questions. If this synthetic noise ends up unfiltered in an evaluation set, a benchmark doesn't measure the reasoning power or domain knowledge of the tested model, but purely its ability to reproduce shallow templates. The decision this article prepares you to make is determining the exact cutoff at which synthetically generated prompt-and-answer pairs have a sufficient signal-to-noise ratio to serve as a formal benchmark.

This article differs strictly from methods that collect datasets from operational applications; read the foundational piece on converting production logs into evaluation data to understand how real interactions differ from purely generated distributions. Where production data struggles with privacy risks and missing annotations, synthetic data struggles with a lack of informational depth and invisible redundancy. By setting up a quantitative filtering process based on information density, you prevent your evaluation infrastructure from processing thousands of tokens that add no discriminating power to the model comparison at all.

The problem of semantic dilution in generated tests

Synthetic data generation via a generator LLM generally produces text with high grammatical fluency but low propositional density. This phenomenon is known as semantic dilution: a prompt uses a hundred tokens to describe a logical relationship that would have fit in twenty. When a model under test is evaluated on such prompts, the benchmark mainly measures whether the model can withstand unnecessary padding, rather than testing complex inference or factual synthesis.

A second complication is the distributional shift toward the middle of the generating model's probability distribution. By default, models generate text that reflects the average patterns of their training data. As a result, the jagged edge cases, obscure jargon, and unpredictable constructions that characterize human data are missing. If we want to build a robust test, we must ruthlessly cut synthetic records that fall below a certain information threshold.

Metric definitions for information density

To purify synthetic test data, objective metrics are required that can be calculated automatically. Three pillars form the basis for this quantitative filtering:

Propositional extraction deconstructs a prompt into minimal statements that can independently be true or false. A question such as "Given that the contractor, which is statutorily based in Utrecht, must respond within fourteen days..." contains three atomic propositions (the contractor is based in Utrecht, a response deadline applies, the deadline is fourteen days). If a 150-word paragraph contains only two propositions, the density is extremely low and the prompt functions as noise in the benchmark.

To prevent models from falling back on shallow structures when generating summarization and evaluation tests, the iterative compression method offers a solution; see the technique of Chain-of-Density for information-dense text to study how entity density is maximized step by step without losing readability. These principles can be directly reversed to assess incoming test items on their entity-to-token ratio.

The filter pipeline: from raw generation to a purified benchmark

The filtering process runs through four consecutive stages. Each stage eliminates a specific class of synthetic degradation, starting with cheap heuristics and ending with computationally heavier semantic analysis.

Stage 1: Surface-level compression and redundancy check

In the first step, prompts are tested for compressibility using standard algorithms such as zlib or gzip. Text with an abnormally high compression ratio contains repetitive token patterns and superficial filler sentences. This is an extremely efficient pre-filter that eliminates 15% to 25% of low-quality synthetic output without any model calls.

Stage 2: Semantic clustering and density measurement

In the second step, the test questions are converted into vectors via an embedding model. We then calculate local density using k-nearest neighbors ($k\text{-NN}$). Test questions that sit in extremely dense clusters are variations on exactly the same template. For each cluster, we select only the item with the highest variation in named entities.

The danger of data contamination lurks constantly here; consult the analysis on building a test set without data leaks to make sure that the examples used as seeds for the synthetic generator weren't already present in the pre-training data of the LLMs being tested.

Stage 3: Logical constraint density

A high-quality benchmark question contains specific constraints: boundary conditions, exclusion criteria, or contradictory source fragments. Using a parser, we check for the presence of logical operators, numeric conditions, and conditional clauses ("unless," "provided that," "only if"). Questions without constraints test only surface-level association and are rejected.

Stage 4: Verifying solvability via an arbiter

When density is artificially boosted, there's a risk that a question becomes logically inconsistent or unsolvable. In this stage, a strictly calibrated model assesses whether the question can be answered unambiguously based on the provided context. For the fine mechanics of this verification step, see the method for LLM-as-a-judge calibration, which covers systematic judge bias and scoring methods.

Practical example: Python filter for propositional density

The script below demonstrates how a dataset can be automatically filtered based on lexical diversity (Type-Token Ratio) and the density of unique content words (nouns, verbs, numbers) relative to function words.

import re
from typing import Dict, List

def analyseer_informatiedichtheid(tekst: str) -> Dict[str, float]:
    woorden = re.findall(r'\b\w+\b', tekst.lower())
    totaal_woorden = len(woorden)
    if totaal_woorden == 0:
        return {"dichtheid": 0.0, "ttr": 0.0, "geslaagd": 0.0}
    
    # Eenvoudige lijst van Nederlandse functiewoorden (stopwoorden)
    functiewoorden = {
        "de", "het", "een", "en", "van", "ik", "te", "dat", "die", "in", 
        "is", "op", "niet", "met", "zijn", "voor", "maar", "er", "om", "als"
    }
    
    inhoudswoorden = [w for w in woorden if w not in functiewoorden]
    unieke_woorden = set(woorden)
    
    # Metrieken berekenen
    inhouds_ratio = len(inhoudswoorden) / totaal_woorden
    ttr = len(unieke_woorden) / totaal_woorden
    
    # Samengestelde informatiedichtheidsscore (0.0 tot 1.0)
    dichtheidsscore = (inhouds_ratio * 0.6) + (ttr * 0.4)
    
    # Filterdrempel: minimaal 55% inhoudelijke lading en variatie
    is_geschikt = 1.0 if (dichtheidsscore >= 0.55 and totaal_woorden >= 20) else 0.0
    
    return {
        "totaal_tokens": float(totaal_woorden),
        "inhouds_ratio": round(inhouds_ratio, 3),
        "ttr": round(ttr, 3),
        "score": round(dichtheidsscore, 3),
        "geslaagd": is_geschikt
    }

# Illustratieve testdata
voorbeelden = [
    "Het is van groot belang dat we kijken naar de manier waarop dingen gebeuren.",
    "Artikel 7:900 BW vereist een vaststellingsovereenkomst ter beëindiging van een bestaand geschil."
]

for vb in voorbeelden:
    resultaat = analyseer_informatiedichtheid(vb)
    print(f"Tekst: {vb[:40]}... -> Score: {resultaat['score']} (Geschikt: {bool(resultaat['geslaagd'])})")

Pitfalls in the Dutch-language context

When filtering Dutch-language synthetic data, specific linguistic complications arise that directly affect density measurements:

First, long compound words (such as aansprakelijkheidsverzekeringsmaatschappij or uitvoeringskwaliteitseisen) lead to distortion in standard word counters. A model that correctly writes compounds as a single word appears, according to simple token counters, to have a lower propositional density than a model that makes English-style spacing errors (aansprakelijkheid verzekering maatschappij). Filters must therefore use morphological decomposition to correctly assess compounds.

Second, the verb-final position in subordinate clauses introduces a syntactic spread that simple n-gram models can mistakenly interpret as 'high entropy.' A strict separation between syntactic complexity and semantic density is essential to prevent grammatical constructions from being confused with factual content.

Third, translated synthetic data often leads to anglicisms and loan translations (such as "maakt zin" instead of "heeft zin" or "aan het einde van de dag" (a calque of "at the end of the day") used as filler). These constructions inflate the token count without any propositional added value. A specific filter for common Dutch AI clichés is therefore a mandatory part of the pipeline.

Reproducibility and measurement setup

To publish an evaluation set that holds up scientifically and operationally, filtering must be fully deterministic and reproducible. This requires fixing four hard parameters:

1. Fixed seed value and determinism: All random sampling within clustering and generation must be pinned to an explicit seed (for example seed=42), with temperature=0.0 for verification steps.

2. Repetitions and stability: Every density measurement via embedding models must be validated for permutation invariance: changing the order of input items must not affect the resulting cluster boundaries.

3. Scoring criteria and cutoff values: The rejection thresholds (such as the 0.55 cutoff in the code example) must be fixed in advance in a configuration file and must not be adjusted afterward to force a specific dataset size.

4. Calibrating the arbiter: If an LLM is used for propositional extraction, the agreement (inter-annotator agreement via Cohen's Kappa) between the model and a human expert must reach at least $\kappa \ge 0.80$ on a minimum of 100 samples before the filter may run autonomously.

Cost and throughput analysis in tokens and time

Implementing an advanced filtering pipeline carries infrastructure costs, but delivers a substantial downstream saving during the actual model benchmarks.

Let's analyze a raw synthetic dataset of 10,000 generated prompts. A complete filter run goes through the following stages:

Stage 1 (heuristics & gzip) processes 10,000 items in about 4 seconds on a standard CPU core, with no API costs, and reduces the set by 20% to 8,000 items.

Stage 2 (embedding clustering) converts 8,000 items into vectors. At an average length of 250 tokens per prompt, this costs about 2,000,000 embedding tokens. Local processing via a compact model takes roughly 45 seconds on a modern accelerator. This reduces the dataset via deduplication to 5,000 items.

Stage 3 & 4 (propositional extraction and validation) processes the remaining 5,000 items via a structured model call. This costs about 1,250,000 input tokens and 500,000 output tokens. The turnaround time is 3 to 5 minutes at typical API throughput rates.

The end result is a purified set of 2,500 high-quality test questions. Although the filtering process requires computation time and tokens once, it saves 75% in inference costs and evaluation time on every subsequent benchmark round (where dozens of candidate models are tested). Moreover, the statistical reliability of the final benchmark scores increases considerably because noise and repetition have been eliminated.

Weaknesses and inherent limitations

Every automated filter introduces systematic blind spots that must be explicitly acknowledged:

Over-penalizing domain-specific conventions: In legal and medical domains, redundant and formal language is sometimes legally required or functionally necessary. An overly aggressive density filter can wrongly classify and remove legitimate contractual clauses or medical disclaimers as 'padding.'

Favoring compressed writing styles: Models that answer extremely concisely score artificially higher on propositional density metrics, even when their answers are too terse for human use. The filter measures information density, not pedagogical or communicative quality.

Circular bias from generator-arbiter overlap: If the same model type is used for both data generation and information-density assessment, a blind spot emerges for that model family's specific stylistic flaws. Always use structurally different architectures for generation and validation, therefore.