Calibrating temperature and top-p for deterministic output

When building evaluation pipelines and automated extraction tasks, how sampling parameters are tuned directly determines whether a language model delivers reliable, repeatable answers. When a benchmark runs two different runs on the same test set, the difference in score should result from a substantive prompt or model change, not from random noise in token selection. In this article, we determine what measurement setup is needed to calibrate sampling parameters for maximum predictability and reproducibility.

This article differs distinctly from general overviews of reproducibility of AI evaluations by not focusing on the entire test environment, but zooming in specifically on the mathematical interaction between softmax temperature, nucleus sampling (top-p), seed values, and floating-point non-determinism on GPU clusters. Where the fundamental theory behind the mathematical definitions can be found in the background article on sampling parameters such as temperature and top-p, this guide focuses on concrete quantitative measurement protocols for eliminating variance.

The interaction between temperature and top-p

At each generation step, a language model produces a vector of raw logit values across the entire vocabulary. Before a token is actually selected, these logits are transformed into a probability distribution via the softmax function, scaled by the parameter temperature (T). As T approaches 0.0, the distribution approaches a sharp peak: the token with the highest logit gets a probability of nearly 100 percent. This is called greedy decoding.

Nucleus sampling, or top-p, operates at a different level: it sorts all tokens by descending probability and selects the smallest set of tokens whose cumulative probability reaches the threshold p. All tokens outside this cumulative threshold are cut off (their effective probability is set to zero), after which the remaining probabilities are renormalized. If temperature is set to 0.0, top-p in theory has no effect anymore, since the highest logit already claims all the probability. In many API implementations, however, manipulating both values simultaneously leads to subtle rounding errors or unexpected sampling behavior if the gateway internally filters first and normalizes afterward.

For evaluations and deterministic tasks, the rule of thumb is: never dynamically adjust both knobs at the same time. Choose either pure greedy decoding (temperature 0.0), or a calibrated temperature combined with a fixed top-p cutoff (top-p 1.0), unless a truncated distribution is explicitly needed to suppress rare outliers in the logit tails.

Why temperature 0 can still show variation

A common assumption among developers is that temperature: 0 by definition leads to 100 percent identical output across repeated runs. In production environments with hosted LLM APIs, this regularly turns out not to be the case. Even with greedy decoding, we sometimes see differing words, different punctuation, or altered JSON structures between consecutive calls with the exact same payload.

This phenomenon has three main technical causes:

• Non-associative floating-point addition: Modern GPU architectures perform matrix multiplications in parallel via vectorized floating-point operations (FP16, BF16, or FP8). Because floating-point addition is not strictly associative (the sum of a and b plus c is not necessarily exactly equal to a plus the sum of b and c, due to rounding differences at the lowest bit level), a varying order of parallel threads can lead to tiny fractional differences in the final logit value. If two competing tokens have nearly identical probabilities, this minimal rounding difference can tip the ranking of the greedy selection.
• Mixture-of-Experts (MoE) routing and dynamic batching: In models that use MoE architectures, tokens are dynamically routed to different expert networks. If the serving engine uses dynamic batching, the exact composition of the batch depends on competing requests on the cluster. Under heavy load, tokens can be distributed across different experts or routed through fallback paths, causing tiny numerical fluctuations.
• Quantization and kernel variation: Large model providers run their workloads across heterogeneous GPU fleets. A request might land on a cluster with a specific Triton kernel one second and on hardware with a different optimization pass the next, resulting in small variations in the softmax outcome.

Quantitative measurement setup: the determinism index

To determine how deterministic a model is under specific parameter settings, we need to use a reproducible measurement setup. We measure consistency by calling the same set of prompts N times over a fixed time interval and calculating the pairwise equality of the generated token sequences.

We define three concrete metrics for this:

1. Exact Match Rate (EMR): The percentage of runs in which a run's output is identical, character-for-character and token-for-token, to the baseline run.
2. Levenshtein Similarity (LS): The normalized edit distance between text sequences, which quantifies how close non-identical outputs are to each other on a scale from 0 to 1.
3. Semantic Cosine Stability (SCS): The cosine similarity between the vector embeddings of the generated texts, used to check whether the actual meaning shifts with small textual mutations.

When performing such analyses, it's essential to ensure statistical reliability by taking a sufficient number of samples. See the foundational piece on statistics for LLM evaluations to calculate how many repetitions per prompt are needed to reliably distinguish random GPU jitter from structural parameter effects.

A step-by-step plan for calibrating sampling parameters

To calibrate a model configuration for a specific pipeline, we follow a systematic four-step protocol. We record all environmental factors before drawing conclusions about the optimal settings.

Step 1: Isolating the baseline via greedy sampling

Start every calibration round by setting the parameter temperature at 0.0 and top-p at 1.0. If the provider supports a seed value (such as the seed parameter), fix it at a constant integer (for example 42). Run the complete evaluation set at least 10 times over a 24-hour window to account for peak hours and changing GPU nodes.

Step 2: Determining the tolerance threshold for the use case

Not every application requires an EMR of 100 percent. Determine in advance what margin of error is acceptable:

• Strict schema extraction: Requires 100 percent syntactic validity. Here, the word choice in free-text fields may vary slightly as long as the JSON parser validates without errors. Always use validated interfaces as described in the guide on reliable structured output and JSON.
• Text classification and labeling: Requires 100 percent label consistency. A small variation in the reasoning (chain-of-thought) is acceptable as long as the final category label remains unchanged.
• RAG synthesis: Requires semantic stability with high embedding similarity. Small synonym swaps are allowed as long as no factual hallucinations occur.

Step 3: Testing top-k and top-p thresholds

If a model still drifts at temperature 0.0 due to logit ties (two tokens with nearly equal probability), test a combination with a tighter top-p threshold (such as top-p 0.85) or top-k filtering (such as k equal to 1 through 5). This forces the decoder to rigorously cut off the tail of the probability distribution before parallel floating-point operations can influence the ranking.

Step 4: Recording it in automated regression tests

Once the optimal configuration has been established, it's stored as an immutable configuration in the test suite. Changes to model checkpoints or provider routing then become immediately visible as soon as the EMR drops below the defined threshold.

Example code: measuring reproducibility and variance in Python

The Python script below shows how we can automatically quantify the variance and Exact Match Rate of a model endpoint across multiple iterations. The script performs repeated calls, compares the outputs, and reports on stability.

import hashlib
from typing import List, Dict, Any

def bereken_determinisme_metrics(outputs: List[str]) -> Dict[str, Any]:
    """
    Kwantificeert de mate van determinisme over een lijst van gegenereerde teksten.
    """
    if not outputs:
        return {"error": "Geen data"}
    
    totaal = len(outputs)
    unieke_hashes = set()
    hash_frequenties: Dict[str, int] = {}
    
    for tekst in outputs:
        # Genereer een SHA-256 hash van de genormaliseerde tekst
        genormaliseerd = tekst.strip()
        h = hashlib.sha256(genormaliseerd.encode("utf-8")).hexdigest()
        unieke_hashes.add(h)
        hash_frequenties[h] = hash_frequenties.get(h, 0) + 1
        
    meest_voorkomende_aantal = max(hash_frequenties.values())
    exact_match_rate = (meest_voorkomende_aantal / totaal) * 100.0
    
    return {
        "totaal_runs": totaal,
        "unieke_varianten": len(unieke_hashes),
        "exact_match_rate_pct": round(exact_match_rate, 2),
        "is_volledig_deterministisch": len(unieke_hashes) == 1
    }

# Simulatie van een testset-run over 5 opeenvolgende calls
test_antwoorden = [
    '{"status": "succes", "categorie": "financieel", "score": 0.95}',
    '{"status": "succes", "categorie": "financieel", "score": 0.95}',
    '{"status": "succes", "categorie": "financieel", "score": 0.95}',
    '{"status": "succes", "categorie": "financieel", "score": 0.95}',
    '{"status": "succes", "categorie": "financieel", "score": 0.95}'
]

resultaat = bereken_determinisme_metrics(test_antwoorden)
print(f"Exact Match Rate: {resultaat['exact_match_rate_pct']}%")
print(f"Aantal varianten: {resultaat['unieke_varianten']}")

When we integrate this script into continuous integration tests, deviations are logged immediately. If the endpoint suddenly returns different outputs for the same input under load, this can indicate a changed backend routing at the API provider. For a broader analysis of concurrency effects, see the article on Evaluating JSON validity under load.

Dutch-language pitfalls in sampling variation

When evaluating Dutch-language prompts, specific linguistic phenomena occur that increase sensitivity to sampling parameters compared to English. This has to do with how the Dutch language is represented in the tokenizers of most large models.

First, Dutch words more often split into multiple subtokens per word than English terms. Compound words such as aansprakelijkheidsverzekering or uitvoeringsbesluit get split into 3 to 5 separate tokens. If the temperature is just slightly above 0.0 (for example, temperature 0.3), the uncertainty over consecutive subtokens accumulates. A small deviation in the second subtoken forces the model to complete an entirely different compound word, resulting in a disproportionately large edit distance.

Second, formal and informal forms of address (such as u versus je/jij) are often semantically equivalent in Dutch, causing their respective logits to lie very close together. With the slightest floating-point jitter, the model can switch forms of address partway through a paragraph:

• Run 1 (temperature 0): "...opsturen naar uw adviseur." (formal "u")
• Run 2 (temperature 0, under GPU jitter): "...opsturen naar je adviseur." (informal "je")

Although the actual content is identical, this results in an Exact Match Rate of zero percent. In test frameworks that analyze prompts for consistency, it's therefore wise to explicitly pin system prompts to one specific style. See the research on measuring system prompt variation and consistency for techniques to contractually constrain style fluctuations.

The role of random seeds and client-side caching

Several major API providers offer a seed parameter in their REST interfaces. Sending a fixed number along with the request (such as seed: 1337) instructs the backend to aim for deterministic sampling within the same hardware configuration. Providers often attach a system_fingerprint response header to the reply.

When the system_fingerprint changes between two calls, the provider is signaling that the backend infrastructure (such as model weights, kernel versions, or quantization levels) has changed. In that case, the loss of determinism is explainable, and the cause doesn't lie in the sampling parameters themselves. For acceptance testing in production environments, monitoring this fingerprint is an essential part of the process; see the overview on setting up acceptance tests for non-deterministic output for organizational frameworks around model releases.

In addition, client-side caching is the most effective tool for guaranteeing determinism where the API falls short. By generating a hash of the payload (including model version, prompt, temperature, top-p, and seed), identical requests can be handled locally without network or GPU overhead. This reduces both latency and token costs to zero for repeated benchmark questions.

Cost and compute time of calibration tests

Systematically calibrating sampling parameters carries operational costs, since each prompt must be evaluated dozens of times across different parameter combinations. To keep the budget under control, you can work with a tiered test matrix.

A standard calibration matrix for a dataset of 100 test questions looks like this:

• Stage 1 (coarse filtering): 100 questions times 3 configurations (temperature 0.0, 0.2, and 0.7) times 3 repetitions results in 900 API calls. This maps out the general spread and sensitivity.
• Stage 2 (fine-tuning): The 20 most volatile prompts from stage 1 are selected for in-depth analysis: 20 questions times 5 top-p variations (between 0.80 and 1.00) times 10 repetitions yields 1,000 API calls.

In total, a solid calibration requires around 1,900 requests. At an average prompt length of 500 input tokens and 200 output tokens, this protocol consumes about 950,000 input tokens and 380,000 output tokens. Running this test once per quarter, or with major model upgrades, prevents hundreds of hours of debugging time in downstream applications.

Conclusion and checklist for production

Deterministic output from language models isn't a given; it's the result of careful parameter calibration and infrastructure control. By systematically testing and fixing temperature, top-p, and seed values, you build evaluations and pipelines that can withstand random noise.

Use the following checklist before putting a deterministic pipeline into production:

• Set temperature to 0.0 for extraction, JSON, and quantitative evaluations.
• Fix top-p at a fixed value (default 1.0 at temperature 0, or 0.9 with light sampling) and avoid changing both variables at the same time.
• Always send an explicit, constant seed along with the request, if supported by the provider.
• Log the system_fingerprint on every API call to detect infrastructure changes immediately.
• Define clear evaluation metrics (Exact Match Rate for code and JSON, Semantic Cosine Stability for free-text generation).
• Account for Dutch language characteristics such as compound words and forms of address when setting tolerance thresholds.