Calibrating G-Eval scoring criteria with human data

Quantitatively evaluating open-ended generative texts poses a fundamental problem for developers: traditional n-gram metrics such as ROUGE and BLEU lack semantic understanding, while large-scale human review is too slow and costly for continuous development cycles. G-Eval offers a framework in which a large language model acts as an assessor using Chain-of-Thought (CoT) reasoning steps and formal scoring rubrics. The decision we make with this calibration measurement is whether an automated evaluation pipeline is reliable enough to autonomously run final regression tests and model selections without direct human oversight.

Without explicit calibration against a vetted human test set, however, G-Eval shows considerable deviations due to prompt sensitivity, scale compression, and implicit biases. This article explicitly distinguishes itself from the basic principles in the general introduction to LLM-as-a-Judge systems and the broad process description in setting up human annotation guidelines. Where those articles focus on, respectively, the abstract workings of assessor models and the logistics of human panels, this document covers the exact mathematical and prompt-engineering alignment between the two worlds.

The anatomy of the G-Eval framework

G-Eval structures qualitative assessment by splitting the evaluation process into four sequential components: the task definition, the detailed assessment criterion, a series of automatically or manually generated evaluation steps (the Chain-of-Thought), and a weighted scoring calculation. Instead of simply asking a model for a score from 1 to 5, G-Eval forces the model to first formulate explicit intermediate steps in which factual deviations, style breaks, or logical gaps are noted.

In the original implementation, G-Eval uses the probability distribution of the output tokens (token log-probabilities) to calculate a continuous score. When the model assigns a score between 1 and 5, the system calculates the normalized sum of the log-probabilities of the tokens '1', '2', '3', '4', and '5'. This yields an expected value with a decimal point (for example, 3.74) instead of a discrete integer, which considerably lowers variance and enables fine-grained ranking.

In modern API environments where direct log-probabilities aren't always consistently available, we enforce determinism via structured outputs. Anyone who wants to understand how parameters and JSON schemas guarantee this validity can consult the documentation on structured output and JSON schemas to prevent faulty data types during evaluation runs.

The calibration process in phases

To calibrate a G-Eval rubric, we go through a systematic process in which human assessments serve as the ground truth. Calibration here means adjusting the prompt, the criteria, and the scoring definitions so that the statistical correlation with human judgment is maximized.

Statistical correlation measures for validation

To quantify the quality of the alignment, a simple average or a percentage of agreement is not enough. We use three complementary statistical metrics: Spearman's rank correlation coefficient ($\rho$), Pearson's correlation coefficient ($r$), and Kendall's Tau ($\tau$).

Spearman's $\rho$ measures how well the relative ranking between assessed texts is preserved, regardless of whether the absolute scale shifts linearly. This is crucial for model selection, where we want to know whether model A consistently scores better than model B. Pearson's $r$, by contrast, tests the linear relationship between the absolute scores. A high Spearman correlation combined with a low Pearson correlation typically indicates scale compression: the model ranks correctly, but compresses all scores together between, say, 3.5 and 4.2.

Before we calculate the correlation between model and human, we must first establish whether the human data itself is reliable enough. See the guide on inter-annotator agreement in judge evaluations to be sure that measurement errors in the human baseline layer are not wrongly attributed to the G-Eval model.

A strict lower bound applies when determining the required sample size for calibration. To statistically distinguish a correlation coefficient of $\rho = 0.80$ from an inadequate $\rho = 0.50$ at $\alpha = 0.05$ and a power of $(1 - \beta) = 0.80$, at least 65 carefully annotated examples are needed. In practice, we recommend a calibration set of at least 120 to 150 items. For a deeper dive into power analyses and confidence intervals, we refer to statistics for LLM evaluations and sample size determination.

Operationalizing scoring rubrics in prompts

A vague instruction such as "Rate the conciseness on a scale of 1 to 5" inevitably leads to arbitrariness. A robust G-Eval prompt defines every discrete anchor point on the scale with measurable conditions and specific exclusion criteria.

Below is an example of an operational prompt configuration in Python, designed to calibrate a summary assessment on the criterion of factual consistency:

import json
from dataclasses import dataclass
from typing import Dict, List, Optional

@dataclass
class GEvalPrompt:
    task_name: str
    criterion_name: str
    criterion_description: str
    evaluation_steps: List[str]
    scoring_rubric: Dict[int, str]

    def render_prompt(self, source_doc: str, candidate_text: str) -> str:
        steps_formatted = "\n".join(
            f"{idx + 1}. {step}" for idx, step in enumerate(self.evaluation_steps)
        )
        rubric_formatted = "\n".join(
            f"Score {score}: {desc}" for score, desc in self.scoring_rubric.items()
        )
        
        return f"""U bent een deskundige evaluator. Beoordeel de kandidaattekst op basis van het brondocument.

Taak: {self.task_name}
Criterium: {self.criterion_name}
Definitie: {self.criterion_description}

Evaluatiestappen:
{steps_formatted}

Scoringscriteria:
{rubric_formatted}

Brondocument:
{source_doc}

Kandidaattekst:
{candidate_text}

Geef uw analyse stap voor stap en eindig met een JSON-object dat exact deze structuur heeft:
{{
  "redenering": "<uw analyse per stap>",
  "gevonden_fouten": ["<fout 1>", "<fout 2>"],
  "score": <geheel getal tussen 1 en 5>
}}"""

# Voorbeeld van een gekalibreerde rubriek
geval_consistentie = GEvalPrompt(
    task_name="Samenvatting valideren",
    criterion_name="Feitelijke Consistentie (Faithfulness)",
    criterion_description="De mate waarin de samenvatting uitsluitend feiten bevat die direct herleidbaar zijn tot het brondocument, zonder toevoegingen of verdraaiingen.",
    evaluation_steps=[
        "Lees het brondocument grondig en identificeer de centrale feitelijke claims.",
        "Isoleer elke afzonderlijke bewering in de kandidaattekst.",
        "Verifieer per bewering of deze direct ondersteund wordt door het brondocument.",
        "Categoriseer eventuele afwijkingen in: (a) directe tegenspraak, (b) niet-ondersteunde extrapolatie, of (c) verwaarloosbare nuanceverschuiving.",
        "Bepaal de eindscore conform de onderstaande rubriek op basis van de zwaarste foutcategorie."
    ],
    scoring_rubric={
        1: "Volledig inaccuraat: bevat ernstige feitelijke tegenspraken met de bron of hallucineert cruciale entiteiten.",
        2: "Grotendeels inaccuraat: bevat minstens één directe feitelijke tegenspraak of meerdere niet-ondersteunde claims.",
        3: "Matig consistent: bevat geen directe tegenspraken, maar introduceert wel speculatieve details die niet in de bron staan.",
        4: "Vrijwel foutloos: alle kernfeiten kloppen exact; bevat hooguit een minimale, niet-storende nuanceafwijking.",
        5: "Volkomen consistent: elke bewering en relatie is direct en ondubbelzinnig verifieerbaar in de brontekst."
    }
)

Neutralizing systematic biases in LLM-as-a-Judge

Even advanced models show specific, reproducible deviations when acting as an evaluator. During the calibration phase, we must actively correct for four persistent phenomena:

• Verbosity bias (length preference): Models structurally assign higher scores to longer, more elaborately worded answers, even when these contain redundant information or minor style errors. Solution: include explicit penalty clauses in the evaluation steps for excessive length.
• Self-preference (self-enhancement bias): A model from family X on average assigns higher scores to texts generated by that same model X than to texts from competing architectures. Solution: anonymize formatting artifacts and evaluate with multiple independent evaluator families.
• Central tendency bias (scale compression): Assessors avoid the extremes (scores 1 and 5) and cluster outputs at 3 and 4. Solution: rescale the rubrics to clearly bounded thresholds or use pairwise ranking as an intermediate step.
• Position bias: In comparative evaluations, the order of presentation influences the outcome. Solution: reverse the order and run the test twice to ensure consistency.

Dutch-language pitfalls in G-Eval calibration

When calibrating evaluation models on Dutch-language texts, specific complications arise that are rarely mentioned in English-language literature. Ignoring these language- and culture-specific characteristics leads to systematically distorted correlations.

First, forms of address introduce noise. In Dutch, the distinction between formal ('u') and informal ('je/jij') is context-dependent. A base model trained in English that acts as an evaluator often assesses a shift from 'u' to 'je' as a trivial variation, while in business or legal contexts this is a serious error. The rubric must therefore explicitly define when style inconsistency leads to point deductions.

Second, compounds and anglicisms pose a risk. Incorrectly writing compound words as separate words (the 'English disease') is rarely penalized by evaluator models because tokenizers mask such errors. When a human panel scores this strictly while the LLM doesn't detect the error, the correlation drops considerably. We solve this by placing formal spelling and grammar checks as a separate automated step before the G-Eval call.

Finally, Flemish Dutch (Belgian Dutch) differs lexically from Dutch as used in the Netherlands. Words like 'gelet op', 'vastgoedmakelaar', or specific official constructions are sometimes wrongly flagged as style errors by models primarily calibrated on Netherlandic Dutch corpora.

Reproducible measurement setup and parameter configuration

To ensure that the calibration measurement is reproducible across successive software releases, all evaluation parameters must be strictly fixed. An evaluation run must not contain any moving variables.

Cost and token analysis

Structurally deploying G-Eval with extensive Chain-of-Thought prompts involves significant compute costs. A realistic estimate of token consumption and turnaround time is necessary to determine whether an evaluation pipeline can run on every pull request or only on nightly builds.

With an average source document of 800 tokens and a candidate text of 250 tokens, the G-Eval system prompt including CoT instructions consumes approximately 650 tokens. The input per evaluation therefore amounts to approximately 1,700 tokens. The generated reasoning steps (output) span an average of 350 to 500 tokens per run.

For a full calibration run over a test set of 150 examples with 3 repetitions, this results in:

• Total input tokens: $150 \times 3 \times 1,700 = 765,000$ tokens.
• Total output tokens: $150 \times 3 \times 450 = 202,500$ tokens.
• Total time at a processing speed of 10 concurrent threads: approximately 4 to 7 minutes.

Thanks to prompt caching on the static part of the rubric (the instructions and scale definitions), the effective number of processed input tokens in production environments can be reduced by 40% to 60%, provided the cache boundaries are precisely aligned.

Conclusion and implementation decision

G-Eval forms a powerful bridge between slow human evaluation and rigid heuristic metrics, provided the scoring criteria are rigorously aligned with vetted human data. A rubric should only be considered production-ready once the Spearman rank correlation with a reliable human panel exceeds the threshold of at least $\rho \ge 0.80$ on an unseen validation set.

Once that threshold is reached, the system delivers a repeatable, continuous quality measurement that directly flags model regressions without needing human annotators for every iteration.