From Production Logs to Evaluation Data: Closing the Feedback Loop

Every developer of AI applications knows the phenomenon: in the test environment, your Large Language Model (LLM) performs perfectly, but as soon as real users start interacting with it, unexpected problems arise. Users ask questions you could never have imagined, use unexpected jargon, or give instructions that completely throw off your prompt.

Building web applications with LLMs requires more than just a good initial prompt or an impressive score on static benchmarks. To guarantee the quality of your application—and prevent regressions when you switch to a newer model or adjust your instructions—you must create a continuous feedback loop. You do this by converting the interactions from your production logs into a living test set in a structured, secure, and automated way.

Why Static Test Sets Fall Short

Many teams start their evaluation process with a manually curated test set. While this is a useful starting point, this data becomes outdated very quickly. This is known as data drift or concept drift. The way people interact with your application evolves as they discover its capabilities (and limitations).

Suppose you have built a financial AI assistant for users doing paper trading via platforms like Alpaca or eToro. Your initial test set probably contained neat questions like: "What is the difference between a market order and a limit order?". In the production environment, however, users type in hurried, context-dependent prompts, such as: "Sell all my AAPL if the VIX goes above 20, ignore previous stop-loss." If your evaluation set only consists of basic definitions, an update to your LLM that accidentally breaks this complex logic will go unnoticed until users start complaining.

Production data is the only source of truth. It contains the edge cases, the typos, the ambiguous intents, and the complex multi-layered instructions that you simply cannot simulate in a synthetic test set.

Step 1: Collecting and Filtering Log Data

The first step is to structurally capture the input and output of your LLM in production. This does not mean simply writing everything to a text file. You need structured logs, ideally in a format like JSON or written to an observability platform. A good log entry contains at least:

Smart filtering instead of labeling everything

It is impossible to manually process tens of thousands of production logs per day. The goal is to draw a representative sample that is valuable for your test set. Use strategies such as stratified sampling for this. Specifically look for:

Privacy Hygiene: Working Safely with User Data

Before we can use these selected logs as evaluation data, we must overcome a crucial hurdle: privacy. Production data can contain personally identifiable information (PII). Think of names, addresses, or in our previous example, specific account balances of a trading portfolio. Storing this raw data directly in a test set—especially if this test set is used by multiple developers or shared via CI/CD pipelines—poses major risks regarding the GDPR.

Implementing strict privacy hygiene is non-negotiable. This process, also called data sanitization, must take place automatically immediately after the log is filtered for the test set.

A commonly used method is deploying Named Entity Recognition (NER) models, such as those from spaCy, or using fast, local SLMs (Small Language Models) specifically trained to mask PII. All names are replaced by placeholders (e.g., [PERSON_1]) and financial or sensitive data is anonymized. You can find more in-depth technical strategies for this in our article on building a test set without data leaks.

Note: Never use complex or black-box cloud LLMs for initial PII scrubbing if your organization requires that sensitive data must not leave its own infrastructure. Use local, deterministic systems or self-hosted open-weights models for this instead.

Step 2: Labeling with Minimal Effort via LLM-as-a-Judge

Now that we have a filtered and anonymized set of real user interactions, we need to determine what the 'perfect' answer would be, or at least establish criteria to evaluate future answers to these questions.

Manual labeling (human-in-the-loop) offers the highest quality, but scales poorly. The modern alternative is the LLM-as-a-judge paradigm. Here, you use a powerful model to evaluate the answers in your logs against a rubric defined by you. In practice, we see that developers like to use heavy models for this, such as Claude Pro (for example, Claude 3.5 Sonnet) for nuanced qualitative analyses, or iterate with models like DeepSeek via open APIs for lightning-fast, cost-effective bulk evaluations.

A typical LLM-judge prompt looks like this:

You are an impartial expert in evaluating AI assistants.
Assess whether the assistant's response below is correct, safe, and helpful given the user query.

User query: {user_query}
Assistant response: {model_response}
Reference knowledge (if RAG): {retrieved_context}

Rate on the following criteria (1-5):
1. Factual correctness based on the context.
2. Free of hallucinations.
3. Tone and helpfulness.

Provide your output strictly in JSON format with the keys 'score_1', 'score_2', 'score_3', and 'explanation'.

If a production log receives a high score from the judge (for example, a perfect 5 across the board), you can directly add this "User query + Response" pair to your test set as a golden reference. Does the response score low? Then this is actually an excellent test case. You add the user query and mark that your current system failed here. This becomes a regression test for your next model update.

Practical Example: Evaluating RAG Applications

When building complex web applications with RAG architectures, the feedback loop becomes even more critical. In RAG (Retrieval-Augmented Generation), an incorrect answer can have two causes: the vector database retrieved the wrong documents (retrieval error), or the LLM drew the wrong conclusion from the correct documents (generation error).

If you collect logs for a RAG application, you must store the retrieved context chunks. By running this production data through your evaluation pipeline, you can measure whether a failing response was caused by your chunking strategy or search algorithm not aligning with the specific way users formulated their queries in production. If you do not have experience setting up such systems yet, it is advisable to first study the basic principles via external sources or our guide on RAG for beginners on the Leren subdomain.

Step 3: Maintaining a Living Test Set

A test set is not a static artifact. It is living documentation of your application's requirements. Once your automated pipeline is running (Collect → Anonymize → Judge Evaluation → Add to Test Set), your dataset will grow continuously.

To prevent the test set from becoming too large and slow for fast iterations, you must apply version control and 'garbage collection':

Step 4: Catching Regressions in Your CI/CD Pipeline

The ultimate goal of this entire exercise is to close the loop. You now have a robust, reality-evolving test set full of complex, redacted, and labeled user queries. How do you deploy it?

Tie the evaluation directly to your deployment process. Whether you tweak your system prompt, experiment with a lower temperature in your API calls, or decide to switch to a completely different LLM provider for better performance: you first run your application against the living test set.

In your CI/CD pipeline, you implement a step we call regression testing (more details in our piece on regression testing for prompts). Your automated pipeline sends all questions from the test set to your new staging environment. The responses are again evaluated by your LLM-as-a-judge. If the overall score drops (for example: the 'helpfulness' score decreases by 15% compared to production), the build fails and you know your change is harmful before a single real user is affected.

Conclusion

Bridging the gap between production monitoring and structured evaluation is what separates mature AI development from quick prototyping. By building an automated pipeline that safely filters, anonymizes, and converts production logs into qualitative test cases via LLM judges, you create a self-improving system.

You are no longer building blindly; you are optimizing based on the raw reality of your users. Take the time today to export, anonymize, and evaluate your first 100 production logs. The insight you gain from this about the actual behavior of your application is often more surprising and valuable than any static benchmark.