Evaluating Embedding Models for Search and RAG Applications

In modern Retrieval-Augmented Generation (RAG) systems, the embedding model forms the bridge between the user's query and the organization's stored knowledge. While Large Language Models (LLMs) excel at generating text, embedding models are designed to convert text into numerical vectors. These vectors capture the semantic meaning of text, making it possible to find related concepts even when the exact keywords are missing.

However, choosing the right model is a complex decision. A model that ranks high on public leaderboards does not automatically perform well on specific Dutch policy documents, medical records, or technical manuals. This article provides an in-depth guide to objectively measuring the performance of embedding models on your own dataset. We discuss how to set up an evaluation pipeline, which metrics truly matter, and how to make an informed choice between model size, speed, and quality.

Why standard benchmarks are not enough

The AI community relies heavily on standardized benchmarks, of which the Massive Text Embedding Benchmark (MTEB) is the most well-known. MTEB tests models on dozens of tasks, including classification, clustering, and retrieval, across multiple languages. While this is an excellent starting point for selecting candidates, it only tells half the story for enterprise applications.

Public benchmarks introduce three major blind spots:

• Domain-specific terminology: A model trained on Wikipedia and news articles (which MTEB heavily relies on) may not understand the nuances of internal company jargon, specific abbreviations, or legal definitions.
• Text length and chunking: Benchmarks use standardized text lengths. If your RAG system works with very long document chunks, some highly-ranked models perform drastically worse because their 'attention span' (context window for embeddings) is limited.
• Language balance: Many models claiming to be 'multilingual' are disproportionately trained on English. Their performance on complex Dutch sentence structures can lag substantially behind the English variant, something that is often masked in an aggregate MTEB score. For an in-depth look at this phenomenon, see our article on testing Dutch LLMs.

The foundation: Precision, Recall, and advanced metrics

When we talk about evaluating embedding models for RAG, we are actually evaluating the retrieval performance: how well does the model retrieve the correct context for a given query? To quantify this, we use specific Information Retrieval (IR) metrics.

Recall@K

In the context of RAG, recall is the most important metric. It answers the question: Is the relevant document in the top K results retrieved by the model?

If we have a RAG system that sends the top 5 most relevant text fragments to the LLM (K=5), we want to know how often the actual, correct fragment is in that top 5. A Recall@5 of 0.85 means that in 85% of the search queries, the required document was successfully found. In a RAG pipeline, recall is crucial: if the information is not retrieved, the LLM simply cannot generate the correct answer, which directly leads to hallucinations.

Precision@K

Precision measures what percentage of the retrieved documents are actually relevant. With Precision@5, we look at the top 5 results and calculate how many of them are useful for the query. Although precision is important for efficiency (you do not want to flood the LLM with irrelevant noise that increases processing time and costs), it is often secondary to recall in a RAG context. After all, an LLM can ignore irrelevant context relatively well, as long as the correct answer is included.

MRR (Mean Reciprocal Rank)

MRR takes into account the position of the first relevant document in the search results. It is calculated as the average of the reciprocal rank (1/rank). If the relevant document is in first place, the score is 1. If it is in second place, the score is 0.5, and so on. MRR is extremely valuable when you want the best information to be as high as possible, which is relevant if you apply strict context limits in your prompts.

NDCG (Normalized Discounted Cumulative Gain)

NDCG is the most nuanced metric. Unlike Recall or MRR, which assume a binary concept (a document is either relevant or irrelevant), NDCG allows for gradations of relevance. A document can be 'highly relevant', 'somewhat relevant', or 'not relevant'. NDCG assigns a higher score to search results where the most relevant documents are at the top, and the score degrades logarithmically as relevant results appear lower in the list. For advanced RAG evaluations, NDCG is the gold standard.

Building a Golden Test Set (Golden Dataset) in 4 steps

To calculate the above metrics, you need a 'golden test set'. This is a collection of queries, linked to the exact document chunks containing the answer. Creating this set manually is time-consuming, but fortunately, we can use LLMs to automate this process through synthetic data generation.

Step 1: Preparing and Chunking Documents

Collect a representative sample of your production data (for example, 500 diverse documents). Apply the exact same chunking strategy that you will use in production. If you use blocks of 500 tokens with an overlap of 50 tokens in production, do the same here. The performance of the embedding model depends heavily on the length and structure of the input.

Step 2: Generating Synthetic Queries

Use a powerful, capable LLM (such as GPT-4, Claude 3.5 Sonnet, or an equivalent open-source model) to generate queries for the text fragments. A good prompt instructs the LLM to create different types of questions:

• Factual questions: Direct questions where the answer is literally in the text.
• Conceptual questions: Questions that ask for an explanation or summary of the concept in the fragment.
• Implicit questions: Questions where the terminology is slightly different from the source text, to truly test the semantic understanding of the embedding model.

Step 3: Manual Validation and Filtering

Although the LLM does the heavy lifting, human review is essential. Randomly review 20% of the generated Q&A pairs. Remove questions that are ambiguous, or questions that could be better answered by another document in your dataset without this being reflected in the mapping. An unchecked synthetic dataset leads to biased evaluation results.

Step 4: Formalizing the Test Set

Save the validated set in a structured format (such as JSONL), where each record contains the following fields: query_id, query_text, relevant_document_id. This is your reference point for all future tests.

The complexity of language: Dutch versus English

A crucial aspect of the evaluation is the language of your source documents and the expected search queries. Many embedding models perform fantastically on English text, but fall short when it comes to Dutch. This has two primary causes:

1. Tokenization efficiency: Models are trained with a specific tokenizer. If this tokenizer is not optimized for the Dutch vocabulary, Dutch words are split into numerous nonsensical sub-tokens. This disrupts the semantic representation, as the model struggles to 'understand' the coherence of the original word.
2. Training data ratio: In inferior models, the vectors (embeddings) for Dutch words are often isolated or fragmented in the vector space, compared to the richly structured cloud of English concepts.

When you have a fully Dutch dataset, it is absolutely necessary to evaluate specific 'multilingual' models (such as the multilingual-e5 series or OpenAI's text-embedding-3) against models that may have been trained with a strong focus on European languages. Pay specific attention to how the model handles typical Dutch compound words (such as "arbeidsongeschiktheidsverzekering" instead of "disability insurance").

Large versus Small: When Does a Heavier Embedding Model Pay Off?

When selecting models, you face a trade-off between the size of the model (number of parameters and vector dimensions) and performance. This has direct implications for infrastructure and costs.

A heavier model with more dimensions picks up subtler relationships between words. However, this also means that each document chunk and each search query takes up more memory in your vector database (for more context on this infrastructure, read the article What is a Vector Database? on our learning platform). For an extensive analysis of these trade-offs, check out our guide on quality versus cost. As a rule of thumb: start small and test whether the Recall@K meets the business requirements. Only scale up to a larger, more expensive model if the evaluation proves that the gain in precision and recall justifies it.

Step-by-Step Plan: Setting Up Your Own Evaluation Pipeline

With the theory and dataset in place, you can set up the technical pipeline. A robust evaluation script typically follows these steps:

1. Loading: Load your corpus (the document chunks) into memory.
2. Embedding: Generate embeddings for all chunks in the corpus using Model A (the candidate).
3. Indexing: Place these vectors in a temporary index (e.g., FAISS or an in-memory instance of a vector database) and configure the desired distance metric. Note: check the model's documentation to see if it is optimized for Cosine Similarity or Dot Product. This makes a significant difference in the score.
4. Querying: Iterate over all queries from your golden test set. Embed the query with Model A, and retrieve the top K nearest document vectors.
5. Scoring: Compare the retrieved document IDs with the relevant_document_id from your test set. Calculate the Recall@5, MRR, and NDCG over the entire set.
6. Repeating: Clear the index and repeat steps 2 to 5 for Model B, C, and so on.

By running this script automatically, you create a repeatable framework. As soon as a new embedding model is launched, you can see within minutes whether switching is worth it for your specific use case.

Common Pitfalls in Embedding Evaluations

During testing and evaluation, mistakes are often made unintentionally that skew the results. Be alert to the following pitfalls:

• Ignoring asymmetric queries: In RAG, you often have a short search query (e.g., 10 words) that needs to be matched with a long answer document (e.g., 300 words). Some models are specially trained for this (asymmetric search), while other models are better at comparing documents of equal length (symmetric search). Make sure your evaluation reflects actual usage.
• Forgetting instructions: Certain models (such as the BAAI/bge models or Instruct models) require a specific prompt prefix for the query, such as "Represent this sentence for searching relevant passages: ". If you omit this instruction during evaluation, the model's performance degrades significantly compared to its true potential.
• Overfitting on the test set: If you optimize the model, or the chunking strategy, solely to achieve a 100% score on your small golden test set, you lose the ability to generalize to new, unseen queries from real users. Always maintain a blind test set (hold-out data) when fine-tuning parameters.

Conclusion

Evaluating embedding models is not a one-time exercise, but a continuous process within the management of RAG applications. By not staring blindly at generic leaderboards, but choosing a systematic approach with a domain-specific golden test set, clear metrics (such as Recall and MRR), and special attention to language nuances, you guarantee the quality of your search results. Ultimately, the quality of your retrieval system determines the intelligence and reliability of your entire AI assistant.