What makes a good answer? Evaluating RAG systems in production

At DareData we have deployed solutions based on retrieval augmented generation (RAG) systems for a number of clients. Our own products GenOS Assistant and GenOS Service include a RAG component so that answers are grounded in proprietary and evolving company knowledge. Believing that without measurement there's no improvement, we care deeply about proper evaluation.

And yet the evaluation of RAG systems is both harder and more important than commonly recognized. Far from settled, the topic still inspires lots of research that we follow closely for new developments. In this post I describe some recent research on evaluation as well as lessons we learned from adapting these ideas to production.

Why you should care about rigorous evaluation

It has never been easier to quickly set up a conversational system with access to your company's internal knowledge. With more powerful language models and better libraries with out-of-the-box retrieval strategies, the time it takes to start producing reasonable answers is short. And yet, "producing reasonable answers" is never the bar in real-world settings. You want to know that the users of your conversational system are consistently getting correct and up-to-date information; that they are getting all the relevant information; and that this will be the case not just today, but also tomorrow, one month from now, and one year from now.

This is one reason you need an evaluation method that you trust and that allows you to remove everyone's vibes from the equation. A second, equally important reason is that a solid evaluation method can and should guide all development decisions, acting as a compass to the explorer. When setting up a RAG system you have to make a number of decisions, including chunking strategy, retrieval strategy, embedding model, language model, which prompts to use, and more. If you trust your evaluation method, then you can quickly experiment with all of these, swapping one piece for another, rolling back changes that did not work out, confident you are moving in the right direction.

Why evaluation of conversational systems is hard

Given a labeled dataset, it is not hard to evaluate an image classification system (it may be hard to build it, but that is a different question!). In time series forecasting there are some decisions involved, such as the forecast horizon you care most about, or whether root mean squared error (RMSE) is a more meaningful metric than mean absolute error (MAE) for your particular problem, but given those, the evaluation itself is also straightforward. Natural language, on the other hand, is hard. Even humans often disagree on which of two candidates is a better summary of a given document, or which of two candidates is a better answer to a given question, forcing us to measure concepts such as the inter-annotator agreement (Thakur et al., 2025).

Another major issue is that evaluating natural language responses manually is slow and (therefore) expensive. Human evaluation is still the gold standard and you should take advantage of it whenever it is available. But it does not scale, so people have been looking for alternatives for decades (Papineni et al., 2002).

It is important to distinguish two different steps in the evaluation pipeline. The first is the construction of reference labels, the ground truth¹. The second is the precise way in which we choose to compare candidate answers with the references. Each of these steps can, independently, be automated or done manually. But they are on completely different scales: the first could be done only once, if you do it right, perhaps updated periodically with long time spans in between; the second should be done multiple times a day, for the whole dataset, at least during development. It would not be shocking if a domain expert wanted to manually write down reference answers for 100 questions, but it would be virtually impossible for someone to manually compare new candidate answers with the references every time a change is introduced in the system.

However you obtain your labeled dataset, there are two approaches to doing the comparison step that we have found insufficient for our purposes. One is the use of classical automatic metrics, such as Exact Match (EM), F1 score (bag-of-tokens), or ROUGE-L, still widely used in benchmarks such as SQuAD, DrQA, ORQA or KILT. These are generally too strict for long-form question answering and so are not well-aligned with human evaluations of RAG systems. The other, more recent, approach is the use of a pure LLM-as-a-Judge, where we simply ask a language model to compare the candidate and reference answers. The advantages are obvious: compared to human evaluations, this is fast and cheap. It is also more flexible than the strict metrics mentioned above. But there are a number of pitfalls, unfortunately not as obvious as the advantages.

1. Biases. A number of biases have been identified when using LLM-as-a-Judge, including the tendency to favor answers generated by the same model or by a model of the same family (self-enhancement bias), the tendency to favor longer responses even when they are not as accurate as shorter alternatives (verbosity bias), or the tendency to favor answers in a given fixed position over others, say in a multiple choice setting (position bias). See Ye et al. (2025) and Wang et al. (2024) for more.
2. Alignment with humans. An LLM-as-a-Judge is often used without any assessment of whether it agrees with human preferences. An automatic metric is only useful if it is well-aligned with humans, and all the other metrics have accompanying empirical studies trying to establish this alignment (e.g. BLEU for translation, ROUGE for summarization, BERTScore for text generation more generally, etc.). There are also empirical studies for LLM-as-a-Judge, but the conclusions are mixed and it is very hard to generalize as the studies often look only at a particular task, a particular language model, a particular collection of prompts. For example, Thakur et al. (2025) found that "only the best (and largest) models show reasonable alignment with humans, though they still differ with up to 5 points from human-assigned scores" and that "even the best judge models differ from humans in this fairly sterile setup, indicating that caution is warranted when applying judge models in more complex scenarios."
3. Metric stability. Metrics are useful because they allow comparisons to be made. They allow us to compare two different RAG systems, or two different versions of the same system (say before and after a particular change). But this only works if the metric itself remains the same. With an LLM-as-a-Judge, the metric is the whole package: the exact version of the language model being used (e.g. claude-sonnet-4-5-20250929), all the prompts involved, all the parameter values being passed in the API call (e.g. temperature). Change one of these and you've changed the metric.
4. Lack of determinism. Language model inference involves sampling a token from a probability distribution at each step, and so the general lack of determinism is not surprising. Importantly, the nondeterminism remains even when doing greedy decoding, i.e., setting the temperature to 0 and always picking the most likely token at each step (see the great blog post by Horace He and the Thinking Machines Lab, as well as the recent developments in the inference libraries vLLM and SGLang that aim to fix this problem). The important point is that the more variance is inherent to your metric, the less useful it is. How can it drive development when evaluating the same thing twice gives two different results?

The nugget evaluation methodology

I will now describe an approach to the evaluation of RAG systems that overcomes some of the problems highlighted above and provides varying levels of control over which parts of the process we want to automate. Most of the ideas here come from Pradeep et al. (2025) and related work, and I encourage you to read that paper for more details.

The paper proposes a framework for evaluating RAG systems such that each step can be done manually or automatically with the help of a language model, and, importantly, they run a human evaluation study to assess the validity of the automations. The authors are involved in organizing the RAG track of the decades-old Text Retrieval Conference (TREC) and so in part their work serves to justify the evaluation approach taken by that conference.

The authors propose an adaptation to the modern day of the nugget evaluation methodology that dates back to 2003 in the evaluation of question answering (QA) systems (Voorhees, 2003). Here a nugget is an atomic fact that should be present in any good answer. Given a document collection and a set of queries, there are essentially four steps needed to evaluate a RAG system in this way, each of which can be done manually or automatically.

1. Create relevance judgments. Manually or automatically by following the UMBRELA framework.
2. Given a query, create nuggets. Here we are interested in creating nuggets, or atomic information units, from the set of relevant documents that we obtained from the previous step. These characterize the information that should be contained in a high-quality answer to the query.
3. Label nugget importance. Each nugget from the previous step is labeled as either vital or okay, where vital nuggets are those that must be present in a good response, while okay nuggets are good to have, but are not absolutely necessary.
4. Given a candidate answer to the query, determine its nugget recall. Here we take a candidate answer and, for each nugget, decide whether it is supported, partially supported, or not supported by the candidate answer. We can then look at the proportion of nuggets (among all nuggets, or just the vital ones) supported by the candidate answer. This is a form of recall, which explains the title of the paper, The Great Nugget Recall.

The paper includes the prompts used for steps 2, 3 and 4 above as well as some implementation details I am not including here, such as limits on the number of nuggets per query or the number of tokens per nugget. In terms of the distinction we made above between dataset construction and candidate scoring, note that steps 1, 2 and 3 belong to the first category, and only step 4 belongs to the second.

The figure below contains a concrete example with a query from the TREC 2024 RAG track. The relevant passages yield three nuggets, here all labeled vital. To evaluate a candidate answer we check each nugget against it and mark it as supported, partially supported, or not supported. The vital-nugget recall is then simply the fraction of vital nuggets the answer covers - with a strict variant that counts only full support, and a relaxed variant that gives partial credit.

To validate this approach, the authors compare human evaluations, made with the help of annotators from the National Institute of Standards and Technology (NIST), with the results of automating some or all of the steps. While the paper has full details, I will just focus on the alignment between human evaluations and the fully automated pipeline, and will consider only the recall of vital nuggets in step 4. The authors used a measure of rank correlation, Kendall's τ, to quantify this alignment. Kendall's τ can take values between -1 and 1, and in our case it is 1 exactly when the two evaluation methods fully agree on the ranks of the candidate RAG systems, i.e., they agree on what is the best RAG system, they agree on what is the second best RAG system, and so on.

Some of the results are good and some are not so good, highlighting the importance of these rigorous empirical studies. When the authors look first at the average performance of each RAG system on the set of all queries and then compute Kendall's τ, they obtain a value of 0.73. This means there is reasonably high agreement between the manual and automatic rankings of RAG systems as a whole. If, however, the authors look first at the Kendall's τ between manual and automatic rankings on each individual query, and only then average over all queries, they obtain the much lower value of 0.36. Intuitively, the automatic approach was reasonably good at reproducing the manual ranking of RAG systems as a whole, but not so good at ranking RAG systems on individual queries.

How we're using these ideas in production

Applying these ideas in a production setting has taught us a number of valuable lessons and changed the way we approach evaluation of conversational systems. Different companies will differ in the degree of involvement they want to have with the evaluation process, but they all want something they can trust, before and after rolling out to real users. For instance, we have found that a good compromise between manual and automatic dataset construction is to automate steps 1 and 2 above, and have a domain expert provide the labels in step 3 through an easy-to-use interface. This relatively small amount of manual work helps catch issues coming from previous steps and contributes massively to establishing confidence in the results. Beyond this, other key lessons include the importance of using real questions from real users, which are often much harder than synthetic ones, and the benefits of doing human reviews of random samples at each automated step. The speed and confidence with which we are able to iterate on a system after investing in a rigorous evaluation far outweigh its initial cost.

Notes

^1. There are reference-free approaches to the evaluation of RAG systems, such as RAGAS, but they have other limitations so we will not discuss them here.