Evaluating an application based on a large language model (LLM) involves testing it systematically, before and after each modification. This guide details how to choose scoring methods, design a reliable LLM judge, and then evaluate RAG systems and agents throughout the publication cycle.
Reading a few responses and finding them satisfactory is not enough. A change to the prompt, a model update, or an alteration to document retrieval can break a behavior without anyone noticing, making it difficult later to determine which change was responsible. Evaluation involves creating test cases, scoring each response against written criteria, and then comparing scores before and after each change, similar to classic non-regression testing in code.
An LLM’s responses vary from one run to another, and the notion of a ‘good answer’ depends on context. A unit test checks for equality, whereas ‘You can return shoes within 30 days’ and ‘Returns are accepted for one month after delivery’ are both correct. The guide distinguishes evaluation, which measures relative quality against fuzzy criteria, from testing, which validates a production release with binary checks, like valid JSON or the absence of a banned phrase. Blocking a deployment over a faithfulness score of 0.7 would, according to the author, amount to debating an arbitrary threshold every week.
Three ways to score a response
A scorer receives a test case, consisting of an input, a response, and sometimes a context or reference, then returns a verdict or a number. Code-based scorers apply deterministic rules to check format, length, or the presence of certain strings. Reference-based scorers compare the response to a known correct answer. Overlap metrics like BLEU and ROUGE, for example, do not recognize a correct paraphrase. Finally, the LLM judge is a model tasked with applying a set of criteria to another model’s response, useful for tone, grounding in sources, or utility.
The guide recommends starting with code wherever a rule suffices, as these scorers are free and always give the same result. Three judge configurations are described: pointwise, where it assigns a label or score to an isolated response; pairwise, where it chooses the better of two responses, a common method for comparing two prompts or two models; and reference-guided, when it has a reference answer available.
Making the judge reliable
A well-written set of criteria determines the judge’s quality. The guide describes it in four parts: the definition of the criterion, the data the judge can use, the procedure to follow, and a strict output format, preferably JSON. Reasoning should precede the verdict, and temperature is set to 0 to reduce sampling noise, without eliminating it. For sensitive criteria, the guide suggests an odd number of runs and taking the majority verdict.
The QAG (question-answer generation) method asks the judge to answer closed yes/no questions, with the calculation then performed by code. The faithfulness of a response containing four statements where three are supported is thus 0.75. For subjective criteria, the G-Eval method calculates a weighted average based on the probabilities of possible scores, so a model split 60/40 between 4 and 5 gets 4.4. Biases are documented in research on judging LLM outputs: position bias, verbosity bias, and self-preference bias for models from the same family. The guide proposes reversing the order of responses, testing cases of comparable length, and using a judge from a different model family.
Calibration remains the most important step. The guide advises manually labeling 30 to 50 responses, then comparing these labels to the judge’s using Cohen’s kappa, which corrects for chance agreement. A score of 0 corresponds to chance-level agreement, and 1 to perfect agreement. Beyond about 0.6, agreement is generally considered substantial.
RAG, agents, and the publication cycle
A RAG (retrieval-augmented generation) system combines two components that can fail separately: the search engine, which retrieves context, and the generator, which writes the response. The guide proposes measuring context precision, its recall, the faithfulness of the response to the retrieved documents, and its relevance to the question. High faithfulness paired with low recall, for example, signals insufficient retrieval.
For agents, the guide starts with task completion, then examines the correctness of the chosen tool, the correctness of its arguments, and the trajectory followed, avoiding redundant calls or loops. Offline evaluations are done before production, on a versioned set of 25 to 50 cases, with release criteria set in advance. Online evaluations are performed on a sample of real traffic. Each failure observed in production can subsequently become a reference test case.
