Most teams evaluate a large language model application once, with a handful of prompts tried by hand, and then ship it. That works until the model changes, the data changes, or a prompt edit quietly breaks a previously working case. In production, LLM behaviour drifts for reasons that have nothing to do with code: a vendor updates a model version, retrieval returns different documents after an index rebuild, or users start asking questions the original test set never covered. Without a repeatable evaluation, you discover these regressions from users, not from your pipeline.

Evaluation is not a single score. A RAG application can fail in retrieval, in generation, or in the interaction between them. It can retrieve the right documents and still hallucinate, retrieve the wrong documents and answer confidently, or retrieve the right documents and refuse to answer. A single benchmark number hides all of that. What production teams need is a layered approach: offline benchmarks for comparison, golden datasets for regression gates, retrieval metrics separated from generation metrics, and live monitoring for the cases no static dataset captures.

This article describes that approach in practical terms: how to build a golden set, which metrics actually help for RAG, how to use LLM-as-judge without fooling yourself, and how to wire evaluation into CI so that a bad prompt or model change fails the build before it reaches users.

#Quick Answer: How Should You Evaluate an LLM Application in Production?

Build a versioned golden dataset of real user questions with expected answers and cited sources, score retrieval and generation separately, and gate releases on that dataset. Use exact-match and lexical metrics only for narrow, factual tasks; for open-ended answers, use structured LLM-as-judge rubrics with human spot-checks, plus faithfulness checks that verify claims against retrieved context. Track the same metrics in production on sampled live traffic. If a model, prompt, or index change moves the golden score beyond an agreed threshold, it does not ship until someone explains why.

#Build a Golden Dataset Before You Optimise Anything

A golden dataset is a curated set of question-and-answer pairs that represent real production use, not textbook examples. Start from production logs, support tickets, and user feedback, anonymised and de-duplicated. Include the hard cases: ambiguous questions, questions with no good answer in the corpus, questions that require synthesis across multiple documents, and questions where the correct answer is a refusal or a clarification. A dataset of only easy, well-formed questions will reward a system that fails on real traffic.

Each item needs more than a question and an answer. Record the expected answer, the documents that support it, the acceptable variants, and metadata: product area, difficulty, language, and whether the answer should cite sources. Version the dataset in git alongside prompts and configuration. When you change the dataset, record why — a new product launch, a corrected answer, a newly discovered failure mode. Without versioning, you cannot tell whether a score changed because the system improved or because the test changed.

Human labelling without gold-plating

Human judgement is ground truth, but labelling is expensive and inconsistent. Use two reviewers for a sample and measure agreement before trusting labels. Write guidelines with borderline examples: what counts as partially correct, when a longer answer is penalised, and when refusal is correct. If reviewers disagree frequently, the task definition is unclear. Clarify it before adding more labels.

#Separate Retrieval Quality From Generation Quality

The most common evaluation mistake is scoring only the final answer. When the final answer is wrong, you need to know whether retrieval failed or generation failed, because the fixes are completely different. Retrieval fixes involve chunking, embeddings, hybrid search, and reranking. Generation fixes involve prompts, context formatting, and model choice. Conflating them wastes effort.

Retrieval metrics that help

For each golden question, check whether the expected documents appear in the retrieved set. Recall at k — the fraction of expected documents found in the top k results — is the workhorse metric. Mean reciprocal rank, which rewards putting the right document first, matters when the generator pays most attention to early context. Also track context precision: how much of the retrieved context is actually relevant. A system that retrieves twenty chunks, of which two are relevant, burdens the generator with noise and increases hallucination risk. A useful diagnostic is to run the generator on gold context — the known-correct documents — versus retrieved context. If gold-context answers are good but retrieved-context answers are bad, retrieval is the bottleneck. If gold-context answers are also bad, the prompt or model is the problem. This single experiment saves weeks of guessing.

Chunking and embedding choices show up here directly. Overly large chunks dilute relevance; overly small chunks lose context. Test chunk sizes and overlap on your golden set rather than copying defaults from a tutorial. Hybrid search — combining keyword search with vector search — often helps for queries with exact identifiers, product codes, or names that embeddings handle poorly. A reranker, which re-scores the top candidates with a more expensive model, frequently improves precision at modest cost. Measure each change on retrieval metrics before judging the final answer, so you know what actually helped.

Generation metrics and their limits

For factual, short answers, exact match and token-level F1 are defensible. For summarisation, ROUGE and similar lexical metrics correlate weakly with quality and penalise good paraphrases. For open-ended RAG answers, the metrics that matter are faithfulness, answer relevance, and completeness. Faithfulness asks whether every claim in the answer is supported by retrieved context. Answer relevance asks whether the answer addresses the question asked, not a nearby question. Completeness asks whether it covers the key points in the expected answer without adding unsupported extras.

These are hard to score automatically without a judge model, which brings its own problems, covered below. A pragmatic middle ground is to combine automated checks — citation coverage, length bounds, refusal behaviour, and format compliance — with sampled human review. Automated checks catch regressions cheaply; human review catches quality drift that automated checks miss.

#LLM-as-Judge: Useful, But Only With Guardrails

Using a strong model to grade answers is now standard practice, and it can work well, provided you treat the judge as a fallible instrument, not an oracle. Judges have known biases: they prefer longer answers, they prefer their own style, they are sensitive to the order in which options are presented, and they can be fooled by confident, fluent hallucinations. An uncalibrated judge will certify a bad system.

Calibrate the judge against human labels first. Run the judge on a few hundred human-graded cases and measure agreement, broken down by answer type. If agreement is low on a category — for example, code answers or refusals — do not use the judge for that category until the rubric improves. Write rubrics as explicit criteria with scoring levels, not vague instructions like "rate quality from 1 to 5". A good rubric says what a score of 5, 3, and 1 look like, with examples. Ask the judge to quote the evidence for its score, and to list unsupported claims, before giving the final grade. That structured output makes judge errors visible and auditable.

Practical judge setup

Use a different, stronger model for judging than for generation where budget allows, and pin the judge model version. A silent judge-model update invalidates trend comparisons just as surely as a generation-model update. Run the judge at low temperature for consistency, and repeat grading on a sample to measure judge variance. If the same answer receives very different scores across runs, the rubric is too subjective or the temperature is too high. Keep human review in the loop permanently: sample a fixed percentage of judged cases for human verification, and track judge-human agreement over time as a metric in its own right. When agreement drops, investigate before trusting new scores.

Example: minimal regression gate in CI

Fails the build if faithfulness or retrieval recall drops

def gate(scores, baseline): assert scores["retrieval_recall@5"] >= baseline["retrieval_recall@5"] - 0.02 assert scores["faithfulness"] >= baseline["faithfulness"] - 0.03 assert scores["refusal_correct_rate"] >= 0.90 return True ```

#Regression Gates, Model Changes, and Live Monitoring

Evaluation only protects production if it runs automatically and blocks bad changes. Wire the golden set into CI: every prompt change, model version change, embedding model change, or index rebuild triggers a full evaluation run, and the results are compared with the previous baseline. Set thresholds per metric, not a single composite score. A composite can hide a retrieval regression behind a generation improvement. Allow small, documented threshold breaches with explicit approval, but require a human sign-off recorded in the pull request. Silent threshold erosion — nudging the baseline down each time — defeats the gate. Track baseline history so erosion is visible.

Model version changes deserve special handling. Vendors update models, deprecate versions, and change behaviour without a major version bump. Pin exact model versions in configuration, and treat any version change as a release that requires a full golden run. Keep a shadow deployment option: route a percentage of live traffic to the new model, compare judge scores and user feedback against the old model, and promote only if the new model is not worse on the metrics that matter.

Monitoring production traffic

Static datasets go stale. Monitor live traffic continuously: sample conversations for human and judge review, track refusal rates, citation rates, user thumbs-down rates, and follow-up question rates. A spike in follow-up questions often signals unclear or incomplete answers. A spike in refusals may signal a retrieval failure or an overly cautious prompt. Segment metrics by intent, language, and product area, because aggregate metrics hide regressions in a minority segment. Feed confirmed failures back into the golden set, closing the loop between monitoring and regression protection. Also monitor cost and latency alongside quality: a reranker or larger judge that improves quality by two percent while doubling latency may not be worth shipping.

For the security side of LLM evaluation — prompt injection, data leakage, and adversarial inputs — a golden set should include adversarial cases, and the principles overlap with OWASP guidance for LLM applications. Quality evaluation and security evaluation are separate gates; passing one does not imply passing the other.

#Frequently Asked Questions

How many test cases do I need for a reliable evaluation?

Start with 50 to 100 cases covering your main intents, and grow to a few hundred as the system matures. Reliability depends more on coverage and label quality than raw count. A small set with diverse, realistic cases and clear labels gives a usable signal; a large set of near-duplicates gives a precise but misleading one. Report confidence intervals or at least score variance across runs, especially for judge-based metrics, so small changes are not over-interpreted.

Is LLM-as-judge reliable enough for production gates?

Yes, if calibrated and monitored. Calibrate against human labels, use structured rubrics with examples, pin the judge model version, and keep sampling human verification permanently. Do not use an uncalibrated judge as the sole gate for high-stakes answers. For those, combine judge scores with human review and deterministic checks such as citation verification and format compliance.

Should I optimise for a public benchmark like MMLU or HumanEval?

Public benchmarks are useful for comparing base models, but they rarely reflect your product's data, users, or failure modes. Use them for initial model selection, then build your own golden set for release decisions. A model that tops a public leaderboard can still fail on your retrieval corpus, your document formats, or your users' phrasing. Your golden set is the benchmark that matters for shipping.

How do I evaluate RAG when there is no single correct answer?

Score the components separately: retrieval recall and precision against known supporting documents, faithfulness against retrieved context, and completeness against a checklist of key points written by a domain expert. Accept multiple valid answers by listing acceptable variants in the golden set. For summarisation or advisory answers, human review of sampled outputs remains essential; automated metrics alone will miss nuance and tone problems.

What is the most common evaluation mistake?

Scoring only the final answer and therefore not knowing whether retrieval or generation failed. The second most common is changing the test set at the same time as the system, which makes score changes uninterpretable. Version prompts, models, indexes, and datasets together, change one thing at a time where possible, and always report which component changed alongside the score.

#Sources and Further Reading

• RAGAS, RAG evaluation framework documentation — https://docs.ragas.io/ — faithfulness, answer relevancy, and context precision metrics.
• TruLens, LLM evaluation documentation — https://www.trulens.org/ — feedback functions and production monitoring for LLM applications.
• LangSmith, Evaluation documentation — https://docs.smith.langchain.com/ — datasets, evaluators, and CI integration concepts.
• NIST, Artificial Intelligence Risk Management Framework — https://www.nist.gov/itl/ai-risk-management-framework — measurement and monitoring guidance for AI systems.
• OWASP Top 10 for LLM Applications — https://owasp.org/www-project-top-10-for-large-language-model-applications/ — security evaluation categories that complement quality evaluation.
• Anthropic, Prompt engineering and evaluation guidance — https://docs.anthropic.com/ — practical advice on test sets and model evaluation.