Evaluating an LLM Feature Before Launch: A Practical Guide

A demo can make an LLM feature look finished. A few polished prompts produce convincing answers, and the feature moves toward launch. Then real users arrive with incomplete requests, contradictory details and questions the system was never designed to answer.

The problem is not that the model suddenly became bad. The team never defined success for the actual task.

To evaluate an LLM application, narrow its job until success can be observed, build a small but representative evaluation set from real inputs, and select checks that match the consequences of failure. Decide what blocks release before looking at the scores. Otherwise, evaluation becomes a collection of numbers that can always be explained away.

This guide evaluates the complete feature—not a leaderboard comparison between foundation models. Include input handling, prompt, model, retrieval or tools, output parsing and fallback behavior.

TL;DR#

• Replace “answer anything” with one bounded user outcome and explicit failure behavior.
• Start with real, privacy-safe inputs and label the expected result or a scoring rubric.
• Include ordinary requests, edge cases, ambiguous inputs and adversarial attempts.
• Combine deterministic checks, task-specific quality review, latency and cost.
• Use an LLM judge only after checking its agreement with human ratings.
• For RAG, score retrieval and answer generation separately.
• Set launch gates in advance and rerun the same evaluation set after every material change.

1. Narrow the task until it can be evaluated#

“Build an assistant that answers questions about our product” is not an evaluation objective. It describes a broad interaction surface, not a testable outcome.

Start by writing a one-sentence task contract:

Given [input], the feature must produce [observable output] for [intended user], while refusing, asking for clarification or escalating when [defined condition] applies.

For example, replace “answer questions about support” with:

Given a customer’s question about an order, return the current order status and the next permitted action using the authenticated account’s order data; if the order cannot be identified, ask for the missing identifier without guessing.

Now the team can test whether the answer uses the correct order, whether the next action is allowed, and whether the system asks a question when the input is incomplete.

Define the boundaries alongside the happy path:

• Inputs: supported formats, languages, fields and context.
• Outputs: required fields, allowed sources, citations, tool calls or response shape.
• Authority: which systems or documents are allowed to support an answer.
• Abstention: when to say “I don’t know,” ask a follow-up or route to a person.
• Prohibited behavior: actions or disclosures the feature must never perform.

A narrow task is a boundary that lets you detect when the system is wrong. If the feature has several materially different jobs—classification, extraction and free-form advice—give each its own evaluation slice. A blended score can hide a serious failure behind easy wins elsewhere.

2. Build the first evaluation set#

An evaluation set is a versioned collection of inputs, context and expected behavior. It should represent the requests you expect the feature to handle, not just the examples that make the demo look good.

Collect inputs from more than one source#

Start with:

• Support tickets, search queries, user-submitted forms and existing workflow records.
• Historical application logs, if the feature already exists in a limited or manual form.
• Human-written examples from domain experts who understand the task.
• Failed or low-rated outputs from a prototype or staged rollout.
• Synthetic examples for rare, sensitive or adversarial conditions that have not appeared in production.

Real inputs reveal vocabulary and missing context that a product brief rarely captures. Synthetic cases exercise gaps but should not replace the real distribution. OpenAI’s evaluation guidance similarly recommends task-specific datasets, production and historical data, edge cases and continuous evaluation rather than generic metrics or “vibe-based” review. OpenAI: Evaluation best practices.

Before using production data, remove or transform personal information, secrets and customer identifiers. Follow retention, access and consent requirements; keep only the context needed for evaluation.

How large should the first set be?#

There is no universal minimum that makes an evaluation valid. Start with a deliberately small set that you can inspect manually—perhaps a few dozen carefully selected cases—then expand it as you learn where the feature fails. It is a starting heuristic, not a statistical guarantee.

Track coverage by scenario, not just row count. A set of ordinary requests may be less useful than a smaller set that includes missing information, unsupported requests, conflicting evidence and harmful instructions. Keep a stable regression subset alongside a working set for new cases.

Store an expected answer or a rubric#

For constrained tasks, store an exact expected value or machine-checkable schema. For open-ended answers, store required facts, allowed sources, disallowed claims and a rubric. Do not force one exact sentence when many answers can be correct.

Hypothetical evaluation set — customer order support

These rows are illustrative, not customer data or results from a deployed system. Each production evaluation row should also record the feature version, expected output or rubric, relevant context, labels, and the reason the case exists.

Add edge cases that stress your task contract: misspellings, partial identifiers, long inputs, duplicated records, stale documents, contradictory context, empty fields and unsupported languages. Add adversarial cases that try to override instructions, extract hidden context, trigger unauthorized tools or persuade the feature to invent missing information. For agentic features, evaluate tool choice and tool arguments—not only the final prose.

3. Choose metrics that match the risk#

There is no universal “LLM quality score.” Pick a small set of measurements that answer distinct questions, and report important failure categories separately.

For structured output, a deterministic validator should check required fields, types, enums and cross-field constraints. A response that sounds good but fails the downstream JSON contract is still a failed request.

For factual tasks, define what counts as a material error. A wrong order status, invented policy or unsupported medical statement is not equivalent to a slightly awkward phrase. Track critical errors independently instead of averaging them into a high overall score.

For latency and cost, measure the complete path—not just generation. Include retrieval, tools, retries and post-processing. Report median and tail latency, segmented by task or request size. The relevant cost unit might be cost per successful resolution rather than cost per raw model call.

Do not select metrics merely because a framework makes them easy to calculate. Text overlap can penalize a valid paraphrase; semantic similarity can reward an answer that sounds close but changes a crucial detail. Inspect examples behind aggregate results.

4. Use model judges carefully#

An LLM judge can scale review of open-ended qualities such as relevance, completeness, groundedness or tone. It is not an oracle, and it should not be the only authority for high-impact outcomes.

Start with a rubric that names observable criteria. For example:

1. Does the response answer the specific question?
2. Are all material claims supported by the supplied order context?
3. Does it avoid promising an action the policy does not permit?
4. Does it ask for clarification when the order is ambiguous?

Define score levels with examples. A score of 1 might mean all criteria pass; a 0 might mean a critical policy or factual failure; intermediate scores should describe partial completion. If a single severe failure must block release, encode it as a separate binary gate rather than letting other strengths compensate.

Then calibrate the judge:

• Have qualified reviewers label a sample independently using the same rubric.
• Run the judge on those exact examples without exposing human labels.
• Compare pass/fail agreement and category-level disagreements.
• Review false passes especially carefully: cases where the judge approves an answer humans consider unsafe or incorrect.
• Revise the rubric and repeat until the judge is useful for the intended screening role.

For subjective comparisons, pairwise review—asking which of two outputs better satisfies the rubric—can be easier to apply consistently than assigning an absolute score. Randomize answer order and watch for length or position bias. OpenAI’s grader guidance describes reference checks, code graders and model graders, and explicitly warns about grader hacking; LangSmith documents code, model, composite and pairwise evaluators for offline and online workflows. OpenAI: Graders · LangSmith: Evaluation types.

Keep humans in the loop for safety-critical decisions, policy interpretation, novel failures and periodic audits. A judge that agrees on routine cases can still miss a rare consequential category. Treat judge prompts and judge model versions as part of the evaluation system.

5. If the feature uses RAG, evaluate retrieval separately#

A retrieval-augmented generation feature has at least two failure surfaces:

1. Retrieval: did the system find the right evidence and rank it usefully?
2. Generation: did the model answer correctly using the evidence it received?

If the answer is wrong, inspect the retrieved chunks before changing the prompt. The retriever may have missed the relevant policy, selected an outdated version or placed irrelevant content above the authoritative source. A better generator cannot reliably use evidence it never received.

For retrieval, evaluate:

• Context recall: did retrieval include the evidence needed to answer?
• Context precision: how much of the retrieved context was relevant, especially near the top?
• Source/version correctness: did it retrieve the current and authorized document?
• Access boundaries: did it exclude documents the current user must not see?

For generation, evaluate factual correctness against the source, whether claims are supported by retrieved context (faithfulness or groundedness), whether citations point to supporting passages, and whether the system abstains when evidence is insufficient. Ragas lists context precision, context recall and faithfulness among its RAG metrics; these describe different parts of the pipeline, not one interchangeable score. Ragas: Available metrics.

Preserve retrieval traces in your evaluation records: query, retrieved document IDs and versions, ranking, chunks passed to the model, final answer and citations. This separates retrieval regressions from generation regressions. Keep authorization tests separate and blocking; a high relevance score does not excuse retrieving another tenant’s private document.

6. Set launch gates before you see the result#

A launch decision should be a policy, not a meeting where the team negotiates thresholds after seeing a disappointing score.

Write down the gates for the feature’s actual risk. A useful checklist:

Launch checklist: (1) task contract defines supported inputs, outputs, scope and abstention; (2) representative real, edge and adversarial cases are included; (3) every case has an expected result, rubric or deterministic assertion; (4) no critical privacy, authorization, safety or policy test fails; (5) task success and factual quality meet pre-agreed thresholds; (6) structured output passes schema and business-rule validation; (7) latency and cost remain within budget; (8) reviewers inspect failures and a sample of passing outputs; (9) automated judges are calibrated against human ratings; (10) rollback, fallback, escalation and monitoring are ready.

Set actual numerical thresholds only after considering the consequence of failure, baseline behavior, expected traffic and available human review capacity. For a low-risk drafting assistant, the team may tolerate occasional stylistic variation. For an assistant that changes account data or provides regulated guidance, the release bar must be stricter and critical failure categories may require zero observed failures in the defined test set. That still does not prove zero risk; it describes the tested boundary.

After launch, monitor feedback, escalations, corrections, task completion, latency, cost and safety events. Sample traces under a privacy-aware retention policy and convert new failures into test cases. Offline regression tests protect against known failures; online monitoring helps discover cases the dataset did not anticipate. LangSmith describes this offline/online loop, including backtesting historical production runs and feeding production anomalies into future datasets. LangSmith: Evaluation types.

Run the evaluation set whenever you change the system prompt, templates, model, model parameters, retrieval configuration, chunking, tools, output schema or relevant application code. Save the configuration and results with each run. Compare against the previous accepted version, inspect regressions by category and update the baseline only through an intentional review.

What to do next#

Treat evaluation as part of feature development, not a final quality check. Narrow the task, collect real examples, define observable success and make critical failures release blockers. A small evaluation set that the team understands is more useful than a large score whose meaning nobody can explain.

Use the ML System Design Checklist to review the wider production system around your model feature—including data, serving, reliability, monitoring and operational concerns. For deeper treatment of agent evaluation boundaries, read Your AI Agent Eval Is a Production Security Boundary. For model and runtime testing, see Benchmark the Model–Harness Pair, Not the Model.

References#

• OpenAI: Evaluation best practices · OpenAI: Graders · LangSmith: Evaluation types · Ragas: Available metrics

Muhammad Azlaan Zubair

Software Engineer and Researcher, writing about AI, developer tooling and automation.

WebsiteGitHubLinkedInInstagramThreadsXORICIDGoogle Scholar