Draft handbook
Audience: AI engineers, software engineers, product builders, technical PMs, and teams shipping LLM features into real user workflows.
Core promise: By the end of this handbook, you will know how to design a practical evaluation system for LLM applications even when there is no single correct answer.
The first time most teams evaluate an LLM application, they do it by vibes. They run ten prompts, read the answers, decide the system “looks pretty good,” and ship it. Then production happens.
A customer asks the same question in a slightly different way. The system retrieves the wrong document. A prompt change fixes one issue and quietly breaks five other workflows. The model gives a confident answer that sounds right but is unsupported by the source material. A new model version improves reasoning but starts ignoring formatting instructions. A support bot that worked beautifully in the demo begins inventing refund policies for angry customers.
The team then discovers the real problem was not only building the LLM app. The real problem was knowing whether the app was working.
Traditional software gives engineers a comfortable testing model. You write a function, define expected outputs, and run unit tests. If the function returns the expected result, the test passes. If it does not, the test fails. LLM applications break this model. Their outputs are open-ended. Two answers can be different but equally acceptable. A short answer may be better than a long answer. A polite refusal may be better than an attempted answer. The system may be correct only if it uses the retrieved context, follows policy, avoids sensitive data, and stays within latency and cost targets.
This is why evaluation is one of the biggest bottlenecks in production AI. Teams can build prototypes faster than they can measure them. They can generate impressive demos faster than they can verify reliability. They can swap models, prompts, retrieval systems, and agent frameworks in a day, but they often have no disciplined way to know whether the change helped.
A production LLM evaluation system is not a single metric. It is a layered process for answering four questions:
If your evaluation process cannot answer those questions, every LLM change becomes a gamble.
This handbook is about replacing that gamble with an engineering workflow.
“No ground truth” does not mean “we cannot evaluate this.” It means there may not be one exact answer that every correct output must match.
For example, suppose a user asks:
Summarize this sales call and identify the next steps.
There may be many acceptable summaries. One might be concise. Another might be more detailed. One might group next steps by owner. Another might group them by date. Exact text matching would mark most good answers as wrong.
Or suppose a customer asks a support assistant:
Can I cancel my plan and get a refund?
A good answer depends on plan type, account status, contract terms, timing, company policy, and the customer’s intent. The assistant may need to ask a clarifying question, retrieve policy, call an internal tool, or escalate to a human. Again, there is no single sentence that represents “the” correct answer.
No-ground-truth tasks usually fall into several categories:
These include writing, summarization, coaching, tutoring, brainstorming, and customer response generation. Quality depends on usefulness, tone, clarity, and completeness.
These include RAG systems, policy assistants, document question answering, legal research assistants, medical literature assistants, and internal knowledge bots. The answer must be supported by context, but the wording may vary.
These include systems that choose tools, plan steps, update records, send messages, create tickets, or operate software. The final answer matters, but so do the intermediate actions.
These include recommendation, ranking, rewrite, and comparison systems. The goal may be to produce the answer that users prefer, not the one that matches an official answer key.
These include refusal behavior, privacy preservation, regulated-domain guidance, and brand safety. The system may need to avoid answering even when it technically can.
The mistake teams make is treating all of these like exact-answer tests. They try to force the output into a rigid expected string. When that fails, they abandon evaluation entirely and return to manual inspection.
The right approach is different. You evaluate against criteria, not exact text. You ask whether the output meets the requirements of the task.
A helpful mental model is:
Traditional test:
input + expected output → pass/fail
LLM evaluation:
input + context + output + rubric → judgment + explanation
The key artifact is the rubric. The rubric defines what good means.
In conventional software, test-driven development means writing tests before implementation. In AI engineering, the equivalent is evaluation-driven development: define evaluation criteria before building or changing the system.
This idea is simple but often ignored. Before you build an LLM app, ask:
If you cannot answer those questions, you are not ready to optimize. You may still prototype, but you should not pretend you can make disciplined production decisions.
Evaluation-driven development prevents several common traps:
Demos reward fluency, speed, and confidence. Production rewards reliability across messy inputs.
Classification tasks are easier to evaluate than open-ended generation. But many valuable AI products are open-ended. Avoiding them because they are hard to evaluate can cause teams to miss high-value opportunities.
One engineer finds one bad answer, changes the prompt, and declares the system fixed. Without regression testing, the team does not know what else changed.
A model that scores higher on public benchmarks may perform worse in your product, on your data, with your latency and cost constraints.
The production mindset is:
Every meaningful LLM system change should be evaluated against representative examples before it ships.
This does not require a massive platform on day one. A spreadsheet, a JSONL file, a simple script, and a judge prompt can be enough to start. The important thing is creating a repeatable loop.
The most important evaluation question is not “What metric should we use?” It is “What does good mean for this product?”
For most LLM apps, good is multi-dimensional. A customer support answer may need to be correct, grounded, empathetic, concise, policy-compliant, and actionable. A code assistant may need to be syntactically valid, secure, idiomatic, and aligned with the repository’s conventions. A RAG assistant may need to retrieve the right sources, cite them, avoid unsupported claims, and say “I don’t know” when evidence is missing.
A practical evaluation framework should include these dimensions:
Did the system solve the user’s problem?
Examples:
Are the claims true?
For grounded systems, correctness should be evaluated relative to provided sources. For open-world tasks, correctness may require reference material, human review, or external verification.
Does the answer stay supported by the available context?
A system can be factually true but still ungrounded if it uses information not present in the retrieved documents. In many enterprise settings, groundedness matters because the assistant is only authorized to answer from approved sources.
Does the answer cover the important parts of the request?
Incomplete answers are common in LLM apps. A support bot might correctly explain cancellation but omit refund timing. A RAG system might answer one part of a multi-part question and ignore the rest.
Does the answer address the actual user intent?
LLMs often answer a nearby question instead of the asked question. Retrieval systems may return topically related but non-answering context.
Did the system follow format, tone, policy, and workflow instructions?
This includes JSON schemas, citation formats, refusal rules, tool-use constraints, and brand voice.
Did the system avoid harmful, disallowed, private, or legally risky content?
Safety evaluation should be domain-specific. A healthcare assistant, finance assistant, HR assistant, and internal developer tool have different risk profiles.
Can the system deliver the required quality within acceptable budget and response time?
A 2% quality improvement may not be worth a 5x latency increase. A slower judge model may be acceptable offline but not in production.
Does the system behave well under failures?
For agents, this includes tool errors, missing context, ambiguous user requests, invalid tool outputs, permission failures, and timeouts.
A mature evaluation system measures several of these dimensions separately. Do not collapse everything into one “quality score” too early. A single score hides the failure mode. A score of 82 does not tell you whether the system is hallucinating, too slow, too verbose, unsafe, or retrieving the wrong context.
Use scorecards, not vibes.
A golden dataset is a versioned collection of examples used to evaluate whether your system is improving or regressing. It is the foundation of practical LLM evaluation.
Many teams delay building one because they think a golden dataset requires perfect reference answers. It does not.
A golden dataset can include:
The important thing is that the dataset represents the work your system must perform.
Good evaluation examples come from multiple sources.
These are the most valuable because they reflect actual language, ambiguity, and edge cases. Remove private information before using them.
Every reported failure is a future regression test. If the system failed once, you want to know if it fails again.
Domain experts can create cases that represent important workflows, policies, and edge cases.
Synthetic data can help expand coverage, especially for rare cases. But synthetic examples should not dominate the dataset because they often miss the messiness of real users.
These include prompt injection, missing context, contradictory context, ambiguous requests, policy-boundary cases, and requests that should be refused.
A useful starting schema:
id: refund_policy_017
input: "I cancelled yesterday. Do I get my money back?"
context:
- document_id: refund_policy_v3
text: "Customers on monthly plans may request..."
expected_behavior: >
Explain refund eligibility based only on the policy. If plan type is unknown,
ask a clarifying question. Do not promise a refund.
rubric:
correctness: "Does not invent eligibility rules."
groundedness: "Uses only supplied policy context."
completeness: "Mentions plan type and cancellation timing."
safety: "Does not provide legal guarantees."
risk_level: high
tags: [support, refunds, policy, clarification]
reference_answer: null
Notice that this example does not require a perfect answer. It defines acceptable behavior.
Start small. A useful first dataset can have 30 to 50 examples. That is enough to reveal obvious regressions and calibrate your judge. Expand toward 100 to 300 examples as the product stabilizes. High-stakes systems may need thousands.
Sample size affects confidence. Large differences can be detected with fewer examples. Small differences require many examples. If one version is dramatically better, a small dataset may show it. If the improvement is subtle, you need more examples to avoid fooling yourself.
A practical rule:
Do not build a dataset of only happy paths. A production evaluation set should include:
Tag everything. Tags let you see whether a change improves one segment while harming another.
Example tags:
refunds, billing, enterprise, no_context, ambiguous, prompt_injection,
privacy, multi_turn, tool_failure, summarization, citation_required
Treat the evaluation dataset as a product asset. Version it like code.
Track:
If the dataset changes every time you run an eval, you cannot compare results. Freeze a baseline set for regression testing, and maintain a separate candidate set for new examples.
LLM-as-judge means using one model to evaluate another model’s output. The judge receives the input, output, optional context, optional reference answer, and rubric. It returns a score, pass/fail decision, preference, or explanation.
LLM judges are popular because human evaluation is expensive and slow. They are useful because many LLM outputs cannot be graded with exact-match metrics. But they are dangerous because the judge is also an LLM: it can be biased, inconsistent, overconfident, and wrong.
The right stance is:
An LLM judge is not ground truth. It is a scalable approximation that must be calibrated, monitored, and periodically audited.
LLM judges tend to work best when:
Examples:
LLM judges struggle when:
For these cases, combine LLM judging with deterministic checks, tool-based verification, specialist review, or human labeling.
There are two common judge patterns.
The judge gives one answer a score, such as 1–5, or pass/fail.
Pros:
Cons:
The judge compares answer A and answer B and chooses the better one.
Pros:
Cons:
A strong production workflow often uses both:
The judge receives a reference answer or expected behavior.
Use this when you have expert-created answers, verified outputs, or required facts.
The judge evaluates based on the input, context, and rubric without a reference answer.
Use this when there are many possible correct answers.
Reference-free judging is common, but it requires better rubrics and more calibration.
Do not ask the judge to “rate the answer.” Ask for structured fields.
Example:
{
"passes": true,
"score": 4,
"failure_categories": [],
"explanation": "The answer is grounded in the supplied policy and asks for plan type before making a refund claim.",
"evidence": ["Mentions monthly vs annual plan", "Avoids promising refund"]
}
Structured outputs make results easier to aggregate, debug, and track.
The quality of your judge depends heavily on the quality of your rubric. A vague rubric creates vague scores.
Bad rubric:
Rate whether the answer is good.
Better rubric:
The answer passes if it directly addresses the user’s question, uses only the provided policy context, states uncertainty when required information is missing, and does not promise actions the company has not authorized.
It describes observable behavior.
Instead of:
The answer should be helpful.
Use:
The answer should include the cancellation deadline, refund eligibility condition, and next action the user should take.
Do not mix everything into one score.
Example dimensions:
correctness: pass/fail
completeness: 1-5
groundedness: pass/fail
clarity: 1-5
policy_compliance: pass/fail
Judges need to know what bad looks like.
Example:
Fail if the answer:
- claims a refund is guaranteed
- uses information not present in the context
- ignores the user's cancellation timing
- fails to ask for plan type when plan type is required
Include examples of passing and failing outputs. This helps both human reviewers and LLM judges align.
A 1–10 score often looks scientific but may be noisy. For many production gates, pass/fail plus failure category is more useful.
Recommended scoring options:
You are evaluating an AI assistant response for a customer support product.
You will receive:
1. The user question
2. The retrieved policy context
3. The assistant answer
4. The evaluation rubric
Your job is to judge only whether the assistant answer satisfies the rubric.
Do not reward information that is not supported by the context.
Do not penalize wording differences if the meaning is correct.
Return JSON with these fields:
- passes: boolean
- correctness: 1-5
- groundedness: 1-5
- completeness: 1-5
- policy_compliance: 1-5
- failure_categories: array of strings
- explanation: short explanation
Rubric:
The answer should:
- address the user's question directly
- rely only on provided context
- ask a clarifying question if eligibility depends on unknown plan type
- avoid promising refunds unless the context clearly supports it
- provide a safe next step
User question:
{{input}}
Retrieved context:
{{context}}
Assistant answer:
{{output}}
Human review does not disappear when you adopt LLM judges. It becomes more focused.
Use humans to:
A calibration set is a small group of examples labeled by trusted reviewers. Use it to test whether the LLM judge agrees with human expectations.
Start with 30–50 examples. Include easy cases, borderline cases, and known failures.
For each example, collect:
Then run the LLM judge and compare.
Ask:
You do not need academic perfection, but you do need signal. Track agreement rates by dimension.
Example:
correctness agreement: 86%
groundedness agreement: 91%
policy compliance agreement: 78%
completeness agreement: 72%
Low agreement is not automatically bad. It may reveal that humans also disagree or that the rubric is unclear.
Disagreement is often where the most valuable learning happens. If reviewers disagree about whether an answer is acceptable, your users may also disagree. The solution may be better product policy, clearer assistant behavior, or a more precise UX flow.
For healthcare, legal, finance, hiring, education, security, and other high-impact domains, LLM judges should not be the only line of defense. Use them for triage and regression detection, but keep humans in the loop for high-risk decisions.
RAG systems require special evaluation because failures can happen in multiple places.
A bad RAG answer may be caused by:
If you only evaluate the final answer, you may not know which component failed.
Separate RAG evaluation into retrieval evaluation and generation evaluation.
Did the retrieved context include the information needed to answer the question?
Low recall means the answer was not available to the model.
Were the retrieved chunks relevant, or did the retriever include distracting noise?
Low precision can cause hallucination or confusion.
Were the most useful chunks near the top?
This matters because models pay more attention to earlier and more salient context.
Did retrieval include all necessary sources for multi-hop or policy-dependent questions?
Some questions require multiple documents.
Are claims supported by retrieved context?
Does the answer address the user’s question?
Are citations present, accurate, and attached to the right claims?
Does the system refuse or ask for clarification when context is insufficient?
Use failure categories such as:
retrieval_miss
retrieval_noise
wrong_source
stale_source
chunk_boundary_failure
reranker_failure
unsupported_claim
citation_mismatch
answer_incomplete
should_have_refused
A failure taxonomy turns evaluation from a score into an engineering roadmap.
Many RAG systems are evaluated only on questions where the answer exists. That is dangerous. Production users ask questions your knowledge base cannot answer.
Include examples where:
The correct behavior may be:
A RAG system that always answers is not reliable. It is just confident.
LLM apps regress silently. A change that improves one example can break another.
Regression testing means running the same evaluation set before and after a change, then comparing results.
Run regression tests when changing:
1. Choose evaluation dataset version.
2. Run current production system as baseline.
3. Run candidate system.
4. Score both outputs with the same evaluators.
5. Compare by dimension and tag.
6. Inspect failures manually.
7. Ship, revise, or reject the change.
Average score can hide important regressions.
Compare:
Example:
Candidate improves average quality from 4.1 to 4.3,
but high-risk policy examples drop from 92% pass to 81% pass.
Decision: do not ship.
For mature systems, add evaluation to CI/CD. A pull request that changes prompts or retrieval should run a smaller smoke eval. Nightly jobs can run the full suite.
Example gates:
- Overall pass rate must not drop by more than 2%.
- High-risk examples must maintain at least 95% pass rate.
- Groundedness must not regress.
- p95 latency must stay below 4 seconds.
- Cost per successful answer must not increase by more than 20% without approval.
Each evaluation run should record:
Without run history, you cannot explain why quality changed.
Quality is not the only metric that matters. Production AI systems operate under product constraints.
Track:
Cost per successful task is more useful than raw cost. A cheap system that fails often may be more expensive operationally than a costly system that resolves issues.
Track:
User experience often depends on tail latency, not average latency.
Track:
The most important evaluation question is whether better eval metrics correlate with better product outcomes.
Examples:
If your eval score improves but business outcomes do not, revisit the rubric.
Offline evaluation tells you whether a change is likely safe. Production monitoring tells you what users actually experience.
Track live signals such as:
Do not judge every production request with an expensive model unless necessary. Sample intelligently.
Prioritize:
Production failures should become evaluation examples.
Workflow:
1. User reports bad answer.
2. Triage and categorize failure.
3. Add anonymized example to candidate dataset.
4. Write expected behavior/rubric.
5. Fix system.
6. Add example to regression suite.
7. Verify future changes do not reintroduce failure.
This loop is how your evaluation system improves over time.
Symptom: Engineers inspect a few examples manually and ship.
Fix: Create a versioned dataset and repeatable eval runner.
Symptom: Judge says outputs are “good” even when they violate product requirements.
Fix: Use product-specific rubrics and failure categories.
Symptom: Judge scores are treated as truth.
Fix: Calibrate against human labels and audit disagreements.
Symptom: Team keeps changing prompts when the real problem is missing context.
Fix: Evaluate retrieval separately.
Symptom: System looks good on answerable questions but hallucinates when context is missing.
Fix: Include no-answer and ambiguous cases.
Symptom: Quality improves but latency, cost, or safety regresses.
Fix: Use a scorecard with guardrail metrics.
Symptom: Model performs suspiciously well on public benchmark-like examples.
Fix: Use private, product-specific examples and track data provenance.
Symptom: Team cannot explain why eval scores changed.
Fix: Version prompts, datasets, models, and judge prompts.
Symptom: Eval set is clean and unrealistic.
Fix: Add real user logs and production failures.
Symptom: Prompt changes repeatedly break old behavior.
Fix: Run baseline/candidate comparisons before shipping.
Deliverables:
Actions:
Deliverables:
Actions:
Deliverables:
Actions:
Deliverables:
Actions:
Abstract advice is useful, but evaluation becomes clearer when you see the entire workflow. This chapter walks through a realistic support assistant for a SaaS company.
Imagine a company sells a subscription analytics product. Customers ask questions about billing, refunds, plan limits, invoices, cancellation, data retention, and account access. The company wants an AI support assistant that can answer common questions using internal help center articles and account metadata.
The product team initially tests the assistant manually. It answers ten sample questions well. The demo looks impressive. But before release, the team identifies several production risks:
These risks cannot be captured by one accuracy score. The team needs an evaluation system.
DimensionWhy it mattersGate?Policy correctnessPrevents wrong refund/cancellation adviceYesGroundednessEnsures answers come from approved documentsYesClarification behaviorPrevents guessing when account details are missingYesEscalation behaviorRoutes sensitive/high-value cases to humansYesToneMaintains support qualityNoLatencyKeeps chat usableYes
Notice what the team did not do. They did not ask, “Was the answer good?” They decomposed “good” into operational requirements.
The first dataset has 50 examples: 20 common billing questions, 10 refund and cancellation edge cases, 5 account-security questions, 5 angry customer messages, 5 no-answer cases where the policy is missing, and 5 prompt-injection or policy-bypass attempts.
Example row:
id: support_refund_004
input: "I cancelled this morning. Can you refund my annual plan?"
context:
account_plan: "annual"
days_since_renewal: 42
policy_excerpt: "Annual plans are refundable within 30 days of renewal unless otherwise specified in the enterprise agreement."
expected_behavior: >
The assistant should explain that the standard annual refund window appears to be 30 days,
note that the user is outside that window, avoid making a final account decision, and offer
escalation to billing support.
rubric:
must:
- mention the 30-day annual-plan window
- not promise a refund
- explain that billing support can review exceptions
- use a calm, professional tone
fail_if:
- says the user definitely qualifies
- invents exception rules
- ignores the annual plan detail
risk_level: high
tags: [billing, refund, annual_plan, high_risk]
This example does not require exact wording. It requires the assistant to respect policy.
MetricResultOverall pass rate74%High-risk pass rate58%Groundedness pass rate82%Clarification pass rate61%Escalation pass rate44%p95 latency3.8s
The average does not look terrible, but high-risk results are unacceptable. Failure categories reveal the real issues:
unsupported_policy_claim: 9
should_have_escalated: 7
should_have_asked_clarifying_question: 6
outdated_context_used: 3
tone_too_casual: 2
Without this eval, the team might have shipped a bot that performed well on easy questions and poorly on risky ones.
The team makes four targeted changes: add refund-policy instructions, require account lookup or clarification when account data is missing, prefer current policy documents during retrieval, and add escalation logic for enterprise, angry, legal, and refund-exception cases.
MetricBaselineCandidateOverall pass rate74%86%High-risk pass rate58%84%Groundedness pass rate82%92%Clarification pass rate61%81%Escalation pass rate44%78%p95 latency3.8s4.4s
The candidate is better, but slower. The team decides the quality gain is worth the latency increase for high-risk flows, but not for simple FAQ flows. They implement routing: low-risk questions use the faster path; high-risk questions use the safer path.
Consider an internal RAG assistant for engineering documentation. Employees ask questions about deployment, incident response, database migrations, and service ownership.
The team first evaluates only final answers. The assistant scores 80%. That sounds promising. But inspection shows three different problems: the retriever missed the right document, the right document was retrieved but buried under irrelevant chunks, and the model ignored the right context and guessed.
These require different fixes. A better prompt will not solve a retrieval miss. A better embedding model may not solve generation hallucination.
Split RAG evaluation into three layers.
Retrieval layer: Did we retrieve at least one document containing the answer? Was it in the top 3? Did we retrieve stale docs? Did we include too much irrelevant context?
Context assembly layer: Did chunk boundaries preserve needed information? Did the context include titles and dates? Did it include enough surrounding context? Did irrelevant text consume the context budget?
Generation layer: Did the model answer using retrieved material? Did it cite the right source? Did it refuse when the answer was missing? Did it distinguish current docs from deprecated docs?
id: deploy_rollback_021
input: "How do I roll back the payments service after a bad deploy?"
expected_sources:
- docs/runbooks/payments/rollback.md
- docs/services/payments/ownership.md
retrieved_sources:
- docs/runbooks/search/rollback.md
- docs/runbooks/payments/rollback.md
- docs/archive/payments_old_deploys.md
retrieval_judgment:
context_recall: pass
context_precision: partial
ranking_quality: fail
answer_judgment:
groundedness: partial
completeness: fail
citation_quality: fail
failure_categories:
- relevant_doc_not_ranked_first
- stale_doc_included
- incomplete_runbook_steps
This record shows why final-answer evaluation is not enough. The right document was retrieved, but not ranked first, and stale context contaminated the answer.
VariantContext recallContext precisionAnswer groundednessp95 latencyBaseline72%81%78%2.1sTop-k 1084%62%73%2.4sHybrid83%79%84%2.5sHybrid + reranker86%88%89%3.7s
The best quality comes from hybrid + reranker, but latency rises. The team chooses hybrid search for all users and reranking only for complex questions.
Evaluation tools help, but they do not solve the hard part. The hard part is defining what good means. A tool can run experiments, store traces, compare versions, and visualize results. It cannot decide your risk tolerance.
Experiment and tracing platforms help store prompts, traces, outputs, datasets, feedback, and run histories. They are useful when multiple people need shared visibility.
Metric libraries help with RAG metrics, LLM-as-judge scorers, faithfulness checks, answer relevance, and test-case management. They are useful when you want a custom harness but do not want to write every metric from scratch.
Custom eval harnesses are scripts that load a dataset, run your application, call scorers, write results, and compare baseline and candidate runs. They are useful when data cannot leave your environment or your product has unusual requirements.
Buy or adopt a platform when multiple people inspect traces, you run frequent experiments, human review workflows matter, or dashboards are needed.
Build a lightweight harness when you are early, the workflow is simple, data is private, or you need deep integration with internal systems.
Many teams start with a custom harness and later add a platform.
A common anti-pattern is installing an eval library, turning on several metrics, and assuming the numbers represent product quality. Metrics must map to product risks. “Answer relevance” may help a support assistant, but it will not tell you whether the assistant violated refund policy. “Faithfulness” may catch unsupported claims, but it will not tell you whether the tone was appropriate for an angry customer.
Use tools as infrastructure, not strategy.
Agents add complexity because they do not only generate text. They choose actions: search documents, call APIs, update records, send emails, create tickets, run code, ask follow-up questions, and decide when to stop.
Evaluating only the final answer misses most of the risk.
Tool selection: Did the agent choose the right tool? A common failure is answering from memory instead of checking the account API.
Tool arguments: Did the agent pass correct arguments? Looking up the wrong customer ID can be worse than refusing to answer.
Tool result interpretation: Did the agent understand the tool response? If an API returns refund_eligible: false, the agent must not tell the customer they qualify.
Step efficiency: Did the agent take a reasonable number of steps, or did it loop?
Recovery behavior: Did the agent handle errors gracefully, or invent results after a timeout?
Stop behavior: Did the agent stop when finished, or continue taking unnecessary actions?
id: agent_invoice_009
user_goal: "Send me my latest invoice"
expected_behavior: >
Authenticate the user, retrieve latest invoice metadata, provide secure invoice link,
and avoid exposing invoice data if authentication fails.
trace:
- step: classify_intent
result: billing_invoice_request
- step: call_tool
tool: get_user_account
args: { user_id: "u_123" }
- step: call_tool
tool: get_latest_invoice
args: { account_id: "acct_456" }
- step: respond
output: "Here is your latest invoice..."
judgment:
tool_selection: pass
tool_arguments: pass
auth_policy: pass
final_answer: pass
latency: pass
wrong_tool
missing_tool_call
bad_tool_arguments
tool_result_misread
unsafe_action
permission_failure
looping
premature_stop
failure_to_escalate
invented_tool_result
For high-stakes agents, gate not only final answer quality but also action correctness. A beautifully written final response is not acceptable if the agent updated the wrong record.
Evaluation is not only a technical artifact. It is a team habit.
Every production LLM feature should have an evaluation owner. This person maintains the dataset, reviews failures, updates rubrics, ensures evals run before release, and communicates quality tradeoffs. Without ownership, evals rot.
For active products, run a weekly or biweekly eval review. Keep it short:
Write release rules before there is pressure to ship.
A prompt/model/retrieval change may ship if:
- high-risk pass rate does not decrease
- overall pass rate improves or stays within 2% of baseline
- policy compliance remains above 95%
- p95 latency remains below 5 seconds
- no new critical failure category appears in manual review
Release rules prevent debates from becoming subjective.
A failure library is a collection of memorable examples that teach the team what can go wrong. Include the user input, bad output, failure category, fix, and eval added to prevent recurrence. This library becomes training material for engineers, reviewers, and product managers.
Not every LLM app needs the same evaluation intensity. A brainstorming assistant and a medical triage assistant should not have the same release bar.
Examples: brainstorming, drafting internal notes, summarizing non-sensitive meetings, creative ideation.
Evaluation focus: usefulness, clarity, latency, and user satisfaction. Human review can be lightweight.
Examples: customer support drafts, sales email generation, internal knowledge search, document summarization.
Evaluation focus: correctness, groundedness, completeness, brand/tone, and escalation behavior. Use regular regression testing and sampled human review.
Examples: healthcare advice, legal guidance, finance recommendations, HR decisions, security operations, autonomous account actions.
Evaluation focus: policy compliance, safety, auditability, human approval, strict refusal behavior, access control, and trace review. LLM judges can assist, but they should not be the final authority.
The goal is not perfect evaluation. The goal is evaluation proportional to risk.
A script can run examples. An evaluation system changes how a team ships.
A real evaluation system has a definition of product quality, a versioned dataset, clear rubrics, calibrated judges, human review where needed, regression testing, production monitoring, failure taxonomy, ownership, release gates, and continuous dataset improvement.
This is what separates production AI teams from demo teams.
The demo team asks, “Does it look good?”
The production team asks:
That is the mindset shift this handbook is designed to create.
id,input,context,expected_behavior,reference_answer,rubric,risk_level,tags,source,notes
Recommended fields:
id
user_input
conversation_history
retrieved_context
expected_behavior
reference_answer_optional
rubric
risk_level
tags
source
created_at
reviewed_by
privacy_status
known_failure_mode
DimensionTypeGate?NotesCorrectness1–5YesMust answer accuratelyGroundednessPass/failYesRequired for RAGCompleteness1–5NoTrack by tagPolicy compliancePass/failYesHard blockerClarity1–5NoProduct qualityLatency p95numericYesUX constraintCost/requestnumericNoBudget constraintTool successpass/failYesAgent reliability
Evaluation run: 2026-07-06
Dataset version: eval_v0.3
Baseline: prompt_v12 + model_a
Candidate: prompt_v13 + model_a
Overall pass rate:
- Baseline: 84%
- Candidate: 88%
High-risk pass rate:
- Baseline: 91%
- Candidate: 87%
Groundedness:
- Baseline: 94%
- Candidate: 93%
Latency p95:
- Baseline: 3.1s
- Candidate: 3.4s
Decision:
Do not ship yet. Candidate improves general cases but regresses high-risk policy cases.
Top failure categories:
1. should_have_asked_clarifying_question
2. unsupported_claim
3. incomplete_answer
You are comparing two assistant answers for the same user request.
Choose the better answer according to this rubric:
1. Correctness
2. Groundedness in provided context
3. Completeness
4. Policy compliance
5. Clarity and usefulness
Do not prefer an answer merely because it is longer.
Do not reward unsupported claims.
If both answers are unacceptable, say both_fail.
If they are equally acceptable, say tie.
Return JSON:
{
"winner": "A" | "B" | "tie" | "both_fail",
"reason": "short explanation",
"key_differences": ["..."],
"failure_categories": ["..."]
}
User request:
{{input}}
Context:
{{context}}
Answer A:
{{answer_a}}
Answer B:
{{answer_b}}
Track:
This handbook synthesizes concepts from production AI engineering practice, the provided AI Engineering reference by Chip Huyen, and current public resources on LLM evaluation. Useful areas to explore further:
Suggested public sources for follow-up reading:
Before you ship an LLM app change, ask:
If the answer is yes, you are no longer shipping by vibes. You are building an evaluation system.
%402x.svg){kind=icon}