Why "it looked good in the demo" is not enough

AI systems are probabilistic. The same question phrased two ways can produce different answers, and a change that improves one type of request can quietly break another. Without systematic evaluation, teams find out about problems from their customers.

Build an evaluation set first

An evaluation set is a collection of realistic inputs with the outcome you expect. Start with a few dozen to a couple of hundred examples drawn from real work: common requests, tricky edge cases, out-of-scope questions and inputs that should be refused.

For each example, record what a good answer must contain, what it must not contain and which sources it should use. Add to the set every time you find a new failure in production. Over time it becomes one of the most valuable assets the project produces.

What to measure

  • Correctness: is the answer or action right?
  • Grounding: is every claim supported by the retrieved sources?
  • Completeness: did it cover what the user needed?
  • Safety and policy: did it avoid restricted topics, personal-data leaks and unauthorised actions?
  • Tone and format: does it follow your style and structure?
  • Latency and cost: how long did it take, and what did it cost per task?

Who does the scoring

Use three layers. Automated checks catch objective failures such as missing citations, broken formats or forbidden content. A second model can grade subjective qualities against a written rubric, which scales well but must itself be checked. Human experts review a sample regularly and settle disagreements; their judgement is the reference everything else is calibrated against.

Treat changes like code releases

Every change to a prompt, model, retrieval setting or tool should run against the evaluation set before release, with results compared side by side with the current version. A new model that is better on average but worse on your most important cases is not an upgrade.

Keep a changelog of what changed and why, and keep the previous version deployable so you can roll back quickly.

Keep measuring in production

  • Sample live conversations or tasks for human review every week.
  • Track escalation and override rates; rising numbers are an early sign of drift.
  • Collect user feedback with a simple rating and an optional comment.
  • Monitor cost and latency per task against an agreed budget.
  • Feed every confirmed failure back into the evaluation set.