Does It Actually Work? Evaluations

AI features are easy to demo and hard to trust, because a model that dazzles on three examples can quietly fail on the fourth. Evaluations are how you replace "it seemed fine" with evidence: a repeatable way to measure whether your system does what it should, across many cases, especially the hard ones.

Start with a test set. Collect real inputs — from your logs, your users, your own work — paired with what a good output looks like. Include the tricky ones, the adversarial ones and the edge cases, not just the happy path. This is the part people skip, and it's the part that matters: an eval built only from examples you already handle well tells you nothing. Twenty genuinely representative cases beat two hundred easy ones.

Grow the set from failure. Every time something goes wrong in production, add that case to the eval before you fix it. Over a year this becomes the most valuable asset you own for the feature — a precise, accumulated record of every way your system has actually broken, which no competitor can copy and no model upgrade can invalidate.

Then define what good means, and be concrete. Some criteria are checkable by code and you should lean on those hardest: did it return valid JSON, include the required field, stay under the length, get the known-correct answer, cite a real source, avoid a forbidden term. Deterministic checks are fast, free and unambiguous.

Others are judgement calls — was the tone right, was it genuinely helpful, did it follow the brief. The standard technique is LLM-as-judge: a model scores outputs against a written rubric. It works well enough to be useful, with caveats worth knowing. Judges favour longer, more confident answers regardless of quality, and tend to favour output from the same model family. Give the judge a specific rubric rather than "rate this 1-10", ask it to state a reason before its score, and spot-check a sample against human judgement so you know how far to trust it.

Pairwise comparison is more reliable than absolute scoring. "Which of these two is better, and why?" produces far more consistent answers than "score this out of ten", both from models and from people. When you're choosing between two prompts or two models, compare them head to head on the same inputs.

The payoff is regression safety. Every time you change a prompt, swap a model, adjust chunking or tweak retrieval, re-run the eval and see in numbers whether it improved. Without this, every change is a gamble and your users are the test suite. With it, you ship deliberately — and you can show a buyer or a boss the evidence rather than an anecdote.

This matters more than it used to, because the ground moves under you. Providers update models, sometimes with the same name, and behaviour shifts. If your quality depends on a model's exact quirks and you have no eval, you find out from complaints. Run the suite on a schedule as well as on every change, and keep the numbers where the team can see them.

Watch cost and latency alongside quality, or you'll optimise one into the ground. A prompt that scores three points higher and costs four times as much is a decision, not an improvement — and it should be made deliberately rather than discovered in a bill.

Start smaller than feels rigorous. Ten cases in a spreadsheet, run by hand, with one automatic check and one rubric, will catch more real problems this month than a perfect framework you never finish building.