Verify, Trust, and Ship: Quality Control on AI Output

Everything you've built in this level is worthless if you can't trust the output enough to ship it — or if you trust it too much and ship something wrong. The practitioner's edge isn't blind faith or blanket suspicion. It's calibrated verification: spending your checking effort exactly where the risk is.

Sort output by two questions. How reversible is it, and how verifiable is it? A throwaway brainstorm is reversible and low-stakes — ship it, no checking. A number in a board deck, a legal clause, a customer-facing claim, a line of code going to production — high-stakes, and must be checked against a source rather than re-read for vibes.

The failure mode to fear is the answer that sounds right. Fluent, confident, well-formatted, and wrong. Your instincts for spotting a weak human answer — hedging, vagueness, bad grammar — are actively misleading here, because the model is fluent regardless of whether it's correct. Polish is not evidence. That is the single hardest habit to build, because it means distrusting the exact signal you've spent your whole life using.

Know where the risk concentrates. Specifics are riskier than generalities: names, dates, figures, quotes, citations, case numbers, legal clauses, API methods. Anything the model would have had to recall precisely rather than reason about is where fabrication lives, and a made-up citation is formatted identically to a real one. Recent events are riskier than settled ones. Niche topics are riskier than mainstream ones. Long chains of reasoning are riskier than single steps.

Your verification toolkit is concrete. Check facts and figures against the primary source, not against a second AI answer. Ask the model to cite where each claim came from, and then actually open the citations — this catches fabrication fast, because invented sources don't resolve. Run an important prompt twice in separate chats and compare: stable answers aren't necessarily right, but answers that vary between runs are a reliable warning. For anything technical, test it rather than trust it — run the code, check the query returns what you expect.

One technique worth using more: ask the model to attack its own output. "What's the weakest claim here?" "What would a subject-matter expert object to?" "Which of these numbers would you want to check?" Models are noticeably better at finding problems than at avoiding them, and this costs one message. It won't catch everything — it can be confidently wrong about its own confidence — but it surfaces plenty.

Be careful about one thing: don't ask the model whether its own answer is true and treat a yes as verification. Asked "are you sure?", a model will often either cave and change a correct answer or reassure you about a wrong one. It has no privileged access to its own reliability. Use it to generate candidate problems, then check those against reality yourself.

For teams, bake this into the workflow as an explicit review step with a written checklist, so quality doesn't depend on whoever happens to be conscientious that day. The checklist should name the specific things that matter for your work — every figure traced to a source, every citation opened, every customer-facing claim approved by whoever owns that claim.

The goal isn't to slow down. It's to move fast safely. Once you know which outputs need verification and which don't, you stop double-checking trivia and start reliably catching the dangerous things. That calibration is the difference between an organisation that has deployed AI and one that is still experimenting with it.