Observability: Seeing What Your AI Does

Traditional software either works or throws an error. AI fails quietly: it returns a fluent, plausible, wrong answer and nothing crashes, nothing alerts, no stack trace appears. That's why observability matters more here than in ordinary systems — without it your feature can degrade for weeks and your first real signal is a customer leaving.

The first layer is per-request tracing, and the rule is to capture the whole chain rather than just the ends. For each request: the user input, what retrieval returned, the prompt actually sent after assembly, the model and version, the parameters, the raw output, any tool calls with their arguments and results, latency broken down by step, tokens in and out, and cost. When something goes wrong you need to see exactly what the model saw, and "the prompt actually sent" is the field teams most often forget to log and most often wish they had.

Version everything that shapes behaviour. Prompts, retrieval configuration, model identifiers, and the eval suite itself should carry versions attached to each trace. Otherwise "quality dropped on Tuesday" is unanswerable, because you can't tell which of the four things that changed on Tuesday did it.

The second layer is aggregate monitoring: latency percentiles, cost per request, error and timeout rates, refusal rates, tool-call failure rates, retrieval hit rates, and quality scores from your evals — tracked over time with alerts on drift. Refusal rate is an underrated signal; a sudden rise usually means either an attack or a prompt change that made the model over-cautious, and both are worth knowing about within hours rather than weeks.

The third layer is the human feedback loop. Capture thumbs up and down, edits users make to generated text, escalations to a human, retries and abandonments. Edits are the richest signal of all — the difference between what you produced and what the user actually wanted is a direct measurement of your gap, and it arrives for free.

Then close the loop, which is the step most teams skip. Feedback that lands in a dashboard nobody acts on is theatre. Route negative signals into a review queue, look at them on a regular cadence, and turn the real failures into permanent cases in your eval suite. That's the mechanism by which a system improves in production rather than ossifying.

Watch specifically for drift, because it's the failure mode this level exists to catch. The world changes, your users change how they phrase things, your data changes, and providers ship model updates — sometimes behind an unchanged name. Run your evals continuously against production samples, not only in CI, so a silent behaviour change surfaces as a number rather than a complaint.

Be deliberate about privacy while you do this. Prompt logs contain whatever users typed, which is frequently personal and occasionally regulated. Decide retention periods, scrub or hash what you don't need, restrict access, and say what you keep in your privacy policy. Full-fidelity logging is enormously useful and is also a liability if you never thought about it.

Observability plus evals is what turns "we think it's working" into "we know it's working, here's the dashboard" — which is the answer a professional gives a boss, an auditor, or an enterprise buyer, and the reason those buyers sign.