Cost and Latency at Scale

In a demo, cost and speed are invisible. At scale they decide whether the feature survives contact with a finance review. Every call costs tokens in and tokens out, every call takes time, and both multiply by your user count. Treat them as design constraints from the start rather than things you discover from a shocking bill or a feature nobody waits for.

Start by knowing your unit economics: what does one request cost, and what does one user cost per month? Until you can answer that, every optimisation is guesswork. Work it out from real traffic, because estimates are reliably wrong — the prompt grew, retrieval attached more context than you thought, and retries are doubling some calls.

The cost levers, roughly in order of return. Right-size the model: using a frontier model to classify a yes/no is the most common waste in production, and a small model that clears your quality bar is often ten times cheaper. Cache the stable prefix: if every call carries the same long system prompt or document, provider-side prompt caching usually pays for itself immediately. Cache whole answers for repeated questions, which in most support-style products is a surprisingly large share of traffic. Trim context to what's needed rather than everything available. And cap output length — output tokens typically cost several times input.

Routing is the pattern that combines these. Send everything to a cheap fast model first, and escalate only the cases it can't handle — detected by low confidence, a validation failure, or a classifier deciding the request is hard. Most workloads are dominated by easy requests, so this often cuts cost by most of it while leaving quality on the hard tail untouched. Your Level 3 evals are what tell you where the escalation threshold belongs.

Latency is a different problem with a different set of levers. Time-to-first-token is what users experience as responsiveness, so stream: it doesn't make anything faster, but it changes the felt experience completely. Do independent work in parallel rather than in sequence — retrieval and a classification call don't need to wait for each other. Keep prompts short, since input length adds real processing time. And prefetch or precompute anything predictable.

Notice the tension. The cheapest option is often the slowest per token, and the fastest is often the priciest. Long chains multiply latency the same way they multiply error, so a five-step agent is five round trips before the user sees anything. Set an explicit budget — a target cost per request and a target p95 latency — and make the trade-offs against it deliberately.

Measure percentiles, not averages. An average response time of two seconds can hide a p99 of thirty, and it's the p99 that generates complaints and abandoned sessions. Same for cost: watch the tail, because a small number of enormous requests often dominates the bill.

Put in the boring protections early. Per-user rate limits, a hard cap on tokens per request, a monthly budget alert, and timeouts. The expensive incidents in production are rarely gradual growth; they're a loop that retried ten thousand times overnight, or one user discovering they can paste a novel into your form.

The through-line from Level 2 returns: match the tool to the job. The cheapest model that clears your quality bar, proven with evals rather than assumed, is usually the right production choice — not the most capable one available.