Calling AI From Code: APIs and SDKs

A chat window is one interface to a model; the API is the one you build on. Instead of typing into a box, your code sends a structured request — a list of messages with roles — and gets back the model's completion. Everything you learned about prompting still applies. You're just doing it programmatically, which makes it repeatable, testable and embeddable in a real product.

The message list has three roles. The system message sets persistent behaviour and constraints. User messages carry input. Assistant messages are the model's previous replies. You send the whole relevant history on every call, because the API is stateless — it remembers nothing between requests. That surprises people coming from chat apps, where the product was quietly maintaining the conversation for them. If your app needs memory, your app stores it.

Authentication is a secret-handling problem, and it's where beginners get burned. Your API key goes in an environment variable or a secrets manager. Never in your source, never in a git commit, and never in client-side code — a key shipped to the browser is a key anyone can extract and spend. Calls to the model belong on your server, with the browser talking to your endpoint instead. Keys that do leak should be rotated immediately rather than quietly reused.

Then cost and latency, which are the same lever. You pay per token in and per token out, with output usually costing several times more than input. So the two things that drive your bill are how much context you send and how much text you ask for. A system prompt that grows to two thousand tokens is paid for on every single call, forever. This is where prompt caching earns its keep: most providers will let you mark a stable prefix — your system prompt, a long document — so repeat calls reuse it at a large discount. If you're sending the same preamble every time, it's usually the single biggest cost saving available.

Latency has its own shape. Time-to-first-token is what a user experiences as responsiveness; total time depends on how much you generate. Streaming the response token by token doesn't make the call faster, but it makes the wait feel dramatically shorter, which is why nearly every chat product streams.

A handful of parameters matter in practice. Temperature controls randomness — near zero for extraction, classification and anything you want reproducible; higher for ideation. Max tokens caps the output, which caps your cost and stops runaway generations. And structured output — asking for JSON matching a schema, which most providers now enforce natively — is what turns a model into something the rest of your code can consume without regex and prayer.

Build for failure from the first version, because networked calls to a busy service fail routinely. Rate limits and overload responses are normal, not exceptional: retry with exponential backoff and jitter. Set a timeout. Decide what your product does when the model is unavailable — degrade to something useful rather than showing a stack trace. And validate what comes back before trusting it; even with structured output, check the shape.

Keep the provider behind a thin layer of your own code. One module that takes your inputs and returns your types, with the vendor's SDK inside it. That way switching model or provider is an afternoon's work rather than a rewrite, and you can run two models side by side when you're evaluating. Given how fast this market moves, cheap switching is worth deliberately paying for.

The mental shift is from asking an AI to designing a call: what context you assemble, what you ask for, how you handle the response, what you do when it fails, and what it costs at scale. Once that clicks, everything else in this level is a variation on the same loop.