Inside a Large Language Model

A large language model does one thing, over and over: given all the text so far, predict what comes next. That's it. Everything impressive it does is that single operation repeated at enormous scale. Understanding this one mechanism explains almost every strange behaviour you'll ever see from an assistant.

First, models don't read letters or words — they read tokens. A token is a chunk of text, usually a short word or a piece of one. "Cat" is one token. "Unbelievable" might split into "un", "believ", "able". Common words get their own token; rare ones get chopped up. As a rough rule, 100 tokens is about 75 English words.

This sounds like trivia until it explains things. Ask a model how many r's are in "strawberry" and it may well get it wrong — not because it's stupid, but because it never saw the letters. It saw two or three tokens. You're asking someone to count brush strokes in a word they only ever heard spoken. Ask it to rhyme or work with individual characters and you're fighting the same limitation. The fix is not a better model, it's a different question: ask it to spell the word out letter by letter first, and accuracy jumps, because now the letters are actually in the text.

Tokens also explain cost and speed. You're billed by the token, and languages that tokenize poorly cost more to process. The same paragraph in English and in Hindi or Thai can differ several-fold in token count, which is a real and under-discussed unfairness in how these systems are priced.

Second, prediction. The model turns your tokens into numbers, runs them through billions of learned parameters, and produces a probability for every possible next token. Then it picks one, appends it, and does the whole thing again. Sentence by sentence, an answer builds itself one token at a time. The model never plans the ending before it starts — which is why asking for a short answer often works better than asking it to trim a long one after the fact.

This also explains why the same prompt gives different answers on different days. The model doesn't always take the single most likely token; there's deliberate randomness in the pick. In developer tools that dial is called temperature — low for consistency, higher for variety. In a chat app it's set for you, which is why you can regenerate a reply and get something meaningfully different.

Third, context. Everything the model can see while it predicts is called the context window: your prompt, the conversation so far, any documents you've pasted, and any instructions the product added invisibly. Modern windows range from tens of thousands of tokens to well over a million, depending on the model — hundreds of pages of text in the larger ones.

Two rules fall out of this and they are the most practically useful things in this lesson. One: the model is only as good as what's in the context. Most disappointing answers are not a model failing, they're a context problem — you knew something relevant and didn't say it. Two: outside the window, nothing exists. Start a new chat and the model has no idea who you are or what you discussed yesterday, unless the product deliberately saves and re-injects it.

Finally, hallucination. Because the model is optimising for what a plausible answer looks like, it will sometimes produce text that is fluent, confident, well-formatted and false. A fabricated citation looks exactly like a real one — same shape, same tone, same confidence — because shape and tone are precisely what the model learned. There is no internal alarm that fires when it doesn't know.

That sounds damning, and it isn't, because the failure is predictable. A tool that fails randomly is unusable; a tool that fails in a known way is one you can design around. You already do this with other tools — you don't distrust a calculator, you distrust your own typing. The rest of this level is about where to put the checks.