How a language model picks its next token

Four habits of the model shape every agent built on it. It can give two different replies to one request. It is billed by the token rather than the word, and one text can come to more tokens on one model than on another. It sounds just as sure when it is wrong as when it is right. And it can ask for tools.

A single mechanism accounts for all four, and a few dozen lines of Go, with no machine learning in them, are enough to watch it work. This page follows chapter 3 of the book, whose experiments live in ch03 of the companion code and run offline.

Next-token prediction

A language model has one job: read some text and assign a probability to every token that could come next. A token is a short piece of text, often a whole word and sometimes a fragment of one; the Tokens section below shows where they come from. To write a reply, the model repeats that step: choose a token, add it to the end of the text, and predict again from the longer text. That repetition is called next-token prediction. Everything a model sends back, a paragraph of advice or a tool call, comes out of it one token at a time.

A model made of counts

The smallest model that works this way just counts. Chapter 3’s toy reads a training text, an early draft of the book’s first two chapters of roughly 7,600 words, and records, for each pair of words, which word followed the pair and how often. The pair “the model” was followed by 38 words in all, 5 of them “sees”, so the toy rates “sees” at 5/38, about 0.13.

Generating means looking up the last two words and choosing from what followed them:

// Generate continues from the words a and b for up to n more words.
// At temperature 0 it always takes the most likely word; above 0 it
// samples from the probabilities, using rng.
func (m *Model) Generate(a, b string, n int, temperature float64,
	rng *rand.Rand) []string {
	out := []string{a, b}
	for range n {
		choices := m.Next(a, b, cmp.Or(temperature, 1))
		if len(choices) == 0 {
			break // a pair the text never had
		}
		word := choices[0].Word
		if temperature > 0 {
			word = sample(choices, rng.Float64())
		}
		out = append(out, word)
		a, b = b, word
	}
	return out
}

The choice can be made in two ways, and each matches something real models do.

Greedy decoding loops. Taking the most probable word every time, the toy reaches “a key and a 401);” and then repeats it forever, because the likeliest continuation of that phrase leads straight back to it. Small models writing long replies can fall into the same trap, which is one reason APIs sample by default.

Sampling differs from run to run. Choose in proportion to the probabilities instead, and three runs from the same starting words go three separate ways. This is the main source of varying answers: every call works out the same probabilities, and the server draws a fresh sample from them.

Fluent and wrong

Here is the toy’s greedy run:

Greedy, temperature 0:
The model ID and the program replays a conversation of your own, run
the finished agent on a real model in its place, and you can reproduce
it without a key. When OPENROUTER_API_KEY is empty, it replays replies
saved from the agent's ToolCall. The agent's Reply gains two fields to
carry this, along with a key and a 401); and a 401);

The sentence about OPENROUTER_API_KEY sounds like the book, since each run of three words in it occurs somewhere in the training text. It is also false: the saved replies come from the API, and ToolCall has nothing to do with them. The toy has no notion of truth; it writes whatever is probable.

A real model is vastly better at probable text, and probable text is true often enough that models are worth using. Still, no stage of the mechanism checks facts, so a false statement that happens to be probable arrives in the same assured voice as a true one. The industry’s word for this is hallucination. The rule it gives an agent is to verify claims with tools. The crash-triage agent in the companion code must produce a test that fails on the old code and passes with its patch; saying it found the bug isn’t enough.

Temperature

Temperature is a setting applied to the probabilities before the choice. Values below 1 sharpen them, so the favorite wins more often, and 0 means always taking the favorite, which is greedy decoding. Values above 1 even them out, and long-shot tokens get chosen more often. At 0.5 the toy’s “sees” climbs from 0.13 to about 0.33. A real model applies temperature by dividing each token’s score by it before the softmax, the step that turns scores into probabilities; chapter 3 shows that the toy’s exponent is that same operation written another way.

Chapter 3 then put a small open model, meta-llama/llama-3.1-8b-instruct, through it: describe Go in exactly three words, ten calls at each of four temperatures. Temperature 0 produced one reply, ten times over. At 0.7 and at 1, all ten replies differed and all of them were sensible. At 1.5, six of the ten fell apart into strings like “Concurrentoons statically”: once an improbable token is chosen, it becomes the start that every later token is predicted from.

Why identical runs differ

Sampling explains most of the variation, but setting the temperature to 0 doesn’t remove it. Providers run many requests together on their hardware, and the arithmetic can come out slightly differently from one batch to the next. When two tokens are nearly tied, that difference is enough to flip the choice, and the text diverges from that token on. A router such as OpenRouter can also pass two requests for one model to two different providers, whose hardware differs. In chapter 3, one provider serving the Llama model at temperature 0 returned the same reply in ten calls out of ten; another, serving the same model at the same setting, returned two different replies across nine calls.

In an agent the effect compounds. A different token early in a turn can mean a different tool call, and every later turn starts from that one. The book’s evals, which run each task many times and report rates, are built around this, and Two identical runs scored 15 points apart shows what happens when you compare single runs. So check what an agent accomplishes, such as the tests passing or the right issue filed, and never the exact words it wrote.

Tokens

Real models work in neither words nor letters. A frequent word gets a token of its own, and a rare one is assembled from fragments. Most tokenizers are built with byte-pair encoding: begin with the 256 possible byte values, then again and again merge the most common neighboring pair into a new token. Chapter 3 implements one in Go and trains it on the toy’s text. An English sentence comes out at about a token per word, while ParseDuration splits into P, ar, se, D, uration.

Production tokenizers handle code far better than that, but the rule survives: text unlike the tokenizer’s training data costs more tokens. In October 2026 the book measured the same two texts on five models. A 1,008-byte Go file came to between 336 and 502 tokens, and a page of English to between 450 and 598. Four of the five landed within 7 percent of each other. The fifth, the book’s default model, anthropic/claude-haiku-5.5, needed about a third more tokens than the others for the English and about 50 percent more for the Go. A price per token can mislead, then; compare models by what a whole task costs. Tokens, cost, and the context window follows that through an agent’s bill.

Attention, briefly

The toy looks two words back; a real model needs a word from thousands of tokens earlier. Attention works like a fuzzy map lookup: each position scores its query against every earlier position’s key and takes a blend of their values, weighted by the match. Because the weights come from a softmax that hands out a fixed total of 1, every token added to the context leaves less attention for the ones that matter. In chapter 3’s small attention example, putting a hundred more words between “test” and “it” cut the weight on “test” to 0.03. That is part of why a model’s accuracy falls as its input grows.

A tool call is text

A model consumes and produces a single sequence of tokens. Your request is a list of messages, so the provider first flattens it into one text with a chat template, using special tokens to mark where each role’s message starts and ends. Chapter 3’s chatml package renders the start of the bug fix the way Qwen3’s template does:

<|im_start|>system
# Tools

You may call one or more functions to assist with the user query.

You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"type": "function", "function": {"name": "read_file", "description": "Read a file in the project.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}}, "required": ["path"]}}}
</tools>

For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call><|im_end|>
<|im_start|>user
The tests in this project fail. Find out why and fix it.<|im_end|>
<|im_start|>assistant
<tool_call>
{"name": "read_file", "arguments": {"path": "duration_test.go"}}
</tool_call><|im_end|>
<|im_start|>user
<tool_response>
package duration
...
</tool_response><|im_end|>
<|im_start|>assistant

The tools are in that text too. read_file appears as one line of JSON in the system message, after English instructions that say to call a function by writing a JSON object between <tool_call> tags. The assistant’s turn shows the result: the tool call is nothing more than text the model wrote in that format, and the provider parses it back out into the reply’s tool_calls field. The tool’s result returns the same way, as a user message wrapped in <tool_response> tags. These tags are Qwen3’s; each model family has its own template, and closed models keep theirs private, but every one of them turns tool calls into text and back.

Three consequences follow for an agent:

• A tool call fails the ways text fails. The JSON can be malformed, an argument can have the wrong type, and the name can belong to no tool at all. The harness, the Go program around the model, validates each call and replies with an error that says how to correct it.
• Instructions and data travel together. A file the agent opened or a crash report it was handed ends up in the same stream as your system prompt. If that text tells the model to ignore its instructions, the model weighs it like any other text. This is prompt injection, and special tokens can’t prevent it.
• Calls don’t teach the model anything. A call leaves the model’s weights unchanged, so whatever it should know about your task must be sent again in each request.

Run it

From the code directory of a clone of the companion repository:

$ go run ./ch03 toy
$ go run ./ch03 bpe
$ go run ./ch03 template
$ go run ./ch03

The first three run the toy model, the tokenizer, and the chat template. The last runs every experiment in order, ending with the probe that asks a real model for its token probabilities; with OPENROUTER_API_KEY unset, the probe replays recorded calls.