Running an agent’s tool calls in parallel with goroutines

When a reply contains several tool calls, the model has only listed them; executing them is up to the harness. Starting them all at the same moment seems like a free speedup, and in Go it takes a few lines. The harder questions are which calls are allowed to overlap, how to keep the conversation valid and the run reproducible, and whether the speedup is real. This page works through those questions with chapter 9 of the book and its runner, ch09/agent/runner.go.

How often models ask for several calls

Models do this on their own, and frequently. In the runs the book recorded from chapter 4 on, between a quarter of replies and every reply requested more than one tool, depending on which model and which task. Chapter 9 counts all four models on both tasks; here are two of them on each:

The book never sets parallel_tool_calls on a request. On models that honor it, setting it to false limits each reply to one call; of the book’s four models only z-ai/glm-5.3-flash accepts it, and Anthropic’s own API has a separate switch for the same purpose. There is no value that makes a model request more than one, and in spite of the name, the flag has no bearing on how your harness executes the calls it gets. Anthropic’s documentation is explicit that execution order is the caller’s business: run the calls however you like, provided each one gets exactly one result.

Which calls may run together

Some pairs must not overlap. In each of the default model’s ten bug-fix runs, a single reply contained an edit to the file followed by a test run. If both start together, the tests can execute before the edit lands, check the old code, and report a failure that the edit had already fixed.

Chapter 9 decides with the read-only flag that every tool has carried since chapter 7:

• A run of adjacent calls to read-only tools executes concurrently.
• Every other call executes by itself, waiting for all earlier calls and holding back all later ones; this is a barrier.

run_tests doesn’t count as read-only, since it executes code, including tests the agent wrote, so it acts as a barrier. A call to a tool the harness doesn’t recognize also gets its own stage, because nothing is known about what it does, and so does the final tool: once it accepts an answer, the calls after it are answered “not run”.

// stages splits a reply's calls, in the model's order, into the groups
// that run together: consecutive calls to read-only tools share a
// stage, and any other call, which may change something a later call
// reads, is a stage of its own, a barrier.
func stages(byName map[string]Tool, calls []ToolCall) [][]int {
	var out [][]int
	for i, c := range calls {
		n := len(out)
		if byName[c.Name].ReadOnly && n > 0 &&
			byName[calls[out[n-1][0]].Name].ReadOnly {
			out[n-1] = append(out[n-1], i)
			continue
		}
		out = append(out, []int{i})
	}
	return out
}

A finer scheme is possible: have each call declare the resources it reads and writes, build a dependency graph, and let two patches to separate files run at once. That scheme depends on every tool describing its effects precisely, and ties would still fall back to the order of the reply. A single bit per tool handles the cases that matter.

Running a stage

The runner launches each call in a stage with sync.WaitGroup.Go, added in Go 1.25, which does the counting that used to need a manual Add(1) and defer Done(), a pair that was easy to misplace. To bound concurrency, a buffered channel acts as a semaphore with eight slots per reply. Only three replies in the recorded runs asked for more than eight calls, and all three came from one model on the triage task. If the run is cancelled while a call is waiting for a slot, the call never starts and its answer is “cancelled”.

Three decisions make the runner correct:

• Each goroutine writes only its own slot. Answers go into a slice at the position the call had in the reply. Since every goroutine owns a distinct index, there is no shared write and no mutex, and the results end up in the model’s order regardless of which call finished first.
• Answers are reported only after Wait. The loop’s goroutine reports them, never a tool’s goroutine, because the loop’s event iterator has to be fed by the goroutine that is ranging over it.
• A panic in one tool is recovered at that tool’s boundary. The panic turns into that call’s answer; the other calls in the stage carry on, and one tool’s failure never cancels its neighbors.

The API accepts results in any order, so why preserve the model’s? Mostly for reproducibility: a run replays identically, and the book’s recorded runs and tests keep passing. It also keeps the next request byte-for-byte equal to what sequential execution would have produced, so the prompt cache still hits. Streaming results back as each call completes wouldn’t gain anything, since the next model call can’t start until every result is in.

The package’s tests check this. They replay the edit-then-test reply using a deliberately slow fake edit and a fake test tool that reports the version of the code it ran against. With every call started at once, the fake tests see the old code; with the staged runner, they see the edited code.

What it saves on a coding agent

Almost nothing, and chapter 9 has the numbers. It took the 87 tool calls from one recorded triage run and replayed them one by one, timing each:

All 60 read-only calls together took about a seventh of a second. Nearly all the time went to two tools: run_tests, and code_patch, which runs the formatter and a compile on every edit. Together they took 33 seconds, and both are barriers, so neither could overlap with anything. For comparison, a short answer from the default model took a median of 3.1 seconds per call. Overlapping the reads on this agent saves a few milliseconds per turn.

The runner stays in the code anyway. It is correct, it costs little, and it pays off for tools that spend their time waiting on remote services, a search API for instance, or a database on another machine.

What was measured, and what wasn’t

Chapter 9 measured three things here: how often each of four models asks for several calls in one reply, across the book’s saved runs; how long each triage tool takes, by replaying one recorded run’s 87 calls one at a time; and, in its tests, that the staged runner keeps an edit ahead of the test run that follows it. It did not measure the runner on tools that wait on a network, where the saving should be largest, since the triage agent has no such tool. The claim that overlapping those calls saves all but the slowest call’s time is arithmetic, not a measurement, and worth checking on your own tools before you count on it. It also didn’t build or measure the finer dependency-graph scheme described above.

A tool that can’t run at all

Some failures leave the model nothing to try. When the workspace has disappeared, every code_read fails, and a model that only sees “no such file” keeps guessing new paths until it hits the turn limit. For this case a tool returns an error wrapping ErrToolFatal. The runner halts at the current stage, gives every call that didn’t run the answer “not run” so the API still accepts the conversation, and the loop returns the error to its caller.

Others choose differently

The OpenAI Agents SDK, as its source stood in October 2026, launches all of a turn’s function tool calls together by default, with an optional limit on how many run at once. Ordering is left to the tools, except that its shell and patch tools each run one call at a time within their own kind. For independent lookups that works well. For a reply that edits a file and then runs tests, the ordering would have to come from the tools themselves.

When to use it

Try it

From the code directory of a clone of the companion repository:

$ go test ./ch09/agent/ -run 'Barrier|Stages' -race -v

TestStages checks how replies split into stages, and TestBarrier runs the edit-then-test reply both ways, every call at once and through the runner, inside a testing/synctest bubble, with the race detector on. The semaphore is in ch09/agent/slots.go.