Structured output: an agent’s answer in a fixed shape

A chatbot’s last message is written for a human. The last step of many agents is written for a program: the crash-triage agent has to say which reports share a cause, where the bug lives, which test proves it, and which tracker issue it belongs to. Code that scrapes those facts out of prose works until the model words things differently, which it will. The answer needs a shape the program can decode.

Chapter 7 of the book gives the triage agent such a shape. Here are the two available approaches, the one the book picks, and the part that counts for more than the format, which is verifying what the answer claims.

Two ways to get structure

A constrained reply. Structured outputs is a request option, response_format, that tells the API to make the reply conform to a JSON Schema. Support varies by model and by provider. On OpenRouter, leaving out require_parameters allows routing to a provider that doesn’t honor the option. Many providers also won’t apply it in a turn that offers tools. The book reserves it for single calls that use no tools, for example condensing the early turns of a long conversation.

A final tool. Give the agent an extra tool, submit_diagnosis in this case, and make its arguments the answer. Nothing about it is special: the schema comes from the same generator as every other tool’s, the arguments go through the same decoder and checks, and a rejected call produces an ordinary tool error. Any model that supports tool calling can use it. The book takes this route by default.

The answer as a Go type

The answer is declared as a Go struct, tagged the same way as the argument structs on the tool design page, so its schema is generated:

// A Diagnosis is the agent's answer for one crash group: which reports
// share a cause, where the code is wrong, the test that shows it, and
// what was done about it.
type Diagnosis struct {
	Fingerprint string   `json:"fingerprint" desc:"the group's fingerprint, from crash_list"`
	Reports     []string `json:"reports" desc:"the IDs of every report in the group"`
	Category    string   `json:"category" enum:"nil_value,out_of_range,concurrency,bad_input,other"`
	RootCause   Location `json:"root_cause" desc:"where the code is wrong, which may not be where it crashed"`
	Evidence    string   `json:"evidence" desc:"why this is the cause: what you read and ran"`
	Reproducer  Location `json:"reproducer,omitzero" desc:"the test you wrote that fails before the fix; leave out for already_fixed and duplicate"`
	Fix         string   `json:"fix,omitzero" desc:"what you changed, for status fixed"`
	Status      string   `json:"status" enum:"fixed,already_fixed,duplicate,not_a_bug,needs_human"`
	Issue       int      `json:"issue,omitzero" desc:"the issue you filed, for status fixed or needs_human"`
	Duplicate   int      `json:"duplicate,omitzero" desc:"for status duplicate: the open issue it duplicates"`
}

// A Location is a function in a file of remind's source.
type Location struct {
	File     string `json:"file" desc:"a file path, such as store/store.go"`
	Function string `json:"function" desc:"a function or method, such as (*Store).Get or TestGetDeleted"`
}

Fields that code will switch on, the status above all, are enumerations. Each field’s description explains it to the model, and the descriptions spell out the easy-to-miss points, such as a root cause that “may not be where it crashed”. A field the model has nothing for is omitted from the JSON; the model never sends null. There is one Diagnosis for each crash group, and the crash catalog’s grader, which chapter 13’s evals use, reads the same record.

How the loop ends on it

Marking a tool Final changes the loop’s stopping rule. When a call to that tool succeeds, the run is over and the call’s arguments are the run’s result:

// finished reports whether a reply held a successful call to the final
// tool, and returns its arguments, the run's answer. A call that failed
// went back to the model as an error, and the run goes on.
func finished(byName map[string]Tool, calls []ToolCall,
	answers []Message) (jsontext.Value, bool) {
	for i, c := range calls {
		if byName[c.Name].Final && i < len(answers) &&
			!answers[i].IsError {
			return c.Arguments, true
		}
	}
	return nil, false
}

If the call is rejected, the model receives the error as a tool result and keeps going. That gives the final tool the power to turn an answer down and lets the model correct it. Once a final call succeeds, nothing else in that reply runs: later calls in the reply get “not run” as their result, so the answer matches the code exactly as it stood at the moment of acceptance.

A model that writes its answer as plain text instead of calling the tool gets a nudge: a user message telling it to call the tool once it is finished, or to carry on with the work. If two nudges don’t help, the run ends with ErrNoFinalAnswer, a sign that the program never received an answer it could use; callers test for it with errors.Is.

Checking what the answer claims

The schema makes sure the answer has the right fields and types. Whether the content is correct is a separate matter, and a model sounds just as sure when it is wrong. submit_diagnosis therefore verifies everything it can before it says yes:

• Each report belongs to one group and only one.
• A group marked duplicate points at an issue that is open, and a group marked fixed points at an issue that exists.
• For a fixed group, the reproducer test is run twice. Against the untouched original plus the tests the agent wrote, it has to fail; against the agent’s changed code it has to pass.

The reproducer check is the one that tests the substance of the claim. If the test already passes before the fix, it says nothing about the crash, no matter how convincing the fix is. A test name that doesn’t match any test gets a dedicated message, because that usually means a typo. Every failed check comes back to the model as an error explaining the problem, and the run continues.

The chapter is clear about what this can’t catch. A test can fail for an unrelated reason, a fix can land in the wrong place, and a flaky test might get through by chance. Chapter 13’s grader narrows that gap with hidden tests the agent never sees.

One run, end to end

go run ./ch07 replays a recorded run of anthropic/claude-haiku-5.5 on one catalog task. The task has four crash reports: two are crashes already in the tracker, and two come from a bug located away from the line that crashed.

The agent got the difficult part right. The crash is a nil pointer dereference inside api.owned, but the defect is in store.Get: for a deleted reminder it returns neither a reminder nor an error. Guarding against nil in owned would have ended the panic while leaving every other caller of Get open to the same bug.

Because the answer was typed, its error was easy to see too. The catalog’s grader checked it against the expected answer 16 ways, and 12 passed. All four failures came from one mistake: the agent marked the two crashes already in the tracker as fixed when they should have been duplicates of the existing issues. The first system prompt gave no instruction for a crash that matches a filed issue, and the next chapter’s prompt work begins with that. In a prose answer the mistake would have been one clause in a paragraph; in a struct it shows up as failed checks. The run used 17 turns and cost $0.034.

When to use which

Whichever you choose, validate in your own code as well: decode strictly, enforce enumerations and required fields, and test every claim in the answer that a program is able to test.

Try it

From the code directory of a clone of the companion repository, with no key set:

$ go run ./ch07

This replays the run described above against a fresh copy of the service; every tool really executes, the reproducer check’s go test runs included, and the grader’s checks print at the end. The final tool and its checks are in ch07/triage/diagnosis.go.