
Durable agent runs: checkpoints, resuming, and doing things exactly once
Surviving a crash without losing work or doing it twice: checkpoints, locks, and idempotency keys.
On the agent compass The whole loop, all four stagesPicture the triage agent partway through a long batch. It has filed one issue and is writing a test for the next crash group when a deploy restarts the machine. If the run simply starts over, it pays a second time for every model call it had made. Worse, if the process died just after the tracker created that issue and before the agent wrote the answer down, the new process has no idea the issue exists and files another. With other tools the same gap means a customer refunded twice or a deploy run twice.
Both gaps close with two pieces: ch15/durable, which stores checkpoints, and ch15/agent, whose loop saves and resumes them.
What to save
Run state is whatever a later process needs in order to carry on: the messages so far, tokens and dollars spent, the turn count, the context policy’s summary, any calls waiting on a person, and a phase recording the loop’s next move.
This loop uses four phases: model when a model call comes next, tools once a reply’s calls are fixed but not all answered, paused while a person decides, and done once the answer exists.
Spending and turn counts are in the state for a reason that is easy to miss.
If a restart forgot them, a run that crashed every few turns would start each life with a fresh budget and never reach its limits.
Restarts get a limit of their own as well, ten by default, after which the run stops with ErrMaxResumes; the count is saved before anything else happens, so a run that dies during every restart still reaches it.
Files the agent changed are part of the state too.
The state can keep a hash (SHA-256) for every file the run has written; on resume, a file that no longer matches stops the run with ErrWorkspaceChanged rather than letting the model reason about code its conversation doesn’t describe.
The sample triage command doesn’t wire that check yet, since only the run’s own process touches its workspace; the finished crash-triage agent records each write and checks the workspace on every resume.
Save at five moments
The natural place to save is the end of each turn, after its tools have run, and it is the wrong place. Tools run in the middle of the turn, and some of them change things outside the process. Lose the turn and the new process asks the model again, and a real model may answer with different calls, which then run on top of the first reply’s calls.
So the loop saves more often:
| # | When | What a crash right after it would otherwise lose |
|---|---|---|
| 1 | Before the first model call | The run itself: it was accepted, and nothing on disk says so |
| 2 | Once the reply is in the conversation, before any of its tools start | The reply, so a restart asks again and may run a second set of calls |
| 3 | Immediately before each call that can change the outside world | The knowledge that the call may have acted |
| 4 | After every tool result | The result, so a finished call runs again |
| 5 | When the run pauses for a person, and when it ends | That it is waiting, or that it already has its answer |
After save point 2, a restarted run only ever runs the calls of the saved reply. A reply that was saved is never paid for twice; a reply that arrived and was lost before its save is requested again.
Write a checkpoint a crash can’t tear
A checkpoint is only useful if, whenever the crash comes, the disk holds either the old copy or the new one, never half of one. Writing over the file in place breaks that. The short version: write a new copy beside the old one and swap it in, so the disk always holds one whole checkpoint. The usual fix has four steps (write a temporary file, sync it, rename it over the old one, sync the directory), written out in the companion code:
// write replaces the checkpoint file with b: write a temporary file,
// sync it, rename it over the old one, then sync the directory.
func (c *Checkpoint) write(b []byte) error {
tmp, err := os.CreateTemp(c.dir, "."+c.run+"-*.tmp")
if err != nil {
return fmt.Errorf("create: %w", err)
}
defer os.Remove(tmp.Name()) // fails harmlessly after the rename
_, err = tmp.Write(b)
if err == nil {
err = c.sync(tmp)
}
if cerr := tmp.Close(); err == nil {
err = cerr
}
if err != nil {
return fmt.Errorf("write: %w", err)
}
c.point("mid-save")
if err := os.Rename(tmp.Name(), c.path); err != nil {
return fmt.Errorf("rename: %w", err)
}
c.point("after rename")
if err := c.syncDir(); err != nil {
return fmt.Errorf("sync directory: %w", err)
}
return nil
}
The temporary file goes in the same directory because a rename can’t cross file systems, and the rename is what swaps old for new in one step.
Syncing the file first keeps the new name from pointing at data that never reached the disk; syncing the directory afterwards keeps the rename itself from vanishing in a power cut.
On macOS, Go’s File.Sync also asks the drive to empty its own cache.
On an M1 Pro, a recorded batch run made 142 saves, with medians between 8 and 12 ms across seven measurements, against model calls whose median was over two seconds. The temporary file is created with permissions 0600, which matters because a checkpoint contains every tool result, and tool results contain whatever the agent read.
Three smaller choices are worth copying, listed below for when you write your own.
- The loop saves with
context.WithoutCancel(ctx). When a shutdown cancels the run, the save that records where it stopped is the one that must happen. - The state is written with
encoding/json/v2with zero fields omitted, so an unset field loads as unset instead of as an empty value that the client would send back to the provider. - Raw JSON values, such as a model’s tool-call arguments, are stored as strings. A model can write invalid arguments, and a checkpoint that embedded them directly would refuse to save the very run that needs saving.
One process per run
Two processes resuming one run would each run its unanswered calls, which brings back the duplicates through another door.
Open takes an flock lock on the run’s lock file before reading anything, and a second process gets ErrLocked at once.
The kernel drops the lock when the process exits by any means, so a crash never leaves a run locked by a process that no longer exists.
That holds on one machine’s local disk.
On a network file system flock means different things on different clients, and a service spread over several machines needs a lease (a claim on the run that expires, kept in a database) instead.
Resume by phase
Resume takes a saved state and the same tools and limits, and continues from the phase:
donereturns the saved answer without calling the model, so restarting a finished run is free.modelmakes the model call it was about to make.toolsandpausedrun the last reply’s unanswered calls in order, except calls with an attempt marker (below), which go to the resolution ladder.
Because turns and spending come from the state, a run’s limits cover its whole life across every process that ran it.
Doing something exactly once
A checkpoint stops lost work. It can’t stop repeated work, because some moment always separates a tool acting from the run recording that it acted. A single process can promise at most once (never retry, and risk losing the action) or at least once (retry until answered, and risk doing it twice). What people call exactly once is the second promise plus a receiver that recognizes a repeat.
An operation is idempotent when doing it twice has the same effect as doing it once.
An idempotency key is the receiver’s half: the receiver is the service the tool calls, and the key lets it recognize a repeat. The key must stay the same whenever one call is sent again, even from a new process, and must differ from every other call’s key. A hash of the run ID and the tool-call ID has both properties, since save point 2 stored the call ID before the call ran:
// Key returns the idempotency key for one tool call of one run. The
// call ID was saved with the reply before the call ran, so every retry
// of the call, in this process or after a restart, gets the same key,
// and no other call gets it.
func Key(run, call string) string {
h := sha256.New()
for _, s := range []string{run, call} {
h.Write(strconv.AppendInt(nil, int64(len(s)), 10))
h.Write([]byte{':'})
h.Write([]byte(s))
}
return hex.EncodeToString(h.Sum(nil)[:16])
}
The run ID carries the uniqueness, so it must never be reused; the triage command builds one from the task, the time, and eight random characters.
The harness sends the key itself, in an Idempotency-Key header over HTTP or in _meta over MCP, never in arguments the model writes.
Keys expire on real services (Stripe’s v1 API may drop one after 24 hours), so a keyed tool declares how long its service keeps them, and an expired key is treated as no key.
An attempt marker needs nothing from the receiver. The loop saves it just before a call that can change something, at save point 3. After a crash, an unanswered call with no marker certainly never started and simply runs; one with a marker may have acted. For a call that needs approval, the marker is written after the yes, so a crash during the decision leaves nothing to resolve.
The triage agent also keeps a refile guard in issue_file: a second filing for a cause the run already filed is refused with the number of the existing issue.
It covers a case keys can’t: the model filing one crash a second time through a fresh call, which carries a fresh key.
The resolution ladder
When a marked call has no saved result, it goes up three steps, each tried only if the one below it can’t decide:
- Run it again when a repeat is harmless: the tool only reads, is idempotent, or sent a key the service still remembers.
- Ask the service through the tool’s lookup, such as a search by key. A lookup may answer “no such record” only when it is certain; a search index that lags behind writes has to return an error and pass the question up.
- Ask a person, through the same approver used for tool approvals, with its own answers: “rerun” if it didn’t happen, or “done” with what they found, which becomes the call’s result.
The third step exists because, for a tool with no key, retrying is a bet that the first call failed, and losing that bet produces the duplicate all of this was built to prevent. With keys that the service honors, a call never gets past the first step.
Does it work? Crash it everywhere
The crash sweep runs a small scripted triage job once to find every point where a crash could land, then reruns it from scratch for each point: crash there, restart, finish, count. It does it again with a second crash during the restart.
Measured result
8 of 18 crash points with duplicates, saving once per turn
Saving once per turn left a duplicate issue or comment at 8 of its 18 crash points. On the agent’s own loop with keys and markers, 910 scenarios ended with no duplicate, nothing missing, and no question for a person.
- Task
- a scripted triage job that files two issues, comments, and searches, crashed at every point a crash could land and then resumed
- Models
- a scripted model, so a repeated call gets the same reply
- Runs
- one scenario per crash point, plus a second crash at every point each restart reaches
- Date
- deterministic;
go test ./ch15/durable/repeats it
Measured with the companion codeThe finding
Without keys, the same loop made no duplicates either, but only by asking a person 624 times over 892 scenarios. The sweep has these limits: a panic is not a power loss, the syncs are off so it runs in seconds, the tracker never crashes, the scripted person always answers correctly, and a scripted model repeats itself where a real one wouldn’t.
Steps for your own agent
- Define a state type that holds the conversation, usage, turns, phase, pending calls, and markers, and test that no field of the loop’s
Resulttype is left out of it. - Save at the five points, with a write, sync, rename, and directory sync, and save with a context that ignores cancellation.
- Lock each run, count restarts before doing anything, and never reuse a run ID.
- Send an idempotency key made from the run and call IDs with every call that changes something, and ask the services you call to honor it.
- Save an attempt marker before each such call, and give each tool a lookup if its service can answer “did this happen?”
- On resume, run unmarked calls, send marked ones up the ladder, and route the last step to a person.
- Crash your own loop at every save point in a test, and count duplicates and questions.
When to use it
Reach for it
- Tools change the world outside the run. With one save per turn, a crash left a duplicate issue or comment at 8 of 18 crash points; keys and markers left none in any scenario.
- The platform stops processes on its own schedule. Kubernetes allows 30 seconds and
docker stop10, and one model call in the recorded runs took 27.5 seconds.
Leave it out
- Markers without keys when nobody is on call. Against a tracker that ignored keys, markers turned crashes inside calls into 624 questions for a person over 892 scenarios of the agent’s loop.
- Short runs with no side effects. No variant in the sweep lost an issue or comment; what a crash costs such a run is the model calls since its last save.
Try it
From the code directory of a clone of the companion repository, with no key set, so the command replays a recorded run:
$ go run ./ch15 -task m1-done-after-delete -state /tmp/triage-state -die-after code_patch
$ go run ./ch15 -task m1-done-after-delete -state /tmp/triage-state -approve off
$ go test ./ch15/durable/
The first command sends itself SIGKILL as soon as the first code_patch returns, before its result is saved.
Open the .json file in /tmp/triage-state: its Phase is tools, Turns is 8, and Attempts holds a marker for the patch.
The second command resumes that run; the ladder’s second step finds the patch already in the workspace, the replay continues, and the run finishes with all 16 of the crash catalog’s checks passed.
-approve off is there because the restart would otherwise stop and ask you to approve the issue it files.
The third runs the crash sweep, which takes about a minute, and compares its table with the saved copy in ch15/durable/testdata/sweep.txt.
var c *Checkpoint
ch15/durable/checkpoint.go:212
var c *Checkpoint. In ch15/durable/checkpoint.go, line 212.
type Checkpoint struct {
dir string
run string
path string
lock *os.File
now func() time.Time
mu sync.Mutex // one save at a time
// For tests: noSync skips the syncs, which cost tens of
// milliseconds each on some machines, and crash, if set, is called
// ...
}
A Checkpoint is one run's state on disk: <dir>/<run>.json, with <dir>/<run>.lock held while a process has the run open.
A Checkpoint is one run's state on disk: <dir>/<run>.json, with <dir>/<run>.lock held while a process has the run open. type Checkpoint struct { dir string run string path string lock *os.File now func() time.Time mu sync.Mutex // one save at a time // For tests: noSync skips the syncs, which cost tens of // milliseconds each on some machines, and crash, if set, is called // ... }. In ch15/durable/checkpoint.go, line 24.
func (c *Checkpoint) write(b []byte) error
write replaces the checkpoint file with b: write a temporary file, sync it, rename it over the old one, then sync the directory.
ch15/durable/checkpoint.go:212
write replaces the checkpoint file with b: write a temporary file, sync it, rename it over the old one, then sync the directory. func (c *Checkpoint) write(b []byte) error. In ch15/durable/checkpoint.go, line 212.
type byte = uint8
byte is an alias for uint8 and is equivalent to uint8 in all ways.
byte is an alias for uint8 and is equivalent to uint8 in all ways. type byte = uint8. Documented at pkg.go.dev, builtin.byte.
type error interface {
Error() string
}
The error built-in interface type is the conventional interface for representing an error condition, with the nil value representing no error.
The error built-in interface type is the conventional interface for representing an error condition, with the nil value representing no error. type error interface { Error() string }. Documented at pkg.go.dev, builtin.error.
var tmp *os.File
ch15/durable/checkpoint.go:213
var tmp *os.File. In ch15/durable/checkpoint.go, line 213.
package os ("os")
Package os provides a platform-independent interface to operating system functionality.
Package os provides a platform-independent interface to operating system functionality. package os ("os"). Documented at pkg.go.dev, os.
func CreateTemp(dir, pattern string) (*File, error)
CreateTemp creates a new temporary file in the directory dir, opens the file for reading and writing, and returns the resulting file.
CreateTemp creates a new temporary file in the directory dir, opens the file for reading and writing, and returns the resulting file. func CreateTemp(dir, pattern string) (*File, error). Documented at pkg.go.dev, os.CreateTemp.
field dir string
field dir string. In ch15/durable/checkpoint.go, line 25.
field run string
field run string. In ch15/durable/checkpoint.go, line 26.
var nil Type
nil is a predeclared identifier representing the zero value for a pointer, channel, func, interface, map, or slice type.
nil is a predeclared identifier representing the zero value for a pointer, channel, func, interface, map, or slice type. var nil Type. Documented at pkg.go.dev, builtin.nil.
package fmt ("fmt")
Package fmt implements formatted I/O with functions analogous to C's printf and scanf.
Package fmt implements formatted I/O with functions analogous to C's printf and scanf. package fmt ("fmt"). Documented at pkg.go.dev, fmt.
func Errorf(format string, a ...any) (err error)
Errorf formats according to a format specifier and returns the string as a value that satisfies error.
Errorf formats according to a format specifier and returns the string as a value that satisfies error. func Errorf(format string, a ...any) (err error). Documented at pkg.go.dev, fmt.Errorf.
func Remove(name string) error
Remove removes the named file or (empty) directory.
Remove removes the named file or (empty) directory. func Remove(name string) error. Documented at pkg.go.dev, os.Remove.
func (f *File) Name() string
Name returns the name of the file as presented to Open.
Name returns the name of the file as presented to Open. func (f *File) Name() string. Documented at pkg.go.dev, os.File.Name.
func (f *File) Write(b []byte) (n int, err error)
Write writes len(b) bytes from b to the File.
Write writes len(b) bytes from b to the File. func (f *File) Write(b []byte) (n int, err error). Documented at pkg.go.dev, os.File.Write.
func (c *Checkpoint) sync(f *os.File) error
ch15/durable/checkpoint.go:257
func (c *Checkpoint) sync(f *os.File) error. In ch15/durable/checkpoint.go, line 257.
var cerr error
ch15/durable/checkpoint.go:222
var cerr error. In ch15/durable/checkpoint.go, line 222.
func (f *File) Close() error
Close closes the [File], rendering it unusable for I/O. On files that support [File.SetDeadline], any pending I/O operations will be canceled and return immediately with an [ErrClosed] error.
Close closes the [File], rendering it unusable for I/O. On files that support [File.SetDeadline], any pending I/O operations will be canceled and return immediately with an [ErrClosed] error. func (f *File) Close() error. Documented at pkg.go.dev, os.File.Close.
func (c *Checkpoint) point(name string)
ch15/durable/checkpoint.go:264
func (c *Checkpoint) point(name string). In ch15/durable/checkpoint.go, line 264.
func Rename(oldpath, newpath string) error
Rename renames (moves) oldpath to newpath.
Rename renames (moves) oldpath to newpath. func Rename(oldpath, newpath string) error. Documented at pkg.go.dev, os.Rename.
field path string
field path string. In ch15/durable/checkpoint.go, line 27.
func (c *Checkpoint) syncDir() error
syncDir makes the rename durable: the new name is an entry in the directory, and until the directory is synced, a power loss can bring back the old entry, or none.
ch15/durable/checkpoint.go:246
syncDir makes the rename durable: the new name is an entry in the directory, and until the directory is synced, a power loss can bring back the old entry, or none. func (c *Checkpoint) syncDir() error. In ch15/durable/checkpoint.go, line 246.
func Key(run, call string) string
Key returns the idempotency key for one tool call of one run.
Key returns the idempotency key for one tool call of one run. func Key(run, call string) string. In ch15/agent/once.go, line 20.
type string string
string is the set of all strings of 8-bit bytes, conventionally but not necessarily representing UTF-8-encoded text.
string is the set of all strings of 8-bit bytes, conventionally but not necessarily representing UTF-8-encoded text. type string string. Documented at pkg.go.dev, builtin.string.
package sha256 ("crypto/sha256")
Package sha256 implements the SHA224 and SHA256 hash algorithms as defined in FIPS 180-4.
Package sha256 implements the SHA224 and SHA256 hash algorithms as defined in FIPS 180-4. package sha256 ("crypto/sha256"). Documented at pkg.go.dev, crypto/sha256.
func New() hash.Hash
New returns a new hash.Hash computing the SHA256 checksum.
New returns a new hash.Hash computing the SHA256 checksum. func New() hash.Hash. Documented at pkg.go.dev, crypto/sha256.New.
func (Writer) Write(p []byte) (n int, err error)
func (Writer) Write(p []byte) (n int, err error). Documented at pkg.go.dev, io.Writer.Write.
package strconv ("strconv")
Package strconv implements conversions to and from string representations of basic data types.
Package strconv implements conversions to and from string representations of basic data types. package strconv ("strconv"). Documented at pkg.go.dev, strconv.
func AppendInt(dst []byte, i int64, base int) []byte
AppendInt appends the string form of the integer i, as generated by [FormatInt], to dst and returns the extended buffer.
AppendInt appends the string form of the integer i, as generated by [FormatInt], to dst and returns the extended buffer. func AppendInt(dst []byte, i int64, base int) []byte. Documented at pkg.go.dev, strconv.AppendInt.
type int64 int64
int64 is the set of all signed 64-bit integers.
int64 is the set of all signed 64-bit integers. type int64 int64. Documented at pkg.go.dev, builtin.int64.
func len(v Type) int
The len built-in function returns the length of v, according to its type:
The len built-in function returns the length of v, according to its type: func len(v Type) int. Documented at pkg.go.dev, builtin.len.
package hex ("encoding/hex")
Package hex implements hexadecimal encoding and decoding.
Package hex implements hexadecimal encoding and decoding. package hex ("encoding/hex"). Documented at pkg.go.dev, encoding/hex.
func EncodeToString(src []byte) string
EncodeToString returns the hexadecimal encoding of src.
EncodeToString returns the hexadecimal encoding of src. func EncodeToString(src []byte) string. Documented at pkg.go.dev, encoding/hex.EncodeToString.
func (Hash) Sum(b []byte) []byte
Sum appends the current hash to b and returns the resulting slice.
Sum appends the current hash to b and returns the resulting slice. func (Hash) Sum(b []byte) []byte. Documented at pkg.go.dev, hash.Hash.Sum.

