An HTTP client for LLM APIs in Go: timeouts, limits, errors, and Retry-After

A model API is an ordinary HTTPS endpoint with unusual traffic. A reply can take half a minute to start and a streamed one minutes to finish; the body can be large; rate limits are part of normal operation; and a request can fail after it has succeeded, with an error inside a 200 OK. Go’s net/http handles all of it, with a few choices made deliberately. This page goes through them, with code from the companion repository, where it is tested. For the chat completions request itself, start with calling a model from Go; for when and how to retry, see retries and observability.

One client, shared

An http.Client and the http.Transport under it are safe for concurrent use, and the transport keeps a pool of open connections. Make one and share it; a new client or transport per request throws the pool away and pays for a new TCP and TLS handshake every time.

The model client takes an *http.Client as a field, and uses http.DefaultClient when it is nil:

ch12/openrouter/client.go

// Client calls a model through OpenRouter.
type Client struct {
	BaseURL string        // DefaultBaseURL if empty
	APIKey  string        // sent as a bearer token
	Model   string        // a model ID from OpenRouter's list
	HTTP    *http.Client  // http.DefaultClient if nil
	Timeout time.Duration // per Generate call; 2 minutes if zero

	// IdleTimeout is the longest a stream may go without sending a
	// byte, including the wait for its headers; 30 seconds if zero
	// (chapter 6). Stream doesn't use Timeout.
	IdleTimeout time.Duration

	// Providers, if set, are the only providers OpenRouter may send
	// requests to, such as "novita" (chapter 3).
	Providers []string
}

Taking the client as a field, instead of building one inside, lets the caller decide what it is: a client with a proxy, one shared by every agent in a process, or one whose transport replays recorded replies in tests.

The default transport is a sensible start: a 30-second dial timeout, a 10-second TLS handshake timeout, HTTP/2 where the server offers it, and idle connections kept for 90 seconds. One default is worth knowing for agents: over HTTP/1.1, it keeps at most two idle connections per host (DefaultMaxIdleConnsPerHost). A program running many calls to one provider in parallel reopens connections constantly unless it uses a transport with a higher MaxIdleConnsPerHost. Over HTTP/2, one connection carries many requests at once, and the limit matters less.

Deadlines go on the context

http.Client has a Timeout field, and it is the wrong tool here. It covers the whole exchange, including reading the body, so it can’t tell a slow start from a long reply: a timeout long enough for a streamed reply that legitimately takes three minutes is far too long to notice a request that never got an answer.

The client instead puts a deadline on each call’s context, two minutes by default, and a stream gets an idle timer that measures silence instead of total time. Both carry their own cause, so a timeout says which one it was. The context page shows both.

Read with a limit

A response body is read with a cap, so a misbehaving server or proxy can’t make the client allocate without bound:

ch12/openrouter/client.go

// readResponse reads and decodes a response, and checks it holds a
// reply.
func readResponse(ctx context.Context,
	resp *http.Response) (chatResponse, error) {
	data, err := io.ReadAll(io.LimitReader(resp.Body, maxBody+1))
	if err != nil {
		err = callError(ctx, err)
		return chatResponse{}, fmt.Errorf("read response: %w", err)
	}
	if len(data) > maxBody {
		return chatResponse{}, errTooLarge
	}
	if resp.StatusCode != http.StatusOK {
		return chatResponse{}, statusError(resp, data)
	}
	var r chatResponse
	if err := json.Unmarshal(data, &r); err != nil {
		return chatResponse{}, fmt.Errorf("decode response: %w", err)
	}
	if e := r.Error; e != nil {
		return chatResponse{}, e.asAPIError()
	}
	if len(r.Choices) == 0 {
		return chatResponse{}, errNoChoices
	}
	if e := r.Choices[0].Error; e != nil {
		return r, fmt.Errorf("%w: %w", agent.ErrProviderFailed,
			e.asAPIError())
	}
	return r, nil
}

The limit is 16 MB, far above any reply, and the client reads one byte past it so it can tell a body that is exactly the limit from one that is over.

The checks after reading are in order, and each catches something real:

A status other than 200 becomes an API error, built below.
A 200 can still carry an error object instead of a reply: the request was accepted, and the provider failed.
A reply with no choices is unusable.
A choice can carry its own error, when a provider failed partway through writing it; the tokens it did write are still billed, so the reply is returned alongside the error.

A client that only checks the status code treats cases 2 and 4 as answers.

When the status isn’t 200

A refused request becomes an *APIError with the status, the API’s own message, and the wait the server asked for:

ch12/openrouter/errors.go

// statusError builds the APIError for a response that wasn't 200, using
// the API's message when the body has one.
func statusError(resp *http.Response, body []byte) *APIError {
	e := &APIError{
		Status:     resp.StatusCode,
		Message:    "no details",
		RetryAfter: agent.RetryAfter(resp.Header, time.Now()),
	}
	var b struct {
		Error apiError `json:"error"`
	}
	switch {
	case json.Unmarshal(body, &b) == nil && b.Error.Message != "":
		e.Message = b.Error.Message
		e.LimitSource = b.Error.Metadata.LimitSource
	case len(body) > 0: // not the API's JSON, such as a proxy's page
		e.Message = string(body[:min(len(body), 200)])
	}
	return e
}

The body isn’t always the API’s JSON. A proxy or load balancer in between can answer with an HTML error page, so the client keeps the first 200 bytes of whatever came back rather than failing to decode it.

The wait comes from the Retry-After header, which has two forms, a number of seconds or an HTTP date:

ch12/agent/retryafter.go

// RetryAfter reads a Retry-After header in either of its forms, a
// number of seconds or an HTTP date, as a wait from now. A wait that is
// negative, in the past, or unreadable is 0; one over a day is capped
// at a day, so the multiplication can't overflow (chapter 12).
func RetryAfter(h http.Header, now time.Time) time.Duration {
	v := strings.TrimSpace(h.Get("Retry-After"))
	if v == "" {
		return 0
	}
	if secs, err := strconv.ParseInt(v, 10, 64); err == nil {
		return time.Duration(min(max(secs, 0), 86400)) * time.Second
	}
	t, err := http.ParseTime(v)
	if err != nil {
		return 0
	}
	return min(max(t.Sub(now), 0), 24*time.Hour)
}

Both forms are clamped: a value in the past or one that doesn’t parse is no wait, and anything over a day is capped at a day. A server’s header is input like any other, and a huge value multiplied into a time.Duration would overflow.

Honoring the wait, within limits

The retry policy uses the server’s wait when there is one, and gives up instead when the wait is longer than the caller is willing to spend:

ch12/agent/retry.go

// wait returns how long to wait before retry n, after err, or a
// *RetryError saying why to give up.
func (p Policy) wait(n int, d Decision, spent time.Duration,
	err error) (time.Duration, error) {
	give := func(why string) (time.Duration, error) {
		return 0, &RetryError{n, fmt.Errorf("%s: %w", why, err)}
	}
	if n >= cmp.Or(p.Attempts, 4) {
		return give("no attempts left")
	}
	if !p.RetryBudget.Allow() {
		return give("retry budget spent")
	}
	wait := p.Backoff.Delay(n)
	if d.After > 0 {
		if d.After > cmp.Or(p.MaxWait, time.Minute) {
			return give(fmt.Sprintf("asked to wait %v", d.After))
		}
		wait = d.After + p.Backoff.Delay(1)
	}
	if spent+wait > cmp.Or(p.Within, 5*time.Minute) {
		return give("out of time")
	}
	if p.OnRetry != nil {
		p.OnRetry(n, err, d, wait)
	}
	return wait, nil
}

A server’s wait replaces the backoff for that retry, plus a random extra of up to a second, so that many clients told to wait the same time don’t all come back in the same instant. A wait longer than MaxWait, one minute by default, ends the retries with an error that names the wait, which is more useful than sleeping through it. Which errors are worth retrying at all is the errors page’s Classify, and the backoff itself is on the retries page.

Keep connections reusable

A connection goes back to the pool only when its response body has been read to the end and closed. defer resp.Body.Close() after every successful request is the rule, and reading the whole body, as readResponse does, is what lets the connection be reused. Code that gives up on a response early, and expects more requests to the same host, reads what is left with io.Copy(io.Discard, resp.Body) before closing it.

RoundTrippers: where tests and faults plug in

http.Client sends every request through its transport, an http.RoundTripper: an interface with one method, RoundTrip(*http.Request) (*http.Response, error). Wrapping or replacing it changes how requests are sent without touching the client code.

The companion code uses that in two ways. Every program runs offline by default with a transport that answers from recorded replies, which testing offline describes. And the retry tests use a transport that injects failures, a status, a dropped connection, a body cut short, in front of a real server:

ch12/agent/fault.go

// RoundTrip applies the next fault to req, or sends it on.
func (f *Faulty) RoundTrip(req *http.Request) (*http.Response, error) {
	fault := f.next()
	if fault.Drop || fault.Status != 0 {
		return fault.instead(req)
	}
	resp, err := f.transport().RoundTrip(req)
	switch {
	case err != nil:
		return nil, err //nolint:wrapcheck // passed on as is
	case fault.Lost:
		_, _ = io.Copy(io.Discard, resp.Body)
		resp.Body.Close()
		return nil, badRecordMAC
	case fault.Reset > 0:
		resp.Body = &cutBody{resp.Body, fault.Reset, resetErr}
	case fault.Truncate > 0:
		resp.Body = &cutBody{resp.Body, fault.Truncate,
			io.ErrUnexpectedEOF}
	}
	return resp, nil
}

When it fakes a lost connection after a real response, it drains and closes that response’s body first, so the real connection isn’t leaked. A RoundTripper owns the request body it is given, too: one that answers without sending a request closes the body itself.

Mistakes to avoid

A new http.Client or http.Transport per request. Each one starts a new connection pool; share one.
http.Client.Timeout on streamed replies. It counts the whole stream; use a deadline per call on the context, and an idle timer for streams.
io.ReadAll(resp.Body) with no limit. Cap it with io.LimitReader.
Checking only the status code. A model API can report a failure inside a 200.
An unclosed body. The connection is never reused, and a long-running agent runs out of file descriptors.
Trusting Retry-After without a cap. Clamp it, and decide how long you’ll wait before you sleep.
The API key anywhere but a header. A failed request’s error includes its URL, so a key in the query string ends up in error messages and logs; and never log request headers.

Try it

From the code directory of a clone of the companion repository:

terminal

$ go test ./ch12/agent -run 'TestDoHonorsRetryAfter|TestFaulty' -v
$ go test ./ch06/openrouter -run 'TestErrorInsideA200|TestAPIErrors' -v

In TestTimeoutSynctest (ch06/openrouter/client_test.go), give the http.Client a Timeout of one second and see which deadline ends the call, and what the error says.
In a test, point the client at an httptest server that returns a 502 with an HTML body, and read the *APIError it produces.
Give RetryAfter a date in the past, an empty header, and 999999999, and check each result.