Retry Budgets for AI Gateways - Recover Without Amplifying Failure

How stacked retries multiply failure#

Consider a document assistant that normally answers in a few seconds. The upstream model slows down. The application retries twice, its SDK retries twice, and the gateway tries a second provider. Each choice looks reasonable alone; together they multiply requests, consume capacity, and deliver an answer after the user has left. A retry policy is really a resource-allocation policy shared across layers.

Clear vocabulary helps:

• A logical operation is what the user asked for, such as summarizing a document.
• An attempt is one actual model request toward that operation.
• A fallback is an attempt on a different route, model, or provider.
• A tool action is a separate operation that may change external state.

Without this separation, a dashboard can show six successful model calls while the user saw one timeout and an unexpected charge.

Budget one: a shared end-to-end deadline#

Choose the deadline from the product experience, not from a library default. In the episode's illustrative example, an interactive operation has twelve seconds. If retrieval takes three and the first model attempt takes six, a fallback has less than three seconds left, and it still needs time for validation and delivery.

Record an operation identifier and deadline at the application boundary. Measure elapsed time with a monotonic clock inside a process, and propagate an agreed deadline or remaining-budget value across services, handling clock skew and transit time explicitly. Before every new attempt, calculate what remains. If it is not enough for a useful response, fail clearly or move to an explicitly supported asynchronous flow; starting work that cannot finish is not resilience.

Budget two: an attempt cap with one owner#

A simple interactive route might allow one initial attempt plus one retry or fallback, but the right count depends on latency, cost, and failure patterns. Ownership matters more than the number: decide which layer retries, and disable redundant retries elsewhere or count them toward the cap. Read SDK and gateway documentation closely, since "maximum retries" may exclude the original attempt while "maximum attempts" usually includes it.

Then classify the failure before choosing an action. Malformed requests, unsupported parameters, and invalid output schemas rarely improve when resent unchanged. Authentication and authorization failures need a configuration fix. A rate limit may justify waiting if the provider's retry guidance fits the remaining deadline. Some connection errors and temporary server failures justify a bounded retry. A timeout is ambiguous: the upstream may still be generating, and billing may already have happened.

Budget three: side effects#

Generating text twice is wasteful; charging a card or sending a message twice is much worse. Do not assume an idempotency header works unless the specific endpoint documents that guarantee. Give every logical external action a stable identifier, record its state durably, and use the downstream system's idempotency mechanism where it exists. If an outcome is unknown, reconcile it before executing again, and never let a conversation retry blindly replay completed tools.

In the episode's support-assistant example, the model drafts a refund explanation and an application service executes the refund after validating the action, checking authorization, and recording its identifier. If the stream later breaks, the explanation can be regenerated without issuing a second refund.

Backoff, service-wide limits, and fallback eligibility#

Exponential backoff with jitter keeps clients from returning at the same instant. Respect retry-after guidance where it applies, but never wait past the remaining deadline, and cap the delay. Also cap total retry traffic across the service, because a struggling provider otherwise receives extra load exactly when it has the least spare capacity. One approach limits retries to a small fraction of recent original requests, with a defined allowance for low traffic, tracked per route or workload. Pair it with concurrency limits, bounded queues, and a circuit breaker whose recovery probes are also limited.

A healthy alternative is not automatically a correct one. Before a fallback, confirm it supports the required tools, structured output, context size, and streaming, meets data-handling requirements, and fits the customer's cost policy. Applications that reach multiple providers through one API key, as with Crazyrouter, should still make this eligibility check explicit. If no eligible route remains, return a transparent failure or a clearly labeled degraded result.

Streaming, telemetry, and failure testing#

Before any output reaches the user, a transparent retry within budget is sometimes possible. After partial text is visible, silently restarting can duplicate or contradict content, so choose a contract: a visible interruption, an explicit regenerate action, or a resumable protocol where the upstream supports one. Closing the client connection should cancel unnecessary work where supported, though cancellation is not proof that upstream processing or billing stopped.

Record telemetry at two linked levels: per operation (final outcome, total latency, total cost, whether recovery helped) and per attempt (route, attempt number, failure category, elapsed time, remaining budget), without storing raw prompts by default. Then test in a controlled environment: verify exact attempt counts, confirm a fallback does not reset the deadline, check that a retry-after longer than the remaining budget stops the client, interrupt a stream after the first token, and simulate an action that succeeded while its acknowledgement was lost.