AI API Gateway Rate Limit Headers - Make Capacity Signals Actionable

Why a yes-or-no 429 is not enough#

Many gateways treat rate limiting as a binary decision: return a 429, let the client wait, and hope the next attempt works. That breaks down for AI workloads, where one request might be a short classification call and the next a long-context generation with a large output budget.

Clients need three things from a rejection: what limit was reached, how much capacity remains, and when a retry has a reasonable chance of succeeding. Without those signals, client code either retries too aggressively and makes congestion worse, or waits far longer than necessary and wastes capacity that was actually available. Rate limit headers are useful only when they describe the decision precisely enough for software to make the next decision well.

Define the unit and the scope#

Request limits, token limits, concurrent-generation limits and spend budgets are different resources. A client told it has ten requests remaining can still fail because its next call needs more input tokens than are left, or because the provider concurrency pool is full. Name the measured resource explicitly.

Scope matters just as much. A limit may apply to an API key, user, tenant, organization, model, provider, region or shared account. Make the relevant scope understandable without exposing other customers' information, and when several buckets could have rejected a request, identify the one that actually did and keep that reason in safe request metadata.

Consistent vocabulary and reset semantics#

Standard HTTP rate limit fields can carry a limit, the remaining allowance and a reset point, and Retry-After tells a rejected client when to try again. If you add gateway-specific fields, document their units, scope and clock semantics. Headers that change meaning between routes create more retry bugs than they fix.

Reset values are easy to misread. A fixed-window reset differs from a sliding-window estimate or a token-bucket refill time; "reset in sixty seconds" could mean the window ends then or that enough tokens should be back by then. Say whether the value is an absolute timestamp, a delay in seconds or an approximation, and base it on server time.

Avoid advertising capacity you have not reserved. Concurrent traffic, provider health, quotas and request size can all change the outcome, so remaining-capacity fields should read as estimates, not promises the scheduler cannot keep.

Account for request cost and coordinate retries#

A long-context chat completion consumes far more capacity than a short call. If the gateway meters weighted tokens, return units that let clients reason about cost, and if a request is rejected because its estimated cost exceeds the available budget, say so rather than returning a generic rate limit message.

Headers and retry policy are one design problem. Document whether Retry-After is an upper bound or an earliest useful retry point. Clients should add jitter, respect deadlines and stop once the user has left; gateways should avoid turning every 429 into a synchronized retry storm. A well-defined reset value is only half the job if thousands of clients act on it at the same instant.

Provider signals and streaming#

Upstream providers return their own limits and reset values, but those often describe the provider account rather than the client's gateway bucket. Normalize them into a clear gateway contract, keep the original signal in internal lineage, and distinguish client throttling from provider throttling, quota exhaustion and temporary overload. A multi-provider gateway like Crazyrouter has to make this translation for every upstream it fronts.

Streaming adds a time dimension: a request can pass admission and then hold capacity for minutes. Decide whether capacity is reserved at admission, metered during generation or reconciled afterward, and release reservations promptly when a stream is cancelled. Initial response headers describe the contract, while usage trailers or an out-of-band usage record can report final consumption where supported.

Test and observe headers as a contract#

Verify field names, types, units, scope, reset behavior and missing-header behavior on both successful and rejected responses. Exercise burst traffic, clock skew, retries, cancellation, oversized requests, provider failures and simultaneous limits. A header that is present but stale, contradictory or uninterpretable is a broken interface, even if every individual field passes a schema check.

Then watch outcomes, not just values: 429 rates by bucket, retry success, retry amplification, time spent waiting, abandoned requests and provider utilization. Compare advertised resets with actual acceptance probability, and if clients consistently retry too early or too late, adjust the header contract and the scheduler together.