AI API Rate Limits: How to Handle 429 Errors in Production (Podcast EP7)

Key takeaways

Treat rate limits as multi-dimensional: you can be under your requests-per-minute cap and still be blocked by a tokens-per-minute or daily token limit.

Retry 429 responses with exponential backoff plus random jitter so your retries do not all hit the provider at the same moment.

Put outbound model calls behind a queue with a rate limiter (for example bottleneck in Node.js or a Python rate-limit library) so you control exactly how many requests leave per second.

Cache answers to repeated questions; even a short-lived cache removes a large share of duplicate calls in support-style chatbots.

Design your app so more than one model can serve a request, and fall back to another provider when one returns 429.

Read the rate-limit headers in every response, alert at around 70% of capacity, and request limit increases days before a launch.

Show notes

What rate limits actually measure#

A rate limit is a cap a provider places on how much you can use its API within a time window. Every major AI provider has them, including OpenAI, Anthropic and Google. OpenAI and Anthropic assign limits by usage tier, so a new account gets far less headroom than an established high-spend one, while Google's Gemini API applies both requests-per-minute and tokens-per-minute caps. The spread is large: at the time of recording (March 2026), OpenAI's lowest tier allowed only a handful of GPT-4o requests per minute, while its top tier allowed thousands.

Think of it as a highway speed limit, except measured in requests per minute, tokens per minute or requests per day. The trap is that limits are tracked on several dimensions at once:

Requests per minute - how many calls you make.
Tokens per minute - how much text flows through those calls.
Daily caps - token or request budgets you may not know exist until you hit them.

You can be comfortably under one limit and still be throttled by another, which is why a 429 often appears to come from nowhere.

Why launches are where apps break#

Picture a customer support chatbot that runs smoothly with ten test users. Then it gets featured on Hacker News and hundreds of people start chatting at once. If each conversation triggers two or three model calls, your backend suddenly fires well over a thousand requests in the same window. On a low usage tier, a large share of those requests come back as errors, and the launch thread shifts from "cool product" to "this doesn't even work."

Do the arithmetic before launch: concurrent users multiplied by calls per conversation gives your peak request rate, and that peak, not your average traffic, is what has to fit under your limits. At the time of recording, OpenAI's Tier 1 allowed on the order of 500 GPT-4o requests per minute, so in the scenario above roughly half of users would have seen errors.

This is common. Teams have burned through an entire daily token budget in the first hour after a Product Hunt launch. The lesson is that rate limits are an infrastructure problem you need to solve before traffic arrives, not after.

Retry, queue and cache#

Three of the five strategies live entirely in your own code.

Exponential backoff with jitter. When you get a 429, do not retry immediately. Wait one second, then two, then four, and add a random offset so that many clients do not retry in lockstep. Most HTTP client libraries already support this.

A request queue. Instead of calling the model the instant a user sends a message, push the job into a queue governed by a rate limiter. You decide how many requests per second leave your system, which turns sudden bursts into a steady flow the provider will accept. Libraries such as bottleneck for Node.js or ratelimit for Python make this straightforward.

Aggressive caching. If fifty users ask about your return policy, you do not need fifty model calls. Caching common answers, even for a few minutes, can remove a large portion of the traffic in a support bot; the episode's estimate was that a five-minute cache can cut API calls by 60 to 70 percent for support chatbots, because so many questions repeat.

Streaming and multi-provider fallback#

The fourth strategy is streaming. Users see text appear immediately instead of staring at a spinner, which keeps the experience feeling responsive while your system is under pressure.

The fifth, and the biggest lever, is not depending on a single provider. Each provider enforces its own limits, so GPT, Claude and Gemini each represent a separate pool of capacity. If your app can serve a request with any of several models, and for most tasks it can, you multiply the headroom available to you. A typical pattern is a primary model with an ordered fallback list: when the primary returns 429, the request moves to the next model instead of surfacing an error to the user. If that model is limited too, it moves on again. The user never sees a failure; they simply get an answer, and for high-traffic applications this is the most effective of the five strategies.

This is much easier when every model sits behind one OpenAI-compatible endpoint. With a gateway such as Crazyrouter, switching from one model to another is a change to the model string rather than a new SDK integration.

Monitor before your users notice#

Providers return rate-limit information in response headers. Log those values, track how close you are to each limit, and set alerts at roughly 70% of capacity so you hear about trouble before customers do.

Plan for known spikes as well. OpenAI and Anthropic both accept requests for higher limits, but approval can take days, so ask well before a launch. Combined with backoff, queuing, caching and a multi-provider setup, this turns rate limits from a launch-day outage into a routine operational metric.

Rate limits feel like a dull infrastructure topic right up until production goes down. Implement backoff, add queuing, cache what repeats, and set up a multi-provider path before you need it rather than during an incident.

Questions from this episode

What does a 429 error mean when calling an AI API?

A 429 means you have exceeded one of the provider's rate limits. That can be requests per minute, tokens per minute or a daily cap, and you can hit one while staying under the others. The fix is to slow down and retry with backoff rather than retrying immediately.

How should I retry requests after hitting an AI API rate limit?

Use exponential backoff with jitter: wait about one second, then two, then four, adding a random offset each time. The jitter prevents all your clients from retrying at the same instant and triggering the limit again. Pair it with a request queue so bursts are smoothed out before they reach the provider.

Can I get more capacity than a single provider's rate limit allows?

Yes, by using more than one provider. OpenAI, Anthropic and Google each enforce separate limits, so an app that can fall back from one model to another when it receives a 429 effectively adds their capacity together. You can also ask a provider for a limit increase, but do it days before you need it.

AI API Rate Limits — What Every Developer Needs to Know