AI API Quotas — Separate Fairness From Mere Rate Limits

Quotas answer a different question than rate limits#

A rate limit says how fast traffic may arrive. A quota says how much of a scarce resource a customer, team, workflow or model may use over a period. AI systems need both. Without quotas, one tenant can drain a shared budget, take all provider capacity, or turn a short burst into a long outage for everyone else.

Start by naming the resource. Requests are easy to explain but not equal: a short classification call and a long generation differ a lot in token cost, latency and provider impact. Track requests, input tokens, output tokens, estimated spend, concurrent executions and queue occupancy separately. A tenant can be under its request limit while consuming most of the token budget, so make the scarce unit explicit.

Separate controls and set clear accounting boundaries#

Use three controls:

• Rate limit: protects a short window, such as requests per second.
• Quota: protects a longer allocation, such as tokens per day or dollars per month.
• Concurrency limit: protects work in progress, especially long streaming generations.

Merging them into one generic limit produces confusing results, such as rejecting a customer who still has budget or admitting work into a queue it cannot drain. Return the control that actually blocked the request, with reset or retry information when it is safe.

Quotas can belong to an API key, user, project, organization, model, route or billing account, and most real systems need a hierarchy. A project may have a daily token budget, each key a concurrency cap, and a premium workflow a separate reserved pool. Enforce the narrowest rule first, then check parent budgets, and record which rule denied the operation so a single key cannot bypass an organization-wide limit.

Atomic reservations and honest estimates#

Before sending work upstream, estimate the maximum commitment, reserve it, and settle when actual usage is known. For streams, reserve an output ceiling and release unused capacity as tokens arrive or the stream ends. For image or video jobs, reserve the estimated cost before submission. If reservation and dispatch are separate steps, two requests can see the same remaining budget and overspend it, so use an atomic store operation or a durable ledger with idempotent reservation IDs.

Estimates are not exact: token counts differ across providers and output length is unknown at admission. Track estimated, reserved, actual and billable usage as distinct values, settle after completion, and reconcile late provider usage reports. If usage is only reported after a failure, hold the reservation until it is accounted for.

Bursts, fairness and noisy neighbors#

A fixed daily counter makes midnight traffic and end-of-day traffic behave oddly. A token bucket gives a refill rate with bounded burst capacity; a rolling window is easier to audit but can spike at boundaries. Choose the model that matches your product promise and document whether unused burst capacity expires, whether quota carries over, and whether cancelled work returns its reservation.

Fairness is not equal allocation. A batch job may need more tokens but tolerate delay; an interactive chat needs low latency and a small budget. Assign pools or priorities by workload class, enforce per-tenant ceilings inside each pool, and use weighted fair queuing on constrained provider routes. Premium traffic still needs a cap, and emergency traffic needs an explicit budget and audit trail.

Tenant limits alone fail if everyone shares one connection pool, worker queue or model semaphore. Partition where practical, or at least consider tenant identity when pulling from a shared queue and apply a maximum service share. Measure queue wait, service time, rejections and provider throttling per tenant, because fair-looking counters mean nothing if one customer absorbs all the latency.

Near-exhaustion behavior, retries and visibility#

When a quota is nearly used up, the gateway can reject, allow a small overdraft, downgrade to a cheaper model, queue for the next refill, or require approval. A user-facing chat might get a clear budget message and a smaller context limit; a background summarization job might wait; a billing-critical operation should not silently switch models. Never hide a downgrade behind a success status: expose the selected route and policy decision in trace metadata and, where appropriate, a response header.

Retries and fallbacks share one logical budget. Group all attempts under an operation ID, cap attempts and spend, and reconcile tokens consumed by failed requests, separating customer-visible work from speculative or failed work in reports.

Show customers current usage, remaining budget, reservation state, reset time and the rule that would block the next request. Give administrators breakdowns by model, route, key, workflow and period, and alert on abnormal burn rate, repeated denials, reservation leaks and large estimate-versus-actual gaps.

Test and roll out quota policy safely#

Before enforcing, send simultaneous requests at the exact budget boundary, crash the gateway between reservation and dispatch, disconnect streams mid-token, delay provider usage reports, replay idempotency keys and change a tenant's plan mid-flight. Verify nothing is counted twice, released capacity never goes negative, and open reservations survive a restart. Load-test with uneven tenants to measure fairness under contention.

Version quota policy like an API contract. Switching from requests to tokens can surprise clients even if the headline number rises, so publish effective dates, expose policy versions, offer a migration window, run a dry-run mode first, roll out by tenant or route, and keep an audited emergency override. A gateway such as Crazyrouter, which sits in front of many providers with one API key, is where this kind of explicit, visible policy can live.