Backpressure for AI Gateways - Admit Only Work You Can Finish

Why queues explode when providers slow down#

The episode uses a fictional coding assistant gateway with forty concurrent generation slots on one route. At roughly five seconds per generation, that is about eight completions per second. If the provider slows to twenty seconds per request, the same slots complete only about two per second. With six requests per second still arriving, the queue grows by roughly four every second, and callers retrying while they wait make it grow faster. These are illustrative averages; real workloads have bursts and long tails.

Autoscaling the gateway does not create provider capacity. An infinite queue does not help either. What helps is a clear admission contract, bounded waiting, and feedback that reaches the caller before resources run out.

Rate, concurrency, and queue limits are different controls#

These three controls are often blurred together, but none substitutes for the others:

• A rate limit controls how fast work arrives, such as requests or tokens per minute.
• A concurrency limit controls how much work is active at once.
• A queue limit controls how much accepted work may wait.

A tenant can stay within its request rate while opening many long-lived streams. A concurrency limit can protect the provider while leaving an unbounded pile of waiting tasks in gateway memory. A small queue can prevent memory exhaustion yet still let one tenant occupy every active slot. Name the resource each control protects before choosing its number.

Admit early, wait briefly, and respect deadlines#

Authenticate, parse within a small bounded budget, validate the request shape, and classify the workload, then decide whether the gateway can make a credible attempt. Don't download a huge document, run retrieval, and build a giant prompt only to discover the generation queue is full. For model work, input size, output cap, route, and stream lifetime can matter more than request count; use conservative reservations and reconcile actual usage.

Waiting must be deadline-aware. If a request has four seconds left, the queue estimates three, and a useful answer needs three more, accepting it just manufactures a timeout. Reject it or offer an explicitly supported service class instead of silently switching models or going asynchronous, and recheck the deadline at dequeue.

Bound queues by item count, retained bytes, and maximum age, because ten waiting requests may be tiny prompts or ten huge contexts. Moving an unbounded queue into a database does not solve the problem. For interactive traffic, a short queue and a quick, explicit overload response usually beat a spinner that runs until the deadline.

Permits and truthful accounting#

A queue reservation means a request may wait; it does not mean a provider slot is taken. At dispatch, acquire execution resources and release the queue reservation exactly once. If acquisition fails, follow a bounded policy instead of spinning. Holding one scarce permit while waiting indefinitely for another invites deadlock or convoys, so allocate compatible resources together through a central scheduler, or use a fixed acquisition order with short waits and rollback. Cancellation and timeout paths must release ownership safely.

Releasing a local socket does not prove remote generation stopped. Track local connection capacity separately from estimated provider work still in flight, and keep a bounded reservation for upstream requests with uncertain terminal states until completion evidence, cancellation confirmation, or a documented expiry resolves them. Otherwise the gateway admits replacements into slots that only look free.

Streaming backpressure for slow clients#

A client may read more slowly than the model generates. If the gateway keeps reading upstream and appending to an unlimited buffer, one slow mobile client can hold megabytes and thousands can exhaust the process. Set per-stream byte limits and stall timers plus a global buffer budget, and await downstream writes where the framework offers flow control. Transport flow control does not mean the model pauses computation or billing; pausing a socket and pausing inference are different capabilities.

At a high-water mark, stop reading upstream where supported and start a bounded stall policy, resuming only after the buffer drains below a lower threshold to avoid oscillation. If the client stays stalled, cancel upstream where possible, end the stream with the protocol's documented failure semantics, and reconcile uncertain usage. Never emit a normal completion marker for truncated output, and never drop arbitrary token chunks to make room, since that can corrupt text, structured output, or tool arguments.

Fairness, overload signals, and testing#

A global limit protects infrastructure, not tenants from each other. Use per-tenant queue bounds and a scheduler with explicit service classes, separate interactive traffic from large background jobs, and use weighted scheduling with a minimum share for everyone. Authorize priority server-side, and monitor fairness at the workflow level as well as per request.

Return overload signals the caller can act on. The episode notes that tenant quota exhaustion commonly maps to HTTP 429 and temporary service-wide overload may map to 503; document one consistent contract with a machine-readable reason and request ID, and send Retry-After only when you have meaningful guidance. Gateways such as Crazyrouter sit between many clients and providers, which is why internal retries and fallbacks must pass admission too: a fallback is not free capacity.

Measure the full funnel (offered, admitted, queued, dispatched, completed, rejected, expired, cancelled) and useful completions within deadline. Test with an open-loop load generator, mix short and long generations, pause readers, throttle the provider, and verify bounded queues, no leaked permits, and recovery without a retry storm.