AI API Tail Latency — Reduce Slow Requests Without Doubling Spend

Why averages hide slow AI API requests#

Tail latency is the small share of requests that wait far longer than the median. An API can look fine on a dashboard of averages and still feel broken to the users who land in that tail.

Measure the full distribution instead: p50, p90, p95, p99 and maximum latency, broken down by route, model, provider, region, workload and response mode. Separate time to first token from time to a complete, accepted result. A streaming chat feels responsive after the first token, but a JSON extraction is useless until the whole validated object arrives, so target the outcome the caller can use.

To find where the tail starts, split traces into queue wait, gateway processing, provider connection, time to first token, generation, validation and fallback time. Compare cold and warm connections, prompt sizes, output lengths and concurrency. The tail is often a small cohort: long contexts, large images, an overloaded region, one model capability, or a queue that provider averages never show.

Set a deadline before you consider hedging#

An HTTP client timeout is not a workflow deadline. The real budget includes queue time, connection setup, time to first token, generation, validation and any tool calls. Attach one absolute deadline to the request, pass the remaining time to every downstream component, and stop work once it is exhausted.

A bounded deadline lets every layer stop spending when the answer is already too late, and stops a slow provider from holding connections, queue slots and user requests indefinitely. When the deadline is hit, return a stable timeout class with a request ID and a useful retry hint rather than a generic 500, so operators can tell "we ran out of time" apart from "the provider rejected the request".

How to hedge without doubling spend#

Hedging starts a second attempt when the first looks unusually slow, before the deadline expires. It helps with stuck connections or overloaded replicas, but it is not free: two attempts can consume twice the tokens, compete for the same capacity and create duplicate side effects.

Choose the trigger from evidence. A fixed delay may suit one route and waste money on another, so use a percentile-based threshold per model and workload, plus a minimum elapsed time and a check that enough deadline remains. Never hedge every request.

• Start with a small fraction of traffic and compare completion time and cost.
• Stop when the extra attempt rate or provider load exceeds its budget.
• Treat a hedge as an exception for the tail, not a second default route.

Cancellation, side effects and choosing a winner#

When one attempt wins, cancel the other at the gateway and pass cancellation to the provider when it is supported. Then confirm that cancellation actually stops generation and releases quota. If it is only advisory, your billing and capacity accounting must assume the loser keeps running.

Only duplicate operations that are safe to duplicate, such as some text completions, embeddings or deterministic classifications. Payments, emails, database writes, tool calls and job submissions need an idempotency key and a durable request state machine, or a split between planning and commit. A timeout does not prove the provider did nothing, so reconcile the original attempt first.

The first response is not always the best one. For structured output, validate before declaring a winner. For retrieval or tool workflows, check that the result belongs to the same request and policy context. If both attempts finish close together, choose deterministically and account for the loser, and never stitch partial streams from two attempts into one answer unless the protocol explicitly supports it.

Budget attempts and coordinate with retries#

Set explicit limits on attempts per request, extra tokens, extra dollars, concurrent hedges and provider-specific rate usage. Show hedge traffic separately from ordinary retries and fallbacks in dashboards and alerts. When the primary route is already degraded, reduce or switch off hedging instead of adding pressure; a slower single attempt is often the right call when capacity is scarce.

Give one component ownership of attempt policy, tag each attempt with an attempt number and parent request ID, and propagate the deadline. A hedge that hits a rate limit should not immediately fire another retry, and upstream layers should not each duplicate a running attempt. Classify the failure first: slow responses, connection errors, rate limits, invalid results and policy blocks each need different handling.

Hedging can look like a sudden traffic multiplier to a provider. Honor rate-limit signals, respect provider-specific limits, and watch the extra request rate, token rate, connection count and error rate per provider. A latency win that triggers throttling for everyone is a regression.

Test the hard cases and make the policy explainable#

Simulate a provider that accepts a request but delays its first token, one that keeps generating after cancellation, a response that arrives after the deadline, two valid responses with different content, a rate limit on the hedge, and a network partition that hides whether the first attempt finished. Add oversized prompts, streaming disconnects, validation failures and a gateway restart mid-flight, and confirm billing reconciliation and request state stay correct.

Each trace should show the deadline, route, hedge threshold, attempt number, trigger, cancellation status, validation result and final cost, with prompts and customer data redacted. Give the policy an owner, a change history, expiry criteria and a visible kill switch. Gateways such as Crazyrouter, which route many models behind one OpenAI-compatible key, are a natural place to apply this discipline consistently across providers.