Deadline Propagation for AI Gateways - Make Every Timeout Share One Clock

Timeouts and deadlines are different tools#

Picture a support application with a fifteen-second response target. Retrieval has a ten-second timeout, the model client thirty seconds, and the fallback another thirty. Each setting sounds defensible, but the user experiences one operation. If the browser gives up at fifteen seconds while the backend keeps generating for a minute, the system spends money and capacity on an answer nobody receives.

A timeout limits how long one activity may wait. A deadline marks when the whole operation stops being useful. You need both:

• A connection timeout guards slow connection setup.
• A response timeout limits a particular exchange.
• An idle timeout detects a stream that stopped making progress.

None of them enforces the total lifetime of the operation. A stream that emits a tiny event just before each idle timeout can run far past the intended deadline.

Define completion and admit with a deadline#

Start at the product boundary. Success might mean the first visible token, a complete validated answer, or an accepted background job, and those are different contracts. An interactive extraction endpoint may need a validated object within its target, a chat interface may measure first meaningful output separately from completion, and long document analysis may belong behind a job ID and status endpoint. Do not silently convert an interactive timeout into a background task.

At admission, create an operation identifier and an effective deadline. A caller-provided deadline is an input to validate: reject malformed values, clamp excessive budgets to a server-side maximum, and keep a tighter caller limit when one is given. Expired requests should never enter the inference queue. And keep time separate from authorization; remaining budget never grants permission to run a tool or use a restricted model.

Carry the clock across processes#

Within a process, measure elapsed time with a monotonic clock, because wall clocks jump when the system synchronizes time. Monotonic readings mean nothing on another host, so choose a documented propagation contract. An absolute deadline needs bounded clock-skew handling; a relative remaining-time value needs explicit accounting for transit and queue delay, or each hop quietly adds time back. Do not assume a custom header reaches the provider or that the provider enforces it; keep a local timer and cancellation handling regardless.

The central rule is that a child operation cannot outlive its parent. Before each downstream call, take the remaining time, subtract the reserve needed to finish locally, and apply the smaller of that and the stage's own maximum. Recompute after waiting for a connection or concurrency permit, and do not dispatch if the result is not positive. Confirm with a timing test what each SDK timeout actually covers: connection setup, reads, pool acquisition, the whole call, or only inactivity.

A worked budget, and its limits#

The episode walks through a hypothetical fifteen-second extraction. Admission and authentication take half a second, queueing two seconds, retrieval two and a half, leaving ten. Reserving one second for validation and delivery leaves nine for model work. If the first attempt fails after six seconds, the retry has three seconds, not a fresh nine. If no eligible fallback can produce a useful result in that window, stop. These figures illustrate accounting, not recommended defaults.

Reservations are not guarantees. Validation can overrun and delivery can stall, so enforce the parent timer through finalization and size reserves from observed latency distributions. Adding each component's 99th percentile does not give an exact end-to-end 99th percentile, so measure end-to-end outcomes under realistic load as well.

Queues, fan-out, and streaming#

Queues can consume the budget before a model sees the request. Carry expiration with queued work, check it at enqueue and again at dequeue, and bound queue length and admission concurrency. Parallel retrieval branches inherit the same outer deadline; parallelism shortens the critical path but does not create more time. Cancel branches the answer no longer needs and bound fan-out so speed does not turn into uncontrolled cost.

Streaming adds a second clock. Track time to first meaningful output and time since useful progress, while keeping an absolute completion bound where the product needs one. Transport heartbeats prove the connection is alive, not that the model is progressing, so define which events reset which timer. Once headers and partial output have gone out, you usually cannot switch to a normal JSON error with a new status code; use the stream protocol's terminal error behavior and show interrupted output as incomplete.

Cancellation, observability, and rollout#

When the user disconnects or the deadline expires, signal child tasks, stop scheduling new attempts, close streams where appropriate, and release local resources. This is best effort: it does not prove a remote provider stopped computing or billing, and a tool may already have committed a side effect. Reconcile unknown outcomes by action identifier instead of assuming cancellation rolled them back.

Capture the initial budget, queue time, remaining budget at each dispatch, stage durations, and the stop reason, distinguishing caller disconnects, local deadlines, upstream timeouts, and idle-stream timeouts. Keep operation IDs out of metric labels and in protected traces. Useful signals include expired-before-dispatch counts, deadline success rate, and wasted downstream work. Test with a fake clock: expire a request in the queue, delay retrieval, trigger a retry, send content-free heartbeats, disconnect mid-generation, and simulate clock skew. Roll out on one endpoint, instrument before tightening limits, canary with rollback, and judge success by valid completed outcomes rather than lower average latency.