AI API Gateway Stream Recovery - Reconnect Without Duplicating Work

Why streaming changes failure handling#

A client might receive half a response, lose its connection, reconnect through a different worker, and ask what happened while the provider is still generating. The gateway needs a recovery contract that tells apart replayable output, in-flight work, completed work, cancellation and unknown provider state.

The goal is simple to state: a broken connection should change the transport, not the meaning or the accounting of the operation.

Stream identity, sequence numbers and replay windows#

Persist the operation ID, attempt ID, stream ID, model revision, route, tenant, policy version and response contract. A reconnect token should locate an authorized stream record bound to the caller, scope, expiry and protocol version; stale or cross-tenant tokens are rejected.

Give each event a monotonic sequence number within the stream. Replay requests name the last sequence the client has durably accepted, and the gateway resends from there without skipping, reordering or inventing events. Sequence numbers aid recovery; they do not prove the provider finished the corresponding work.

Replay buffers must be bounded in time and size, and privacy policy may forbid retaining content at all. When the requested point is gone, return a clear recovery boundary so the client can choose a fresh request, a completed result or an explicit failure. Acknowledgements should reflect what the client has durably processed, not just what reached a local buffer.

Replay is not resume#

Replay resends output that was already produced. Resume asks the provider or gateway to continue from a checkpoint, which may be unsupported or semantically unsafe. Unless the provider contract guarantees continuation, a new provider attempt is a restart or reconstruction, and it should be labeled that way.

Partial responses need the same honesty. Record whether a stream ended normally, was cancelled, expired, failed validation or stopped after a provider error, and on reconnect tell the client whether it is getting replayed partial output, a final result or an error boundary.

Disconnects, reconnect races and cancellation#

One dropped TCP connection does not necessarily mean the caller gave up. Use a short grace period, an explicit cancellation endpoint and a policy for whether generation continues; if nobody is consuming and continuing has no value, cancel and record the provider outcome.

Reconnects race. A client may reconnect while the old connection is alive, or two workers may handle the same recovery request. A stream lease or generation number with atomic takeover ensures only one writer appends a given sequence.

Cancellation can land just after the final token, during provider shutdown or while a tool awaits approval. Use an explicit state machine with compare-and-set transitions and report whether cancellation was accepted, too late or unknown. Never claim a clean cancel when an effect may already have happened.

Tools, billing and provider uncertainty#

Tokens that describe a tool call can be replayed; the tool execution cannot. Persist tool-call identity, an arguments hash, approval, dispatch, observed result and owner epoch, and never dispatch an effect just because the client missed the earlier tool event.

Billing follows attempts. A reconnect with replay should not create a new provider charge, while a real restart or fallback creates new usage linked to the original operation. Customers should be able to see whether recovery reused work or paid for a new attempt. Behind a multi-provider gateway like Crazyrouter, that linkage is what makes a recovered request's cost explainable.

If the gateway loses the provider connection before a terminal event, the outcome may be unknown. Do not blindly retry requests with tools or side effects; mark the attempt uncertain, reconcile through provider status where available, or return an explicit unknown result.

Security, failover, backpressure and testing#

Stream buffers, reconnect tokens and traces can hold private prompts and output. Encrypt or restrict them per tenant policy, redact credentials, and avoid revealing provider names, queue state or another tenant's streams through recovery errors. If a reconnect reaches another region, confirm that the stream record, buffer, policy and model revision exist there; if only the final output is durable, offer replay or retrieval instead of pretending live continuation works. Bound replay rate and buffer growth so recovery traffic cannot degrade unrelated streams.

Test disconnects before the first token, mid-event, after a tool proposal and after tool execution, plus duplicate reconnects, stale tokens, missing buffers, lost acknowledgements, provider timeouts, worker restarts and region failover. Track replay gaps, takeover conflicts, unknown outcomes and extra cost, and assert that events stay ordered and effects never repeat.