AI API Gateway Session Affinity - Keep Stateful Work on the Right Route

Hidden state behind stateless requests#

A provider may hold a session, a worker may keep a conversation cache, an adapter may maintain state, or a workflow may need every step inside one regional boundary. When a follow-up lands somewhere unrelated, users see lost context, extra cost, higher latency or subtly inconsistent answers.

Start by naming what state actually exists instead of calling it all "the session": conversation history, provider session identifiers, connection pools, tool state, safety decisions, cached prefixes, rate-limit buckets and workflow checkpoints. Each needs an owner, lifetime, access policy, replication rule and recovery behavior.

Write an explicit affinity contract#

Specify the stable session identity, eligible regions and workers, required capabilities, maximum idle time and lifetime, migration policy, and what happens when the preferred location is unavailable. Be clear whether affinity is required for correctness, useful for performance, or best-effort.

The affinity key should be opaque. Never put provider credentials, conversation content or sensitive tenant data in a routing cookie or header. Bind the key to tenant, workflow, policy and expiry, and validate issuer, audience, scope and version. A session identifier locates state; it does not grant access. Authorization is rechecked on every request and after every migration, because yesterday's key must not bypass today's model allowlist, residency rule or tool permission.

External state and the right level of stickiness#

Stateful workers lower latency but complicate placement, draining, restarts and scaling. When recovery is part of the contract, keep durable conversation or workflow state in an access-controlled store and treat local caches as accelerators with explicit invalidation.

Provider sessions deserve honesty: they can expire, be region-bound, or be tied to a model revision. Record their scope and last verification, and if one expires, rebuild from an approved durable representation or return a clear continuity error rather than silently attaching a new session.

Choose the narrowest affinity that works. Some workloads need worker affinity, others region or model-revision affinity. Overly sticky routing wastes capacity and hurts balancing; overly loose routing loses the state you meant to keep.

Migration without split brain#

Treat migration as a protocol, not a reroute:

• Quiesce new work or establish a new epoch
• Flush durable state and transfer or rehydrate approved caches
• Verify policy and capability at the destination
• Publish the new location and reject or safely retry requests carrying the old epoch

Use fencing tokens, leases, epochs or a single authoritative owner record so two workers never both believe they own a mutable session. If the state store cannot provide the ordering you need, downgrade the contract or refuse migration rather than relying on timing luck.

Session-routed requests still carry their own deadlines, so do not wait forever for a preferred worker when a compatible fallback exists. A cancelled request must not leave a half-written turn behind. Tool effects need idempotency, authorization and an ownership epoch, and the same effect must never run twice because two workers saw the same pending message.

Failover, cleanup, streaming and billing#

Define which state can cross regions and which data boundaries forbid it. A failover may keep conversation history but lose hidden provider cache state or exact sampling continuity, so tell the caller which case applies.

Sessions need idle timeouts, absolute lifetimes, size limits and verifiable deletion across replicas and derived caches, and a stale key must not keep sensitive data alive. Affinity maps, session stores and migration streams must be tenant-scoped, and a recycled worker must never serve old state to a new identity.

A stream may pin a worker for its lifetime while the next turn lands elsewhere; never append a partial assistant message as a completed turn. Migrations that rebuild context can consume extra tokens, so usage should show whether cost came from user work, recovery or maintenance. For a gateway such as Crazyrouter that routes across many providers, that attribution is what keeps recovery explainable.

Measure and test ownership failure#

Track stickiness, state-store lookups, cache hit rate, migration duration, rebuilds, stale-owner conflicts, failover continuity, extra tokens and first-request latency after migration. High stickiness is not success if it creates hotspots or slows recovery.

Test worker restarts, lease expiry, duplicate owners, concurrent turns, provider session expiry, model revision changes, region outages, policy changes, cancellation during migration, partial streams and clock skew. Assert that no request reads unauthorized state and no side effect runs twice. Affinity is reliable when continuity is explicit, bounded, recoverable and never mistaken for authorization.