AI API Gateway Failure Domains - Make Fallbacks Truly Independent

Different names, same dependency#

A fallback is only useful if it avoids the thing that failed. Two routes with different provider names can still share a cloud region, inference cluster, network edge, account quota, identity service, model host or billing control plane. When the primary fails and the fallback shares that dependency, the gateway has changed labels without changing risk.

Reliable routing needs diversity that is proven, not assumed. The goal of failure-domain-aware routing is to make sure a fallback changes the dependency that actually failed, not merely the endpoint name, and to be honest about how confident you are in that claim.

Build a dependency graph and label domains#

Map every route to its provider, model host, region, availability zone, network path, account, credential issuer, quota pool, storage, policy service, adapter and gateway control plane, including third-party dependencies where known. An incomplete graph should carry explicit uncertainty instead of an invented claim of independence.

A route belongs to several failure domains at once: physical location, provider organization, cloud platform, account, network, model family, deployment pipeline and shared gateway components. Keep these labels versioned and auditable, because a single provider field cannot describe multi-layer correlation.

Define the independence each request needs#

Critical workloads might require a fallback in a different provider and region; lower-risk traffic might accept another cluster in the same region. Residency, model quality, latency and cost further constrain the eligible set. Spell out which dimensions must differ.

Build ordered fallback sets before anything fails, using policy, capability, quality, deadline, price and failure-domain separation. Validate that each candidate can actually satisfy the request, since an independent route is useless if it lacks the modality, tool semantics, context length or data boundary the request requires.

Learn from telemetry and scope health precisely#

Compare error spikes, latency changes, rate limits, network failures and control-plane events across routes. Routes that degrade together may share a dependency missing from your inventory. Correlation is a clue, not proof, but repeated joint failure should lower your confidence that they are independent.

Separate common cause from shared demand: two providers may slow down together because a network path failed, or because a launch raised traffic everywhere. Keep route, tenant, region, network and workload dimensions in telemetry, and do not rewrite the graph over one coincidental spike.

Health should be scoped to domains. An account quota failure should mark routes on that account, not every route from the provider, and a regional network event should affect only routes crossing that path.

Hidden correlations: retries, credentials, software and models#

Several kinds of shared risk are easy to miss:

• Retry convergence. During an outage, independent workers may all pick the same fallback and overload it. Coordinate weights, reservations, retry budgets and adaptive concurrency, and reserve independent capacity for critical work.
• Credentials and quotas. Separate API keys can share one organization quota or billing account, and a revoked identity provider can disable credentials across vendors.
• Software. Operationally independent providers may share the same adapter, tokenizer, validation library, prompt template or rollout, so one translation bug breaks both.
• Model family. Endpoints serving the same weights or architecture can reproduce the same quality defect or malformed tool behavior, so operational diversity is not semantic diversity.

Third-party infrastructure is often opaque. Attach confidence levels and expiry to independence claims, route conservatively when guarantees depend on unknown internals, and do not advertise multi-provider resilience when two routes may share a hidden backend.

Policy, caches, billing and proving survivability#

Never silently relax residency, privacy, retention or customer allowlists during failover. If no compliant independent route exists, return a clear unavailable result and record which constraint blocked failover. Bind cached routing decisions to dependency-graph and health epochs so they are invalidated when labels, health or confidence change.

Failover can change price, latency, model behavior and tokenization, so record the original route, failed domain, fallback, attempts, usage and pricing version. Customers should not pay for uncontrolled retry storms, and a gateway such as Crazyrouter that fronts many providers should be able to explain the tradeoff it made.

Simulate provider outages, quota exhaustion, credential revocation, regional network loss, shared adapter bugs, control-plane outages, model regressions and dependency-map errors. Verify that the fallback avoids the affected domain, protected traffic keeps its capacity, and non-compliant routes stay excluded. Run game days with undocumented shared dependencies to see whether telemetry exposes them, then update the graph and confidence from the evidence.

Measure the share of requests with a valid independent fallback, dependency-map coverage, correlated error events, fallback success, retry amplification and recovery time. Count proven survivability, not provider logos in your config.