AI API Gateway Health Probes — Test Readiness Before You Route Production Traffic

Health is not one boolean for AI routes#

A provider endpoint may answer a ping while streaming is broken, structured output is rejected, a model capability has changed, or a region accepts requests but cannot finish them within your deadline. Traditional liveness and readiness checks only ask whether a process is running.

The useful question for an AI gateway is narrower: can this provider, model, region and adapter deliver this workload under its current policy and deadline? Text completion, vision, tool calling and long-running video routes can all have different readiness states at the same moment.

Build a route inventory and separate three levels#

Give every route an identity made of provider, model family, region, adapter version and protocol mode. Then list the capabilities it claims: streaming, JSON or schema-constrained output, images, audio, tool calls, maximum input size, context limits and expected latency class. Without that inventory, a probe can show that something responded but not that the response proves anything.

Then separate three kinds of checks:

• Liveness: is the gateway or adapter process alive?
• Readiness: are the upstream connection, credentials, quota and basic protocol path available?
• Suitability: does a specific workload contract work, such as accepting the request shape, streaming expected events, returning valid structure or handling an image input?

Keeping these apart stops a green dashboard from hiding a broken capability.

Design safe synthetic probes that cover the real path#

Probes should use a tiny fixed prompt, no customer content, no external tools and no side effects. For structured output, request a small known object and validate it with the same parser production uses. For streaming, require the expected event sequence and a terminal marker. For vision, use a non-sensitive fixture with a known classification; for embeddings, check dimensions and numeric validity. Version and review fixtures, and keep them deliberately boring.

A direct provider ping can pass while DNS, a proxy, TLS, authentication, the adapter, response normalization or the client-facing stream fails. Run probes through the same meaningful layers as production, mark them clearly as synthetic, and record each stage so operators can tell an upstream refusal from a parser failure or a gateway policy rejection.

Workload-specific readiness and stable state transitions#

A provider can be ready for short non-streaming text and not for long streaming requests. Track readiness separately for protocol, capability, capacity, latency, quality guardrails and policy, so a route can stay eligible for low-priority batch work while being pulled from interactive traffic, rather than removing a whole provider because one feature degraded.

Measure more than success: probe duration, time to first byte, time to first token, total completion time, event counts, output validation, upstream status and failure reason, compared with each workload's thresholds. A probe that succeeds after thirty seconds is not ready for a two-second interactive deadline, and a fast response that fails schema validation is a failed probe for a structured workflow.

Use a sliding window and conservative transitions. One failure should not eject a healthy route during a network blip, but repeated failures should lower confidence quickly. Require several consecutive successes before re-admitting traffic and add cooldowns to prevent flapping. Named states such as unknown, ready, degraded, quarantined and recovering are easier to operate than an unexplained score.

Keep probes cheap, realistic and tied to routing#

Probes must not multiply an outage. Bound their frequency, concurrency and timeout, use a dedicated low-cost model or provider test entitlement when one exists, and never retry a probe indefinitely. Account for regional time zones, planned maintenance and the difference between cold and warm routes.

Credentials and quotas are part of readiness. A probe that runs on a privileged production key can look healthy while customer keys have different scopes or limits. Probe with credentials and policy that represent the traffic class being admitted, keep permissions minimal, attribute probe usage separately, give it a budget, and include it in capacity planning.

The router should consume a compact state with an expiry time, not a log. Decisions can then read like "eligible for streaming text in Tokyo" or "excluded from tool calling because the last three schema probes failed", with the probe version and evidence timestamp attached.

Probe on change, guard against false confidence#

Run probes whenever an adapter, model alias, routing policy, credential, region or feature flag changes, in addition to scheduled checks that catch drift. Keep new routes at zero traffic or a tiny canary until required probes pass, and compare replacement routes with the incumbent so "available" does not quietly mean "less capable".

Synthetic prompts cannot prove answer quality for every application, so pair readiness with offline evaluations and sampled production quality checks. Test the probe system itself by expiring credentials, corrupting a fixture, breaking a stream terminator, exceeding a context limit and simulating a slow but successful region. Expose safe route status, capability availability, last probe time and a stable reason code to developers, so they can tell "model does not support this feature" from "route is recovering". Multi-provider gateways such as Crazyrouter are where this route-level signal matters most.