AI API SLOs and Error Budgets — Reliability You Can Actually Operate

Start from the user journey#

A provider can report healthy availability while your users see authentication failures, queue delays, malformed output or a model that cannot complete the requested tool call. Service-level indicators should measure the experience your product is responsible for, not the provider's dashboard.

That requires defining success precisely. For a chat request, success might mean the first token arrives within three seconds and the response completes without an application error. For a structured extraction workflow, success may require valid schema output. For an agent task, the final tool action and the user-visible result may matter more than the initial model response.

Separate objectives and measure percentiles#

Interactive chat, batch summarization, image generation, coding agents and payment-linked operations have very different latency and correctness needs. A single availability target across every endpoint produces a number that is easy to publish but hard to operate. Give each workload class its own objective.

Measure latency with percentiles, because averages hide the users who wait longest. Track time to first token and total completion time at the 50th, 95th and 99th percentiles. For asynchronous jobs, track queue age and time to final artifact rather than pretending that accepting a request equals completing it.

An image generation job and a payment-linked operation should not share one target simply because they share an API. Each deserves indicators that match what its users actually notice.

Quality failures are failures#

Empty responses, invalid JSON, missing citations, failed tool calls, language mismatches and unusable media are all user-visible failures, even when the HTTP status is 200. Not every quality issue belongs in a pager alert, but important product contracts should be part of the objective wherever they can be measured.

The goal is coverage of the contracts users rely on. If a workflow promises structured output, invalid structure is an outage for that workflow, whatever the status code says. If an agent promises to complete a tool action, a model reply that never reaches the action is a failure too.

Error budgets and burn-rate alerts#

An error budget turns an objective into an operating allowance. If the target permits a small fraction of failed or excessively slow requests, that allowance guides decisions: when the budget is healthy, teams can ship faster and try new routes; when it burns too quickly, reliability work takes priority over risky changes.

Do not wait for the monthly target to fail. Burn-rate alerts react to how fast the budget is being consumed. A short window detects severe outages quickly, while a longer window catches gradual degradation such as rising tail latency or repeated fallback use. Combining both reduces noise while still surfacing urgent incidents.

Used this way, the error budget becomes a shared language between product and engineering: it tells both sides when there is room to experiment and when stabilizing has to come first.

Attribution, fallbacks, cost and policy#

Honest reliability accounting depends on knowing where failures come from and what recovery really delivered:

• Attribute budget consumption by layer. Separate application errors, gateway policy failures, provider failures, queue timeouts, schema failures and customer cancellations, so the team responds correctly instead of blaming the upstream provider for everything.
• Treat fallbacks honestly. A fallback can preserve availability but change latency, context limits, tool support, safety behavior or output quality. Without a per-workload decision on how fallbacks count, the dashboard stays green while the product quietly gets worse.
• Connect SLOs to cost. Aggressive retries can raise apparent availability while doubling spend and tail latency, and a premium fallback may rescue important requests but waste money on low-value batch jobs. Monitor cost per successful outcome alongside reliability.
• Protect tenant and regional policy. A response that violates data residency, model allowlists or customer routing rules is not a success. Include policy enforcement in synthetic tests and incident reviews.

A unified OpenAI-compatible gateway such as Crazyrouter can give you consistent routing and fallback signals across models, but the success definitions remain yours to set.

Dashboards and ongoing review#

Build dashboards around decisions: current objective compliance, remaining error budget, burn rate, affected workloads, fallback volume, latency percentiles and representative trace IDs. Operators should be able to move from a budget alert straight to the routes and failure modes consuming it.

Revisit objectives as the product changes. A target designed for a simple chatbot may not fit an agent that performs financial actions. Review indicators after major model changes, new regions, new tools and significant traffic shifts, and keep historical definitions so trend reports stay understandable.