AI API Gateway Semantic Caching - Reuse Results Without Reusing Risk

Decide what is allowed to be reused#

Reusing an answer saves time and money, but similarity is not permission and a plausible answer is not necessarily the right one. Start by sorting workloads into three buckets: cacheable (deterministic, public), conditionally cacheable, and never cacheable. Requests that depend on tenant data, user permissions, the current time, tools, retrieval state or hidden system context need extra care.

When the classification is unknown or the input is sensitive, the safe default is a miss. A shared cache entry should never be created by accident.

Build cache keys from behavior, not prompt text#

Two identical prompts can mean different things under different policies. If a dimension can change the meaning or authorization of a result, it belongs in the cache key or in an invalidation rule. Typical dimensions include:

• Tenant or public scope, and project
• Model, provider contract and adapter version
• System instructions and policy revision
• Region and data classification
• Tool permissions and retrieval corpus version
• Output format, temperature, seed and other relevant parameters

Similarity search only proposes candidates. An embedding match cannot prove equivalent intent, permissions, freshness or safety, so combine thresholds, reranking, structured filters and a final compatibility check. Where a wrong hit is expensive, require a stricter exact or semantic contract instead of relying on distance alone.

Isolation, permissions and freshness#

Private output must never land in a global namespace. Enforce scope on the server for both reads and writes, and bind each entry to the identity and policy context of the request that created it. A client-supplied cache scope is a hint to validate, not an authority.

Permissions also change after the fact. A user can lose access to a document or tool while an old answer stays cached, so decide whether authorization is rechecked on every hit or tied to a permission epoch, and make sure a hit never skips current policy evaluation. Freshness is question-specific: a definition can stay useful for months, while balances, model availability or retrieved documents go stale fast. Store creation time, source versions, expiry and a freshness class, and disable stale-while-revalidate where stale data could cause harm.

Model, prompt and policy changes alter meaning too. Tie entries to a tested model contract so a route alias change does not silently serve output from an incompatible model, and support immediate bypass for urgent safety or deny rules.

Tools, streaming and honest billing#

Answers built from retrieval or tool calls depend on the source they read. Reference a source version or freshness token, and generally bypass the cache for mutating tools unless the result is explicitly read-only and safely scoped.

Streaming needs its own decision: cache the assembled result, the event sequence, or nothing. Preserve finish reason, structured output, usage and cancellation state. A client that saw a partial stream must not later receive a cached complete answer as though the original call had succeeded, and hits should say whether they are replays.

On billing, a hit may skip provider usage but still consume gateway resources. Choose whether cached responses are charged, discounted or shown as a separate category, record hits, misses and bypasses against the logical operation, and never invent provider tokens for work that was not sent upstream.

Warming, eviction, residency and poisoning#

Precomputing popular prompts can send sensitive requests to providers and create charges before anyone asks. Warm only approved public inputs with bounded budgets and known policy, region, model and freshness, and never use customer traffic as an unreviewed warming source.

Eviction is part of correctness. Remove entries when credentials, permissions, source data, model contract or residency rules change, and propagate tenant deletion across replicas, indexes, backups and derived embeddings. A cache that drops the answer but keeps a vector or debug copy is not fully deleted. Store entries only in eligible regions and fail closed when location or classification is unknown.

Assume attackers will try high-similarity prompts, weak normalization or cross-tenant reuse. Validate provenance, scope and content before insertion, rate-limit warming, isolate namespaces and watch for unusual hit patterns.

Observability, failure modes and rollout#

Every hit should be explainable: record an opaque key reference, scope, similarity and compatibility decision, model and policy revisions, age, source version and outcome, without logging raw prompts or answers by default. Keep the hit on the request lineage so support can answer why a request skipped the provider and whether the entry was still authorized.

If the cache is down, fall back to the normal provider path only when policy, capacity and budget allow. If validation is down, treat candidates as misses. Roll out on a narrow tenant, public workload, model and region, comparing hits against fresh generations, and remember that disabling writes is not a rollback while old entries stay readable. A semantic cache is a second inference path; whether you run it yourself or behind a gateway such as Crazyrouter, reuse a result only when it still belongs to this request, tenant, policy and moment.