AI API Gateway Context Compaction - Preserve Meaning Under Token Limits

Why trimming is a correctness problem#

Long-context failures are quiet. The request is accepted, the model answers fluently, and yet an earlier constraint, a tool result, a user correction, or a safety instruction disappeared during trimming. That makes context management a correctness and policy boundary, not just a way to save tokens.

The first step is to stop treating the prompt as one undifferentiated string. Separate it into classes, because each has different retention, trust, privacy, and compression rules:

• System policy and developer instructions
• User intent and conversation turns
• Tool calls and tool results
• Retrieved documents
• Model reasoning fields and gateway metadata

Protect hard facts and keep structured history#

Some facts cannot be paraphrased away: safety policy, authorization boundaries, the current task, user constraints, tool approvals, financial limits, data residency, and unresolved commitments. Attach provenance and priority to them. A summary that merely sounds equivalent is not enough when the original wording carries a hard guarantee.

Underneath, store messages as structured records with stable IDs, parent relationships, roles, timestamps, tool-call identity, citations, and content hashes. Compaction should produce a new context version that points back to its source records. That keeps rollback, audit, and dispute investigation possible without resending the full history to the model.

Compress by meaning, with bounded summaries#

Each compression technique fails differently. Recent-window trimming is simple but can drop early constraints. Summarization is flexible but can hallucinate or omit details. Retrieval recovers selected facts but may miss implicit context. A workable policy combines protected facts, hierarchical summaries, recency, relevance, unresolved tasks, and explicit user preferences.

Summaries themselves should be typed and bounded. Record what each one covers, which source messages it includes, what it intentionally omits, the model and prompt version that produced it, its confidence, creation time, and token size. Avoid open-ended chains of summaries of summaries that no longer reference their sources.

Tool state, trust labels, and contradictions#

A tool call and its result are not interchangeable with a sentence saying a tool was used. Keep the call ID, arguments digest, approval, result status, side-effect status, and citations. Verbose read-only output can be compressed aggressively, but an unknown or irreversible effect must never become an unverified fact.

Compaction is also a prompt-injection surface. Retrieved documents, tool output, and old assistant messages can contain instructions that must not outrank current policy, so trust labels and authority boundaries have to survive summarization. Likewise, record corrections and changed preferences as supersession with validity intervals instead of blending contradictions into a vague statement. When a conflict touches a hard requirement, ask for clarification or fail safely.

Tokens, caching, privacy, and policy#

System additions, tool schemas, media references, summary headers, and output allowance all consume the window. Estimate with the selected route's tokenizer where possible and reserve space for the response and tool rounds; a compacted prompt with no room for the required output is not a success.

Rewriting a summary or moving messages can break provider prefix caching, so keep stable prefixes and version summary boundaries, but never reuse a cached prompt after authorization, policy, or user state has changed. Raw history, summaries, embeddings, and audit references may need different retention; redact credentials before storing derived context and keep private history out of shared summaries or cross-tenant indexes.

Finally, make compaction policy-aware. Tenant plan, residency, safety mode, tool grants, and quality tier change what may be kept. If your application reaches several model families through one gateway such as Crazyrouter, remember that a cheaper route with a smaller window must not receive a weaker safety context. If the policy cannot fit, reject or choose a compliant route.

Validate, expose, and test before dispatch#

Before sending, check for missing protected facts, broken tool references, orphaned citations, duplicated instructions, invalid role transitions, overflow, and summary drift. On failure, expand from source records, switch strategy, or return an explicit context-limit error. The context used by each attempt should be immutable; a retry or stream reconnect may use a newer version only if the contract allows it and the change is recorded.

Honor user controls such as pinning, correcting, deleting, or no-memory handling, including removal of derived summaries. Record compaction decisions and costs (tokens before and after, summary generation, retrieval, cache hits) so operators can tell whether a bad answer came from the model or from the gateway. Test cases should include early safety instructions, late corrections, irreversible tool effects, injected retrieval content, deleted and pinned messages, tokenizer mismatches, retries, and tenant isolation.