Tokenizer Governance: Keep Token Counts and Billing Aligned (Podcast EP338)

Key takeaways

Name the counting contract for every estimate: model, tokenizer family and revision, chat template revision, adapter version, counting method, and observation time.

Count the effective provider request after semantic assembly, including role markers, tool definitions, schemas, and media, not just the visible prompt text.

Pin tokenizer assets with checksums and version chat templates independently so a library update cannot silently change production counts.

Reserve room for requested output, tool-call continuation, a safety margin, and provider overhead, and recount for every eligible route including fallbacks.

Keep estimated, provider-reported, gateway-measured, and billed quantities separate until reconciliation, and use calibration to improve future estimates without rewriting past ones.

Truncate only by source and priority under explicit policy, never by blindly cutting bytes, JSON, protected instructions, or artifact references.

Show notes

Tokens are not a universal unit#

Identical text can tokenize differently across model families, tokenizer revisions, Unicode handling, chat templates, and provider wrappers. A gateway built on the wrong assumptions will admit requests that fail upstream, reject requests that would have fit, and explain charges incorrectly.

The fix starts with naming the counting contract. An estimate is only reproducible if it records:

Model identity
Tokenizer family and revision
Chat template revision
Provider adapter version
Counting method and observation time

A bare "estimated tokens" number without these dimensions cannot be explained later.

Count the effective request, not the visible text#

System and role markers, message separators, tool definitions, JSON schemas, images, audio, files, reasoning controls, and provider metadata can all add hidden or model-specific units. Admission has to reason about the effective provider request.

Three habits keep those counts honest. Pin tokenizer assets (vocabulary, merge rules, special-token maps, normalization, template code) as immutable, checksum-verified files. Version chat templates separately, because message framing can change while the tokenizer stays the same. And count after semantic assembly: early estimates help with planning, but a final bounded check should run on the assembled request, using the same structured representation and adapter rules that will build it. Flattening messages into one string can change both the count and the semantics.

Tools, media, and output reservation#

Tool names, descriptions, schemas, strictness flags, and allowed choices can consume a large share of context. Include the tool registry version and rendered schema in the count, and invalidate cached estimates when a description or schema changes.

Multimodal inputs need their own accounting. Providers may meter images by tiles, resolution, detail level, duration, frames, or proprietary units. Record each media input, its transformation, the provider rule, and the estimated units, and avoid converting everything into fake text tokens unless the contract defines that conversion.

Input that exactly fills the window leaves no room for the answer, so reserve requested output tokens, tool-call continuation, a safety margin, and provider overhead, and make it visible when the gateway reduces or rejects an output limit. For unknown models, use a documented upper-bound estimator with a margin and a confidence flag; a characters-divided-by-something heuristic is not exact enough near a hard limit.

Calibrate without rewriting history#

Compare estimated input, output, cached, reasoning, media, and total units with what providers report, keyed by attempt, model, route, and revision. Calibration should update future confidence and policy, not edit the estimate already attached to a completed request.

Provider reports are not automatically authoritative either: fields can be missing, asynchronous, rounded, or redefined. Keep estimated, provider-reported, gateway-measured, and billed quantities separate until reconciliation decides the charge. Bind count caches to a canonical request identity, so two transport encodings of the same request share a count while any change to model, messages, tools, artifacts, template, or policy creates a new one.

Routing, truncation, and streaming#

A request may fit one model and overflow another, so count or bound against each eligible route before selecting it, and repeat the check on fallback rather than reusing the primary model's number. When one gateway such as Crazyrouter gives an application access to several model families, this per-route check applies on every fallback.

When a request exceeds its budget, trim under explicit policy: optional examples, low-ranked retrieval, old history, or verbose tool output first. Never cut byte sequences, JSON structures, protected instructions, or artifact references blindly, and record what was removed, the before and after counts, the tokenizer revision, and the reason. A shortened request is a different input with its own cache and audit identity.

For streams, the final output count may arrive only in the last event. Track reserved, emitted, provider-reported, accepted, and delivered units separately, and remember that a client disconnect does not prove generation stopped.

Operating the counting layer#

Counting is itself an attack surface. Bound input size, nesting, schema complexity, and tokenizer CPU, and cache immutable tool and template components so a hostile request cannot burn memory just to learn it is too large.

Roll out tokenizer changes by comparing old and new revisions on representative traffic, measuring count deltas and admission changes, staging workers, and keeping rollback. Never mix old-revision admission with new-revision rendering for the same request. Return typed errors such as unsupported tokenizer, unknown template, input limit exceeded, output reservation conflict, media limit exceeded, tool schema too large, or count changed after assembly, without exposing hidden prompts. Test mixed scripts, emoji, combining characters, malformed Unicode, special-token text, nested schemas, and fallback models, and track estimation error, false gateway rejects, template drift, and reconciliation gaps by model and revision.

Questions from this episode

Why does the same prompt have different token counts on different models?

Token counts depend on the model family, tokenizer revision, Unicode handling, chat template, and provider wrapper. Role markers, tool schemas, and media also add model-specific units, so a count is only meaningful alongside the exact tokenizer and template that produced it.

How much space should I reserve for output when checking context limits?

Reserve the requested output tokens plus room for tool-call continuation, a safety margin, and provider overhead. The episode gives no universal number; the key point is that input fitting exactly into the window still leaves no room for the result, and any reduction of the output limit should be visible to the caller.

Should billing use my own token estimate or the provider's usage report?

Keep both, plus any gateway-measured quantity, separate until reconciliation determines the authoritative charge. Providers can omit fields, report late, round values, or change semantics, so use their reports to calibrate future estimates without rewriting the estimate stored on a completed request.

AI API Gateway Tokenizer Governance - Keep Counting, Limits, and Billing Aligned