AI API Request Logging in Python: Usage, Retries, and Validation

AI API request logging should create one record per network attempt and connect those attempts to one application operation. Capture status, elapsed time, request IDs, usage when supplied, and application validation without logging prompts, credentials, or full model outputs by default.

Using Crazyrouter at https://cn.crazyrouter.com/v1, we ran the Python recorder below against text-embedding-3-small on October 11, 2026. It recorded HTTP 200, one validated 512-dimension vector, and 5 reported input tokens. The fixture contains synthetic text and measures one request, not a latency distribution or a monthly invoice.

Which fields belong in an AI API request log?#

Use low-cardinality dimensions such as endpoint, model family and outcome in aggregated metrics. Request IDs and task IDs belong in logs or traces, not unrestricted metric labels.

Python: record a real embedding request without saving its content#

Install requests, set CRAZYROUTER_API_KEY, save this as observe_embedding.py, and run python observe_embedding.py. The example uses a direct connection; configure the session if your deployment requires a proxy.

"""Emit one content-free record for one API attempt, including failures."""
import json
import os
import time
import uuid
from datetime import datetime, timezone
import requests

def observe():
    session = requests.Session()
    session.trust_env = False  # This fixture uses a direct connection.
    record = {"at": datetime.now(timezone.utc).isoformat(),
              "operation_id": str(uuid.uuid4()), "attempt": 1,
              "requested_model": "text-embedding-3-small", "endpoint": "/embeddings",
              "status": None, "provider_request_id": None,
              "usage": None, "outcome": "error", "vectors": None, "dimensions": None}
    start = time.perf_counter()
    try:
        base = os.getenv("CRAZYROUTER_BASE_URL", "https://cn.crazyrouter.com/v1").rstrip("/")
        response = session.post(base + record["endpoint"], timeout=(10, 60),
            headers={"Authorization": "Bearer " + os.environ["CRAZYROUTER_API_KEY"]},
            json={"model": record["requested_model"], "input": ["Synthetic observability fixture"],
                  "dimensions": 512})
        record["status"] = response.status_code
        record["provider_request_id"] = response.headers.get("x-request-id") or response.headers.get("request-id")
        response.raise_for_status()
        body = response.json()
        record["reported_model"] = body.get("model")
        record["usage"] = body.get("usage")  # Missing usage is unknown, not zero cost.
        rows = body.get("data", [])
        if len(rows) != 1 or len(rows[0].get("embedding", [])) != 512:
            raise ValueError("Embedding shape did not match the requested contract")
        record.update(vectors=1, dimensions=512, outcome="validated")
        return record
    except Exception as exc:
        record["error_type"] = type(exc).__name__
        raise
    finally:
        record["duration_ms"] = round((time.perf_counter() - start) * 1000)
        print(json.dumps(record, ensure_ascii=False))

if __name__ == "__main__":
    observe()

The finally block emits a record even when the request or shape check raises an exception. Exceptions remain failures to the caller. The code logs neither the authorization header nor the embedding array nor the input text.

Selected fields from the verified run:

{
  "requested_model": "text-embedding-3-small",
  "endpoint": "/embeddings",
  "attempt": 1,
  "status": 200,
  "usage": {
    "prompt_tokens": 5,
    "total_tokens": 5
  },
  "outcome": "validated",
  "vectors": 1,
  "dimensions": 512,
  "duration_ms": 826
}

The complete record also included a locally generated operation ID and a service response request ID. The milliseconds above are one observed end-to-end duration; they should not be used as p50, p95, an SLA, or a cross-provider ranking. See the embedding API guide for batching, dimension selection and retrieval validation.

How should retries and tool calls appear in traces?#

Put the operation ID outside the retry loop. Increment the attempt number for each real HTTP request, then emit a separate final task outcome after validation. If an SDK also retries, instrument or disable that retry layer so request counts remain interpretable.

A task that needs three model calls and one database lookup should have one parent task and separate child operations. Record time spent waiting, executing tools and retrying separately; otherwise an apparent “slow model” may really be a slow tool or backoff delay.

The recorder intentionally sends one attempt. Use the Python retry implementation to add bounded retries, and use the tool protocol comparison when adapting tool messages across native APIs.

How do usage tokens become a defensible cost estimate?#

Usage and billing are related measurements, not interchangeable fields. The response may omit usage, represent cached input differently, or report an amount before route discounts. A timeout may leave processing and charge status unknown. Keep missing values as unknown and reconcile with your actual usage or billing ledger.

For a token-priced route, calculate an estimate from the current input, output and cache rates, retaining the pricing version used. For image or video routes, use their actual billing unit instead of inventing token counts. Reconcile estimated spend with debited spend before setting budget alerts.

Do not calculate cost per successful task using only the successful attempt. Include eligible failed attempts, retries and tool work in the numerator; count validated completed tasks in the denominator. If there are no successful tasks, report the metric as undefined rather than zero.

Check current pricing and inspect the usage console before using an estimate for purchasing or budget limits.

Use logs to improve the workload, not to accumulate private content#

For repeated embedding jobs, a high attempt count may indicate that unchanged documents are being processed repeatedly. Track changed-document counts and checkpoint outcomes before changing the model. The incremental embedding guide explains that workflow.

Set access controls and retention for operational logs. Even IDs and hashes may be sensitive or linkable. This fixture uses a synthetic string, but a production system needs a deliberate decision about which metadata is necessary and who can read it.

Frequently asked questions#

Is HTTP 200 enough to count an AI request as successful?#

It confirms HTTP success, not a usable result. The example additionally validates the vector count and dimension. A tool workflow needs argument and business-result checks appropriate to that operation.

Should missing usage be stored as zero?#

No. Store unknown usage explicitly, then reconcile if provider records become available. Zero and unavailable have different meanings for budgets.

Should request IDs be metric labels?#

Usually not. Unique IDs produce unbounded cardinality. Store them in logs or traces and use bounded labels for dashboards.

Is this a replacement for an observability platform?#

No. It is a runnable instrumentation boundary that can feed your existing log collector. Aggregation, alerting, retention, tenant isolation and billing reconciliation remain separate responsibilities.