Back to Blog
EnglishTutorial

AI Structured Output in Python: JSON Mode, Schema, and Live Validation

Four live GPT-4.1-mini and Gemini requests show why HTTP 200 is not enough. Includes a runnable schema example, one fenced-JSON failure, and eight local validation checks.

S
Sophie M
April 8, 2026 / 2438 views
Share:
AI Structured Output in Python: JSON Mode, Schema, and Live Validation

Structured output needs three checks: the response finished, the text parses into the expected shape, and the extracted values match the source. JSON mode and JSON Schema requests address different parts of that contract. Neither an HTTP 200 nor a schema-shaped object proves that the facts are correct.

Crazyrouter is an AI API gateway where this four-request fixture produced three directly parseable, correct JSON objects and one Markdown-wrapped response that failed strict JSON parsing. The requests used GPT-4.1-mini and Gemini 2.5 Flash through Chat Completions. This small test describes these routes and inputs, not a provider reliability ranking.

By Sophie M · Tested November 3, 2026

JSON mode versus structured outputs#

The OpenAI structured-output documentation distinguishes JSON mode from schema adherence. The model, endpoint and supported schema subset still matter. Do not assume that sending an OpenAI-shaped parameter through a compatibility gateway guarantees identical native-provider behavior.

Request or validation layerPurposeWhat you still check
response_format: {"type":"json_object"}Ask the compatible endpoint for JSONParsing, required keys, types, and facts
response_format: {"type":"json_schema", ...}Supply a schema to a supported endpointCompletion/refusal, local validation, and facts
Local application validatorReject objects outside your contractWhether accepted values are supported by the source
Crazyrouter test routeExercise these request shapes on selected modelsActual response behavior per model and route

A successful schema request is useful evidence of the tested path, but it does not establish that every possible schema is supported or how enforcement is implemented internally.

Four live requests with one synthetic source#

We first checked that gpt-4.1-mini and gemini-2.5-flash appeared in the live model catalog, then called the Chat Completions endpoint (/v1/chat/completions) at https://api.crazyrouter.com once for each model/format pair. Each request used an output budget of 160 tokens and the same prompt:

text
Extract JSON from this synthetic fixture: ticket DEMO-42 has high priority and contains 3 items. Return ticket_id, priority, count only.

The expected object was {"ticket_id":"DEMO-42","priority":"high","count":3}. Passing required a completed response, no refusal, direct json.loads parsing, exactly those fields, an integer count, and the expected values. We did not strip Markdown or repair output before scoring it.

ModelRequested formatHTTPDirect parsing and fixture checkObserved wall time
GPT-4.1-minijson_object200Passed1.138 s
GPT-4.1-minijson_schema, strict200Passed1.397 s
Gemini 2.5 Flashjson_object200Failed: Markdown fences3.619 s
Gemini 2.5 Flashjson_schema, strict200Passed3.316 s

These times include the observed request path and are single samples, not a speed benchmark. The Gemini JSON-mode response began with a fenced code block even though the enclosed fields were correct. Its finish_reason was stop; that still did not make the full response valid JSON. We did not trace which component in the routed stack caused this behavior.

Three of four live requests passed direct JSON parsing and exact fixture validation

Runnable Python example with explicit validation#

Save this program as structured_ticket.py, set CRAZYROUTER_API_KEY in the environment, and run python structured_ticket.py. It uses the Python standard library. The default model is GPT-4.1-mini; MODEL selects another model for a deliberate retest. A different model is not automatically covered by the results above.

python
"""Validate completion, JSON shape, and fixture facts independently."""
import json
import os
import urllib.request

SCHEMA = {'type': 'object', 'properties': {
    'ticket_id': {'type': 'string'},
    'priority': {'type': 'string', 'enum': ['low', 'high']},
    'count': {'type': 'integer'}},
    'required': ['ticket_id', 'priority', 'count'], 'additionalProperties': False}

def validate_ticket(value):
    if not isinstance(value, dict) or set(value) != {'ticket_id','priority','count'}:
        raise ValueError('missing or extra keys')
    if not isinstance(value['ticket_id'], str) or value['priority'] not in ('low','high'):
        raise ValueError('invalid ticket ID type or priority')
    if type(value['count']) is not int:  # bool is an int subclass in Python.
        raise ValueError('count must be an integer')
    return value

def parse_completion(response):
    choice = response['choices'][0]
    message = choice['message']
    if choice.get('finish_reason') != 'stop' or message.get('refusal'):
        raise ValueError('refused, truncated, or incomplete response')
    return validate_ticket(json.loads(message.get('content') or ''))

def main():
    model = os.getenv('MODEL', 'gpt-4.1-mini')
    base = os.getenv('CRAZYROUTER_BASE_URL', 'https://api.crazyrouter.com/v1')
    payload = {'model': model, 'max_tokens': 160,
        'response_format': {'type':'json_schema','json_schema':{
            'name':'ticket','strict':True,'schema':SCHEMA}},
        'messages':[{'role':'user','content':
            'Extract JSON from this synthetic fixture: ticket DEMO-42 has high '
            'priority and contains 3 items. Return ticket_id, priority, count only.'}]}
    request = urllib.request.Request(base.rstrip('/')+'/chat/completions',
        data=json.dumps(payload).encode(), headers={
            'Authorization':'Bearer '+os.environ['CRAZYROUTER_API_KEY'],
            'Content-Type':'application/json'})
    with urllib.request.urlopen(request,timeout=45) as response:
        envelope = json.load(response)
    data = parse_completion(envelope)
    if data != {'ticket_id':'DEMO-42','priority':'high','count':3}:
        raise ValueError('valid shape, but fixture facts do not match')
    print(json.dumps({'response_id':envelope['id'],'data':data}))

if __name__ == '__main__':
    main()

The exact standalone program also completed successfully in a separate run, with response ID chatcmpl-Eg5MOUBhDy4zAlsqhMQGg3XI6E2TX. That run is additional validation of the example, not another row in the four-request matrix. Its output matched all three fixture facts.

The helper validates this ticket contract only; it is not a general JSON Schema engine. type(value['count']) is int intentionally rejects booleans, since Python treats bool as a subclass of int. For a larger contract, use a maintained schema validator and test its coercion and strictness settings. Keep required fields and unknown-field handling explicit.

Negative cases the application rejected#

We executed eight local validation cases against the same parser: one valid object and seven invalid or incomplete envelopes. These are controlled application tests, separate from the four model requests.

CaseApplication result
Completed response with correct fields and typesAccepted
Correct object inside Markdown fencesRejected
Missing required fieldsRejected
String instead of integer countRejected
Boolean instead of integer countRejected
Additional unexpected fieldRejected
finish_reason: lengthRejected
Explicit refusalRejected

Eight local validation cases separated acceptable output from malformed, wrong-type, truncated and refused responses

The code additionally compares the parsed object with the known fixture facts. In a real extraction task, choose domain checks appropriate to the source: reference matching, allowed identifiers, ranges, totals or a review path. A plausible number can pass structural validation and still be false.

OpenAI, Claude, and Gemini use different native contracts#

The current official documents describe structured-output options for all three providers. Check the native request format before moving a schema between endpoints.

Native interfaceDocumented output configurationCoverage in this article
OpenAI Chat Completionsresponse_format with supported JSON SchemaTested compatible route with GPT-4.1-mini
OpenAI ResponsesIts own text-format configurationDocumentation only; not tested here
Claude Messagesoutput_config.format for JSON outputsDocumentation only; no native Claude schema test here
Gemini native APIStructured output with a supported JSON SchemaDocumentation only; the live test used a compatible Chat Completions route

Use the Claude structured-output guide and Gemini structured-output guide for model availability and schema limitations. Those documents were retrieved on November 3, 2026. Native parameters should not be copied into an unrelated endpoint unchanged. The API protocol comparison provides context for endpoint selection.

Tool arguments and final structured answers are also different outputs. If the model requests a tool, validate the arguments and authorization before execution, then preserve the correct tool-result correlation. See tool-use ID validation for a separately tested native tool round trip.

Failure handling without silent repair#

Classify the failure before deciding to retry. Unsupported schemas and invalid requests require a configuration change. Authentication failures require credential repair. Truncated output may require a smaller task or a suitable output budget. A refusal is a distinct outcome; do not convert it to an empty successful object.

If your product deliberately accepts Markdown-wrapped JSON, implement that as a separate, logged normalization policy, then validate the normalized object. Do not label normalization as successful strict JSON output in a benchmark. The example here rejects fences so that unexpected behavior remains visible.

Record the endpoint, requested model, response ID, finish reason, parsing outcome and validation outcome. Avoid logging private source text by default. The request logging guide distinguishes HTTP success from completed tasks, while the retry guide covers deadlines and safe replay. If you cache results, cache only after your validator accepts them; the exact-response cache example demonstrates that boundary.

Frequently asked questions#

Is JSON mode the same as JSON Schema enforcement?#

No. JSON mode and schema adherence are distinct contracts. Check the chosen endpoint and model, and retain local validation for the application's requirements.

Does HTTP 200 mean I can call json.loads safely?#

No. One live response in this fixture returned Markdown-wrapped JSON with HTTP 200. Direct parsing rejected it.

Do these results rank GPT and Gemini reliability?#

No. Each row is one request on one routed path. A reliability comparison requires a representative dataset, repeated runs and a controlled configuration.

Does Claude support native structured outputs?#

Its current documentation describes JSON outputs using output_config.format. Check current supported models and constraints. This article does not claim to have tested that native interface.

Can I discard unexpected fields and call the result valid?#

That is a transformation policy, not strict acceptance of the returned contract. Decide it explicitly, retain relevant diagnostics, and validate after transformation if your application permits it.

Why validate facts after validating types?#

A ticket ID can be a string and still reference the wrong ticket. A count can be an integer and still contradict the source. Structural validity is necessary for this workflow but does not prove correctness.

Implementation Guides

Podcast

AI Dev Tools: The Crazyrouter Podcast

Short episodes for developers on AI APIs, gateways, cost control and coding tools.

Latest episodes

  • EP196 · Oct 29, 2026 · 6:53

    AI API Security Posture — Turn Controls Into Continuous Evidence

    A practical guide to AI API security posture: inventory exposure, verify controls continuously, protect keys and data, constrain tools, and turn security assumptions into evidence that operators can act on.

  • EP195 · Oct 28, 2026 · 7:21

    AI API Policy Drift — Detect When Production Rules Stop Matching Intent

    A practical guide to policy drift in AI APIs: version routing and safety rules, detect stale enforcement, compare intended and observed behavior, and restore control before small gaps become incidents.

Episodes with written show notes →

Topics

ComparisonsTutorial

Related Articles

Gemini Advanced Review 2026: Is It Worth It for Coding, Research, and API Teams?Tutorial

Gemini Advanced Review 2026: Is It Worth It for Coding, Research, and API Teams?

If you searched for **gemini advanced review**, you probably do not need another shallow feature list. You need to know what Gemini Advanced is, how it compares with alternatives, how to use it in a d...

May 26
KYC AI Workflow: Tested Extraction, Clarification, and Human ReviewTutorial

KYC AI Workflow: Tested Extraction, Clarification, and Human Review

A tested KYC AI workflow using synthetic records: text and image extraction, source checks, an address mismatch, a clarification draft, and human-review handoff. Includes the first-run unsupported inference, seven revised API calls, eight local checks, and complete Python code.

Nov 3
Streaming API Implementation Guide: SSE for AI Applications in 2026Tutorial

Streaming API Implementation Guide: SSE for AI Applications in 2026

Implement production AI streaming with Server-Sent Events, cancellation, backpressure, reconnects, and usage tracking.

Aug 23
AI API Retries in Python: Retry-After, Timeouts, and Safe ReplayTutorial

AI API Retries in Python: Retry-After, Timeouts, and Safe Replay

Build bounded AI API retries in Python with Retry-After, jitter, attempt limits, and an overall deadline. Includes tested failure cases and safe replay rules.

Sep 19
One API Key for Multiple AI Models: Catalog Access and Live Route TestsTutorial

One API Key for Multiple AI Models: Catalog Access and Live Route Tests

Check model visibility and actual generation with one gateway credential. A three-model probe produced two completed outputs and one timeout; runnable Python included.

Oct 13
AI API Response Caching in Python: TTL, Isolation, and a Live TestTips

AI API Response Caching in Python: TTL, Isolation, and a Live Test

Run an exact response cache with trusted identity scope, TTL and output validation. One live API call served two reads; nine local checks exercised isolation and invalidation.

Sep 19