Open-Weight vs Commercial Model APIs: Test the Output Contract Before Switching

A deployment choice has separate dimensions: the model’s license, who operates inference and whether the output satisfies your application. Test the last one with the same fixtures before using labels such as open or commercial as a quality shortcut.

Crazyrouter was the gateway for this 4-case experiment: 4 cases met the declared complete-output criterion; the results below retain the failures and their limits.

By Brian J · Tested November 3, 2026

How should I compare open-weight and commercial model API outputs?#

The two routes are gpt-5-mini and deepseek-v4-flash, both accessed through Crazyrouter. One fixture extracts an ID and integer quantity; the other requires null for an omitted quantity. Each complete JSON object must match the independently declared expected result. All four responses passed in this run.

A missing value is useful here because an invented zero would look syntactically valid while changing meaning. Markdown fences are not stripped, so the test measures compatibility with a strict JSON consumer rather than a human reader’s ability to recover the answer.

November 3 experiment and observed results#

The scripts ran on Windows with Python 3.10.2 and synthetic inputs. Gateway generation and catalog requests use https://api.crazyrouter.com; paths and complete payloads appear in the code. Each result describes this run only. A test program exiting successfully means its observations were recorded, not that every model response passed.

Observed successful-envelope response IDs (failed acceptance can still have an ID):

resp_0a177c3645f940e6016ae9eb506c108196b8362b80b9eb92c1
resp_0277036a775d5218016ae9eb521b1c819587d7b60deb634861
27395da6-d342-4681-8627-43b0e07681f7
2a6bd7f4-6bc9-47d4-984f-7c6fb6350305

Do not confuse hosted access with self-hosting#

Both calls used a hosted gateway. Neither tested a local model server, GPU memory, installation, license compliance or provider data retention. A model family’s publication policy does not establish the license of an arbitrary artifact or prove which weights a gateway executed.

DeepSeek’s native documentation also describes versioned routes and legacy aliases. The requested identifier here is recorded exactly; no equivalence to a local checkpoint is claimed. Review the actual artifact and intended use separately when moving to self-hosted inference.

Expand the decision beyond two easy cases#

Use your own labelled cases, difficult missing fields, long inputs and downstream error costs. Evaluate each required feature separately: ordinary JSON replies do not verify streaming, tool calls or vision. Keep latency and cost measurements tied to an accepted result, including failed attempts.

This page focuses on output contracts. The related deployment-evidence guide covers operational prerequisites, and the break-even guide covers illustrative cost arithmetic. Those separate questions should not be collapsed into another generic ranking.

Reproduce the experiment#

Install requests, set CRAZYROUTER_API_KEY in the process environment, save this complete script and run it with Python 3.10 or later. Requests use your authorized credential and can consume usage. Authorization headers are not printed.

import json
import os
import time
import requests

BASE = "https://api.crazyrouter.com/v1"

def catalog(models):
    with requests.Session() as s:
        s.trust_env=False
        r=s.get(BASE+"/models",headers={"Authorization":"Bearer "+os.environ["CRAZYROUTER_API_KEY"]},timeout=(10,40),allow_redirects=False)
        r.raise_for_status();names={x["id"] for x in r.json()["data"]}
    return {m:m in names for m in models}

def chat(model,prompt,**options):
    payload={"model":model,"messages":[{"role":"user","content":prompt}],**options}
    record={"model":model,"request":payload};start=time.perf_counter()
    try:
        with requests.Session() as s:
            s.trust_env=False
            r=s.post(BASE+"/chat/completions",headers={"Authorization":"Bearer "+os.environ["CRAZYROUTER_API_KEY"]},json=payload,timeout=(10,65),allow_redirects=False)
        record["http"]=r.status_code
        if r.status_code==200:
            d=r.json();c=d["choices"][0];record.update(response_id=d.get("id"),output=c["message"].get("content"),finish=c.get("finish_reason"),usage=d.get("usage",{}))
        else:record["error"]="HTTP_"+str(r.status_code)
    except (requests.RequestException,ValueError,KeyError,IndexError) as e:record["error"]=type(e).__name__
    record["seconds"]=round(time.perf_counter()-start,3)
    return record

def parsed(row):
    if row.get("finish")!="stop":return None
    try:return json.loads(row.get("output") or "null")
    except (ValueError,TypeError):return None

def emit(**data):print(json.dumps(data,ensure_ascii=False))

MODELS = ['gpt-5-mini', 'deepseek-v4-flash']
CASES = [('extract', 'Return only JSON with id R-407 and quantity 3. Use keys id, quantity.', {'id': 'R-407', 'quantity': 3}), ('missing data', 'Return only JSON with keys id and quantity. Source: id R-408, quantity not supplied. Use null for missing quantity.', {'id': 'R-408', 'quantity': None})]
OPTIONS = {'gpt-5-mini': {'max_completion_tokens': 768, 'reasoning_effort': 'minimal'}}
availability=catalog(MODELS);rows=[]
for model in MODELS:
    if not availability[model]:continue
    for name,prompt,expected in CASES:
        r=chat(model,prompt,**OPTIONS.get(model,{"max_tokens":1024}))
        r.update(case=name,expected=expected)
        r["passed"]=parsed(r)==expected
        rows.append(r)
emit(catalog=availability,requests=rows,criterion="Complete stop plus exact parsed JSON; no fence stripping",local_tests=[])

Sources and related guides#

Sources checked November 3, 2026: DeepSeek model and pricing reference. Official descriptions and gateway observations are separate evidence.

• Self-Hosted Model Readiness: Test an Evidence Gate Before Deployment
• Self-Hosted vs Hosted AI Costs: Test the Break-Even Formula
• AI Structured Output in Python: JSON Mode, Schema, and Live Validation
• AI API Reseller vs Official API: What a Synthetic Probe Can Verify
• AI SaaS Unit Economics: Calculate Cost per Usable Output

FAQ#

Did both routes pass?#

Yes, two extraction fixtures per route passed the exact JSON criterion.

Which model is better overall?#

This four-call sample cannot answer that.

Was DeepSeek self-hosted?#

No. Both routes were called through the gateway.

Does the API name prove a checkpoint identity?#

No. It identifies the requested route.

Why include a missing value?#

It checks that the model preserves uncertainty rather than inventing a quantity.

Does this settle licensing?#

No. Review the actual artifact and terms for your intended deployment.

Last actually tested: November 3, 2026. This API-dependent article is due for review December 3, 2026.