Back to Blog
EnglishComparison

Haiku 5.5 vs Sonnet 4.6: Recheck JSON, Totals and Supporting IDs

Four requests test tied maxima and filtered totals. Markdown wrappers and one unnecessary clarification prevent every reply from passing the declared strict JSON contract.

B
Boluo Z
October 8, 2026 / 580 views
Share:
Haiku 5.5 vs Sonnet 4.6: Recheck JSON, Totals and Supporting IDs

Evaluate the answer and its supporting record IDs together, then separately check whether the consumer can parse the response. This November recheck uses two small tasks to expose the difference between plausible content and an accepted JSON result.

Crazyrouter was the gateway for this 4-case experiment: 0 cases met the declared complete-output criterion; the results below retain the failures and their limits.

By Boluo Z · Tested November 3, 2026

How do Haiku 5.5 and Sonnet 4.6 handle these exact JSON tasks?#

For the maximum task, both B and C attain eleven, so returning only one ID is incomplete. For the filtered-total task, only two accepted rows count and their values sum to eight. The expected objects include every required field and no extras.

Haiku 5.5 returned the correct objects inside Markdown fences for both tasks. Sonnet 4.6 asked for more context on the maximum task despite the prompt containing the data, and returned fenced JSON for the total. All four failed the strict raw-JSON acceptance criterion.

November 3 experiment and observed results#

The scripts ran on Windows with Python 3.10.2 and synthetic inputs. Gateway generation and catalog requests use https://api.crazyrouter.com; paths and complete payloads appear in the code. Each result describes this run only. A test program exiting successfully means its observations were recorded, not that every model response passed.

Requested model IDVisible to this credential
claude-haiku-5-5Yes
claude-sonnet-4-6Yes
Route / caseHTTP or turnsFinish / outcomeAcceptedObserved output or error
claude-haiku-5-5 / tied maxima200stopNo```json ⏎ {"max": 11, "rows": ["B", "C"]} ⏎ ```
claude-haiku-5-5 / filtered total200stopNo```json ⏎ {"count": 2, "total": 8} ⏎ ```
claude-sonnet-4-6 / tied maxima200stopNoI need more context about the data you're referring to. Could you please provide the input data/table that contains the IDs and values for A=6, B=11, C=11, D=4?
claude-sonnet-4-6 / filtered total200stopNo```json ⏎ { ⏎ "count": 2, ⏎ "total": 8 ⏎ } ⏎ ```

Observed successful-envelope response IDs (failed acceptance can still have an ID):

text
msg_011Cgg8SkNTez4DCx8jEJHoJ
msg_011Cgg8SwD5G1idB944E4d7t
msg_011Cgg8T5gp1auMnKEtF5H9Z
msg_011Cgg8TFtEswkWfwm8BQTti

Exact output contract: Complete response and declared JSON or marker required

Keep content errors separate from formatting errors#

The Haiku responses and the Sonnet total contain recoverable expected values, but direct json.loads rejects their wrappers. The Sonnet clarification supplies no result to recover. These are different failure modes and should lead to different application diagnostics.

This test does not enable JSON mode or a schema-constrained endpoint. It measures an ordinary text request with an explicit formatting instruction. A different output mode may change the result and needs its own complete test.

Treat the older benchmark as a different experiment#

The earlier article described a broader task collection. This revision presents the new dated probe without extrapolating its four calls to overall intelligence, all languages or every model setting. It removes current-tense performance claims that this run cannot establish.

For a rollout, include more tied cases, missing fields and large inputs, predeclare acceptance and retain failing raw responses. A single aggregate accuracy score hides whether the next fix belongs in prompting, parsing or task reasoning.

Reproduce the experiment#

Install requests, set CRAZYROUTER_API_KEY in the process environment, save this complete script and run it with Python 3.10 or later. Requests use your authorized credential and can consume usage. Authorization headers are not printed.

python
import json
import os
import time
import requests

BASE = "https://api.crazyrouter.com/v1"

def catalog(models):
    with requests.Session() as s:
        s.trust_env=False
        r=s.get(BASE+"/models",headers={"Authorization":"Bearer "+os.environ["CRAZYROUTER_API_KEY"]},timeout=(10,40),allow_redirects=False)
        r.raise_for_status();names={x["id"] for x in r.json()["data"]}
    return {m:m in names for m in models}

def chat(model,prompt,**options):
    payload={"model":model,"messages":[{"role":"user","content":prompt}],**options}
    record={"model":model,"request":payload};start=time.perf_counter()
    try:
        with requests.Session() as s:
            s.trust_env=False
            r=s.post(BASE+"/chat/completions",headers={"Authorization":"Bearer "+os.environ["CRAZYROUTER_API_KEY"]},json=payload,timeout=(10,65),allow_redirects=False)
        record["http"]=r.status_code
        if r.status_code==200:
            d=r.json();c=d["choices"][0];record.update(response_id=d.get("id"),output=c["message"].get("content"),finish=c.get("finish_reason"),usage=d.get("usage",{}))
        else:record["error"]="HTTP_"+str(r.status_code)
    except (requests.RequestException,ValueError,KeyError,IndexError) as e:record["error"]=type(e).__name__
    record["seconds"]=round(time.perf_counter()-start,3)
    return record

def parsed(row):
    if row.get("finish")!="stop":return None
    try:return json.loads(row.get("output") or "null")
    except (ValueError,TypeError):return None

def emit(**data):print(json.dumps(data,ensure_ascii=False))

MODELS = ['claude-haiku-5-5', 'claude-sonnet-4-6']
CASES = [('tied maxima', 'Return only JSON with max and rows for A=6, B=11, C=11, D=4. rows must include every ID with maximum, in input order.', {'max': 11, 'rows': ['B', 'C']}), ('filtered total', 'Return only JSON with count and total for accepted records: R1 accepted 3, R2 rejected 8, R3 accepted 5.', {'count': 2, 'total': 8})]
OPTIONS = {}
availability=catalog(MODELS);rows=[]
for model in MODELS:
    if not availability[model]:continue
    for name,prompt,expected in CASES:
        r=chat(model,prompt,**OPTIONS.get(model,{"max_tokens":1024}))
        r.update(case=name,expected=expected)
        r["passed"]=parsed(r)==expected
        rows.append(r)
emit(catalog=availability,requests=rows,criterion="Complete stop plus exact parsed JSON; no fence stripping",local_tests=[])

Catalog visibility and attempted cases: Absent IDs were not silently substituted

Sources checked November 3, 2026: Anthropic model overview. Official descriptions and gateway observations are separate evidence.

FAQ#

Did either model pass the strict fixture?#

No. All four requests failed the complete raw-JSON criterion.

Did Haiku return the right values?#

Yes, inside Markdown fences, which the declared parser does not accept.

What was different about Sonnet’s maximum answer?#

It asked for context instead of returning the requested result.

Was JSON mode enabled?#

No. These were ordinary text requests with a formatting instruction.

Does this replace an overall benchmark?#

No. It is a dated contract recheck with two small tasks.

Should I strip fences automatically?#

Only if you deliberately define and test that as your application’s accepted output format.

Last actually tested: November 3, 2026. This API-dependent article is due for review December 3, 2026.

Implementation Guides

Podcast

AI Dev Tools: The Crazyrouter Podcast

Short episodes for developers on AI APIs, gateways, cost control and coding tools.

Latest episodes

  • EP196 · Oct 29, 2026 · 6:53

    AI API Security Posture — Turn Controls Into Continuous Evidence

    A practical guide to AI API security posture: inventory exposure, verify controls continuously, protect keys and data, constrain tools, and turn security assumptions into evidence that operators can act on.

  • EP195 · Oct 28, 2026 · 7:21

    AI API Policy Drift — Detect When Production Rules Stop Matching Intent

    A practical guide to policy drift in AI APIs: version routing and safety rules, detect stale enforcement, compare intended and observed behavior, and restore control before small gaps become incidents.

Episodes with written show notes →

Topics

ComparisonsComparison

Related Articles

Gemini 3.5 Flash vs Claude Response-Tier Models: Which One Should Developers Use?Comparison

Gemini 3.5 Flash vs Claude Response-Tier Models: Which One Should Developers Use?

A practical comparison of Gemini 3.5 Flash against Claude Haiku, Sonnet, and Opus-style response tiers for latency, cost, coding, reasoning, and production API routing.

May 21
Claude Fable 5 vs GPT-5.5: How a max_tokens Misread Changed the Model ComparisonComparison

Claude Fable 5 vs GPT-5.5: How a max_tokens Misread Changed the Model Comparison

A real Crazyrouter OpenAI-compatible API comparison of claude-fable-5 and gpt-5.5 across math reasoning, physics reasoning, and a long Canvas animation task, with a focus on max_tokens, finish_reason=length, completion_tokens, and browser validation.

Jul 6
GLM-5.3 vs GLM-5.2: A Hard Benchmark with Real API CallsComparison

GLM-5.3 vs GLM-5.2: A Hard Benchmark with Real API Calls

Six hard, verifiable API tasks comparing GLM-5.3 and GLM-5.2. GLM-5.3 led first-delivery coverage 4/6 to 2/6, while GLM-5.2 won the hidden code test.

Aug 14
Haiku 5.5 vs DeepSeek V4.1 Flash: Test Scheduling Gaps and No-Slot CasesComparison

Haiku 5.5 vs DeepSeek V4.1 Flash: Test Scheduling Gaps and No-Slot Cases

Two models solve the same available-slot and impossible-slot fixtures using integer minutes and half-open intervals. All four strict JSON checks pass; this is not a general scheduling benchmark.

Oct 9
Haiku 5.5 vs Haiku 4.5: Test Strict JSON, False, Zero and NullComparison

Haiku 5.5 vs Haiku 4.5: Test Strict JSON, False, Zero and Null

Two structured text fixtures distinguish exact JSON from fenced objects. Haiku 5.5 passes both; Haiku 4.5 returns matching values inside Markdown fences and fails the raw-JSON contract.

Oct 11
AI API Pricing Comparison 2026: OpenAI vs Claude vs Gemini vs RoutersComparison

AI API Pricing Comparison 2026: OpenAI vs Claude vs Gemini vs Routers

A developer-focused June 2026 guide to AI API pricing, alternatives, implementation patterns, pricing tradeoffs, and when to use Crazyrouter for unified AI API access.

Jun 4