Haiku 5.5 vs Sonnet 4.6: Recheck JSON, Totals and Supporting IDs
Four requests test tied maxima and filtered totals. Markdown wrappers and one unnecessary clarification prevent every reply from passing the declared strict JSON contract.

Evaluate the answer and its supporting record IDs together, then separately check whether the consumer can parse the response. This November recheck uses two small tasks to expose the difference between plausible content and an accepted JSON result.
Crazyrouter was the gateway for this 4-case experiment: 0 cases met the declared complete-output criterion; the results below retain the failures and their limits.
By Boluo Z · Tested November 3, 2026
How do Haiku 5.5 and Sonnet 4.6 handle these exact JSON tasks?#
For the maximum task, both B and C attain eleven, so returning only one ID is incomplete. For the filtered-total task, only two accepted rows count and their values sum to eight. The expected objects include every required field and no extras.
Haiku 5.5 returned the correct objects inside Markdown fences for both tasks. Sonnet 4.6 asked for more context on the maximum task despite the prompt containing the data, and returned fenced JSON for the total. All four failed the strict raw-JSON acceptance criterion.
November 3 experiment and observed results#
The scripts ran on Windows with Python 3.10.2 and synthetic inputs. Gateway generation and catalog requests use https://api.crazyrouter.com; paths and complete payloads appear in the code. Each result describes this run only. A test program exiting successfully means its observations were recorded, not that every model response passed.
| Requested model ID | Visible to this credential |
|---|---|
| claude-haiku-5-5 | Yes |
| claude-sonnet-4-6 | Yes |
| Route / case | HTTP or turns | Finish / outcome | Accepted | Observed output or error |
|---|---|---|---|---|
| claude-haiku-5-5 / tied maxima | 200 | stop | No | ```json ⏎ {"max": 11, "rows": ["B", "C"]} ⏎ ``` |
| claude-haiku-5-5 / filtered total | 200 | stop | No | ```json ⏎ {"count": 2, "total": 8} ⏎ ``` |
| claude-sonnet-4-6 / tied maxima | 200 | stop | No | I need more context about the data you're referring to. Could you please provide the input data/table that contains the IDs and values for A=6, B=11, C=11, D=4? |
| claude-sonnet-4-6 / filtered total | 200 | stop | No | ```json ⏎ { ⏎ "count": 2, ⏎ "total": 8 ⏎ } ⏎ ``` |
Observed successful-envelope response IDs (failed acceptance can still have an ID):
msg_011Cgg8SkNTez4DCx8jEJHoJ
msg_011Cgg8SwD5G1idB944E4d7t
msg_011Cgg8T5gp1auMnKEtF5H9Z
msg_011Cgg8TFtEswkWfwm8BQTti

Keep content errors separate from formatting errors#
The Haiku responses and the Sonnet total contain recoverable expected values, but direct json.loads rejects their wrappers. The Sonnet clarification supplies no result to recover. These are different failure modes and should lead to different application diagnostics.
This test does not enable JSON mode or a schema-constrained endpoint. It measures an ordinary text request with an explicit formatting instruction. A different output mode may change the result and needs its own complete test.
Treat the older benchmark as a different experiment#
The earlier article described a broader task collection. This revision presents the new dated probe without extrapolating its four calls to overall intelligence, all languages or every model setting. It removes current-tense performance claims that this run cannot establish.
For a rollout, include more tied cases, missing fields and large inputs, predeclare acceptance and retain failing raw responses. A single aggregate accuracy score hides whether the next fix belongs in prompting, parsing or task reasoning.
Reproduce the experiment#
Install requests, set CRAZYROUTER_API_KEY in the process environment, save this complete script and run it with Python 3.10 or later. Requests use your authorized credential and can consume usage. Authorization headers are not printed.
import json
import os
import time
import requests
BASE = "https://api.crazyrouter.com/v1"
def catalog(models):
with requests.Session() as s:
s.trust_env=False
r=s.get(BASE+"/models",headers={"Authorization":"Bearer "+os.environ["CRAZYROUTER_API_KEY"]},timeout=(10,40),allow_redirects=False)
r.raise_for_status();names={x["id"] for x in r.json()["data"]}
return {m:m in names for m in models}
def chat(model,prompt,**options):
payload={"model":model,"messages":[{"role":"user","content":prompt}],**options}
record={"model":model,"request":payload};start=time.perf_counter()
try:
with requests.Session() as s:
s.trust_env=False
r=s.post(BASE+"/chat/completions",headers={"Authorization":"Bearer "+os.environ["CRAZYROUTER_API_KEY"]},json=payload,timeout=(10,65),allow_redirects=False)
record["http"]=r.status_code
if r.status_code==200:
d=r.json();c=d["choices"][0];record.update(response_id=d.get("id"),output=c["message"].get("content"),finish=c.get("finish_reason"),usage=d.get("usage",{}))
else:record["error"]="HTTP_"+str(r.status_code)
except (requests.RequestException,ValueError,KeyError,IndexError) as e:record["error"]=type(e).__name__
record["seconds"]=round(time.perf_counter()-start,3)
return record
def parsed(row):
if row.get("finish")!="stop":return None
try:return json.loads(row.get("output") or "null")
except (ValueError,TypeError):return None
def emit(**data):print(json.dumps(data,ensure_ascii=False))
MODELS = ['claude-haiku-5-5', 'claude-sonnet-4-6']
CASES = [('tied maxima', 'Return only JSON with max and rows for A=6, B=11, C=11, D=4. rows must include every ID with maximum, in input order.', {'max': 11, 'rows': ['B', 'C']}), ('filtered total', 'Return only JSON with count and total for accepted records: R1 accepted 3, R2 rejected 8, R3 accepted 5.', {'count': 2, 'total': 8})]
OPTIONS = {}
availability=catalog(MODELS);rows=[]
for model in MODELS:
if not availability[model]:continue
for name,prompt,expected in CASES:
r=chat(model,prompt,**OPTIONS.get(model,{"max_tokens":1024}))
r.update(case=name,expected=expected)
r["passed"]=parsed(r)==expected
rows.append(r)
emit(catalog=availability,requests=rows,criterion="Complete stop plus exact parsed JSON; no fence stripping",local_tests=[])

Sources and related guides#
Sources checked November 3, 2026: Anthropic model overview. Official descriptions and gateway observations are separate evidence.
- Claude Jupiter, Opus 4.7 and Sonnet 4.6: Catalog and JSON Recheck
- Haiku 5.5 vs DeepSeek V4.1 Flash: Test Scheduling Gaps and No-Slot Cases
- Haiku 5.5 vs Haiku 4.5: Test Strict JSON, False, Zero and Null
- AI Structured Output in Python: JSON Mode, Schema, and Live Validation
- AI Guardrails in Python: Reject Invalid Outputs After Prompt Injection
FAQ#
Did either model pass the strict fixture?#
No. All four requests failed the complete raw-JSON criterion.
Did Haiku return the right values?#
Yes, inside Markdown fences, which the declared parser does not accept.
What was different about Sonnet’s maximum answer?#
It asked for context instead of returning the requested result.
Was JSON mode enabled?#
No. These were ordinary text requests with a formatting instruction.
Does this replace an overall benchmark?#
No. It is a dated contract recheck with two small tasks.
Should I strip fences automatically?#
Only if you deliberately define and test that as your application’s accepted output format.
Last actually tested: November 3, 2026. This API-dependent article is due for review December 3, 2026.
Implementation Guides
Podcast
AI Dev Tools: The Crazyrouter Podcast
Short episodes for developers on AI APIs, gateways, cost control and coding tools.
Latest episodes
EP196 · Oct 29, 2026 · 6:53
AI API Security Posture — Turn Controls Into Continuous Evidence
A practical guide to AI API security posture: inventory exposure, verify controls continuously, protect keys and data, constrain tools, and turn security assumptions into evidence that operators can act on.
EP195 · Oct 28, 2026 · 7:21
AI API Policy Drift — Detect When Production Rules Stop Matching Intent
A practical guide to policy drift in AI APIs: version routing and safety rules, detect stale enforcement, compare intended and observed behavior, and restore control before small gaps become incidents.





