Three Agent Routes, One Tool Contract: Completed Turns Can Still Fail JSON
Opus 4.8 and Sonnet 4.6 complete synthetic stock lookups but wrap the final JSON; GPT-5.5 returns HTTP 404. Six executor checks pass, while no route passes the complete strict contract.

An agent workflow needs more than a successful tool call: the final response must also satisfy the consumer. This three-route probe separates model turns, executor validation and final JSON acceptance so partial success remains visible.
Crazyrouter was the gateway for this 3-case experiment: 0 cases met the declared complete-output criterion; the results below retain the failures and their limits.
By Sophie M · Tested November 3, 2026
Does the same agent workflow pass on all three model routes?#
Each route is asked to look up DEMO-7 stock and return exactly the object containing sku and available. The local tool returns seven and has no external side effect. The same six executor fixtures reject unknown functions, malformed arguments, extra fields, wrong SKU and duplicate call IDs.
Opus 4.8 and Sonnet 4.6 each completed a tool-call turn followed by a stop response. Both final answers were wrapped in Markdown, with additional prose in the Sonnet response, so neither passed the strict parser. The table preserves this distinction instead of collapsing it into a binary “tool support” claim.
November 3 experiment and observed results#
The scripts ran on Windows with Python 3.10.2 and synthetic inputs. Gateway generation and catalog requests use https://api.crazyrouter.com; paths and complete payloads appear in the code. Each result describes this run only. A test program exiting successfully means its observations were recorded, not that every model response passed.
| Requested model ID | Visible to this credential |
|---|---|
| claude-opus-4-8 | Yes |
| claude-sonnet-4-6 | Yes |
| gpt-5.5 | Yes |
| Route / case | HTTP or turns | Finish / outcome | Accepted | Observed output or error |
|---|---|---|---|---|
| claude-opus-4-8 / stock tool loop | 200, 200 | final | No | ```json ⏎ {"sku":"DEMO-7","available":7} ⏎ ``` |
| claude-sonnet-4-6 / stock tool loop | 200, 200 | final | No | Here is the result sourced directly from the stock lookup tool: ⏎ ⏎ ```json ⏎ {"sku": "DEMO-7", "available": 7} ⏎ ``` |
| gpt-5.5 / stock tool loop | 404 | http_failure | No | No usable content |
Observed successful-envelope response IDs (failed acceptance can still have an ID):
msg_011Cgg8ZMv3CWaPWdsZd2Nzr
msg_011Cgg8ZYcjMyG7oxKvi2DuW
msg_011Cgg8Zgu3rd8GbPXFnKVa6
msg_011Cgg8ZsCgJ4DoFLe49pT2b
| Executed local case | Expected behavior met |
|---|---|
| valid | Yes |
| duplicate | Yes |
| unknown | Yes |
| bad JSON | Yes |
| extra field | Yes |
| wrong SKU | Yes |

A catalog entry is only the first check#
GPT-5.5 was visible but returned HTTP 404 before a tool loop completed. After consulting its current documentation, the final test used reasoning_effort none rather than minimal; the HTTP failure persisted. No fallback model was substituted.
The final script caps each route at three turns and has no hidden retry. The result is a dated integration check, not an availability percentage or an assertion that the route never works.
Keep workflow scores tied to an end state#
A tool result, a finished text turn and an accepted application object are three separate milestones. Only the last one satisfies this fixture. If your application accepts fenced JSON, define and test that parser before making a different comparison.
The old article used longer workflows and timing rankings. Those measurements are not mixed with this small November contract test. A production comparison needs repeated runs, representative tools, safe retry behavior and a cost denominator based on accepted outcomes.
Reproduce the experiment#
Install requests, set CRAZYROUTER_API_KEY in the process environment, save this complete script and run it with Python 3.10 or later. Requests use your authorized credential and can consume usage. Authorization headers are not printed.
import json
import os
import time
import requests
BASE = "https://api.crazyrouter.com/v1"
def catalog(models):
with requests.Session() as s:
s.trust_env=False
r=s.get(BASE+"/models",headers={"Authorization":"Bearer "+os.environ["CRAZYROUTER_API_KEY"]},timeout=(10,40),allow_redirects=False)
r.raise_for_status();names={x["id"] for x in r.json()["data"]}
return {m:m in names for m in models}
def chat(model,prompt,**options):
payload={"model":model,"messages":[{"role":"user","content":prompt}],**options}
record={"model":model,"request":payload};start=time.perf_counter()
try:
with requests.Session() as s:
s.trust_env=False
r=s.post(BASE+"/chat/completions",headers={"Authorization":"Bearer "+os.environ["CRAZYROUTER_API_KEY"]},json=payload,timeout=(10,65),allow_redirects=False)
record["http"]=r.status_code
if r.status_code==200:
d=r.json();c=d["choices"][0];record.update(response_id=d.get("id"),output=c["message"].get("content"),finish=c.get("finish_reason"),usage=d.get("usage",{}))
else:record["error"]="HTTP_"+str(r.status_code)
except (requests.RequestException,ValueError,KeyError,IndexError) as e:record["error"]=type(e).__name__
record["seconds"]=round(time.perf_counter()-start,3)
return record
def parsed(row):
if row.get("finish")!="stop":return None
try:return json.loads(row.get("output") or "null")
except (ValueError,TypeError):return None
def emit(**data):print(json.dumps(data,ensure_ascii=False))
def execute(call,seen):
if call.get("id") in seen:return {"error":"duplicate"}
if call.get("function",{}).get("name")!="stock_lookup":return {"error":"unknown_tool"}
try:args=json.loads(call["function"]["arguments"])
except (ValueError,KeyError,TypeError):return {"error":"invalid_json"}
if type(args) is not dict or set(args)!={"sku"} or args["sku"]!="DEMO-7":return {"error":"invalid_arguments"}
seen.add(call["id"])
return {"sku":"DEMO-7","available":7}
def loop(model):
messages=[{"role":"user","content":'Use stock_lookup for SKU DEMO-7, then return only JSON {"sku":"DEMO-7","available":7}. The tool is the source of stock data.'}]
tool={"type":"function","function":{"name":"stock_lookup","description":"Read synthetic stock, no external operation","parameters":{"type":"object","properties":{"sku":{"type":"string"}},"required":["sku"],"additionalProperties":False}}}
seen=set();turns=[];outcome="turn_limit";output=None
with requests.Session() as s:
s.trust_env=False
for turn in range(3):
payload={"model":model,"messages":messages,"tools":[tool],"tool_choice":"auto"}
if model.startswith("gpt-5"):payload.update(max_completion_tokens=1024,reasoning_effort="none" if model=="gpt-5.5" else "minimal")
else:payload["max_tokens"]=1024
started=time.perf_counter()
try:
response=s.post(BASE+"/chat/completions",headers={"Authorization":"Bearer "+os.environ["CRAZYROUTER_API_KEY"]},json=payload,timeout=(10,65),allow_redirects=False)
row={"turn":turn+1,"http":response.status_code,"request":json.loads(json.dumps(payload))}
if response.status_code!=200:outcome="http_failure";turns.append(row);break
data=response.json();choice=data["choices"][0];msg=choice["message"]
row.update(response_id=data.get("id"),finish=choice.get("finish_reason"),output=msg.get("content"),usage=data.get("usage",{}),seconds=round(time.perf_counter()-started,3));turns.append(row)
calls=msg.get("tool_calls") or []
if calls:
if choice.get("finish_reason")!="tool_calls" or len(calls)!=1:outcome="invalid_tool_turn";break
result=execute(calls[0],seen);row["tool_result"]=result
if "error" in result:outcome="executor_rejected";break
messages.append({"role":"assistant","content":msg.get("content"),"tool_calls":calls})
messages.append({"role":"tool","tool_call_id":calls[0]["id"],"content":json.dumps(result)})
else:
output=msg.get("content");outcome="final" if choice.get("finish_reason")=="stop" else "incomplete";break
except (requests.RequestException,ValueError,KeyError,IndexError) as e:
outcome=type(e).__name__;turns.append({"turn":turn+1,"error":outcome});break
try:answer=json.loads(output or "null")
except ValueError:answer=None
return {"model":model,"case":"stock tool loop","turns":turns,"outcome":outcome,"output":output,"passed":outcome=="final" and len(seen)==1 and answer=={"sku":"DEMO-7","available":7}}
valid={"id":"call-test","function":{"name":"stock_lookup","arguments":'{"sku":"DEMO-7"}'}}
tests=[]
cases=[("valid",valid,set(),None),("duplicate",valid,{"call-test"},"duplicate"),("unknown",{"id":"x","function":{"name":"delete"}},set(),"unknown_tool"),("bad JSON",{"id":"x","function":{"name":"stock_lookup","arguments":"{"}},set(),"invalid_json"),("extra field",{"id":"x","function":{"name":"stock_lookup","arguments":'{"sku":"DEMO-7","tenant":"other"}'}},set(),"invalid_arguments"),("wrong SKU",{"id":"x","function":{"name":"stock_lookup","arguments":'{"sku":"OTHER"}'}},set(),"invalid_arguments")]
for name,call,seen,error in cases:
result=execute(call,seen);assert result.get("error")==error;tests.append({"case":name,"passed":True})
models=['claude-opus-4-8', 'claude-sonnet-4-6', 'gpt-5.5'];available=catalog(models)
emit(catalog=available,requests=[loop(m) for m in models if available[m]],local_tests=tests,scope="At most three model turns; read-only synthetic dictionary, no external side effect")

Sources and related guides#
Sources checked November 3, 2026: OpenAI function calling, Anthropic model overview, GPT-5.5 model page. Official descriptions and gateway observations are separate evidence.
- GPT Agent Workflows: A Bounded Tool Loop with a Tested Stop Condition
- Function Calling Across AI Models: Two Completed Loops and One Timeout
- Authorize LLM Tool Calls: Tenant Isolation with a Tested Python Executor
- Claude tool_use_id Errors: Fix Missing and Misordered tool_result Blocks
- Kimi Agent Action Verification: Test State Versions Before a Write
FAQ#
Did any route pass the full strict contract?#
No in the final run.
Did the Claude routes execute the lookup?#
Yes. Both returned the synthetic tool result and a completed final turn, but the final JSON was wrapped.
What happened to GPT-5.5?#
The first model request returned HTTP 404.
Are the local executor checks passing?#
Yes, all six passed.
Does this test real side effects?#
No. The lookup is a local synthetic read.
Can I infer a speed winner?#
No. Different completion and failure outcomes do not support a meaningful speed ranking here.
Last actually tested: November 3, 2026. This API-dependent article is due for review December 3, 2026.
Implementation Guides
Podcast
AI Dev Tools: The Crazyrouter Podcast
Short episodes for developers on AI APIs, gateways, cost control and coding tools.
Latest episodes
EP196 · Oct 29, 2026 · 6:53
AI API Security Posture — Turn Controls Into Continuous Evidence
A practical guide to AI API security posture: inventory exposure, verify controls continuously, protect keys and data, constrain tools, and turn security assumptions into evidence that operators can act on.
EP195 · Oct 28, 2026 · 7:21
AI API Policy Drift — Detect When Production Rules Stop Matching Intent
A practical guide to policy drift in AI APIs: version routing and safety rules, detect stale enforcement, compare intended and observed behavior, and restore control before small gaps become incidents.





