Back to Blog
EnglishAI Coding

Three Agent Routes, One Tool Contract: Completed Turns Can Still Fail JSON

Opus 4.8 and Sonnet 4.6 complete synthetic stock lookups but wrap the final JSON; GPT-5.5 returns HTTP 404. Six executor checks pass, while no route passes the complete strict contract.

S
Sophie M
June 3, 2026 / 792 views
Share:
Three Agent Routes, One Tool Contract: Completed Turns Can Still Fail JSON

An agent workflow needs more than a successful tool call: the final response must also satisfy the consumer. This three-route probe separates model turns, executor validation and final JSON acceptance so partial success remains visible.

Crazyrouter was the gateway for this 3-case experiment: 0 cases met the declared complete-output criterion; the results below retain the failures and their limits.

By Sophie M · Tested November 3, 2026

Does the same agent workflow pass on all three model routes?#

Each route is asked to look up DEMO-7 stock and return exactly the object containing sku and available. The local tool returns seven and has no external side effect. The same six executor fixtures reject unknown functions, malformed arguments, extra fields, wrong SKU and duplicate call IDs.

Opus 4.8 and Sonnet 4.6 each completed a tool-call turn followed by a stop response. Both final answers were wrapped in Markdown, with additional prose in the Sonnet response, so neither passed the strict parser. The table preserves this distinction instead of collapsing it into a binary “tool support” claim.

November 3 experiment and observed results#

The scripts ran on Windows with Python 3.10.2 and synthetic inputs. Gateway generation and catalog requests use https://api.crazyrouter.com; paths and complete payloads appear in the code. Each result describes this run only. A test program exiting successfully means its observations were recorded, not that every model response passed.

Requested model IDVisible to this credential
claude-opus-4-8Yes
claude-sonnet-4-6Yes
gpt-5.5Yes
Route / caseHTTP or turnsFinish / outcomeAcceptedObserved output or error
claude-opus-4-8 / stock tool loop200, 200finalNo```json ⏎ {"sku":"DEMO-7","available":7} ⏎ ```
claude-sonnet-4-6 / stock tool loop200, 200finalNoHere is the result sourced directly from the stock lookup tool: ⏎ ⏎ ```json ⏎ {"sku": "DEMO-7", "available": 7} ⏎ ```
gpt-5.5 / stock tool loop404http_failureNoNo usable content

Observed successful-envelope response IDs (failed acceptance can still have an ID):

text
msg_011Cgg8ZMv3CWaPWdsZd2Nzr
msg_011Cgg8ZYcjMyG7oxKvi2DuW
msg_011Cgg8Zgu3rd8GbPXFnKVa6
msg_011Cgg8ZsCgJ4DoFLe49pT2b
Executed local caseExpected behavior met
validYes
duplicateYes
unknownYes
bad JSONYes
extra fieldYes
wrong SKUYes

Full application contract: Tool execution plus exact final JSON is required

A catalog entry is only the first check#

GPT-5.5 was visible but returned HTTP 404 before a tool loop completed. After consulting its current documentation, the final test used reasoning_effort none rather than minimal; the HTTP failure persisted. No fallback model was substituted.

The final script caps each route at three turns and has no hidden retry. The result is a dated integration check, not an availability percentage or an assertion that the route never works.

Keep workflow scores tied to an end state#

A tool result, a finished text turn and an accepted application object are three separate milestones. Only the last one satisfies this fixture. If your application accepts fenced JSON, define and test that parser before making a different comparison.

The old article used longer workflows and timing rankings. Those measurements are not mixed with this small November contract test. A production comparison needs repeated runs, representative tools, safe retry behavior and a cost denominator based on accepted outcomes.

Reproduce the experiment#

Install requests, set CRAZYROUTER_API_KEY in the process environment, save this complete script and run it with Python 3.10 or later. Requests use your authorized credential and can consume usage. Authorization headers are not printed.

python
import json
import os
import time
import requests

BASE = "https://api.crazyrouter.com/v1"

def catalog(models):
    with requests.Session() as s:
        s.trust_env=False
        r=s.get(BASE+"/models",headers={"Authorization":"Bearer "+os.environ["CRAZYROUTER_API_KEY"]},timeout=(10,40),allow_redirects=False)
        r.raise_for_status();names={x["id"] for x in r.json()["data"]}
    return {m:m in names for m in models}

def chat(model,prompt,**options):
    payload={"model":model,"messages":[{"role":"user","content":prompt}],**options}
    record={"model":model,"request":payload};start=time.perf_counter()
    try:
        with requests.Session() as s:
            s.trust_env=False
            r=s.post(BASE+"/chat/completions",headers={"Authorization":"Bearer "+os.environ["CRAZYROUTER_API_KEY"]},json=payload,timeout=(10,65),allow_redirects=False)
        record["http"]=r.status_code
        if r.status_code==200:
            d=r.json();c=d["choices"][0];record.update(response_id=d.get("id"),output=c["message"].get("content"),finish=c.get("finish_reason"),usage=d.get("usage",{}))
        else:record["error"]="HTTP_"+str(r.status_code)
    except (requests.RequestException,ValueError,KeyError,IndexError) as e:record["error"]=type(e).__name__
    record["seconds"]=round(time.perf_counter()-start,3)
    return record

def parsed(row):
    if row.get("finish")!="stop":return None
    try:return json.loads(row.get("output") or "null")
    except (ValueError,TypeError):return None

def emit(**data):print(json.dumps(data,ensure_ascii=False))

def execute(call,seen):
    if call.get("id") in seen:return {"error":"duplicate"}
    if call.get("function",{}).get("name")!="stock_lookup":return {"error":"unknown_tool"}
    try:args=json.loads(call["function"]["arguments"])
    except (ValueError,KeyError,TypeError):return {"error":"invalid_json"}
    if type(args) is not dict or set(args)!={"sku"} or args["sku"]!="DEMO-7":return {"error":"invalid_arguments"}
    seen.add(call["id"])
    return {"sku":"DEMO-7","available":7}

def loop(model):
    messages=[{"role":"user","content":'Use stock_lookup for SKU DEMO-7, then return only JSON {"sku":"DEMO-7","available":7}. The tool is the source of stock data.'}]
    tool={"type":"function","function":{"name":"stock_lookup","description":"Read synthetic stock, no external operation","parameters":{"type":"object","properties":{"sku":{"type":"string"}},"required":["sku"],"additionalProperties":False}}}
    seen=set();turns=[];outcome="turn_limit";output=None
    with requests.Session() as s:
        s.trust_env=False
        for turn in range(3):
            payload={"model":model,"messages":messages,"tools":[tool],"tool_choice":"auto"}
            if model.startswith("gpt-5"):payload.update(max_completion_tokens=1024,reasoning_effort="none" if model=="gpt-5.5" else "minimal")
            else:payload["max_tokens"]=1024
            started=time.perf_counter()
            try:
                response=s.post(BASE+"/chat/completions",headers={"Authorization":"Bearer "+os.environ["CRAZYROUTER_API_KEY"]},json=payload,timeout=(10,65),allow_redirects=False)
                row={"turn":turn+1,"http":response.status_code,"request":json.loads(json.dumps(payload))}
                if response.status_code!=200:outcome="http_failure";turns.append(row);break
                data=response.json();choice=data["choices"][0];msg=choice["message"]
                row.update(response_id=data.get("id"),finish=choice.get("finish_reason"),output=msg.get("content"),usage=data.get("usage",{}),seconds=round(time.perf_counter()-started,3));turns.append(row)
                calls=msg.get("tool_calls") or []
                if calls:
                    if choice.get("finish_reason")!="tool_calls" or len(calls)!=1:outcome="invalid_tool_turn";break
                    result=execute(calls[0],seen);row["tool_result"]=result
                    if "error" in result:outcome="executor_rejected";break
                    messages.append({"role":"assistant","content":msg.get("content"),"tool_calls":calls})
                    messages.append({"role":"tool","tool_call_id":calls[0]["id"],"content":json.dumps(result)})
                else:
                    output=msg.get("content");outcome="final" if choice.get("finish_reason")=="stop" else "incomplete";break
            except (requests.RequestException,ValueError,KeyError,IndexError) as e:
                outcome=type(e).__name__;turns.append({"turn":turn+1,"error":outcome});break
    try:answer=json.loads(output or "null")
    except ValueError:answer=None
    return {"model":model,"case":"stock tool loop","turns":turns,"outcome":outcome,"output":output,"passed":outcome=="final" and len(seen)==1 and answer=={"sku":"DEMO-7","available":7}}

valid={"id":"call-test","function":{"name":"stock_lookup","arguments":'{"sku":"DEMO-7"}'}}
tests=[]
cases=[("valid",valid,set(),None),("duplicate",valid,{"call-test"},"duplicate"),("unknown",{"id":"x","function":{"name":"delete"}},set(),"unknown_tool"),("bad JSON",{"id":"x","function":{"name":"stock_lookup","arguments":"{"}},set(),"invalid_json"),("extra field",{"id":"x","function":{"name":"stock_lookup","arguments":'{"sku":"DEMO-7","tenant":"other"}'}},set(),"invalid_arguments"),("wrong SKU",{"id":"x","function":{"name":"stock_lookup","arguments":'{"sku":"OTHER"}'}},set(),"invalid_arguments")]
for name,call,seen,error in cases:
    result=execute(call,seen);assert result.get("error")==error;tests.append({"case":name,"passed":True})
models=['claude-opus-4-8', 'claude-sonnet-4-6', 'gpt-5.5'];available=catalog(models)
emit(catalog=available,requests=[loop(m) for m in models if available[m]],local_tests=tests,scope="At most three model turns; read-only synthetic dictionary, no external side effect")

Model turns observed: HTTP failures are retained; at most three turns per route

Sources checked November 3, 2026: OpenAI function calling, Anthropic model overview, GPT-5.5 model page. Official descriptions and gateway observations are separate evidence.

FAQ#

Did any route pass the full strict contract?#

No in the final run.

Did the Claude routes execute the lookup?#

Yes. Both returned the synthetic tool result and a completed final turn, but the final JSON was wrapped.

What happened to GPT-5.5?#

The first model request returned HTTP 404.

Are the local executor checks passing?#

Yes, all six passed.

Does this test real side effects?#

No. The lookup is a local synthetic read.

Can I infer a speed winner?#

No. Different completion and failure outcomes do not support a meaningful speed ranking here.

Last actually tested: November 3, 2026. This API-dependent article is due for review December 3, 2026.

Implementation Guides

Podcast

AI Dev Tools: The Crazyrouter Podcast

Short episodes for developers on AI APIs, gateways, cost control and coding tools.

Latest episodes

  • EP196 · Oct 29, 2026 · 6:53

    AI API Security Posture — Turn Controls Into Continuous Evidence

    A practical guide to AI API security posture: inventory exposure, verify controls continuously, protect keys and data, constrain tools, and turn security assumptions into evidence that operators can act on.

  • EP195 · Oct 28, 2026 · 7:21

    AI API Policy Drift — Detect When Production Rules Stop Matching Intent

    A practical guide to policy drift in AI APIs: version routing and safety rules, detect stale enforcement, compare intended and observed behavior, and restore control before small gaps become incidents.

Episodes with written show notes →

Related Articles

Claude Code Skills vs Subagents vs Dynamic Workflows: Which One Should You Use?AI Coding

Claude Code Skills vs Subagents vs Dynamic Workflows: Which One Should You Use?

Claude Code, Codex, Cursor, and modern AI coding tools now expose multiple workflow primitives: skills, subagents, background agents, and dynamic workflows. This guide explains when to use each, with a small selector tool and Crazyrouter model-routing examples.

Jun 3
Claude Code Dynamic Workflows, Rebuilt: A Practical Ultracode-Style Orchestration TemplateAI Coding

Claude Code Dynamic Workflows, Rebuilt: A Practical Ultracode-Style Orchestration Template

Dynamic workflows in Claude Code are trending because they turn one prompt into orchestration, subagents, and verification gates. We rebuilt the useful pattern as a reproducible local workflow with model routing through Crazyrouter.

Jun 3
Background Coding Agents Without Lock-In: Recreating the Cursor Pattern with Git WorktreesAI Coding

Background Coding Agents Without Lock-In: Recreating the Cursor Pattern with Git Worktrees

Cursor background agents are useful because they move long-running AI coding work out of your local single-branch loop. This guide rebuilds the same workflow pattern with plain git worktrees, task packets, trace logs, and Crazyrouter model routing.

Jun 4
"Claude Plans, Codex Reviews: Rebuilding the Viral Two-Agent Coding Workflow with Crazyrouter"AI Coding

"Claude Plans, Codex Reviews: Rebuilding the Viral Two-Agent Coding Workflow with Crazyrouter"

"Twitter is full of Codex-in-Claude-Code workflows. We rebuilt the useful part: one agent plans or implements, a second agent reviews adversarially, and the whole process becomes reproducible packets instead of copy-paste chaos."

Jun 3
Function Calling Across AI Models: Two Completed Loops and One TimeoutGuide

Function Calling Across AI Models: Two Completed Loops and One Timeout

Run the same tool-call loop across GPT, Claude and Gemini routes. Two complete round trips, one timeout and six executor fixtures show the tested compatibility limits.

Sep 19
AI API Reseller vs Official API: What a Synthetic Probe Can VerifyComparison

AI API Reseller vs Official API: What a Synthetic Probe Can Verify

Two synthetic gateway requests returned HTTP 200, but only one matched the expected answer. Separate functional checks from unverified retention, upstream identity and contractual claims.

Oct 15