Claude Opus 5 vs GPT-5.6-SOL: Both Score 10/10 on Core Answers in API Validation

Claude Opus 5 vs GPT-5.6-SOL: Core Accuracy Ties at 10/10 in a Reproducible Model Test#

Model leaderboards often compress very different capabilities into a single score. That may be convenient, but it is not enough for engineering decisions. A useful comparison needs reproducible prompts, objective graders, executable tests, and separate accounting for reasoning correctness, task completion, and output format.

Network latency is deliberately excluded from this intelligence score. End-to-end API time depends on routing, upstream load, rate limits, caching, and the service instance handling a request. Without a fixed upstream, enough repeated measurements, and confidence intervals, latency is an operational observation—not evidence that one model reasons better than another.

This test sent 10 objectively gradable tasks to Claude Opus 5 and GPT-5.6-SOL through the same OpenAI-compatible Crazyrouter endpoint. Strict JSON was evaluated separately as an instruction-following test.

The results resist a simple leaderboard narrative:

• Core-answer accuracy: Claude Opus 5 scored 10/10; GPT-5.6-SOL scored 10/10.
• Complete delivery of every sub-requirement: Claude Opus 5 scored 9/10; GPT-5.6-SOL scored 10/10.
• Hidden tests across two Python tasks: both models scored 2/2.
• Strict JSON on the first attempt: Claude Opus 5 scored 0/1; GPT-5.6-SOL scored 1/1.
• Additional strict-JSON retries: Opus remained noncompliant in two extra retries by appending text or using code fences.

The defensible conclusion is that core correctness was tied. The models did not, however, behave identically when evaluated for complete task delivery and machine-readable output.

Create an API key and rerun the test with your own workload

Quick answers#

For workloads dominated by objectively checked mathematics, physics, or algorithms, this sample does not show a meaningful accuracy gap. For systems that require directly parseable JSON or complete delivery of every instruction in a single response, GPT-5.6-SOL was more consistent in this run.

Why network latency is not part of the score#

API response time is not a clean measure of model intelligence. It can vary because of:

• the network path from the gateway to each upstream provider;
• channel load and account rate limits at the time of the request;
• whether a cache is hit;
• which service instance receives the request;
• how the upstream provider counts or conceals reasoning tokens.

A request completing several seconds earlier does not establish stronger reasoning. A credible latency comparison would require a controlled upstream, enough repeated requests, and reported confidence intervals.

For that reason, no response-time metric appears in the result table, and this article does not name a speed winner. Latency remains important for operations and user experience, but it should be benchmarked separately from correctness.

For current model availability, see the Crazyrouter model list. Cost planning should likewise be handled separately using Crazyrouter pricing.

Test environment and grading methodology#

Before testing, the model-list endpoint was queried to confirm that both exact model IDs were available:

GET https://cn.crazyrouter.com/v1/models

claude-opus-5
gpt-5.6-sol

All benchmark requests used:

POST https://cn.crazyrouter.com/v1/chat/completions

Within each test batch, both models received the same system prompt, user prompt, temperature, and max_tokens. No external tools were enabled. An HTTP 200 response was not treated as evidence that the task had passed.

Results were graded on three separate layers:

1. Core-answer accuracy: Were the key values, conclusions, and algorithmic behaviors correct?
2. Complete task delivery: Did the response satisfy every listed sub-requirement?
3. Machine-verifiable delivery: Did generated code pass hidden tests, and could JSON be parsed directly?

Automated string checks were followed by manual review. This matters because equivalent LaTeX, capitalization differences, exact fractions, and rounded decimals can otherwise produce false negatives.

For example, GPT-5.6-SOL did not reproduce the reference fraction verbatim on the probability task, but it supplied an equivalent formula and the correct decimal. Opus reported the physics result 0.302 m as 0.30 m to two significant figures; that was not counted as an error.

Results across the 10 intelligence tasks#

These results produce two different—and equally important—scores:

• Core correctness: 10/10 for both models.
• Complete task delivery: 9/10 for Opus 5 and 10/10 for GPT-5.6-SOL.

Treating those as the same metric would conceal the most useful operational difference in the test.

The hardest probability task: a correct answer can still be incomplete#

The target pattern was HHTH, with a biased coin defined by P(H)=0.62 and P(T)=0.38.

The task required the models to construct equations using longest prefix-suffix matching states. It also requested explicit explanations of two common sources of error:

• Why does another H after state HH leave the process in state HH?
• Why can the expected waiting time not be written simply as 1/P(HHTH)?

Both models reached the correct expected waiting time:

E[N] ≈ 12.6547

GPT-5.6-SOL completed the full derivation and explained that the pattern overlaps with itself, giving:

E[N] = 1 / P(HHTH) + 1 / P(H)

Opus 5 also supplied the correct states, equations, exact fraction, and decimal result. However, the response ended with:

finish_reason=length

The generation reached the max_tokens=3200 boundary while listing additional state expectations. It therefore omitted the final requested explanation of why inverse pattern probability alone is insufficient.

This distinction is central to the evaluation:

A correct final value counts toward core-answer accuracy. A response that omits a requested explanation does not count as complete task delivery.

The finding reflects the same methodological issue examined in the earlier max_tokens truncation retest: a reliable evaluator should retain the answer, finish_reason, and output budget. A partially correct response should not be declared complete merely because the visible portion looks convincing.

Executable-code validation: run the code instead of grading its style#

Code comparisons are particularly vulnerable to subjective judgments. Longer output, more comments, and cleaner-looking abstractions do not prove that an implementation is correct.

For both programming tasks, the generated code was saved as a .py file and executed in isolation against the same hidden tests.

Log aggregation#

The required function had to handle:

• half-open time windows;
• different counting rules for successful and failed requests;
• missing users and models;
• cache hit rate;
• lexicographic user ordering when costs were equal;
• preservation of the input objects without mutation.

Both models passed the hidden tests.

Stable routing with reliability constraints#

The routing function had to apply the following priorities and constraints:

• minimize total cost;
• when costs tie, minimize latency;
• when both tie, maximize reliability;
• when all three tie, select by lexicographic path order;
• support banned nodes, maximum hops, minimum reliability, and invalid edges.

Both models passed every hidden test for this task as well.

The executable-code result is therefore a tie: 2/2 hidden code tests for Claude Opus 5 and 2/2 for GPT-5.6-SOL. Code length or presentation style was not used to infer a winner.

For another GPT-5.6-SOL test involving difficult physics and dependency algorithms, see GPT-5.6-SOL vs GPT-5.5 高难物理与代码实测.

Strict JSON: an instruction-following test, not a mathematics test#

The strict-JSON task asked each model to condense incident data into one object. The output constraint was explicit:

exactly one JSON object and no Markdown

Both models correctly calculated the failure rate, assigned channel ownership, and determined the number of failures that remained unrecovered after retries. The difference was the response envelope:

• GPT-5.6-SOL returned JSON only on the first attempt, and the output could be passed directly to json.loads.
• Opus 5 added a code fence and a Verification section on the first attempt.
• On the first additional retry, Opus returned compact JSON but still appended Verification text.
• On the second additional retry, Opus again added a code fence and explanatory text.

The first-run score was therefore:

• Claude Opus 5: 0/1
• GPT-5.6-SOL: 1/1

Opus remained noncompliant in both extra retries. This should be described as “correct data, invalid delivery format,” not as a calculation failure.

Combining strict formatting and reasoning correctness into one undifferentiated score would obscure the actual engineering issue.

A basic local validator could look like this:

import json

REQUIRED_KEYS = [
    "window",
    "total_requests",
    "failed_requests",
    "failure_rate_pct",
    "provider_owned_failures",
    "customer_owned_failures",
    "recovered_by_retry",
    "unrecovered_failures",
    "root_cause",
    "action",
]


def parse_incident_json(raw: str) -> dict:
    payload = json.loads(raw)
    if list(payload) != REQUIRED_KEYS:
        raise ValueError("unexpected schema or key order")
    if payload["failed_requests"] != 84:
        raise ValueError("failed request count mismatch")
    return payload

A system prompt alone does not guarantee structured output. A safer production design is to use any structured-output controls supported by the provider, validate the schema locally, and retry or switch models when parsing fails.

Reproducing the comparison through one API#

The following minimal Python example uses the OpenAI-compatible chat completions endpoint. UTM parameters belong in website links, not API endpoint URLs.

import os
import requests

BASE_URL = "https://cn.crazyrouter.com/v1"
API_KEY = os.environ["CRAZYROUTER_API_KEY"]


def ask(model: str, prompt: str, max_tokens: int = 4000) -> dict:
    response = requests.post(
        f"{BASE_URL}/chat/completions",
        headers={
            "Authorization": f"Bearer {API_KEY}",
            "Content-Type": "application/json",
        },
        json={
            "model": model,
            "messages": [
                {"role": "system", "content": "Answer accurately and follow every requested constraint."},
                {"role": "user", "content": prompt},
            ],
            "temperature": 0.2,
            "max_tokens": max_tokens,
        },
        timeout=600,
    )
    response.raise_for_status()
    return response.json()


for model in ("claude-opus-5", "gpt-5.6-sol"):
    result = ask(model, "Your benchmark prompt here")
    choice = result["choices"][0]
    print(model, choice.get("finish_reason"), choice["message"].get("content", ""))

For a production-grade evaluation, also retain:

• the response ID;
• the returned model;
• the unmodified response content;
• finish_reason;
• request settings, including max_tokens;
• local grader results;
• code-test or schema-validation output.

Check current availability in the Crazyrouter model list, then run a small regression suite using prompts drawn from your own workload.

Production guidance#

Workloads where both models belong on the shortlist#

Both models performed well for:

• mathematics and physics problems with explicit reference answers;
• Python algorithms that can be checked with unit or hidden tests;
• tasks requiring detection of false premises or unsupported statistical conclusions;
• workflows with local validators that can determine whether a response succeeded.

Workloads where GPT-5.6-SOL had an advantage in this test#

GPT-5.6-SOL was more consistent when:

• every sub-requirement had to be completed in one response;
• the raw response body had to contain pure JSON;
• the application needed to avoid extracting structured data from explanatory prose.

These are findings from this test rather than a claim of universal superiority.

Additional safeguards for Opus 5#

Based on the observed failures, an Opus 5 integration should:

• allocate more max_tokens for long derivations;
• always inspect finish_reason;
• parse and validate JSON before sending it downstream;
• define explicit recovery behavior for Markdown fences or explanatory text;
• retry or switch models when output fails structural validation.

None of these precautions implies that Opus 5 reasoned less accurately here. It answered all 10 core tasks correctly and passed both hidden code tests. The observed difference was primarily whether every requirement fit into—and remained isolated within—the final response.

For another comparison of delivery behavior within the Claude family, see Claude Opus 5 vs Claude Fable 5 真实 API 测试.

FAQ#

Which is more intelligent, Claude Opus 5 or GPT-5.6-SOL?#

This test does not establish an overall intelligence winner. Across 10 objectively gradable tasks, both models achieved 10/10 core-answer accuracy.

Why was complete task delivery not a tie?#

Opus reached the max_tokens=3200 boundary during the difficult probability task. Its expected value and state equations were correct, but it omitted the final requested explanation. That produced 9/10 complete tasks for Opus versus 10/10 for GPT-5.6-SOL.

Was the Opus probability answer wrong?#

No. Opus correctly obtained:

E[N] ≈ 12.6547

The failure was incomplete delivery, not an incorrect probability result. The response ended with finish_reason=length.

Should invalid JSON count as an intelligence error?#

Not automatically. In this case, Opus calculated the JSON fields and values correctly but added material forbidden by the output contract. It is more precise to classify that as a format and instruction-following failure.

What happened during the two additional Opus JSON retries?#

Opus remained noncompliant in both. One retry appended Verification text after compact JSON, while the other again used a code fence and added explanatory text.

How were the code tasks graded?#

Each model’s output was saved as executable Python and run against the same hidden tests. The tests covered edge cases, ordering rules, invalid input, and input immutability. Both models passed both programming tasks, scoring 2/2.

Why is there no speed winner?#

End-to-end latency can reflect network routes, gateway behavior, caching, rate limits, upstream load, and service-instance selection. Because those variables were not controlled through a fixed upstream and sufficient repeated measurements, network latency was excluded from the intelligence score.

Are 10 tasks enough for a long-term model-selection decision?#

No. This is a small, reproducible capability sample. Before deployment, run real workload prompts repeatedly—ideally 20–50 times—and track core accuracy, complete task delivery, code-test pass rate, and format-violation rate separately.

How should I design my own model comparison?#

Start with 10–30 representative production prompts. Define an executable or objectively reviewable pass condition for each task, then submit identical inputs and settings through the same API. Choose the grading rules before inspecting the model outputs.

Should a production system trust a model’s JSON without validation?#

No. Parse the output locally, validate it against the expected schema, and define retry or fallback behavior. Even explicit instructions such as exactly one JSON object and no Markdown may not guarantee compliant output.

Final verdict#

The most useful result is not “GPT wins” or “Opus wins.” It is a more precise engineering conclusion:

Claude Opus 5 and GPT-5.6-SOL both achieved 10/10 core-answer accuracy across the 10 intelligence tasks. GPT-5.6-SOL was more consistent at completing every sub-requirement and delivering strict JSON, while Opus 5 requires closer management of output budgets and format validation.

Both models deserve consideration when tasks have reference answers, executable tests, or local validators. If downstream software consumes JSON directly, schema validation, retries, and model fallback should be treated as required system components rather than optional improvements.

Create a Crazyrouter account and run your own Opus 5 / GPT-5.6-SOL test