CrazyrouterClaude Opus 5 vs GPT-5.6 Luna: 18 Verifiable Tasks, No Speed Leaderboard
A correctness-first benchmark of Claude Opus 5 and GPT-5.6 Luna across math, physics, executable algorithms, false-premise resistance, constraints, and instruction following.
Quick answer#
Claude Opus 5 scored 17/18 (94.4%) and GPT-5.6 Luna scored 16/18 (88.9%) on the harder round. That is a one-task lead in this sample—not proof that Opus is universally stronger. The useful result is the shape of the failures: Opus delivered all three algorithm files successfully but broke a strict JSON contract; Luna followed all three formatting instructions but shipped two algorithm files that failed during import.
Why the benchmark had to become harder#
The first eight tasks were too easy to separate the models: both scored 8/8. The second round therefore used 18 tasks, with hard, harder, and extreme variants in each of six categories. The questions required exact fractions, multi-stage physical modeling, executable files, resistance to misleading assumptions, unique logical solutions, and byte-level output discipline. The goal was not to reward impressive prose. Every decisive claim had to be checked by deterministic rules, independent calculations, JSON parsing, or Python execution.
How the 18-task round was scored#
Both models received the same task text, system instruction, temperature, and task-level output budget. Mathematical and physics answers were checked against exact values or numerical tolerances. Algorithm responses were saved as delivered and imported into the same hidden-test harness. Strict JSON and CSV tasks were validated as artifacts, not read charitably. A file that only works after deleting model-written assertions is useful diagnostic evidence, but it is not a successful first delivery. Route latency was retained in the raw JSON for infrastructure diagnosis and was excluded from every winner decision in this article.
Results across six dimensions#
| Dimension | Opus 5 | GPT-5.6 Luna |
|---|---|---|
| Mathematical reasoning | 3/3 | 3/3 |
| Complex physics | 3/3 | 3/3 |
| Algorithm delivery | 3/3 | 1/3 |
| False-premise resistance | 3/3 | 3/3 |
| Constraint reasoning | 3/3 | 3/3 |
| Instruction following | 2/3 | 3/3 |
| Total | 17/18 (94.4%) | 16/18 (88.9%) |
The algorithm gap was a delivery gap#
Luna's two failed algorithm answers contained core functions that passed hidden tests after their own bottom-of-file assertions were removed. The problem was that the assertions encoded incorrect expected values, so importing the original files raised AssertionError before a caller could use the functions. Under a strict complete-file contract, those are failures. Opus's three original algorithm files imported and passed the shared harness. This distinction matters in production: a plausible function body is not the same thing as a deployable artifact.
The strict-JSON failure points in the other direction#
Opus's only hard-round failure was almost the mirror image. The JSON keys and values were correct, but the model wrapped the object in a Markdown code fence even though the instruction required exactly one JSON object and nothing else. The same behavior reappeared in the independent retry. Luna returned valid strict output in all three instruction-following tasks. If your pipeline feeds model output directly into a parser, wrapper discipline is part of correctness, not a cosmetic preference.
Output budgets were separated from intelligence errors#
Some early responses ended with finish_reason=length because hidden reasoning consumed the available output budget. Those attempts were preserved, but they were not counted as intelligence failures. The same effective budget was increased for both models before scoring. This avoids confusing capacity configuration with reasoning quality while still documenting a real integration risk: long reasoning can leave too little room for the visible artifact.
What this means for production routing#
Use Luna first for strict JSON, CSV, short extraction, and high-volume structured work—but keep schema validation. Use Opus first for long algorithm files, defensive engineering, and tasks where the complete executable artifact matters more than brevity. For math, complex physics, false-premise resistance, and constraint reasoning, this sample was a tie, so cost, interface compatibility, and output style are better routing variables. In every case, validate the task artifact: parse JSON, check key numbers, and run generated code exactly as received.
Reproduction and raw evidence#
The benchmark runner is scripts/opus5_vs_luna_hard_benchmark.py. The complete hard-round result is .tmp/opus5-vs-luna-hard-benchmark-results.json; independent failure retries are in .tmp/opus5-hard-failure-retry.json and .tmp/luna-hard-failure-retry.json. The API endpoint used was https://cn.crazyrouter.com/v1/chat/completions. Latency fields remain in those raw files for route diagnosis only.
FAQ#
Does 17/18 make Opus 5 the overall winner?#
No. It gives Opus a one-task lead on this specific 18-task set. The sample does not cover vision, tools, long-context retrieval, or multi-turn agents.
Why count faulty self-tests as algorithm failures?#
Because the requested deliverable was a complete Python file. If the original file crashes on import, downstream systems cannot use it without manual repair.
Why is latency absent from the verdict?#
Latency reflects the model, gateway route, upstream load, and channel state. It is valuable operational evidence but not a reliable measure of intelligence correctness.
Final verdict#
In this correctness-first benchmark, Opus 5 led 17/18 to 16/18 because it delivered all algorithm artifacts successfully. Luna was stronger at strict instruction compliance. The practical conclusion is not a universal champion; it is a routing rule backed by reproducible failure modes.
Run both models through Crazyrouter with your own production prompts
