GLM-5.3 vs GLM-5.2: A Hard Benchmark with Real API Calls

GLM-5.3 vs GLM-5.2: A Hard Benchmark with Real API Calls#

The first benchmark showed that GLM-5.3 was more likely to deliver visible answers within a fixed output budget. This follow-up raised the difficulty across six objectively verifiable tasks: extreme mathematics, causal calibration, constraint reasoning, nested JSON, multi-stage physics, and executable optimization code.

The first-delivery score was 4/6 for GLM-5.3 and 2/6 for GLM-5.2. That headline does not tell the whole story: GLM-5.2 produced the only dependency optimizer that passed the hidden test suite, while both models failed the extreme math task.

Test setup#

Before the run, GET https://cn.crazyrouter.com/v1/models listed both exact model IDs. All 12 first-run responses returned the requested model identity.

Results#

GLM-5.3 completed four first-run tasks in 334.1 seconds total. GLM-5.2 completed two in 665.0 seconds. Latency was retained as operational evidence but did not change correctness scores.

The extreme math task defeated both models#

The task used four coupon probabilities: 1/2, 1/4, 1/8, 1/8. It required two exact inclusion-exclusion results:

E[T] = 1339/105 ≈ 12.752380952
P(T<=8) = 46179/131072 ≈ 0.352317810

Both models consumed the full 9,000-token reasoning budget and returned no visible answer. A 20,000-token retry still produced an empty body for GLM-5.3; GLM-5.2 exceeded the runner's 420-second read timeout.

This is a useful production warning: raising max_tokens does not guarantee a deliverable answer. A gateway should inspect finish reason, visible content, and timeout state together.

Where GLM-5.3 was stronger#

On the Simpson task, GLM-5.3 correctly recovered the reversal:

Small stones: A 81/87 > B 234/270
Large stones: A 192/263 > B 55/80
Crude total: A 273/350 < B 289/350
50/50 standardized rates: A=0.8305, B=0.7771

It also separated standardization from causal identification, noting the need for randomization or adequate adjustment, positivity, no unmeasured confounding, and uncertainty estimates.

The minimal-UNSAT task required exactly one JSON object. GLM-5.3 returned [1,2,3] and correctly proved that the constraints force C and D into the same position. The response parsed without cleanup.

The physics task required switching system boundaries after the hanging mass hit the ground and the rope went slack. GLM-5.3 matched all four reference values:

a ≈ 0.847 m/s²
v1 ≈ 1.59 m/s
v2 ≈ 1.18 m/s
x ≈ 0.0835 m

Where GLM-5.2 won#

The code task asked for optimize_release_plan(items, capacity_by_day, dependencies) with transitive dependencies, per-day capacities, value maximization, risk and lexical tie-breaks, invalid references, and cycle detection.

GLM-5.2 returned a complete Python file at the 16k budget and passed the external hidden test suite without edits. GLM-5.3 returned no body at 16k. At 24k it finally produced code, but selected a plan worth 24 instead of the correct optimum worth 29, so the retry still failed.

This is why code generation must be executed. Version numbers, code length, and a polished explanation are not substitutes for tests.

OpenAI-compatible request#

import os
import requests

r = requests.post(
    "https://cn.crazyrouter.com/v1/chat/completions",
    headers={"Authorization": f"Bearer {os.environ['CRAZYROUTER_API_KEY']}"},
    json={
        "model": "glm-5.3",
        "messages": [{"role": "user", "content": "Your acceptance task"}],
        "temperature": 0.2,
        "max_tokens": 6000,
    },
    timeout=600,
)

choice = r.json()["choices"][0]
assert choice["finish_reason"] == "stop"
assert choice["message"]["content"].strip()

Production guidance#

1. Prefer GLM-5.3 for causal analysis, constraint reasoning, and multi-stage derivations based on this run.
2. Keep GLM-5.2 in the candidate pool for executable algorithm generation.
3. Treat HTTP 200 + empty content as a business failure.
4. Parse and validate JSON instead of accepting visually plausible output.
5. Run generated code in an isolated test harness.
6. Track first-delivery success separately from retry-assisted success.

FAQ#

Is GLM-5.3 universally better?#

No. It had better first-run coverage, but GLM-5.2 won the executable code task.

Did more output budget fix the failures?#

Not reliably. GLM-5.2 physics still returned an empty body at 20k. GLM-5.3 math remained empty at 20k, and its 24k code retry failed hidden tests.

Why count HTTP 200 responses as failures?#

Because multiple responses consumed the entire reasoning budget and delivered no usable content.

How was code graded?#

The original generated file was imported and executed against the same hidden harness. No manual fixes were allowed.

Was latency part of the score?#

No. It was kept as operational evidence only.

What is the practical model-selection rule?#

Route by task type and enforce real acceptance tests. A single aggregate model ranking loses the most important result from this benchmark.

Verdict#

GLM-5.3 won first-delivery coverage, 4/6 to 2/6. GLM-5.2 delivered the best executable algorithm. The defensible production conclusion is task-aware routing with strict validation, not automatic replacement of the older model.

Run the same models through Crazyrouter's OpenAI-compatible API.