Kimi K3 vs GPT-5.6-SOL: High-Difficulty Tests in Math, Physics, and Programming

Kimi K3 vs GPT-5.6-SOL: Stress Tests in Math, Physics, and Programming#

This time I didn’t benchmark simple Q&A. Instead, I fixed three problems that require chained reasoning: a probability problem with overlapping patterns and a second moment, a physics problem involving pulley inertia plus a rope slack transition, and a Python problem that depends on closures, multi-day capacity, and a three-level tie-break. Both models used the same interface, the same prompt, and roughly the same output budget, with tools and web access disabled.

Bottom line first:

• gpt-5.6-sol produced a complete visible answer on all three tasks, and its reference results for the math and physics problems were correct.
• Under the initial 6500 token cap, kimi-k3 ended both the math and physics tasks with finish_reason=length, leaving the visible answer empty; even after raising the math cap to 10000, it was still truncated. The programming task timed out on read after about 245 seconds.
• GPT-5.6-SOL’s programming solution passed separate extra checks for dependency closure, capacity, tie-breaks, invalid dependencies, and cyclic dependencies. However, the first assertion sample bundled with the model itself had an incorrect expected value. That means “the core code is correct” and “all example tests pass” have to be validated separately.

The latency figures in this article are just what I observed for this request; they do not imply any fixed SLA. In the initial concurrent run, GPT’s math and physics requests received HTTP 408, and I retried them sequentially afterward. So the timing numbers here are only for call experience, not for claiming an absolute ranking.

Test Setup#

The tests were run from Beijing time 2026-07-17 to 2026-07-18, using:

POST https://cn.crazyrouter.com/v1/chat/completions
model: kimi-k3 / gpt-5.6-sol
temperature: 0.2

The first round used the same limits for both models: max_tokens=6500 for math and physics, and max_tokens=7500 for programming. Each task asked the model to provide a derivation that could be verified or code that could be run directly, and I recorded finish_reason, completion/reasoning tokens, request latency, numerical correctness, and whether the code ran independently in Python 3.11.

Summary Table#

For Kimi’s math and physics requests, the API reported reasoning tokens of roughly 6497 each; the math retry was close to 9997 reasoning tokens. That does not mean Kimi lacks reasoning ability. It means that under this prompt and the current routing, the reasoning budget was very easy to exhaust, and the caller never got the final answer.

Math Problem: Pattern Overlap Affects Not Just the Expectation, but the Variance Too#

The problem is: with heads probability p=3/5 and tails probability q=2/5, keep tossing until HHTH appears for the first time, allowing overlaps. Find E[T] and Var(T).

The correct prefix automaton states are S0=empty prefix, S1=H, S2=HH, S3=HHT, and S4=HHTH (absorbing state). The key transition is S2 --H--> S2: after HH, another H still leaves the longest suffix as HH, so you must not incorrectly fall back to the empty state.

GPT-5.6-SOL set up the first-moment equations as:

M0 = 1 + p M1 + q M0
M1 = 1 + p M2 + q M0
M2 = 1 + p M2 + q M3
M3 = 1 + q M0

Continuing with the second-moment equations gives:

E[T]   = 715/54       ≈ 13.2407407407
E[T²]  = 195335/729   ≈ 267.9492455418
Var(T) = 270115/2916  ≈ 92.6320301783

The expectation can also be checked independently using the boundary formula. The only non-empty proper border of HHTH is the single character H, so E[T] = 1/p + 1/(p³q) = 715/54. GPT’s full answer covered the state transitions, the second-moment expansion, and a sanity check. Under the 6500 and 10000 limits, Kimi produced no visible derivation, so there is no way to score its mathematical correctness in this run; it can only be recorded as “not completed within budget.”

Physics Problem: Once the Mass Hits the Ground, You Cannot Keep Using the Same Energy Equation#

The setup is: m_A=4.0 kg on a rough 25° incline with μ_k=0.18; m_B=3.0 kg hanging; pulley M_p=1.2 kg, R=0.10 m; after B drops 1.50 m and hits the ground, the rope immediately goes slack; A then slides up another 0.10 m before contacting a k=250 N/m spring; g=9.8 m/s².

Before B hits the ground, the rope is taut and there is no slip, so the pulley’s effective inertial mass is I/R²=(1/2)M_p=0.60 kg. Solving the translational equations for the two blocks together with the pulley rotation equation gives:

a  ≈ 0.847 m/s²
v1 ≈ 1.59 m/s
v2 ≈ 1.18 m/s
x  ≈ 0.0835 m = 8.35 cm

After B hits the ground, B’s velocity is altered by the collision with the floor, while A is still moving up the incline, and the pulley may still be spinning. The problem also explicitly says the rope immediately goes slack, so you can no longer use v_A=v_B=Rω. From that point on, you should analyze only A:

v2² = v1² - 2g(sinθ + μ_k cosθ)d
1/2 m_A v2² = 1/2 kx² + m_A g(sinθ + μ_k cosθ)x

GPT-5.6-SOL also checked dimensional consistency, the stopping distance without a spring, and the sum of spring energy plus friction/gravity losses, and the numbers were self-consistent. Kimi never produced a visible answer in the first round, so there is no way to compare its intermediate modeling quality.

Programming Problem: The Core Algorithm Passed Independent Checks, but the Model’s Own Sample Has a Bug#

The programming task was to implement optimize_release_plan(items, capacity_by_day, dependencies), handling daily capacity, zero capacity for unconfigured dates, direct and transitive dependency closure, invalid dependencies, cycle detection, and a three-level tie-break of value → risk → sorted id list. Since items <= 18, bitmask enumeration is a reasonable baseline.

GPT-5.6-SOL used dependency-closure caching plus 2^n subset enumeration. I extracted the code and ran separate checks in Python 3.11: dependency closure, multi-day capacity samples, value/risk/id three-level tie-breaks, unconfigured-date capacity, missing dependencies, and cyclic dependencies. All 6 checks passed.

However, the first assertion bundled at the end of the model’s code failed: it wrote A -> B -> C while also including an item X with value 11. Under the given capacity, B + C + X has a total value of 13, which is higher than A + B + C at 12, so returning ['B', 'C', 'X'] is correct; the assertion ['A', 'B', 'C'] is a test fixture bug.

This is a good reminder that in code tasks, you cannot judge only the function body, and you also cannot assume a test suite is trustworthy just because it includes 8 assertions. Model-generated test data still needs to be checked manually or against a reference implementation. Kimi K3’s programming request timed out on read after about 245 seconds, and no complete code was obtained.

Overall Assessment#

The more accurate conclusion is not “one model is absolutely smarter,” but:

1. Under this routing and these budgets, GPT-5.6-SOL was more likely to collapse complex reasoning into a final answer, making it a better fit for workflows that need a readable derivation or runnable code right away.
2. Kimi K3’s main issue here was the conversion efficiency between reasoning budget and visible output; even when the math limit was raised to 10000 tokens, it still did not finish.
3. GPT-5.6-SOL’s code should not be trusted blindly either. Its function logic passed independent checks, but the attached test fixture had a value-calculation error.
4. For production selection, you should record finish_reason, reasoning tokens, latency, and local test results together, instead of relying only on the final sentence saying the answer is correct.

Reproducing the Experiment#

The test script and sequential retry results are stored here, and the summary of the initial concurrent run is also written into the script output:

.tmp/kimi_k3_vs_gpt56sol_test.py
.tmp/kimi-k3-vs-gpt56sol-results.json
.tmp/retry-gpt56sol-math.json
.tmp/retry-gpt56sol-physics.json
.tmp/retry-gpt56sol-programming.json
.tmp/retry-kimi-k3-math.json

For reruns, it is best to pin the model IDs, prompt, temperature, max_tokens, and concurrency level, and save each output to a separate file. Upstream traffic, cache hits, and rate-limit state all affect latency; in this article, HTTP 408, length, and read timeouts were all recorded as part of the test results rather than being hidden.