LocalsOnly · measured2026-10-22

SWE-bench Verified

swe-bench-verified · 100 tasks. Compare only matched task sets and protocols.

GLM-5.3-Flash · NVFP4 (LibertAIDAI) · 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total · SGLang

Needs two desk boxes: DGX Spark (serve head) + ASUS GX10 (tensor-parallel worker), 128GB each.

● Tested by LocalsOnly

Download weights · LibertAIDAI/GLM-5.3-Flash-NVFP4 @ caca4e6a ↗Serve recipe · randomllama/GLM-5.3-Flash-DFlash2-SGLang-2x-DGX-Spark@18520bfb ↗

WALL-CLOCK TIME49.2 h

INPUT TOKENS57,165,257

Recorded effort for this run. Wall time and token counts are not a dollar cost; hardware power and utilization also matter.

Reported score60.0%60 of 100 tasks passed

Throughput—

Tokens per second · see measurement basis

Concurrency1

As recorded on this receipt

6 infrastructure outcomes are recorded. Failure counts may overlap with task outcomes.

Read before comparing

RECEIPT label: Harbor terminus-2 thin (max_turns=50) on the frozen first-100 slice of 500; not Claude Code or an official SWE-bench Verified compare.
Engine A/B, three ways: SGLang (this row) 60/100; TensorFold 71/100; vLLM 69/100. Same seed slice, dataset ref sha256:b934b0cc, terminus-2 settings (max_turns 50, k=1 n=1, client concurrency 1, 100,000-token cap, amd64) and the same two boxes; config.json differs only in job name, model name and endpoint.
Different weights AND a different engine: this row serves LibertAIDAI/GLM-5.3-Flash-NVFP4 @caca4e6a (NVFP4) on SGLang TP=2; the TensorFold and vLLM rows serve the Mia-AiLab EXL3 TR3 4bpw quant. Any gap is not an engine-only effect.
Thinking flag ignored: enable_thinking=false was sent (as on the TensorFold and vLLM runs), but this SGLang stack does not honor it, so the model reasoned anyway. Reasoning followed by a literal </think> reached the agent inside the reply in 311 of 1974 agent steps (63 of 94 trajectories). The TensorFold and vLLM Verified runs were thinking off, so this is a further protocol difference.
Overlap: 52 tasks resolved on both SGLang and TensorFold, 8 SGLang only, 19 TensorFold only; vs vLLM 50 both, 10 SGLang only, 19 vLLM only. On the 93 tasks where neither SGLang nor TensorFold had a container error, SGLang resolved 59 and TensorFold 66. One try per task (standard error about 4.9 points).
Error breakdown: 33 of 100 trials errored. 27 AgentTimeoutError (agent ran past Harbor's time limit); 8 of them had already produced a passing patch and are resolved, 19 score 0. 6 RuntimeError (django__django-11490, django__django-13033, django__django-16263, psf__requests-2317, scikit-learn__scikit-learn-14053, sympy__sympy-20428): a task-container command timed out after 120 s; no verifier reward, scored 0 (unresolved) and kept in the 100 denominator. TensorFold had 0 timeouts and 1 RuntimeError; vLLM had 6 setup errors.
Wall time 49h14m (8:33 PM ET Oct 20 to 9:48 PM ET Oct 22), one serial segment with 0 retries and no resume, vs 12.6 h TensorFold and 19.2 h vLLM. Not a speed measurement: timeouts and effective thinking dominate. 57.2M input / 2.65M output tokens (TensorFold 81.1M / 1.05M).
Speculative decoding: DFlash2 drafter incoai/GLM-5.3-Flash-DFlash2 @bf582e4e (8 draft tokens); the recipe authors do not guarantee it is lossless. The drafter is licensed cc-by-nc-nd-4.0: research / evaluation use only, not commercial. Target weights MIT.
bf16 KV cache (the authors' fp8-KV 131k config is not published, so not reproduced); context 100,000 with a 102,720-token KV pool; chunked prefill 4096; max-running-requests 2 (server); Harbor ran one trial at a time. Sampling: server defaults, temperature 1.0 / top_p 0.95.
Raw image inspect stored with the job files: jobs/sglang-glm53/image-inspect-spark.json and image-inspect-asus.json (same image id sha256:a45e44f78f3d on both boxes, tag glm53-flash-dflash:c4d5d45e5-gb10tile, created 2026-10-16) and image-inspect-base-spark.json (base lmsysorg/sglang:glm-5.3-flash-arm64, RepoDigest lmsysorg/sglang@sha256:e88340d6). The inspect labels record SGLANG_BUILD_COMMIT=unknown; the reported version string sglang 0.0.0.dev1+g033446bb05 (not the recipe's pinned c4d5d45e5) and the overlay of PR #36708 glm5_next.py plus a GB10 tilelang retune (recipe rev 18520bfb) come from Model Lab's gate doc.
Weights are LibertAIDAI's community NVFP4 quant, not Z.ai's official FP8. Needs two desk boxes: serve head DGX Spark spark-5c2f (:8901) + worker ASUS GX10 gx10-e789 (2× GB10, 256GB total). Not two DGX Sparks and not a single 128GB desk box.

Run configuration & provenance +

Context: 100,000
Temperature: Not recorded
Wall time: 2954.2 minutes
Weight file: LibertAIDAI/GLM-5.3-Flash-NVFP4 @ caca4e6a (multi-file) ↗
Revision: caca4e6a4ebbd66f159d3d2fc256683fd6e27177
Reasoning: enable_thinking=false sent via chat_template_kwargs (as on TensorFold/vLLM) but NOT honored by this SGLang stack: effective thinking ON. Reasoning text followed by a literal </think> reached terminus-2 inside content in 311 of 1974 agent steps (63 of 94 trajectories).
Agent version: Not reported
Output token limit: Not reported
Dataset revision: sha256:b934b0cc3dc800fe945eaf9f1623329db97ee3133c706d20644524c7759fb341
Harness host: mac
Model host: 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total
Input tokens: 57,165,257
Output tokens: 2,647,054
Cached input tokens: 0
Agent turn / step limit: 50
Concurrency: 1
Weight file size: 194.70 GB

Throughput measurement: Measurement method not separately reported. This should not be treated as standardized decode-only speed.

Harbor terminus-2 on Kris Mac mini (docker, DOCKER_DEFAULT_PLATFORM=linux/amd64) -> SGLang TP=2 OpenAI-compat [local endpoint] (LibertAIDAI/GLM-5.3-Flash-NVFP4 @caca4e6a, bf16 KV, context 100000 / KV pool 102720, DFlash2 drafter incoai/GLM-5.3-Flash-DFlash2 @bf582e4e, image glm53-flash-dflash:c4d5d45e5-gb10tile id a45e44f78f3d on lmsysorg/sglang@sha256:e88340d6; DGX Spark spark-5c2f head + ASUS GX10 gx10-e789 worker over ConnectX-7; enable_thinking=false sent but ignored by this SGLang stack, so effective thinking ON; max_turns=50, k=1 n=1, client concurrency 1). Job swever-mac-dgx-glm53flash-sglang-nvfp4-n100-20261018a. RECEIPT, same protocol and seed as the TensorFold and vLLM Verified RECEIPTs; NOT Claude Code official-compare.

Run ID: measured-2x-gb10-glm53flash-nvfp4-sglang-swe-bench-verified-n100-2026-10-22

Tasks

Task explorer

Passed tasks first, then other outcomes grouped by recorded issue. Use the filters to inspect a particular outcome.

100 task records

60 Passed40 Not passed

All 100 recorded tasks, before filters or pagination. 25 of 100 matching tasks shown below. Recorded issues can also occur on scored tasks.

PassedAstropy astropy 8707+

Task ID: astropy__astropy-8707
Recorded name: astropy__astropy-8707

Reward: 1

Recorded issue: No failure explanation in this task record.