LocalsOnly · measured2026-10-13
SWE-bench Verified
swe-bench-verified · 100 tasks. Compare only matched task sets and protocols.
GLM-5.3-Flash · EXL3 TR3 4bpw · 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total · TensorFold
Needs two desk boxes: DGX Spark (serve head) + ASUS GX10 (tensor-parallel worker), 128GB each.
● Tested by LocalsOnlyRecorded effort for this run. Wall time and token counts are not a dollar cost; hardware power and utilization also matter.
Tokens per second · see measurement basis
As recorded on this receipt
- RECEIPT label: Harbor terminus-2 thin (max_turns=50, thinking OFF) on the frozen first-100 slice of 500; not Claude Code or an official SWE-bench Verified compare.
- Engine A/B, same weights: this row and the vLLM Verified row (2026-10-02, 69/100) use the same Mia-AiLab EXL3 TR3 4bpw weights @25a44fdb, the same seed slice, terminus-2 settings (thinking off, max_turns 50, k=1 n=1, 100,000-token cap, amd64) and the same two boxes. The serving stack changed: TensorFold v1.10 instead of vLLM.
- Not a pure engine comparison: TensorFold serves the dense/shared weights at 4-bit, while the vLLM run kept them at BF16.
- 71 vs 69 with one try per task is within noise (standard error about 4.6 points). 59 tasks pass on both engines and 19 fail on both; 12 pass only on TensorFold and 10 only on vLLM.
- Infra differs between the runs: the vLLM run lost 6 tasks to tmux-install timeouts before any model call (scored 0); TensorFold passed 4 of those 6. TensorFold lost 1 task (django__django-11951, tmux send-keys failure, scored 0), which vLLM passed. On the 93 tasks where neither run had an infra error, TensorFold passed 67 and vLLM 68.
- Errors count as 0: 1 of 100 trials (django__django-11951) failed with RuntimeError and has no verifier reward, so the headline 71/100 is Harbor's mean; 71 of 99 verified trials passed.
- Weights are Mia-AiLab EXL3 TR3 4bpw (community quant) with a DFlash2 speculator, fp8 KV cache; not Z.ai's official weights. Server-default sampling; thinking off passed explicitly.
- Wall time 12.6 h vs 19.2 h for the vLLM run, which included a serve crash and 26 re-run trials; not a speed measurement.
- Needs two desk boxes: serve head DGX Spark spark-5c2f (:8888) + worker ASUS GX10 gx10-e789 (2× GB10, 256GB total). Not a single 128GB desk box.
Run configuration & provenance +
- Context
- 100,000
- Temperature
- Not recorded
- Wall time
- 753.9 minutes
- Revision
- 25a44fdbf16862a46b7cc9921142c6c81350af2f
- Reasoning
- thinking OFF · enable_thinking=false passed explicitly via chat_template_kwargs (GLM template defaults ON)
- Agent version
- Not reported
- Output token limit
- Not reported
- Dataset revision
- sha256:b934b0cc3dc800fe945eaf9f1623329db97ee3133c706d20644524c7759fb341
- Harness host
- mac
- Model host
- 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total
- Input tokens
- 81,109,888
- Output tokens
- 1,047,582
- Cached input tokens
- 77,284,736
- Agent turn / step limit
- 50
- Concurrency
- 1
- Weight file size
- 175.72 GB
Throughput measurement: Measurement method not separately reported. This should not be treated as standardized decode-only speed.
Harbor terminus-2 on Kris Mac mini (docker, DOCKER_DEFAULT_PLATFORM=linux/amd64) -> TensorFold v1.10 OpenAI-compat [local endpoint] (GLM-5.3-Flash-EXL3 TR3 4bpw, dense/shared 4-bit, fp8 KV, DFlash2 bf582e4e, context 100000, image v0.6.0-a1897d591f70, recipe 549fdc5; DGX Spark spark-5c2f head + ASUS GX10 gx10-e789 worker, single RoCE link; thinking OFF, max_turns=50, k=1 n=1). Job swever-mac-dgx-glm53flash-tf110-n100-20261012a. RECEIPT, same protocol and seed as the vLLM Verified RECEIPT; NOT Claude Code official-compare. Hardware evidence id gb10-2x = DGX Spark spark-5c2f :8888 head + ASUS GX10 gx10-e789 worker.
Run ID: measured-2x-gb10-glm53flash-exl3-4bpw-tensorfold-swe-bench-verified-n100-2026-10-13Task explorer
Passed tasks first, then other outcomes grouped by recorded issue. Use the filters to inspect a particular outcome.
All 100 recorded tasks, before filters or pagination. 25 of 100 matching tasks shown below. Recorded issues can also occur on scored tasks.
PassedAstropy astropy 8707+
Task ID: astropy__astropy-8707
Recorded name: astropy__astropy-8707
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedAstropy astropy 13579+
Task ID: astropy__astropy-13579
Recorded name: astropy__astropy-13579
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedAstropy astropy 14096+
Task ID: astropy__astropy-14096
Recorded name: astropy__astropy-14096
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedAstropy astropy 14995+
Task ID: astropy__astropy-14995
Recorded name: astropy__astropy-14995
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 7530+
Task ID: django__django-7530
Recorded name: django__django-7530
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 10554+
Task ID: django__django-10554
Recorded name: django__django-10554
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 11099+
Task ID: django__django-11099
Recorded name: django__django-11099
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 11179+
Task ID: django__django-11179
Recorded name: django__django-11179
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 11490+
Task ID: django__django-11490
Recorded name: django__django-11490
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 12262+
Task ID: django__django-12262
Recorded name: django__django-12262
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 12419+
Task ID: django__django-12419
Recorded name: django__django-12419
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 13033+
Task ID: django__django-13033
Recorded name: django__django-13033
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 13121+
Task ID: django__django-13121
Recorded name: django__django-13121
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 13406+
Task ID: django__django-13406
Recorded name: django__django-13406
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 13417+
Task ID: django__django-13417
Recorded name: django__django-13417
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 13933+
Task ID: django__django-13933
Recorded name: django__django-13933
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 14122+
Task ID: django__django-14122
Recorded name: django__django-14122
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 14311+
Task ID: django__django-14311
Recorded name: django__django-14311
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 14376+
Task ID: django__django-14376
Recorded name: django__django-14376
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 14434+
Task ID: django__django-14434
Recorded name: django__django-14434
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 14631+
Task ID: django__django-14631
Recorded name: django__django-14631
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 14999+
Task ID: django__django-14999
Recorded name: django__django-14999
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 15022+
Task ID: django__django-15022
Recorded name: django__django-15022
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 15037+
Task ID: django__django-15037
Recorded name: django__django-15037
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 15561+
Task ID: django__django-15561
Recorded name: django__django-15561
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.