LocalsOnly · measured2026-10-12
Terminal-Bench 2.1
Version 2.1 · 89 tasks. Compare only matched task sets and protocols.
GLM-5.3-Flash · EXL3 TR3 4bpw · 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total · TensorFold
Needs two desk boxes: DGX Spark (serve head) + ASUS GX10 (tensor-parallel worker), 128GB each.
● Tested by LocalsOnlyRecorded effort for this run. Wall time and token counts are not a dollar cost; hardware power and utilization also matter.
Tokens per second · see measurement basis
As recorded on this receipt
- Engine A/B, same weights: this row and the vLLM Terminal-Bench 2.1 row (2026-10-03, 39/89) use the same Mia-AiLab EXL3 TR3 4bpw weights @25a44fdb, the same Harbor terminus-2 settings (thinking off, max_turns 40, agent-timeout x4, timeout x2, k=1 n=1, full 89, 100,000-token cap) on the same two boxes. The serving stack changed: TensorFold v1.10 instead of vLLM.
- Not a pure engine comparison: TensorFold serves the dense/shared weights at 4-bit, while the vLLM run kept them at BF16.
- Same total score, different tasks: both engines pass 39/89. 29 tasks pass on both; 10 pass only on TensorFold and 10 only on vLLM. With k=1 n=1, single-task flips are within run-to-run noise.
- Wall time: 17.8 h here vs 25.4 h for the vLLM run (about 1.4x faster). The gap includes run-to-run variance in which tasks run long, so it is not a clean speed measurement. See the separate 10/9 speed test on the AutomationBench comparison card for like-for-like tok/s.
- Errors count as 0: 4 trials failed with infra RuntimeErrors and have no verifier reward (qemu-startup and qemu-alpine-ssh: tmux session failed to start; compile-compcert: tmux send-keys failed; regex-log: a command timed out after 120 s). 1 trial (regex-chess) hit the 4-hour agent timeout and was then verified at reward 0. All 5 score 0, so the headline is 39/89; 39 of the 85 verified trials passed (45.9%). The vLLM run had 2 infra errors (the two qemu tasks) and no agent timeouts.
- MEASURED with thinner settings than the official Terminal-Bench 2.1 card protocol (thinking on, 256k context, no turn cap). Not directly comparable to the official row or to published thinking-on figures.
- Weights are Mia-AiLab EXL3 TR3 4bpw (community quant) with a DFlash2 speculator, fp8 KV cache; not Z.ai's official weights. Server-default sampling; thinking off passed explicitly.
- Needs two desk boxes: serve head DGX Spark spark-5c2f (:8888) + worker ASUS GX10 gx10-e789 (2× GB10, 256GB total). Not a single 128GB desk box.
Run configuration & provenance +
- Context
- 100,000
- Temperature
- Not recorded
- Wall time
- 1069.4 minutes
- Revision
- 25a44fdbf16862a46b7cc9921142c6c81350af2f
- Reasoning
- off
- Agent version
- 2.0.0
- Output token limit
- Not reported
- Dataset revision
- sha256:7d7bdc1cbedad549fc1140404bd4dc45e5fd0ea7c4186773687d177ad3a0699a
- Harness host
- Mac mini · 64GB
- Model host
- 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total
- Input tokens
- 58,722,980
- Output tokens
- 1,184,834
- Cached input tokens
- 55,395,008
- Agent turn / step limit
- 40
- Concurrency
- 1
- Weight file size
- 175.72 GB
Throughput measurement: Measurement method not separately reported. This should not be treated as standardized decode-only speed.
Harbor terminus-2 on Kris Mac mini (docker, native images) -> TensorFold v1.10 OpenAI-compat [local endpoint] (GLM-5.3-Flash-EXL3 TR3 4bpw, dense/shared 4-bit, fp8 KV, DFlash2 bf582e4e, context 100000, image v0.6.0-a1897d591f70, recipe 549fdc5; DGX Spark spark-5c2f head + ASUS GX10 gx10-e789 worker, single RoCE link; thinking OFF, max_turns=40, agent-timeout×4, timeout×2, k=1 n=1). Job tb21-mac-dgx-glm53flash-tf110-20261011a. Same Harbor settings as the vLLM run tb21-mac-dgx-glm53flash-20261002a. Not the official Terminal-Bench 2.1 card protocol. Hardware evidence id gb10-2x = DGX Spark spark-5c2f :8888 head + ASUS GX10 gx10-e789 worker.
Run ID: measured-2x-gb10-glm53flash-exl3-4bpw-tensorfold-terminal-bench-2.1-2026-10-12Task explorer
Passed tasks first, then other outcomes grouped by recorded issue. Use the filters to inspect a particular outcome.
All 89 recorded tasks, before filters or pagination. 25 of 89 matching tasks shown below. Recorded issues can also occur on scored tasks.
PassedBreak filter js from html+
Task ID: break-filter-js-from-html
Recorded name: break-filter-js-from-html
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedBuild cython ext+
Task ID: build-cython-ext
Recorded name: build-cython-ext
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedBuild pmars+
Task ID: build-pmars
Recorded name: build-pmars
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedBuild pov ray+
Task ID: build-pov-ray
Recorded name: build-pov-ray
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedCancel async tasks+
Task ID: cancel-async-tasks
Recorded name: cancel-async-tasks
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedCobol modernization+
Task ID: cobol-modernization
Recorded name: cobol-modernization
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedConfigure git webserver+
Task ID: configure-git-webserver
Recorded name: configure-git-webserver
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedConstraints scheduling+
Task ID: constraints-scheduling
Recorded name: constraints-scheduling
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedCustom memory heap crash+
Task ID: custom-memory-heap-crash
Recorded name: custom-memory-heap-crash
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDistribution search+
Task ID: distribution-search
Recorded name: distribution-search
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedExtract elf+
Task ID: extract-elf
Recorded name: extract-elf
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedFilter js from html+
Task ID: filter-js-from-html
Recorded name: filter-js-from-html
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedFinancial document processor+
Task ID: financial-document-processor
Recorded name: financial-document-processor
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedFix code vulnerability+
Task ID: fix-code-vulnerability
Recorded name: fix-code-vulnerability
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedFix git+
Task ID: fix-git
Recorded name: fix-git
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedFix ocaml gc+
Task ID: fix-ocaml-gc
Recorded name: fix-ocaml-gc
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedGit leak recovery+
Task ID: git-leak-recovery
Recorded name: git-leak-recovery
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedGit multibranch+
Task ID: git-multibranch
Recorded name: git-multibranch
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedHf model inference+
Task ID: hf-model-inference
Recorded name: hf-model-inference
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedLarge scale text editing+
Task ID: large-scale-text-editing
Recorded name: large-scale-text-editing
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedLargest eigenval+
Task ID: largest-eigenval
Recorded name: largest-eigenval
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedLog summary date ranges+
Task ID: log-summary-date-ranges
Recorded name: log-summary-date-ranges
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedModernize scientific stack+
Task ID: modernize-scientific-stack
Recorded name: modernize-scientific-stack
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedMteb retrieve+
Task ID: mteb-retrieve
Recorded name: mteb-retrieve
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedMulti source data merger+
Task ID: multi-source-data-merger
Recorded name: multi-source-data-merger
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.