LocalsOnly · measured2026-10-03
Terminal-Bench 2.1
Version 2.1 · 89 tasks. Compare only matched task sets and protocols.
GLM-5.3-Flash · EXL3 TR3 4bpw · 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total · vllm
Needs two desk boxes: DGX Spark (serve head) + ASUS GX10 (tensor-parallel worker), 128GB each.
● Tested by LocalsOnlyRecorded effort for this run. Wall time and token counts are not a dollar cost; hardware power and utilization also matter.
Tokens per second · see measurement basis
As recorded on this receipt
- Weights are Mia-AiLab EXL3 TR3 4bpw (community quant) with a DFlash2 speculator (k=7), fp8 KV cache; not Z.ai's official weights.
- Server-default sampling (generation_config temp 1.0 / top_p 0.95); the harness passes no sampling overrides. Thinking OFF: enable_thinking=false is passed explicitly (the GLM chat template defaults to ON), matching the Flash-Next thin Terminal-Bench receipt.
- Thin RECEIPT, not the official Terminal-Bench 2.1 card protocol. The official card runs Terminus with thinking on, a 256k context and no turn cap; this run is Harbor terminus-2, thinking off, max_turns=40, agent-timeout x4, timeout x2, k=1, n=1, full 89 tasks. It is not comparable to the official card or to published thinking-on figures.
- Context capped at 100,000 tokens (serve max_model_len). The Flash-Next thin receipt ran with n_ctx 65,536.
- Errors count as 0: 2 of 89 trials (qemu-alpine-ssh, qemu-startup) failed with RuntimeError (Failed to start tmux session) in agent setup, have no verifier reward, and are scored 0, so the headline 39/89 is Harbor's mean. 39 of the 87 trials with a verifier reward passed (44.8%). No AgentTimeoutError.
- First launch (2026-10-02 3:01 PM ET) was aborted after about 30 minutes with no scored trials: after a serve restart the GLM serve emitted degenerate output. The serve was restarted, sanity prompts were clean, and the job was relaunched fresh with the same job id at 3:39 PM ET. Only the relaunch is scored.
- Needs two desk boxes: serve head DGX Spark spark-5c2f (:8888) + tensor-parallel worker ASUS GX10 gx10-e789 (2× GB10, 256GB total). Not a single 128GB desk box.
Run configuration & provenance +
- Context
- 100,000
- Temperature
- Not recorded
- Wall time
- 1521.2 minutes
- Revision
- 25a44fdbf16862a46b7cc9921142c6c81350af2f
- Reasoning
- off
- Agent version
- 2.0.0
- Output token limit
- Not reported
- Dataset revision
- sha256:7d7bdc1cbedad549fc1140404bd4dc45e5fd0ea7c4186773687d177ad3a0699a
- Harness host
- Mac mini · 64GB
- Model host
- 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total
- Input tokens
- 56,154,105
- Output tokens
- 1,231,939
- Cached input tokens
- 0
- Agent turn / step limit
- 40
- Concurrency
- 1
- Weight file size
- 175.72 GB
Throughput measurement: Measurement method not separately reported. This should not be treated as standardized decode-only speed.
Harbor terminus-2 on Kris Mac mini (docker, native images, no platform override) -> vLLM OpenAI-compat [local endpoint] (GLM-5.3-Flash-EXL3 4bpw, TP=2 across DGX Spark spark-5c2f head + ASUS GX10 gx10-e789 worker, fp8 KV, DFlash2 k=7, max_model_len 100000, thinking OFF, max_turns=40, agent-timeout×4, timeout×2, k=1 n=1). Job tb21-mac-dgx-glm53flash-20261002a. RECEIPT — thin parity with the Flash-Next thin Terminal-Bench 2.1 terminus-2 path; NOT thinking-on official compare. Hardware evidence id gb10-2x = DGX Spark spark-5c2f :8888 head + ASUS GX10 gx10-e789 worker (never swap labels).
Run ID: measured-2x-gb10-glm53flash-exl3-4bpw-terminal-bench-2.1-2026-10-03Task explorer
Passed tasks first, then other outcomes grouped by recorded issue. Use the filters to inspect a particular outcome.
All 89 recorded tasks, before filters or pagination. 25 of 89 matching tasks shown below. Recorded issues can also occur on scored tasks.
PassedBuild pmars+
Task ID: build-pmars
Recorded name: build-pmars
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedBuild pov ray+
Task ID: build-pov-ray
Recorded name: build-pov-ray
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedCobol modernization+
Task ID: cobol-modernization
Recorded name: cobol-modernization
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedCode from image+
Task ID: code-from-image
Recorded name: code-from-image
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedConstraints scheduling+
Task ID: constraints-scheduling
Recorded name: constraints-scheduling
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedCount dataset tokens+
Task ID: count-dataset-tokens
Recorded name: count-dataset-tokens
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDistribution search+
Task ID: distribution-search
Recorded name: distribution-search
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedExtract elf+
Task ID: extract-elf
Recorded name: extract-elf
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedFinancial document processor+
Task ID: financial-document-processor
Recorded name: financial-document-processor
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedFix code vulnerability+
Task ID: fix-code-vulnerability
Recorded name: fix-code-vulnerability
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedGit leak recovery+
Task ID: git-leak-recovery
Recorded name: git-leak-recovery
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedGit multibranch+
Task ID: git-multibranch
Recorded name: git-multibranch
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedHeadless terminal+
Task ID: headless-terminal
Recorded name: headless-terminal
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedHf model inference+
Task ID: hf-model-inference
Recorded name: hf-model-inference
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedKv store grpc+
Task ID: kv-store-grpc
Recorded name: kv-store-grpc
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedLargest eigenval+
Task ID: largest-eigenval
Recorded name: largest-eigenval
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedLog summary date ranges+
Task ID: log-summary-date-ranges
Recorded name: log-summary-date-ranges
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedMerge diff arc agi task+
Task ID: merge-diff-arc-agi-task
Recorded name: merge-diff-arc-agi-task
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedModernize scientific stack+
Task ID: modernize-scientific-stack
Recorded name: modernize-scientific-stack
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedMteb retrieve+
Task ID: mteb-retrieve
Recorded name: mteb-retrieve
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedMulti source data merger+
Task ID: multi-source-data-merger
Recorded name: multi-source-data-merger
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedNginx request logging+
Task ID: nginx-request-logging
Recorded name: nginx-request-logging
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedOpenssl selfsigned cert+
Task ID: openssl-selfsigned-cert
Recorded name: openssl-selfsigned-cert
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedOverfull hbox+
Task ID: overfull-hbox
Recorded name: overfull-hbox
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedPassword recovery+
Task ID: password-recovery
Recorded name: password-recovery
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.