LocalsOnly · measured2026-10-14
GPQA Diamond
Diamond · version 1.0 · 198 tasks. Compare only matched task sets and protocols.
GLM-5.3-Flash · EXL3 TR3 4bpw · 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total · TensorFold
Needs two desk boxes: DGX Spark (serve head) + ASUS GX10 (tensor-parallel worker), 128GB each.
● Tested by LocalsOnlyRecorded effort for this run. Wall time and token counts are not a dollar cost; hardware power and utilization also matter.
Tokens per second · see measurement basis
As recorded on this receipt
- Engine A/B: this row and the vLLM GPQA Diamond row (2026-09-29, mean 0.8586, 170/198) use the same Mia-AiLab EXL3 TR3 4bpw weights file @25a44fdb and an identical Harbor config (only job_name differs): terminus-2, thinking on, max_turns 50, k=1 n=1, full 198, 100,000-token cap, same two boxes. The serving stack changed: TensorFold v1.10 instead of vLLM.
- Not a pure engine comparison: TensorFold serves the dense/shared weights at 4-bit, while the vLLM run kept them at BF16. TensorFold's server-default sampling was not separately verified to match vLLM's.
- Mostly the same tasks: 166 tasks pass on both; 7 pass only on TensorFold and 4 only on vLLM. With k=1 n=1 the 3-task gap (173 vs 170) is within run-to-run noise (standard error about 2.4 points).
- Timeout errors count as scored: 17 AgentTimeoutErrors (3600 s each); 16 scored 0 and 1 had already written a correct answer and scored 1. 6 of them (tasks 25, 48, 71, 94, 102, 121) hung with no model output. The vLLM run had 23 (20 scored 0, 3 scored 1).
- Wall time: 29h53m here vs 38h32m for the vLLM run. Both were clean serial runs (0 retries, no resume), but the gap is not a speed measurement: timeout waits dominate and TensorFold generated more tokens (3.40M vs 2.23M output).
- Speed: a separate post-run single-stream probe on 10/14 (same speed.py prompts and settings as the 10/9 speed test, thinking off, 512-token generations, 3 runs) recorded TensorFold 49.7/49.7/49.8 tok/s, in line with the 10/9 TensorFold 50.4. The vLLM side was not re-probed; its 10/9 figure was 24.0 tok/s (mean of 23.2, 25.8, 23.1). Not measured during this GPQA run.
- Weights are Mia-AiLab EXL3 TR3 4bpw (community quant) with a DFlash2 speculator, fp8 KV cache; not Z.ai's official weights. Harbor terminus-2 agentic harness (writes /app/answer.txt), not a one-shot multiple-choice prompt.
- Needs two desk boxes: serve head DGX Spark spark-5c2f (:8888) + worker ASUS GX10 gx10-e789 (2× GB10, 256GB total). Not a single 128GB desk box.
Reported category outcomes
Each category keeps its own recorded denominator. Some categories exclude unscored trials. Mean reward, when present, is partial credit.
Run configuration & provenance +
- Context
- 100,000
- Temperature
- Not recorded
- Wall time
- 1793.1 minutes
- Revision
- 25a44fdbf16862a46b7cc9921142c6c81350af2f
- Reasoning
- thinking on · server-default reasoning effort (not passed by harness)
- Agent version
- Not reported
- Output token limit
- Not reported
- Dataset revision
- harbor gpqa-diamond@1.0 (198 tasks) · harbor-datasets 1983ac5c4d43f43cb7a9af9f89c54d09025589ec
- Harness host
- mac
- Model host
- 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total
- Input tokens
- 13,174,753
- Output tokens
- 3,404,783
- Cached input tokens
- 0
- Agent turn / step limit
- 50
- Concurrency
- 1
- Weight file size
- 175.72 GB
Throughput measurement: Not recorded as a run throughput. Job wall time includes 17 x 3600 s timeout waits (6 of them hung with no model output), so wall-clock throughput is not a speed measurement. See speed_probe for a separate post-run single-stream probe.
Harbor terminus-2 on Kris Mac mini (docker) -> TensorFold v1.10 OpenAI-compat [local endpoint] (GLM-5.3-Flash-EXL3 TR3 4bpw, dense/shared 4-bit, fp8 KV, DFlash2 bf582e4e, context 100000, image v0.6.0-a1897d591f70, recipe 549fdc5; DGX Spark spark-5c2f head + ASUS GX10 gx10-e789 worker over ConnectX-7; thinking on, max_turns=50, k=1 n=1, server-default sampling, 3600 s task timeout). Job gpqa-mac-dgx-glm53flash-tf110-20261013a. config.json identical to the vLLM job gpqa-mac-dgx-glm53flash-20260928a except job_name (diffed). Dataset gpqa-diamond@1.0.
Run ID: measured-2x-gb10-glm53flash-exl3-4bpw-tensorfold-gpqa-diamond-1.0-2026-10-14Task explorer
Passed tasks first, then other outcomes grouped by recorded issue. Use the filters to inspect a particular outcome.
All 198 recorded tasks, before filters or pagination. 25 of 198 matching tasks shown below. Recorded issues can also occur on scored tasks.
PassedTask ID 0+
Task ID: 0
Recorded name: 0
Reward: 1
Tokens in / out: 2,282 / 1,079
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 1+
Task ID: 1
Recorded name: 1
Reward: 1
Tokens in / out: 2,354 / 3,860
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 2+
Task ID: 2
Recorded name: 2
Reward: 1
Tokens in / out: 2,370 / 791
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 3+
Task ID: 3
Recorded name: 3
Reward: 1
Tokens in / out: 2,259 / 1,201
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 4+
Task ID: 4
Recorded name: 4
Reward: 1
Tokens in / out: 8,321 / 2,046
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 5+
Task ID: 5
Recorded name: 5
Reward: 1
Tokens in / out: 2,478 / 2,709
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 6+
Task ID: 6
Recorded name: 6
Reward: 1
Tokens in / out: 2,634 / 3,298
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 7+
Task ID: 7
Recorded name: 7
Reward: 1
Tokens in / out: 2,933 / 17,456
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 9+
Task ID: 9
Recorded name: 9
Reward: 1
Tokens in / out: 2,307 / 1,596
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 10+
Task ID: 10
Recorded name: 10
Reward: 1
Tokens in / out: 1,191,168 / 67,378
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 11+
Task ID: 11
Recorded name: 11
Reward: 1
Tokens in / out: 4,300 / 1,522
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 14+
Task ID: 14
Recorded name: 14
Reward: 1
Tokens in / out: 2,708 / 1,064
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 16+
Task ID: 16
Recorded name: 16
Reward: 1
Tokens in / out: 2,476 / 1,498
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 18+
Task ID: 18
Recorded name: 18
Reward: 1
Tokens in / out: 3,176 / 20,373
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 19+
Task ID: 19
Recorded name: 19
Reward: 1
Tokens in / out: 2,360 / 915
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 20+
Task ID: 20
Recorded name: 20
Reward: 1
Tokens in / out: 2,640 / 11,862
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 21+
Task ID: 21
Recorded name: 21
Reward: 1
Tokens in / out: 2,238 / 7,747
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 22+
Task ID: 22
Recorded name: 22
Reward: 1
Tokens in / out: 2,563 / 13,036
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 23+
Task ID: 23
Recorded name: 23
Reward: 1
Tokens in / out: 2,492 / 27,551
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 24+
Task ID: 24
Recorded name: 24
Reward: 1
Tokens in / out: 2,742 / 4,817
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 26+
Task ID: 26
Recorded name: 26
Reward: 1
Tokens in / out: 2,350 / 1,078
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 27+
Task ID: 27
Recorded name: 27
Reward: 1
Tokens in / out: 76,459 / 74,057
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 28+
Task ID: 28
Recorded name: 28
Reward: 1
Tokens in / out: 7,676 / 4,671
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 30+
Task ID: 30
Recorded name: 30
Reward: 1
Tokens in / out: 2,216 / 26,346
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 31+
Task ID: 31
Recorded name: 31
Reward: 1
Tokens in / out: 2,665 / 5,964
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.