LocalsOnly · measured2026-09-29
GPQA Diamond
Diamond · version 1.0 · 198 tasks. Compare only matched task sets and protocols.
GLM-5.3-Flash · EXL3 TR3 4bpw · 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total · vllm
Needs two desk boxes: DGX Spark (serve head) + ASUS GX10 (tensor-parallel worker), 128GB each.
● Tested by LocalsOnlyRecorded effort for this run. Wall time and token counts are not a dollar cost; hardware power and utilization also matter.
Tokens per second · see measurement basis
As recorded on this receipt
- Weights are Mia-AiLab EXL3 TR3 4bpw (community quant) with a DFlash2 speculator, not Z.ai's official weights.
- Server-default sampling: no harness overrides (generation_config temp 1.0 / top_p 0.95; thinking on, default reasoning effort).
- Context capped at 100,000 tokens (serve max_model_len). Flash-Next GPQA used 65,536 and DeepSeek V4 Flash 32,768; not binding on prompts of roughly 1–3k tokens.
- Harbor terminus-2 agentic harness (writes /app/answer.txt, max_turns=50), not a one-shot multiple-choice prompt.
- Timeout errors count as 0: 23 AgentTimeoutErrors (3600 s each), of which 20 scored 0 and 3 had already written a correct answer and scored 1. Flash-Next had 14 timeouts and DeepSeek V4 Flash 43 under the same rule.
- Hung trials: tasks 48 and 94 produced no model output at all and sat for the full 3600 s before timing out; both scored 0.
- Z.ai publishes no GPQA Diamond number for GLM-5.3-Flash; no vendor cite is compared here.
- Needs two desk boxes: serve head DGX Spark spark-5c2f + tensor-parallel worker ASUS GX10 gx10-e789 (2× GB10, 256GB total). Not a single 128GB desk box.
Reported category outcomes
Each category keeps its own recorded denominator. Some categories exclude unscored trials. Mean reward, when present, is partial credit.
Run configuration & provenance +
- Context
- 100,000
- Temperature
- Not recorded
- Wall time
- 2312.2 minutes
- Revision
- 25a44fdbf16862a46b7cc9921142c6c81350af2f
- Reasoning
- thinking on · server-default reasoning effort (not passed by harness)
- Agent version
- Not reported
- Output token limit
- Not reported
- Dataset revision
- harbor gpqa-diamond@1.0 (198 tasks) · harbor-datasets 1983ac5c4d43f43cb7a9af9f89c54d09025589ec
- Harness host
- mac
- Model host
- 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total
- Input tokens
- 7,468,272
- Output tokens
- 2,230,401
- Cached input tokens
- 0
- Agent turn / step limit
- 50
- Concurrency
- 1
- Weight file size
- 175.72 GB
Throughput measurement: Harbor job n_output_tokens / job wall seconds (2230401 / 138733 s), the same method as the DeepSeek V4 Flash GPQA receipt. End-to-end agent throughput including the 23 x 3600 s timeout waits; not decode-only.
Harbor terminus-2 on Kris Mac mini (docker) → vLLM OpenAI-compat [local endpoint] (GLM-5.3-Flash-EXL3, thinking on, max_turns=50). Two-box serve: DGX Spark spark-5c2f head + ASUS GX10 gx10-e789 TP worker (2× GB10 · 256GB total). vLLM EXL3 4bpw, TP=2, fp8 KV cache, DFlash2 speculator (k=7), max_model_len 100000, server-default sampling. Job gpqa-mac-dgx-glm53flash-20260928a. Dataset gpqa-diamond@1.0.
Run ID: measured-2x-gb10-glm53flash-exl3-4bpw-gpqa-diamond-1.0-2026-09-29Task explorer
Passed tasks first, then other outcomes grouped by recorded issue. Use the filters to inspect a particular outcome.
All 198 recorded tasks, before filters or pagination. 25 of 198 matching tasks shown below. Recorded issues can also occur on scored tasks.
PassedTask ID 0+
Task ID: 0
Recorded name: 0
Reward: 1
Tokens in / out: 2,343 / 1,106
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 1+
Task ID: 1
Recorded name: 1
Reward: 1
Tokens in / out: 2,364 / 1,652
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 2+
Task ID: 2
Recorded name: 2
Reward: 1
Tokens in / out: 2,332 / 778
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 3+
Task ID: 3
Recorded name: 3
Reward: 1
Tokens in / out: 2,343 / 957
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 4+
Task ID: 4
Recorded name: 4
Reward: 1
Tokens in / out: 2,702 / 1,107
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 5+
Task ID: 5
Recorded name: 5
Reward: 1
Tokens in / out: 2,485 / 1,119
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 6+
Task ID: 6
Recorded name: 6
Reward: 1
Tokens in / out: 2,339 / 2,101
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 7+
Task ID: 7
Recorded name: 7
Reward: 1
Tokens in / out: 2,932 / 7,294
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 9+
Task ID: 9
Recorded name: 9
Reward: 1
Tokens in / out: 3,690 / 967
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 11+
Task ID: 11
Recorded name: 11
Reward: 1
Tokens in / out: 2,315 / 1,698
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 14+
Task ID: 14
Recorded name: 14
Reward: 1
Tokens in / out: 2,656 / 838
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 16+
Task ID: 16
Recorded name: 16
Reward: 1
Tokens in / out: 2,342 / 885
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 18+
Task ID: 18
Recorded name: 18
Reward: 1
Tokens in / out: 3,070 / 17,478
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 19+
Task ID: 19
Recorded name: 19
Reward: 1
Tokens in / out: 2,419 / 1,050
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 20+
Task ID: 20
Recorded name: 20
Reward: 1
Tokens in / out: 2,538 / 5,920
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 21+
Task ID: 21
Recorded name: 21
Reward: 1
Tokens in / out: 2,222 / 4,337
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 22+
Task ID: 22
Recorded name: 22
Reward: 1
Tokens in / out: 2,579 / 15,463
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 23+
Task ID: 23
Recorded name: 23
Reward: 1
Tokens in / out: 2,509 / 7,880
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 24+
Task ID: 24
Recorded name: 24
Reward: 1
Tokens in / out: 2,718 / 5,856
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 25+
Task ID: 25
Recorded name: 25
Reward: 1
Tokens in / out: 2,835 / 9,441
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 26+
Task ID: 26
Recorded name: 26
Reward: 1
Tokens in / out: 2,382 / 996
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 27+
Task ID: 27
Recorded name: 27
Reward: 1
Tokens in / out: 127,878 / 40,677
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 28+
Task ID: 28
Recorded name: 28
Reward: 1
Tokens in / out: 7,514 / 5,832
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 29+
Task ID: 29
Recorded name: 29
Reward: 1
Tokens in / out: 2,302 / 8,789
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedTask ID 30+
Task ID: 30
Recorded name: 30
Reward: 1
Tokens in / out: 2,327 / 9,485
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.