LocalsOnly · measured2026-10-15
IFBench
Loose prompt grading · 300 tasks. Compare only matched task sets and protocols.
GLM-5.3-Flash · EXL3 TR3 4bpw · 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total · TensorFold
Needs two desk boxes: DGX Spark (serve head) + ASUS GX10 (tensor-parallel worker), 128GB each.
● Tested by LocalsOnlyTokens per second · see measurement basis
As recorded on this receipt
- Engine A/B: this row and the vLLM IFBench row (2026-10-01, loose 0.6767, 203/300) use the same Mia-AiLab EXL3 TR3 4bpw weights file @25a44fdb and a byte-identical generate.py: single-turn IFBench_test, 300 prompts, temp 0, thinking on, reasoning stripped, max_tokens 90,000, 100,000-token cap, one request at a time, same two boxes. The serving stack changed: TensorFold v1.10 instead of vLLM.
- Not a pure engine comparison: TensorFold serves the dense/shared weights at 4-bit, while the vLLM serve ran stock recipe defaults with the dense-FP8 opt-in off (BF16 dense). top_p stays at the server default on both and was not separately verified to match.
- Quality: TensorFold is slightly lower. Loose prompt-level 199/300 (66.3%) vs vLLM 203/300 (67.7%), a gap of 4 prompts that is inside run-to-run noise (standard error about 2.7 points; 182 prompts pass on both, 17 only on TensorFold, 21 only on vLLM). Strict prompt-level 162/300 (54.0%) vs 176/300 (58.7%), a 14-prompt gap larger than one standard error (about 2.9 points; 151 both, 11 TensorFold only, 25 vLLM only). Instruction-level loose 236/344 (68.6%) vs 240/344 (69.8%); strict 197/344 (57.3%) vs 212/344 (61.6%). k=1 n=1, one run each.
- Empty answers count as failures: 27 responses (finish_reason length at 90,000 tokens, thinking used the whole budget) vs 18 on vLLM; 11 prompts are empty on both. Among prompts that returned an answer, loose pass is 199/273 (72.9%) on TensorFold vs 203/282 (72.0%) on vLLM, so the loose gap is accounted for by the 9 extra empties.
- Run integrity: TensorFold 17.46 h of summed per-item generation time as one uninterrupted serial segment, 0 HTTP errors, 0 timeouts, every request succeeded on the first attempt. The vLLM run had one request timeout (key 191) during a 9/30 DGX Spark / ASUS GX10 outage and was resumed from the checkpoint; its wall was 38.0 h including about 3.5 h lost.
- Time: summed per-item generation time 17.46 h on TensorFold vs 33.23 h on vLLM (same script, one request at a time), about 1.9x less; on the 266 prompts both engines answered it is 10.81 h vs 20.01 h (1.85x). This is an observation from two single runs on different days with different token counts (TensorFold generated 5.23M completion tokens vs 4.44M), not a controlled speed test.
- Speed: per-item decode over answered prompts (completion tokens / request seconds, prefill included, median) was 66.2 tok/s on TensorFold vs 32.1 on vLLM, computed the same way from each job's checkpoint; long-reasoning rates differ from the short single-stream probe. The controlled single-stream numbers remain the 10/9 speed test (TensorFold 50.4, vLLM 24.0 tok/s) and a post-run TensorFold probe on 10/15 (thinking off 49.8/49.2/49.6 tok/s; thinking on 51.0); vLLM was not re-probed.
- Weights are Mia-AiLab EXL3 TR3 4bpw (community quant) with a DFlash2 speculator, fp8 KV cache; not Z.ai's official weights. Single-turn IFBench_test, no Harbor; headline is prompt-level loose accuracy, as on the vLLM and Flash-Next rows. Not the published IFBench card protocol; Z.ai publishes no IFBench figure for GLM-5.3-Flash.
- Needs two desk boxes: serve head DGX Spark spark-5c2f (:8888) + worker ASUS GX10 gx10-e789 (2× GB10, 256GB total). Not a single 128GB desk box.
Run configuration & provenance +
- Context
- 100,000
- Temperature
- 0
- Wall time
- Not recorded
- Revision
- 25a44fdbf16862a46b7cc9921142c6c81350af2f
- Reasoning
- Not reported
- Agent version
- Not reported
- Output token limit
- 90,000
- Dataset revision
- IFBench_test single-turn n=300
- Harness host
- mac
- Model host
- 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total
- Input tokens
- Not reported
- Output tokens
- Not reported
- Cached input tokens
- Not reported
- Agent turn / step limit
- Not reported
- Concurrency
- 1
- Weight file size
- 175.72 GB
Throughput measurement: Measurement method not separately reported. This should not be treated as standardized decode-only speed.
IFBench_test single-turn n=300 (no multi-turn, no Harbor). Generate on Kris Mac mini -> TensorFold v1.10 OpenAI-compat [local endpoint] (GLM-5.3-Flash-EXL3 TR3 4bpw, dense/shared 4-bit, fp8 KV, DFlash2 bf582e4e, context 100000, image v0.6.0-a1897d591f70, recipe 549fdc5; DGX Spark spark-5c2f head + ASUS GX10 gx10-e789 worker over ConnectX-7). temp=0 (request), thinking ON, reasoning stripped before eval, max_tokens 90000, one request at a time. Headline = prompt-level loose accuracy, same as the vLLM row. Job ifbench-mac-dgx-glm53flash-tf110-20261014a. MEASURED local IFBench; NOT the published frontier IFBench card row.
Run ID: measured-2x-gb10-glm53flash-exl3-4bpw-tensorfold-ifbench-2026-10-15Task explorer
Passed tasks first, then other outcomes grouped by recorded issue. Use the filters to inspect a particular outcome.
No task records in this receipt.
The headline or summary is available above. Individual outcomes have not been provided.
Explore test coverage →