LocalsOnly · measured2026-10-20
IFBench
Loose prompt grading · 300 tasks. Compare only matched task sets and protocols.
GLM-5.3-Flash · NVFP4 (LibertAIDAI) · 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total · SGLang
Needs two desk boxes: DGX Spark (serve head) + ASUS GX10 (tensor-parallel worker), 128GB each.
● Tested by LocalsOnlyTokens per second · see measurement basis
As recorded on this receipt
- Engine A/B, three ways: SGLang (this row) loose prompt-level 205/300 (68.3%), strict 168/300 (56.0%); TensorFold 199/300 and 162/300; vLLM 203/300 and 176/300. Same single-turn IFBench_test script and settings: 300 prompts, temp 0, thinking on, reasoning stripped, max_tokens 90,000, 100,000-token cap, one request at a time, same two boxes; generate.py differs only in endpoint and model name.
- Different weights AND a different engine: this row serves LibertAIDAI/GLM-5.3-Flash-NVFP4 @caca4e6a (NVFP4) on SGLang TP=2; the TensorFold and vLLM rows serve the Mia-AiLab EXL3 TR3 4bpw quant. Any gap is not an engine-only effect.
- Quality: SGLang is level with vLLM and slightly above TensorFold on loose (205 vs 203 vs 199; standard error about 2.7 points, so within noise), and between them on strict (168 vs 176 vLLM, 162 TensorFold). Loose overlap: 173 pass on both SGLang and TensorFold, 32 SGLang only, 26 TensorFold only; vs vLLM 176 both, 29 SGLang only, 27 vLLM only. Strict vs TensorFold 146/22/16; vs vLLM 149/19/27. Instruction-level loose 241/344 (70.1%), strict 202/344 (58.7%) vs TensorFold 236 / 197 and vLLM 240 / 212. k=1 n=1, one run each.
- Empty answers count as failures: 12 responses (finish_reason length at 90,000 tokens, thinking used the whole budget; keys 7, 63, 88, 191, 200, 201, 203, 206, 207, 212, 267, 272) vs 27 on TensorFold and 18 on vLLM. 7 are empty on both SGLang and TensorFold. Among prompts that returned an answer, loose pass is 205/288 (71.2%) vs TensorFold 199/273 and vLLM 203/282.
- Run integrity: one uninterrupted serial segment, 26.90 h of summed per-item generation time, wall 26h54m (5:39 PM ET Oct 19 to 8:33 PM ET Oct 20), 0 HTTP errors, 0 timeouts, every request succeeded on the first attempt, no resume. Concurrency: the client sends one request at a time (serial loop); max-running-requests 2 is only the server setting.
- Time and speed (observations only, not a controlled speed test): summed generation 26.90 h vs TensorFold 17.46 h and vLLM 33.23 h; SGLang generated 3.43M completion tokens (TensorFold 5.23M, vLLM 4.44M). Per-item decode over answered prompts (completion tokens / request seconds, prefill included, median) was 32.0 tok/s vs TensorFold 66.2 and vLLM 32.1, computed the same way. The only single-stream probe on this serve is the 2026-10-16 setup test (mean 21.4 tok/s, 512 forced tokens), not measured during this run.
- Speculative decoding: DFlash2 drafter incoai/GLM-5.3-Flash-DFlash2 @bf582e4e (8 draft tokens); the recipe authors do not guarantee it is lossless. The drafter is licensed cc-by-nc-nd-4.0: research / evaluation use only, not commercial. Target weights MIT.
- bf16 KV cache (the authors' fp8-KV 131k config is not published, so not reproduced); context 100,000 with a 102,720-token KV pool; chunked prefill 4096; max-running-requests 2. Thinking on: reasoning arrived in reasoning_content on all 300 responses and was stripped before scoring, so the ignored thinking-off flag does not affect this suite. temperature 0 sent by the client; top_p server default 0.95.
- Raw image inspect stored with the job files: jobs/sglang-glm53/image-inspect-spark.json and image-inspect-asus.json (same image id sha256:a45e44f78f3d on both boxes, tag glm53-flash-dflash:c4d5d45e5-gb10tile, created 2026-10-16) and image-inspect-base-spark.json (base lmsysorg/sglang:glm-5.3-flash-arm64, RepoDigest lmsysorg/sglang@sha256:e88340d6). The inspect labels record SGLANG_BUILD_COMMIT=unknown; the reported version string sglang 0.0.0.dev1+g033446bb05 (not the recipe's pinned c4d5d45e5) and the overlay of PR #36708 glm5_next.py plus a GB10 tilelang retune (recipe rev 18520bfb) come from Model Lab's gate doc.
- Weights are LibertAIDAI's community NVFP4 quant, not Z.ai's official FP8. Single-turn IFBench_test, no Harbor; headline is prompt-level loose accuracy, as on the TensorFold and vLLM rows. Not the published IFBench card protocol; Z.ai publishes no IFBench figure for GLM-5.3-Flash. Needs two desk boxes: serve head DGX Spark spark-5c2f (:8901) + worker ASUS GX10 gx10-e789 (2× GB10, 256GB total). Not a single 128GB desk box.
Run configuration & provenance +
- Context
- 100,000
- Temperature
- 0
- Wall time
- Not recorded
- Revision
- caca4e6a4ebbd66f159d3d2fc256683fd6e27177
- Reasoning
- Not reported
- Agent version
- Not reported
- Output token limit
- 90,000
- Dataset revision
- IFBench_test single-turn n=300
- Harness host
- mac
- Model host
- 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total
- Input tokens
- Not reported
- Output tokens
- Not reported
- Cached input tokens
- Not reported
- Agent turn / step limit
- Not reported
- Concurrency
- 1
- Weight file size
- 194.70 GB
Throughput measurement: Measurement method not separately reported. This should not be treated as standardized decode-only speed.
IFBench_test single-turn n=300 (no multi-turn, no Harbor). Generate on Kris Mac mini -> SGLang TP=2 OpenAI-compat [local endpoint] (LibertAIDAI/GLM-5.3-Flash-NVFP4 @caca4e6a, bf16 KV, context 100000 / KV pool 102720, max-running-requests 2, DFlash2 drafter incoai/GLM-5.3-Flash-DFlash2 @bf582e4e, image glm53-flash-dflash:c4d5d45e5-gb10tile id a45e44f78f3d on lmsysorg/sglang@sha256:e88340d6; DGX Spark spark-5c2f head + ASUS GX10 gx10-e789 worker over ConnectX-7). temp=0 (request), thinking ON, reasoning stripped before eval, max_tokens 90000, client sends one request at a time (serial loop, no threads). Headline = prompt-level loose accuracy, same as the TensorFold and vLLM rows. Job ifbench-mac-dgx-glm53flash-sglang-nvfp4-20261018a. MEASURED local IFBench; NOT the published frontier IFBench card row.
Run ID: measured-2x-gb10-glm53flash-nvfp4-sglang-ifbench-2026-10-20Task explorer
Passed tasks first, then other outcomes grouped by recorded issue. Use the filters to inspect a particular outcome.
No task records in this receipt.
The headline or summary is available above. Individual outcomes have not been provided.
Explore test coverage →