LocalsOnly · measured2026-10-19

GPQA Diamond

Diamond · version 1.0 · 198 tasks. Compare only matched task sets and protocols.

GLM-5.3-Flash · NVFP4 (LibertAIDAI) · 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total · SGLang

Needs two desk boxes: DGX Spark (serve head) + ASUS GX10 (tensor-parallel worker), 128GB each.

● Tested by LocalsOnly

Download weights · LibertAIDAI/GLM-5.3-Flash-NVFP4 @ caca4e6a ↗Serve recipe · randomllama/GLM-5.3-Flash-DFlash2-SGLang-2x-DGX-Spark@18520bfb ↗

WALL-CLOCK TIME33.4 h

INPUT TOKENS3,133,326

Recorded effort for this run. Wall time and token counts are not a dollar cost; hardware power and utilization also matter.

Reported score86.9%172 of 198 tasks passed

Throughput—

Tokens per second · see measurement basis

Concurrency1

As recorded on this receipt

0 infrastructure outcomes are recorded. Failure counts may overlap with task outcomes.

Read before comparing

Engine A/B, three ways: SGLang (this row) 172/198, mean 0.8687; TensorFold 173/198, 0.8737; vLLM 170/198, 0.8586. Same Harbor gpqa-diamond@1.0 config (terminus-2, thinking on, max_turns 50, k=1 n=1, full 198, 100,000-token cap, same two boxes); only job_name, model name and endpoint differ.
Different weights AND a different engine: this row serves LibertAIDAI/GLM-5.3-Flash-NVFP4 @caca4e6a (NVFP4) on SGLang TP=2; the TensorFold and vLLM rows serve the Mia-AiLab EXL3 TR3 4bpw quant. Any gap is not an engine-only effect.
Mostly the same tasks: 165 pass on both SGLang and TensorFold; 7 pass only on SGLang and 8 only on TensorFold. Against vLLM: 165 both, 7 SGLang only, 5 vLLM only. With k=1 n=1 a 1-2 task gap is well within run-to-run noise (standard error about 2.4 points).
Timeout errors count as scored: 16 AgentTimeoutErrors (3600 s each); 14 scored 0 and 2 (tasks 8 and 10) had already written a correct answer and scored 1. 7 of them (tasks 48, 71, 90, 93, 94, 102, 147) recorded no model output. TensorFold had 17 (16 scored 0, 1 scored 1); vLLM had 23 (20 scored 0, 3 scored 1).
Length stops: Harbor does not record finish_reason, so length-terminated calls cannot be counted from the raw files. The largest recorded single call is 14,429 output tokens and the largest prompt 56,585 tokens, far below the 100,000-token context, and no trial log shows a context-exceeded error.
Wall time: 33h23m here (8:15 AM ET Oct 18 to 5:39 PM ET Oct 19) vs 29h53m TensorFold and 38h32m vLLM. Not a speed measurement: timeout waits dominate. SGLang generated 1.67M output tokens (TensorFold 3.40M, vLLM 2.23M).
Speed (observation only, not measured during this run): the single-stream setup probe on 2026-10-16 right after boot (6 runs, 512 forced tokens, same speed.py method) recorded a mean of 21.4 tok/s (median 21.8; runs 19.78-22.86). TensorFold probes about 50 tok/s; vLLM 24.0 (10/9). No new probe was taken for GPQA.
Speculative decoding: DFlash2 drafter incoai/GLM-5.3-Flash-DFlash2 @bf582e4e (8 draft tokens); the recipe authors do not guarantee it is lossless. The drafter is licensed cc-by-nc-nd-4.0: research / evaluation use only, not commercial. Target weights MIT.
bf16 KV cache (the authors' fp8-KV 131k config is not published, so not reproduced); context 100,000 with a 102,720-token KV pool; chunked prefill 4096; max-running-requests 2. Thinking on (same as the TensorFold and vLLM GPQA runs, so the ignored thinking-off flag does not affect this suite). Sampling: server defaults, temperature 1.0 / top_p 0.95.
Raw image inspect stored with the job files: jobs/sglang-glm53/image-inspect-spark.json and image-inspect-asus.json (same image id sha256:a45e44f78f3d on both boxes, tag glm53-flash-dflash:c4d5d45e5-gb10tile, created 2026-10-16) and image-inspect-base-spark.json (base lmsysorg/sglang:glm-5.3-flash-arm64, RepoDigest lmsysorg/sglang@sha256:e88340d6). The inspect labels record SGLANG_BUILD_COMMIT=unknown; the reported version string sglang 0.0.0.dev1+g033446bb05 (not the recipe's pinned c4d5d45e5) and the overlay of PR #36708 glm5_next.py plus a GB10 tilelang retune (recipe rev 18520bfb) come from Model Lab's gate doc.
Weights are LibertAIDAI's community NVFP4 quant, not Z.ai's official FP8. Harbor terminus-2 agentic harness (writes /app/answer.txt), not a one-shot multiple-choice prompt. Needs two desk boxes: serve head DGX Spark spark-5c2f (:8901) + worker ASUS GX10 gx10-e789 (2× GB10, 256GB total). Not a single 128GB desk box.

Reported category outcomes

Each category keeps its own recorded denominator. Some categories exclude unscored trials. Mean reward, when present, is partial credit.

gpqa-diamond

86.9%

172 / 19886.9% passed · mean reward 86.9%

Run configuration & provenance +

Context: 100,000
Temperature: Not recorded
Wall time: 2003.3 minutes
Weight file: LibertAIDAI/GLM-5.3-Flash-NVFP4 @ caca4e6a (multi-file) ↗
Revision: caca4e6a4ebbd66f159d3d2fc256683fd6e27177
Reasoning: thinking on · server-default reasoning effort (not passed by harness)
Agent version: Not reported
Output token limit: Not reported
Dataset revision: harbor gpqa-diamond@1.0 (198 tasks) · harbor-datasets 1983ac5c4d43f43cb7a9af9f89c54d09025589ec
Harness host: mac
Model host: 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total
Input tokens: 3,133,326
Output tokens: 1,666,416
Cached input tokens: 0
Agent turn / step limit: 50
Concurrency: 1
Weight file size: 194.70 GB

Throughput measurement: Not recorded as a run throughput. Job wall time includes 16 x 3600 s timeout waits (7 of them with no model output recorded), so wall-clock throughput is not a speed measurement. See speed_observation for the separate 2026-10-16 single-stream setup probe.

Harbor terminus-2 on Kris Mac mini (docker) -> SGLang TP=2 OpenAI-compat [local endpoint] (LibertAIDAI/GLM-5.3-Flash-NVFP4 @caca4e6a, bf16 KV, context 100000 / KV pool 102720, DFlash2 drafter incoai/GLM-5.3-Flash-DFlash2 @bf582e4e, image glm53-flash-dflash:c4d5d45e5-gb10tile id a45e44f78f3d on lmsysorg/sglang@sha256:e88340d6; DGX Spark spark-5c2f head + ASUS GX10 gx10-e789 worker over ConnectX-7; thinking on, max_turns=50, k=1 n=1, server-default sampling temp 1.0 / top_p 0.95, 3600 s task timeout). Job gpqa-mac-dgx-glm53flash-sglang-nvfp4-20261018a. config.json identical to the TensorFold job gpqa-mac-dgx-glm53flash-tf110-20261013a except job_name, model name and endpoint (diffed). Dataset gpqa-diamond@1.0.

Run ID: measured-2x-gb10-glm53flash-nvfp4-sglang-gpqa-diamond-1.0-2026-10-19

Tasks

Task explorer

Passed tasks first, then other outcomes grouped by recorded issue. Use the filters to inspect a particular outcome.

198 task records

172 Passed26 Not passed

All 198 recorded tasks, before filters or pagination. 25 of 198 matching tasks shown below. Recorded issues can also occur on scored tasks.

PassedTask ID 0+

Task ID: 0
Recorded name: 0

Reward: 1

Tokens in / out: 2,325 / 1,269

Recorded issue: No failure explanation in this task record.