LocalsOnly · measured2026-10-28
Terminal-Bench 2.1
Version 2.1 · 89 tasks. Compare only matched task sets and protocols.
GLM-5.3-Flash · NVFP4/MXFP8 CSF-QAD · 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total · Flare
Needs two desk boxes: DGX Spark (serve head) + ASUS GX10 (tensor-parallel worker), 128GB each.
● Tested by LocalsOnlyRecorded effort for this run. Wall time and token counts are not a dollar cost; hardware power and utilization also matter.
Tokens per second · see measurement basis
As recorded on this receipt
- Different engine, quant and context: NVIDIA Flare (commit 47581cc, flare 0.1.0) serving the local-inference-lab NVFP4/MXFP8 CSF-QAD checkpoint @dec48abd at a 1,048,576-token advertised context, vs EXL3 TR3 4bpw at a 100,000-token cap on the vLLM and TensorFold rows. Same suite and harness, so the rows sit together, but deltas are not engine-only.
- Thinking forced ON: enable_thinking=false was sent (same as prior runs) but Flare ignores it; reasoning_content is present in all 1,551 agent steps across the 86 trajectories. The vLLM and TensorFold rows ran thinking off. This is the biggest recipe difference.
- Sampling: server default temperature 1.0 / top_p 0.95; Flare clamps temperature to 0.95-1.0. Prior servers used their own defaults.
- Single RoCE link between the boxes (B12X_ROCE_HCA=rocep1s0f1 pinned in the flare-agent drop-in) because the second link had transport retry errors.
- CUDA 13.2 forward-compat libcuda (~/flare/cuda-compat via LD_LIBRARY_PATH) on host driver 580.173.02 (CUDA 13.0).
- 3 harness errors (RuntimeError, no verifier reward) scored 0 and kept in the 89 denominator: overfull-hbox, qemu-alpine-ssh, qemu-startup. Excluding them gives 62/86 = 72.1% (secondary figure only, not the score).
- Harbor flags 11 errored trials: the 3 RuntimeErrors plus 8 AgentTimeoutErrors (4x agent timeout). 7 timeouts verified at reward 0; torch-pipeline-parallelism timed out but verified at reward 1.
- Endpoint reached through an SSH tunnel (Mac [local address]:18003 -> spark-5c2f [local address]:8003). Wall time includes tunnel and harness overhead; not a speed figure.
- Not comparable to the official Terminal-Bench 2.1 card (Terminus, 256k, thinking on, no turn cap): this run keeps the 40-turn cap to mirror the prior local runs.
Run configuration & provenance +
- Context
- Not recorded
- Temperature
- 1
- Wall time
- 2678.8 minutes
- Revision
- dec48abd33efa73c3bb7c95b74eee10cad34f9be
- Reasoning
- on (forced by Flare)
- Agent version
- Not reported
- Output token limit
- Not reported
- Dataset revision
- sha256:7d7bdc1cbedad549fc1140404bd4dc45e5fd0ea7c4186773687d177ad3a0699a
- Harness host
- Not reported
- Model host
- 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total
- Input tokens
- Not reported
- Output tokens
- Not reported
- Cached input tokens
- Not reported
- Agent turn / step limit
- 40
- Concurrency
- 1
- Weight file size
- 177.75 GB
Throughput measurement: Measurement method not separately reported. This should not be treated as standardized decode-only speed.
Harbor terminus-2 on Kris Mac mini (docker, native images) -> SSH tunnel [local address]:18003 -> NVIDIA Flare OpenAI-compat on spark-5c2f [local address]:8003 (GLM-5.3-Flash NVFP4/MXFP8 CSF-QAD @dec48abd, TP2, MTP draft tokens 3, flare-prep image 47581cc; DGX Spark spark-5c2f head + ASUS GX10 gx10-e789 worker over one RoCE link)
Run ID: measured-2x-gb10-glm53flash-nvfp4-flare-terminal-bench-2.1-2026-10-28Task explorer
Passed tasks first, then other outcomes grouped by recorded issue. Use the filters to inspect a particular outcome.
All 89 recorded tasks, before filters or pagination. 25 of 89 matching tasks shown below. Recorded issues can also occur on scored tasks.
PassedBn fit modify+
Task ID: bn-fit-modify
Recorded name: bn-fit-modify
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedBreak filter js from html+
Task ID: break-filter-js-from-html
Recorded name: break-filter-js-from-html
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedBuild cython ext+
Task ID: build-cython-ext
Recorded name: build-cython-ext
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedBuild pmars+
Task ID: build-pmars
Recorded name: build-pmars
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedBuild pov ray+
Task ID: build-pov-ray
Recorded name: build-pov-ray
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedCaffe cifar 10+
Task ID: caffe-cifar-10
Recorded name: caffe-cifar-10
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedCancel async tasks+
Task ID: cancel-async-tasks
Recorded name: cancel-async-tasks
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedCircuit fibsqrt+
Task ID: circuit-fibsqrt
Recorded name: circuit-fibsqrt
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedCobol modernization+
Task ID: cobol-modernization
Recorded name: cobol-modernization
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedCode from image+
Task ID: code-from-image
Recorded name: code-from-image
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedConfigure git webserver+
Task ID: configure-git-webserver
Recorded name: configure-git-webserver
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedConstraints scheduling+
Task ID: constraints-scheduling
Recorded name: constraints-scheduling
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedCount dataset tokens+
Task ID: count-dataset-tokens
Recorded name: count-dataset-tokens
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedCrack 7z hash+
Task ID: crack-7z-hash
Recorded name: crack-7z-hash
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedCustom memory heap crash+
Task ID: custom-memory-heap-crash
Recorded name: custom-memory-heap-crash
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDb wal recovery+
Task ID: db-wal-recovery
Recorded name: db-wal-recovery
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDistribution search+
Task ID: distribution-search
Recorded name: distribution-search
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDna insert+
Task ID: dna-insert
Recorded name: dna-insert
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedExtract elf+
Task ID: extract-elf
Recorded name: extract-elf
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedFeal differential cryptanalysis+
Task ID: feal-differential-cryptanalysis
Recorded name: feal-differential-cryptanalysis
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedFeal linear cryptanalysis+
Task ID: feal-linear-cryptanalysis
Recorded name: feal-linear-cryptanalysis
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedFix code vulnerability+
Task ID: fix-code-vulnerability
Recorded name: fix-code-vulnerability
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedFix git+
Task ID: fix-git
Recorded name: fix-git
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedFix ocaml gc+
Task ID: fix-ocaml-gc
Recorded name: fix-ocaml-gc
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedGit leak recovery+
Task ID: git-leak-recovery
Recorded name: git-leak-recovery
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.