Published full-model reference, then measured local recipes.
| Recipe | GPQA | Terminal | IFBench | Auto | Verified |
|---|---|---|---|---|---|
| Z.ai reference / publishedPublished reference | — | 84.3 | — | 48.8 | — |
| 87.4 | 43.8 | 66.3 | 48.7 | 71.0 | |
| 85.9 | 43.8 | 67.7 | 46.3 | 69.0 | |
| 86.9 | DNQ† | 68.3 | 40.3 | 60.0 |
The published full-model reference is the top row — or the dashed gold tick on Graphic — not rank #1. A blank cell, missing tick, or missing bead has no recorded cite. Scores are never invented.
† DNQ (did not qualify): the SGLang NVFP4 + DFlash2 Terminal-Bench 2.1 run was stopped by choice at 32 of 89 trials, because a full run would take several more days (full runs took about 25.4 h on vLLM and 17.8 h on TensorFold). Partial results are not a score and are not ranked. Hardware: DGX Spark head + ASUS GX10 worker.
AutomationBench · public 600 48.8% (Z.ai): Protocols differ from the LocalsOnly GLM-5.3-Flash run (official FP8 vs EXL3 4bpw; full set vs 300-task subset; PR #13 applied vs not; Z.ai agent, toolset and step cap unpublished) — do not rank one against the other.
ON THIS SETUP · MEASURED BY LOCALSONLY
Local 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total · TensorFold
Needs two desk boxes: DGX Spark (serve head) + ASUS GX10 (tensor-parallel worker), 128GB each.
500 human-verified repository issues. Public-comparable headline for repo software engineering. Agent and harness belong on the cite.
swe-bench-verified · 100 tasks
Repository tests verify issue resolution. Task subsets should only be compared when their selection and evaluation protocol match.
Multi-step tasks in a terminal environment.
Version 2.1 · 89 tasks
The task must pass its verification in a terminal environment. Tools, harness, context budget and execution failures all contribute to the result.
Follow verifiable instruction constraints.
Loose prompt grading · 300 tasks
This receipt uses loose prompt-level grading. It is not the strict score, and the local and published protocols are not matched.
Sales, marketing, operations, support, finance and HR workflows.
Local evaluation · 300 tasks
A workflow is complete only when every assertion passes. Mean reward gives partial credit for progress; it is not the share of jobs finished. Task sets and evaluation budgets can differ between receipts.
Mean reward 82.8% · partial credit, not completed jobs.
Graduate-level science questions.
Diamond · version 1.0 · 198 tasks
Credit comes from answering the science question correctly. Multiple-choice accuracy does not establish research or experimental competence.
BEYOND THE HEADLINE
Each block represents one recorded task, in receipt order from left to right, then top to bottom. Fill shows the outcome; work-category colors are not used here.
Tested by LocalsOnly · EXL3 TR3 4bpw · 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total. Partial progress still requires further work before a task is complete.
Inspect individual task outcomes →Completed the recorded checks.
7 / 7 assertions · 14 steps
Follow this exact task ↗sales.negative_selection · ID 2
sales · 7 / 7 assertions · 14 steps / 25 tools · passed
Did not complete the recorded checks. The record does not establish a cause.
2 / 3 assertions · 16 steps
Follow this exact task ↗sales.multi_hop_lookup · ID 1
sales · 2 / 3 assertions · 16 steps / 32 tools · not passed
Context, sampling, harness settings and dataset versions can vary by benchmark. Each evidence link above contains its run-specific configuration.