LocalsOnly · measured2026-10-18

AutomationBench · local 300*

Local evaluation · 300 tasks. Task sets and protocols differ. A local sample is not established as a subset of the published public set; scores remain separate.

GLM-5.3-Flash · NVFP4 (LibertAIDAI) · 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total · SGLang

Needs two desk boxes: DGX Spark (serve head) + ASUS GX10 (tensor-parallel worker), 128GB each.

● Tested by LocalsOnly

Download weights · LibertAIDAI/GLM-5.3-Flash-NVFP4 @ caca4e6a ↗Serve recipe · randomllama/GLM-5.3-Flash-DFlash2-SGLang-2x-DGX-Spark@18520bfb ↗

WALL-CLOCK TIME37.5 h

INPUT TOKENS113,375,607

Recorded effort for this run. Wall time and token counts are not a dollar cost; hardware power and utilization also matter.

Fully completed40.3%121 of 300 tasks fully completed

Mean reward76.3%

Partial credit. Not completion.

Concurrency1

As recorded on this receipt

0 infrastructure outcomes are recorded. Failure counts may overlap with task outcomes.

Read before comparing

Different weights AND a different engine: this row serves LibertAIDAI/GLM-5.3-Flash-NVFP4 (NVFP4, 4-bit) on SGLang; the TensorFold and vLLM rows serve the Mia-AiLab EXL3 TR3 4bpw quant. Any score gap is not a pure engine effect.
Fully completed 121/300 (40.3%), mean reward 0.763. Same 300-task seed, harness commit, flags and 100k context cap as the TensorFold row (146/300, mean 0.828) and the vLLM row (139/300, mean 0.824). Overlap with TensorFold: 94 passed by both, 27 SGLang only, 52 TensorFold only. Overlap with vLLM: 95 both, 26 SGLang only, 44 vLLM only. The mean-reward gap (-0.065 vs TensorFold, -0.061 vs vLLM) is about three standard errors of the difference (unpaired, one run each, sampling temperature 1.0), so it is unlikely to be sampling noise alone.
Run integrity: 6 tasks (finance.invoice_dispute, finance.overdue_invoice_followup, hr.offboarding_multi_system, marketing.voice_of_customer, sales.email_zoom_fuzzy, support.gorgias_refund_processing) are flagged aborted by the harness: a model call hit finish_reason=length in a runaway generation (the tasks total 74k-99k output tokens; one task spent 98,883 in a single call), cutting the rollout off mid-work. They were not re-run and are scored as exported (reward 0 for four, 0.077 and 0.556 for the others). 11 tasks in total include a length-terminated model call (harness stop_reasons). TensorFold had 0 and vLLM had 0.
17 tasks hit the 50-step cap (TensorFold 1, vLLM 0); 19 tasks scored zero (TensorFold 7, vLLM 6); average 19.4 turns per task (TensorFold 14.6, vLLM 14.3); 37.1 tool calls per task from the export (the checkpoint metrics total 11,168, 37.2 per task). No exceptions or empty responses.
Speed test (separate, 2026-10-16 right after boot, not measured during this AutomationBench run): single stream, 512 forced tokens, 6 runs on the same speed.py method as the TensorFold and vLLM probes (those banked 3 runs): SGLang 19.78, 19.81, 22.02, 22.71, 21.51, 22.86 tok/s (mean 21.4; median 21.8; the first two runs were slower), time to first token 0.37-0.42 s. TensorFold probes: 50.4 (10/9), 49.7 (10/13), 49.5 (10/15); vLLM 24.0 (10/9). Single stream only; no multi-stream SGLang probe is banked. Speed here is a setup test, not a run throughput.
Raw image inspect stored with the job files: jobs/sglang-glm53/image-inspect-spark.json and image-inspect-asus.json (same image id sha256:a45e44f78f3d on both boxes, tag glm53-flash-dflash:c4d5d45e5-gb10tile, created 2026-10-16) and image-inspect-base-spark.json (base lmsysorg/sglang:glm-5.3-flash-arm64, RepoDigest lmsysorg/sglang@sha256:e88340d6). The inspect labels record SGLANG_BUILD_COMMIT=unknown; the reported version string sglang 0.0.0.dev1+g033446bb05 (not the recipe's pinned c4d5d45e5) and the overlay of PR #36708 glm5_next.py plus a GB10 tilelang retune (recipe rev 18520bfb) come from Model Lab's gate doc.
Speculative decoding: the DFlash2 drafter (incoai/GLM-5.3-Flash-DFlash2 @bf582e4e, 8 draft tokens) is on; the recipe authors do not guarantee it is lossless (they saw 1 of 5 outputs diverge at a formatting token). The drafter is licensed cc-by-nc-nd-4.0: research / evaluation use only, not commercial. The target weights are MIT.
bf16 KV cache (the authors' fp8-KV 131k config is not published, so not reproduced); context 100,000 with a 102,720-token KV pool (boot log); chunked prefill 4096; max-running-requests 2; flashinfer_cutlass MoE, dsa/tilelang attention with the draft worker falling back to flashinfer attention (boot log).
Thinking on per server default (reasoning_effort null; the chat template defaults to max effort). SGLang with this recipe does not honor enable_thinking=false: a validation call with thinking off failed (reasoning leaked into content), so any thinking-off test on this stack is not comparable. Sampling: server defaults, temperature 1.0 / top_p 0.95 (boot log: default chat sampling params from model generation config).
300-task subset: frozen 50×6 Zapier seed (first 50 tasks per domain), not the full AutomationBench public set. Harness is AutomationBench v1.0.6 @ 4a8e106 + local pause/continue patch, without PR #13; Z.ai's 48.8 includes that fix.
Weights are LibertAIDAI's community NVFP4 quant (MIT), not Z.ai's official FP8. Needs two desk boxes: serve head DGX Spark spark-5c2f (:8901) + worker ASUS GX10 gx10-e789 (2× GB10, 256GB total). Not a single 128GB desk box.

Reported category outcomes

Each category keeps its own recorded denominator. Some categories exclude unscored trials. Mean reward, when present, is partial credit.

sales

34.0%

17 / 5034.0% passed · mean reward 62.4%

marketing

48.0%

24 / 5048.0% passed · mean reward 86.1%

operations

52.0%

26 / 5052.0% passed · mean reward 79.8%

support

32.0%

16 / 5032.0% passed · mean reward 82.6%

finance

40.0%

20 / 5040.0% passed · mean reward 75.0%

36.0%

18 / 5036.0% passed · mean reward 71.8%

Run configuration & provenance +

Context: 100,000
Temperature: Not recorded
Wall time: 2252.5 minutes
Weight file: LibertAIDAI/GLM-5.3-Flash-NVFP4 @ caca4e6a (multi-file) ↗
Revision: caca4e6a4ebbd66f159d3d2fc256683fd6e27177
Reasoning: thinking on · server-default reasoning effort (harness reasoning_effort=null; chat template default max); enable_thinking=false is not honored by this SGLang stack
Agent version: Not reported
Output token limit: Not reported
Dataset revision: 1.0.6 · 4a8e1061254004d9dac807054eed33fad7d1ff14
Task selection: First 50 tasks in source order per department; six departments, 300 tasks.
Harness host: mac
Model host: 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total
Input tokens: 113,375,607
Output tokens: 3,774,170
Cached input tokens: 0
Agent turn / step limit: 50
Concurrency: 1
Weight file size: 194.70 GB

Throughput measurement: Not recorded as a run throughput on purpose. Only a SETUP-TEST single-stream probe (2026-10-16, 6 runs, mean 21.4 tok/s) exists; see disclosures and engine_ab.speed_setup_test. Export totals (3,774,170 output tokens / 135,082 s model time) are end-to-end agent numbers, inflated by 6 runaway generations, and are not used as a speed claim.

Native AutomationBench toolset=api, max_steps=50, serial (max_concurrent=1), save-every=1, frozen 50×6 Zapier seed (300), benchmark 1.0.6, AutomationBench git 4a8e106 + local pause/continue patch; PR #13 NOT applied. Kris Mac mini → SGLang OpenAI-compat [local endpoint] (served id glm-5.3-flash-dflash2, context 100000). Two-box serve: DGX Spark spark-5c2f head + ASUS GX10 gx10-e789 worker (2× GB10 · 256GB total). SGLang TP=2 with LibertAIDAI/GLM-5.3-Flash-NVFP4 @caca4e6a, DFlash2 drafter bf582e4e (8 draft tokens), bf16 KV, chunked prefill 4096, max-running-requests 2. Server-default sampling, thinking on.

Run ID: measured-2x-gb10-glm53flash-nvfp4-sglang-automationbench-full300-2026-10-18

Tasks

Task explorer

Passed tasks first, then other outcomes grouped by recorded issue. Use the filters to inspect a particular outcome.

300 task records

121 Passed179 Not passed

All 300 recorded tasks, before filters or pagination. 25 of 300 matching tasks shown below. Recorded issues can also occur on scored tasks.

PassedNegative selection · sales+

Task ID: 2
Recorded name: sales.negative_selection

Reward: 1

Assertions: 7 / 7

Steps: 14

Tool calls: 29

Model calls: 14

Tokens in / out: 339,846 / 5,917

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedFormat ambiguity · sales+

Task ID: 5
Recorded name: sales.format_ambiguity

Reward: 1

Assertions: 1 / 1

Steps: 12

Tool calls: 31

Model calls: 12

Tokens in / out: 154,205 / 9,819

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedImplicit rules · sales+

Task ID: 6
Recorded name: sales.implicit_rules

Reward: 1

Assertions: 4 / 4

Steps: 22

Tool calls: 30

Model calls: 22

Tokens in / out: 430,379 / 5,722

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedCross reference validation · sales+

Task ID: 7
Recorded name: sales.cross_reference_validation

Reward: 1

Assertions: 4 / 4

Steps: 20

Tool calls: 34

Model calls: 20

Tokens in / out: 373,808 / 8,428

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedSoft matching · sales+

Task ID: 8
Recorded name: sales.soft_matching

Reward: 1

Assertions: 3 / 3

Steps: 15

Tool calls: 20

Model calls: 15

Tokens in / out: 253,991 / 3,530

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedCalendar crm meeting · sales+

Task ID: 11
Recorded name: sales.calendar_crm_meeting

Reward: 1

Assertions: 3 / 3

Steps: 13

Tool calls: 19

Model calls: 13

Tokens in / out: 107,249 / 5,561

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedFive level conditional · sales+

Task ID: 13
Recorded name: sales.five_level_conditional

Reward: 1

Assertions: 5 / 5

Steps: 14

Tool calls: 20

Model calls: 14

Tokens in / out: 183,595 / 6,022

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedZoom calendar conflict · sales+

Task ID: 15
Recorded name: sales.zoom_calendar_conflict

Reward: 1

Assertions: 4 / 4

Steps: 12

Tool calls: 18

Model calls: 12

Tokens in / out: 144,192 / 3,733

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedCreate important draft · sales+

Task ID: 17
Recorded name: sales.create_important_draft

Reward: 1

Assertions: 2 / 2

Steps: 35

Tool calls: 58

Model calls: 35

Tokens in / out: 884,509 / 19,143

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedApply project label · sales+

Task ID: 19
Recorded name: sales.apply_project_label

Reward: 1

Assertions: 7 / 7

Steps: 17

Tool calls: 32

Model calls: 17

Tokens in / out: 171,751 / 6,005

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedZoom recording distribution · sales+

Task ID: 40
Recorded name: sales.zoom_recording_distribution

Reward: 1

Assertions: 13 / 13

Steps: 27

Tool calls: 55

Model calls: 27

Tokens in / out: 558,572 / 11,324

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedCalendly capacity management · sales+

Task ID: 41
Recorded name: sales.calendly_capacity_management

Reward: 1

Assertions: 4 / 4

Steps: 24

Tool calls: 48

Model calls: 24

Tokens in / out: 477,373 / 15,100

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedZoom attendance pipeline · sales+

Task ID: 43
Recorded name: sales.zoom_attendance_pipeline

Reward: 1

Assertions: 8 / 8

Steps: 8

Tool calls: 16

Model calls: 8

Tokens in / out: 90,255 / 5,706

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedSlack channel for new account · sales+

Task ID: 44
Recorded name: sales.slack_channel_for_new_account

Reward: 1

Assertions: 7 / 7

Steps: 13

Tool calls: 19

Model calls: 13

Tokens in / out: 114,171 / 2,539

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedDocusign void resend · sales+

Task ID: 45
Recorded name: sales.docusign_void_resend

Reward: 1

Assertions: 12 / 12

Steps: 25

Tool calls: 46

Model calls: 25

Tokens in / out: 627,249 / 10,433

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedDocusign sequential signing · sales+

Task ID: 48
Recorded name: sales.docusign_sequential_signing

Reward: 1

Assertions: 5 / 5

Steps: 17

Tool calls: 29

Model calls: 17

Tokens in / out: 319,033 / 3,622

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedZoom customer health · sales+

Task ID: 49
Recorded name: sales.zoom_customer_health

Reward: 1

Assertions: 4 / 4

Steps: 50

Tool calls: 117

Model calls: 50

Tokens in / out: 3,020,878 / 29,761

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedSocial engagement response · marketing+

Task ID: 51
Recorded name: marketing.social_engagement_response

Reward: 1

Assertions: 7 / 7

Steps: 19

Tool calls: 37

Model calls: 19

Tokens in / out: 242,817 / 7,368

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedContact data cleanup · marketing+

Task ID: 53
Recorded name: marketing.contact_data_cleanup

Reward: 1

Assertions: 10 / 10

Steps: 22

Tool calls: 79

Model calls: 22

Tokens in / out: 547,822 / 16,810

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedAd platform audit · marketing+

Task ID: 55
Recorded name: marketing.ad_platform_audit

Reward: 1

Assertions: 10 / 10

Steps: 21

Tool calls: 26

Model calls: 21

Tokens in / out: 273,772 / 12,651

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedConversion tracking · marketing+

Task ID: 56
Recorded name: marketing.conversion_tracking

Reward: 1

Assertions: 5 / 5

Steps: 16

Tool calls: 27

Model calls: 16

Tokens in / out: 201,842 / 5,734

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedCampaign launch checklist · marketing+

Task ID: 57
Recorded name: marketing.campaign_launch_checklist

Reward: 1

Assertions: 4 / 4

Steps: 29

Tool calls: 43

Model calls: 29

Tokens in / out: 671,887 / 18,536

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedReferral processing · marketing+

Task ID: 60
Recorded name: marketing.referral_processing

Reward: 1

Assertions: 15 / 15

Steps: 24

Tool calls: 55

Model calls: 24

Tokens in / out: 371,316 / 11,181

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedPartner comarketing · marketing+

Task ID: 61
Recorded name: marketing.partner_comarketing

Reward: 1

Assertions: 16 / 16

Steps: 9

Tool calls: 23

Model calls: 9

Tokens in / out: 72,560 / 5,068

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedLead scoring · marketing+

Task ID: 63
Recorded name: marketing.lead_scoring

Reward: 1

Assertions: 11 / 11

Steps: 10

Tool calls: 16

Model calls: 10

Tokens in / out: 89,652 / 8,534

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

Showing 25 of 300 tasks