LocalsOnly · measured2026-10-10

AutomationBench · local 300*

Local evaluation · 300 tasks. Task sets and protocols differ. A local sample is not established as a subset of the published public set; scores remain separate.

GLM-5.3-Flash · EXL3 TR3 4bpw · 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total · TensorFold

Needs two desk boxes: DGX Spark (serve head) + ASUS GX10 (tensor-parallel worker), 128GB each.

● Tested by LocalsOnly

Download weights · Mia-AiLab/GLM-5.3-Flash-EXL3-TR3-4bpw @ 25a44fdb ↗Serve recipe · MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks-TensorFold@549fdc5 ↗

WALL-CLOCK TIME13.6 h

INPUT TOKENS66,830,706

Recorded effort for this run. Wall time and token counts are not a dollar cost; hardware power and utilization also matter.

Fully completed48.7%146 of 300 tasks fully completed

Mean reward82.8%

Partial credit. Not completion.

Concurrency1

As recorded on this receipt

0 infrastructure outcomes are recorded. Failure counts may overlap with task outcomes.

Read before comparing

Speed test (separate, 10/9, not measured during this AutomationBench run): single stream, thinking off, 512-token generations, 3 runs: TensorFold 50.4 tok/s (50.4, 50.3, 50.4) vs vLLM 24.0 (23.2, 25.8, 23.1), about 2.1×. 4 streams aggregate, 2 runs: TensorFold 83.5 / 86.0 vs vLLM 39.3 / 40.7. Single stream, thinking on, 1 run: 51.7 vs 24.3. Time to first token: 0.18–0.20 s vs 0.25–0.40 s. Same two boxes, 100k cap. TensorFold ran dense weights at 4-bit, vLLM at BF16.
Engine A/B, same weights: this row and the vLLM row (2026-09-27, mean reward 0.824, 139/300) use the same Mia-AiLab EXL3 TR3 4bpw weights @25a44fdb, the same 300-task seed in the same harness commit, flags and 100k context cap. Only the serving stack changed: TensorFold v1.10 instead of vLLM.
Not a pure engine comparison: TensorFold serves the dense/shared weights at 4-bit (vLLM kept them BF16), and TensorFold's default reasoning effort when the request sets none was not verified to match the vLLM run's default. The harness sends no effort setting in either run.
The gap to the vLLM row (+0.4 points mean reward, +7 full passes) is smaller than one standard error of either mean (about 1.3 points); treat the two engines as tied on quality from this pair.
300-task subset: frozen 50×6 Zapier seed (first 50 tasks per domain), not the full AutomationBench public set.
Weights are Mia-AiLab EXL3 TR3 4bpw (community quant), not Z.ai's official FP8.
Harness is AutomationBench v1.0.6 @ 4a8e106 without PR #13 (null-type fix); Z.ai's 48.8 includes that fix.
Context capped at 100,000 tokens for parity with the vLLM run (TensorFold default is larger).
Server-default sampling: no harness overrides (temp 1.0 / top_p 0.95; thinking on by server default).
Single RoCE link between the two boxes (the second link was not used).
Needs two desk boxes: serve head DGX Spark spark-5c2f + worker ASUS GX10 gx10-e789 (2× GB10, 256GB total). Not a single 128GB desk box.

Reported category outcomes

Each category keeps its own recorded denominator. Some categories exclude unscored trials. Mean reward, when present, is partial credit.

sales

52.0%

26 / 5052.0% passed · mean reward 81.2%

marketing

52.0%

26 / 5052.0% passed · mean reward 87.0%

operations

50.0%

25 / 5050.0% passed · mean reward 81.1%

support

36.0%

18 / 5036.0% passed · mean reward 85.1%

finance

48.0%

24 / 5048.0% passed · mean reward 80.4%

54.0%

27 / 5054.0% passed · mean reward 82.1%

Run configuration & provenance +

Context: 100,000
Temperature: Not recorded
Wall time: 817.1 minutes
Weight file: Mia-AiLab/GLM-5.3-Flash-EXL3-TR3-4bpw @ 25a44fdb (multi-file) ↗
Revision: 25a44fdbf16862a46b7cc9921142c6c81350af2f
Reasoning: thinking on · server-default reasoning effort (harness reasoning_effort=null; TensorFold default not verified equal to vLLM run)
Agent version: Not reported
Output token limit: Not reported
Dataset revision: 1.0.6 · 4a8e1061254004d9dac807054eed33fad7d1ff14
Task selection: First 50 tasks in source order per department; six departments, 300 tasks.
Harness host: mac
Model host: 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total
Input tokens: 66,830,706
Output tokens: 2,669,919
Cached input tokens: 66,967,936
Agent turn / step limit: 50
Concurrency: 1
Weight file size: 175.72 GB

Throughput measurement: Not recorded as a run throughput on purpose. Only SETUP-TEST speeds (separate speed test 10/9) exist for the engine A/B (single stream 50.4 tok/s TensorFold vs 24.0 vLLM; 4 streams ~85 vs ~40); see disclosures and engine_ab.speed_setup_test. Export totals (2,669,919 output tokens / 48,965 s model time) are end-to-end agent numbers and are not used as a speed claim.

Native AutomationBench toolset=api, max_steps=50, serial (max_concurrent=1), save-every=1, frozen 50×6 Zapier seed (300), benchmark 1.0.6, AutomationBench git 4a8e106 + local pause/continue patch; PR #13 NOT applied. Kris Mac mini → TensorFold OpenAI-compat [local endpoint] (served id GLM-5.3-Flash-EXL3, context 100000). Two-box serve: DGX Spark spark-5c2f head + ASUS GX10 gx10-e789 worker (2× GB10 · 256GB total). TensorFold v1.10 (image v0.6.0-a1897d591f70), EXL3 TR3 4bpw experts, dense/shared 4-bit, fp8 KV, DFlash2 bf582e4e, single RoCE link. Server-default sampling, thinking on.

Run ID: measured-2x-gb10-glm53flash-exl3-4bpw-tensorfold-automationbench-full300-2026-10-10

Tasks

Task explorer

Passed tasks first, then other outcomes grouped by recorded issue. Use the filters to inspect a particular outcome.

300 task records

146 Passed154 Not passed

All 300 recorded tasks, before filters or pagination. 25 of 300 matching tasks shown below. Recorded issues can also occur on scored tasks.

PassedNegative selection · sales+

Task ID: 2
Recorded name: sales.negative_selection

Reward: 1

Assertions: 7 / 7

Steps: 14

Tool calls: 25

Model calls: 14

Tokens in / out: 338,056 / 4,812

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedFormat ambiguity · sales+

Task ID: 5
Recorded name: sales.format_ambiguity

Reward: 1

Assertions: 1 / 1

Steps: 11

Tool calls: 19

Model calls: 11

Tokens in / out: 110,657 / 4,858

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedImplicit rules · sales+

Task ID: 6
Recorded name: sales.implicit_rules

Reward: 1

Assertions: 4 / 4

Steps: 13

Tool calls: 22

Model calls: 13

Tokens in / out: 164,300 / 3,021

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedCross reference validation · sales+

Task ID: 7
Recorded name: sales.cross_reference_validation

Reward: 1

Assertions: 4 / 4

Steps: 23

Tool calls: 38

Model calls: 23

Tokens in / out: 623,293 / 7,458

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedSoft matching · sales+

Task ID: 8
Recorded name: sales.soft_matching

Reward: 1

Assertions: 3 / 3

Steps: 10

Tool calls: 24

Model calls: 10

Tokens in / out: 146,125 / 4,123

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedDependency chain · sales+

Task ID: 10
Recorded name: sales.dependency_chain

Reward: 1

Assertions: 2 / 2

Steps: 18

Tool calls: 29

Model calls: 18

Tokens in / out: 269,711 / 6,725

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedCalendar crm meeting · sales+

Task ID: 11
Recorded name: sales.calendar_crm_meeting

Reward: 1

Assertions: 3 / 3

Steps: 10

Tool calls: 17

Model calls: 10

Tokens in / out: 62,099 / 2,098

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedSheets reconciliation · sales+

Task ID: 12
Recorded name: sales.sheets_reconciliation

Reward: 1

Assertions: 3 / 3

Steps: 9

Tool calls: 15

Model calls: 9

Tokens in / out: 94,188 / 4,788

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedFive level conditional · sales+

Task ID: 13
Recorded name: sales.five_level_conditional

Reward: 1

Assertions: 5 / 5

Steps: 16

Tool calls: 22

Model calls: 16

Tokens in / out: 235,042 / 6,204

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedZoom crm meeting · sales+

Task ID: 14
Recorded name: sales.zoom_crm_meeting

Reward: 1

Assertions: 2 / 2

Steps: 10

Tool calls: 17

Model calls: 10

Tokens in / out: 130,529 / 2,847

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedZoom calendar conflict · sales+

Task ID: 15
Recorded name: sales.zoom_calendar_conflict

Reward: 1

Assertions: 4 / 4

Steps: 11

Tool calls: 19

Model calls: 11

Tokens in / out: 142,024 / 3,642

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedCreate important draft · sales+

Task ID: 17
Recorded name: sales.create_important_draft

Reward: 1

Assertions: 2 / 2

Steps: 14

Tool calls: 31

Model calls: 14

Tokens in / out: 230,545 / 13,747

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedMark vip emails read · sales+

Task ID: 18
Recorded name: sales.mark_vip_emails_read

Reward: 1

Assertions: 6 / 6

Steps: 21

Tool calls: 90

Model calls: 21

Tokens in / out: 405,031 / 11,925

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedApply project label · sales+

Task ID: 19
Recorded name: sales.apply_project_label

Reward: 1

Assertions: 7 / 7

Steps: 16

Tool calls: 39

Model calls: 16

Tokens in / out: 185,532 / 5,949

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedCreate contact for account · sales+

Task ID: 23
Recorded name: sales.create_contact_for_account

Reward: 1

Assertions: 3 / 3

Steps: 17

Tool calls: 30

Model calls: 17

Tokens in / out: 225,073 / 7,147

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedCreate new opportunity · sales+

Task ID: 26
Recorded name: sales.create_new_opportunity

Reward: 1

Assertions: 1 / 1

Steps: 14

Tool calls: 34

Model calls: 14

Tokens in / out: 158,440 / 7,120

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedCreate note on contact · sales+

Task ID: 27
Recorded name: sales.create_note_on_contact

Reward: 1

Assertions: 10 / 10

Steps: 12

Tool calls: 19

Model calls: 12

Tokens in / out: 89,683 / 2,792

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedZoom meeting followup · sales+

Task ID: 36
Recorded name: sales.zoom_meeting_followup

Reward: 1

Assertions: 8 / 8

Steps: 22

Tool calls: 41

Model calls: 22

Tokens in / out: 471,551 / 7,573

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedDocusign compliance template · sales+

Task ID: 39
Recorded name: sales.docusign_compliance_template

Reward: 1

Assertions: 13 / 13

Steps: 16

Tool calls: 36

Model calls: 16

Tokens in / out: 353,445 / 6,244

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedZoom recording distribution · sales+

Task ID: 40
Recorded name: sales.zoom_recording_distribution

Reward: 1

Assertions: 13 / 13

Steps: 21

Tool calls: 45

Model calls: 21

Tokens in / out: 547,443 / 9,876

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedZoom attendance pipeline · sales+

Task ID: 43
Recorded name: sales.zoom_attendance_pipeline

Reward: 1

Assertions: 8 / 8

Steps: 10

Tool calls: 22

Model calls: 10

Tokens in / out: 176,025 / 5,881

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedSlack channel for new account · sales+

Task ID: 44
Recorded name: sales.slack_channel_for_new_account

Reward: 1

Assertions: 7 / 7

Steps: 10

Tool calls: 20

Model calls: 10

Tokens in / out: 85,527 / 2,382

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedDocusign void resend · sales+

Task ID: 45
Recorded name: sales.docusign_void_resend

Reward: 1

Assertions: 12 / 12

Steps: 17

Tool calls: 32

Model calls: 17

Tokens in / out: 277,320 / 6,952

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedDocusign sequential signing · sales+

Task ID: 48
Recorded name: sales.docusign_sequential_signing

Reward: 1

Assertions: 5 / 5

Steps: 19

Tool calls: 36

Model calls: 19

Tokens in / out: 356,537 / 4,927

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedZoom customer health · sales+

Task ID: 49
Recorded name: sales.zoom_customer_health

Reward: 1

Assertions: 4 / 4

Steps: 24

Tool calls: 170

Model calls: 24

Tokens in / out: 739,717 / 28,943

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

Showing 25 of 300 tasks