LocalsOnly · measured2026-09-27

AutomationBench · local 300*

Local evaluation · 300 tasks. Task sets and protocols differ. A local sample is not established as a subset of the published public set; scores remain separate.

GLM-5.3-Flash · EXL3 TR3 4bpw · 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total · vllm

Needs two desk boxes: DGX Spark (serve head) + ASUS GX10 (tensor-parallel worker), 128GB each.

● Tested by LocalsOnly

Download weights · Mia-AiLab/GLM-5.3-Flash-EXL3-TR3-4bpw @ 25a44fdb ↗Serve recipe · MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks@f4970207 ↗

WALL-CLOCK TIME29.4 h

INPUT TOKENS30,036,238

Recorded effort for this run. Wall time and token counts are not a dollar cost; hardware power and utilization also matter.

Fully completed46.3%139 of 300 tasks fully completed

Mean reward82.4%

Partial credit. Not completion.

Throughput24.1 t/s

Tokens per second · see measurement basis

0 infrastructure outcomes are recorded. Failure counts may overlap with task outcomes.

Read before comparing

300-task subset: frozen 50×6 Zapier seed (first 50 tasks per domain), not the full AutomationBench public set.
Weights are Mia-AiLab EXL3 TR3 4bpw (community quant), not Z.ai's official FP8.
Harness is AutomationBench v1.0.6 @ 4a8e106 without PR #13 (null-type fix); Z.ai's 48.8 includes that fix.
Context capped at 100,000 tokens (serve max_model_len). DeepSeek V4 Flash and Flash-Next AutomationBench runs used 65,536.
Server-default sampling: no harness overrides (generation_config temp 1.0 / top_p 0.95; thinking on, default reasoning effort).
Crash and resume: the TP worker (ASUS GX10) ran out of host memory at seed task 169 when a stray Ollama model load hit it (since fixed; bridge job unloaded). Rows 1–168 kept; tainted row 169 (connection-failure abort) dropped; tasks 169–300 re-run in a resume job with the same model, endpoint, flags and seed.
Needs two desk boxes: serve head DGX Spark spark-5c2f + tensor-parallel worker ASUS GX10 gx10-e789 (2× GB10, 256GB total). Not a single 128GB desk box.

Reported category outcomes

Each category keeps its own recorded denominator. Some categories exclude unscored trials. Mean reward, when present, is partial credit.

sales

48.0%

24 / 5048.0% passed · mean reward 77.5%

marketing

50.0%

25 / 5050.0% passed · mean reward 87.1%

operations

40.0%

20 / 5040.0% passed · mean reward 78.6%

support

46.0%

23 / 5046.0% passed · mean reward 87.3%

finance

48.0%

24 / 5048.0% passed · mean reward 82.9%

46.0%

23 / 5046.0% passed · mean reward 81.0%

Run configuration & provenance +

Context: 100,000
Temperature: Not recorded
Wall time: 1761.1 minutes
Weight file: Mia-AiLab/GLM-5.3-Flash-EXL3-TR3-4bpw @ 25a44fdb (multi-file) ↗
Revision: 25a44fdbf16862a46b7cc9921142c6c81350af2f
Reasoning: thinking on · server-default reasoning effort (harness reasoning_effort=null)
Agent version: Not reported
Output token limit: Not reported
Dataset revision: 1.0.6 · 4a8e1061254004d9dac807054eed33fad7d1ff14
Task selection: First 50 tasks in source order per department; six departments, 300 tasks.
Harness host: mac
Model host: 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total
Input tokens: 30,036,238
Output tokens: 1,281,140
Cached input tokens: 0
Agent turn / step limit: 50
Concurrency: 1
Weight file size: 175.72 GB

Throughput measurement: resume132 export summary.total_output_tokens / summary.total_model_time_s (1,281,140 / 53,189 s). Parent checkpoint has no token/time totals. End-to-end agent throughput incl. prefill, not decode-only.

Native AutomationBench toolset=api, max_steps=50, serial (max_concurrent=1), save-every=1, frozen 50×6 Zapier seed (300), benchmark 1.0.6, AutomationBench git 4a8e106 + local pause/continue patch; PR #13 NOT applied. Kris Mac mini → vLLM OpenAI-compat [local endpoint] (served id GLM-5.3-Flash-EXL3, max_model_len 100000). Two-box serve: DGX Spark spark-5c2f head + ASUS GX10 gx10-e789 TP worker (2× GB10 · 256GB total). vLLM EXL3 4bpw, TP=2, fp8 KV cache, DFlash2 speculator (k=7). Server-default sampling. Merged from parent168 + resume132 after an ASUS worker OOM crash at task 169.

Run ID: measured-2x-gb10-glm53flash-exl3-4bpw-automationbench-full300-2026-09-27

Tasks

Task explorer

Passed tasks first, then other outcomes grouped by recorded issue. Use the filters to inspect a particular outcome.

300 task records

139 Passed161 Not passed

All 300 recorded tasks, before filters or pagination. 25 of 300 matching tasks shown below. Recorded issues can also occur on scored tasks.

PassedNegative selection · sales+

Task ID: 2
Recorded name: sales.negative_selection

Reward: 1

Steps: 19

Tool calls: 33

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedFormat ambiguity · sales+

Task ID: 5
Recorded name: sales.format_ambiguity

Reward: 1

Steps: 10

Tool calls: 14

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedImplicit rules · sales+

Task ID: 6
Recorded name: sales.implicit_rules

Reward: 1

Steps: 15

Tool calls: 22

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedCross reference validation · sales+

Task ID: 7
Recorded name: sales.cross_reference_validation

Reward: 1

Steps: 14

Tool calls: 27

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedSoft matching · sales+

Task ID: 8
Recorded name: sales.soft_matching

Reward: 1

Steps: 11

Tool calls: 21

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedDependency chain · sales+

Task ID: 10
Recorded name: sales.dependency_chain

Reward: 1

Steps: 15

Tool calls: 26

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedCalendar crm meeting · sales+

Task ID: 11
Recorded name: sales.calendar_crm_meeting

Reward: 1

Steps: 11

Tool calls: 15

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedSheets reconciliation · sales+

Task ID: 12
Recorded name: sales.sheets_reconciliation

Reward: 1

Steps: 10

Tool calls: 15

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedFive level conditional · sales+

Task ID: 13
Recorded name: sales.five_level_conditional

Reward: 1

Steps: 22

Tool calls: 28

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedZoom crm meeting · sales+

Task ID: 14
Recorded name: sales.zoom_crm_meeting

Reward: 1

Steps: 15

Tool calls: 24

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedZoom calendar conflict · sales+

Task ID: 15
Recorded name: sales.zoom_calendar_conflict

Reward: 1

Steps: 10

Tool calls: 17

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedCreate important draft · sales+

Task ID: 17
Recorded name: sales.create_important_draft

Reward: 1

Steps: 16

Tool calls: 41

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedMark vip emails read · sales+

Task ID: 18
Recorded name: sales.mark_vip_emails_read

Reward: 1

Steps: 20

Tool calls: 54

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedApply project label · sales+

Task ID: 19
Recorded name: sales.apply_project_label

Reward: 1

Steps: 16

Tool calls: 35

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedAdvance opportunity stage · sales+

Task ID: 22
Recorded name: sales.advance_opportunity_stage

Reward: 1

Steps: 24

Tool calls: 53

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedQualify lead · sales+

Task ID: 25
Recorded name: sales.qualify_lead

Reward: 1

Steps: 11

Tool calls: 27

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedCreate new opportunity · sales+

Task ID: 26
Recorded name: sales.create_new_opportunity

Reward: 1

Steps: 13

Tool calls: 22

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedCreate note on contact · sales+

Task ID: 27
Recorded name: sales.create_note_on_contact

Reward: 1

Steps: 11

Tool calls: 17

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedDocusign compliance template · sales+

Task ID: 39
Recorded name: sales.docusign_compliance_template

Reward: 1

Steps: 13

Tool calls: 27

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedZoom recording distribution · sales+

Task ID: 40
Recorded name: sales.zoom_recording_distribution

Reward: 1

Steps: 18

Tool calls: 46

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedCalendly capacity management · sales+

Task ID: 41
Recorded name: sales.calendly_capacity_management

Reward: 1

Steps: 15

Tool calls: 27

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedZoom attendance pipeline · sales+

Task ID: 43
Recorded name: sales.zoom_attendance_pipeline

Reward: 1

Steps: 13

Tool calls: 27

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedSlack channel for new account · sales+

Task ID: 44
Recorded name: sales.slack_channel_for_new_account

Reward: 1

Steps: 11

Tool calls: 18

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedDocusign sequential signing · sales+

Task ID: 48
Recorded name: sales.docusign_sequential_signing

Reward: 1

Steps: 16

Tool calls: 33

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedSocial engagement response · marketing+

Task ID: 51
Recorded name: marketing.social_engagement_response

Reward: 1

Steps: 14

Tool calls: 26

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

Showing 25 of 300 tasks