LocalsOnly · measured2026-09-19

AutomationBench · local 300*

Local evaluation · 300 tasks. Task sets and protocols differ. A local sample is not established as a subset of the published public set; scores remain separate.

Qwen 3.8 Flash-Next · UD-IQ3_XXS · GB10 · 128GB · llama.cpp

● Tested by LocalsOnly

Weight pin not recorded on this receipt. No download URL.

WALL-CLOCK TIME41.7 h

INPUT TOKENSNot recorded

Recorded effort for this run. Wall time and token counts are not a dollar cost; hardware power and utilization also matter.

Fully completed41.7%125 of 300 tasks fully completed

Mean reward72.2%

Partial credit. Not completion.

Throughput20.6 t/s

Tokens per second · see measurement basis

0 infrastructure outcomes are recorded. Failure counts may overlap with task outcomes.

Reported category outcomes

Each category keeps its own recorded denominator. Some categories exclude unscored trials. Mean reward, when present, is partial credit.

sales

38.0%

19 / 5038.0% passed · mean reward 67.5%

marketing

58.0%

29 / 5058.0% passed · mean reward 86.0%

operations

40.0%

20 / 5040.0% passed · mean reward 77.0%

support

36.0%

18 / 5036.0% passed · mean reward 83.1%

finance

44.0%

22 / 5044.0% passed · mean reward 66.5%

34.0%

17 / 5034.0% passed · mean reward 53.1%

Run configuration & provenance +

Context: 65,536
Temperature: Not recorded
Wall time: 2500.8 minutes
Weight file: Provider-managed / not reported
Revision: Not reported
Reasoning: Not reported
Agent version: Not reported
Output token limit: Not reported
Dataset revision: 1.0.6 · 4a8e1061254004d9dac807054eed33fad7d1ff14
Task selection: First 50 tasks in source order per department; six departments, 300 tasks.
Harness host: Not reported
Model host: ASUS GX10 · 128GB
Input tokens: Not reported
Output tokens: Not reported
Cached input tokens: Not reported
Agent turn / step limit: 50
Concurrency: 1
Weight file size: Not reported

Throughput measurement: export summary.total_output_tokens / summary.total_model_time_s

Native AutomationBench toolset=api, max_steps=50, serial, frozen 50×6 Zapier seed (300). Kris Mac → Asus gx10-e789 llama.cpp OpenAI-compat :8081 (qwen3.8-flash-next UD-IQ3_XXS, n_ctx=65536). Job ab-mac-asus-flashnext-20260917a. Benchmark 1.0.6. MEASURED — same recipe/seed as 27B AutomationBench bank.

Run ID: measured-spark-1x-unsloth-ud-iq3_xxs-automationbench-50x6-2026-09-19

Tasks

Task explorer

Passed tasks first, then other outcomes grouped by recorded issue. Use the filters to inspect a particular outcome.

300 task records

125 Passed175 Not passed

All 300 recorded tasks, before filters or pagination. 25 of 300 matching tasks shown below. Recorded issues can also occur on scored tasks.

PassedUpdate contact phone · sales+

Task ID: 1
Recorded name: sales.update_contact_phone

Reward: 1

Assertions: 4 / 4

Steps: 10

Tool calls: 15

Model calls: 10

Tokens in / out: 110,377 / 3,659

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedCreate note on contact · sales+

Task ID: 7
Recorded name: sales.create_note_on_contact

Reward: 1

Assertions: 10 / 10

Steps: 11

Tool calls: 19

Model calls: 11

Tokens in / out: 93,991 / 3,445

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedApply project label · sales+

Task ID: 11
Recorded name: sales.apply_project_label

Reward: 1

Assertions: 7 / 7

Steps: 18

Tool calls: 43

Model calls: 18

Tokens in / out: 223,017 / 6,476

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedCreate draft reply · sales+

Task ID: 12
Recorded name: sales.create_draft_reply

Reward: 1

Assertions: 5 / 5

Steps: 34

Tool calls: 73

Model calls: 34

Tokens in / out: 1,058,519 / 14,308

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedMulti hop lookup · sales+

Task ID: 13
Recorded name: sales.multi_hop_lookup

Reward: 1

Assertions: 3 / 3

Steps: 22

Tool calls: 44

Model calls: 22

Tokens in / out: 416,919 / 14,838

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedFormat ambiguity · sales+

Task ID: 17
Recorded name: sales.format_ambiguity

Reward: 1

Assertions: 1 / 1

Steps: 9

Tool calls: 16

Model calls: 9

Tokens in / out: 103,984 / 2,681

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedImplicit rules · sales+

Task ID: 18
Recorded name: sales.implicit_rules

Reward: 1

Assertions: 4 / 4

Steps: 18

Tool calls: 29

Model calls: 18

Tokens in / out: 305,429 / 4,646

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedSoft matching · sales+

Task ID: 20
Recorded name: sales.soft_matching

Reward: 1

Assertions: 3 / 3

Steps: 11

Tool calls: 23

Model calls: 11

Tokens in / out: 86,980 / 5,171

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedSheets reconciliation · sales+

Task ID: 24
Recorded name: sales.sheets_reconciliation

Reward: 1

Assertions: 3 / 3

Steps: 12

Tool calls: 28

Model calls: 12

Tokens in / out: 139,803 / 7,207

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedFive level conditional · sales+

Task ID: 25
Recorded name: sales.five_level_conditional

Reward: 1

Assertions: 5 / 5

Steps: 32

Tool calls: 65

Model calls: 32

Tokens in / out: 858,436 / 15,063

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedZoom crm meeting · sales+

Task ID: 26
Recorded name: sales.zoom_crm_meeting

Reward: 1

Assertions: 2 / 2

Steps: 10

Tool calls: 15

Model calls: 10

Tokens in / out: 79,380 / 2,528

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedZoom calendar conflict · sales+

Task ID: 27
Recorded name: sales.zoom_calendar_conflict

Reward: 1

Assertions: 4 / 4

Steps: 11

Tool calls: 23

Model calls: 11

Tokens in / out: 136,235 / 3,320

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedSlack channel for new account · sales+

Task ID: 30
Recorded name: sales.slack_channel_for_new_account

Reward: 1

Assertions: 7 / 7

Steps: 11

Tool calls: 17

Model calls: 11

Tokens in / out: 106,329 / 2,086

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedDocusign contract send · sales+

Task ID: 34
Recorded name: sales.docusign_contract_send

Reward: 1

Assertions: 11 / 11

Steps: 19

Tool calls: 36

Model calls: 19

Tokens in / out: 376,155 / 5,271

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedDocusign compliance template · sales+

Task ID: 41
Recorded name: sales.docusign_compliance_template

Reward: 1

Assertions: 13 / 13

Steps: 15

Tool calls: 35

Model calls: 15

Tokens in / out: 275,736 / 6,088

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedCalendly capacity management · sales+

Task ID: 43
Recorded name: sales.calendly_capacity_management

Reward: 1

Assertions: 4 / 4

Steps: 18

Tool calls: 27

Model calls: 18

Tokens in / out: 226,607 / 6,845

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedZoom attendance pipeline · sales+

Task ID: 45
Recorded name: sales.zoom_attendance_pipeline

Reward: 1

Assertions: 8 / 8

Steps: 14

Tool calls: 29

Model calls: 14

Tokens in / out: 252,473 / 7,131

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedDocusign sequential signing · sales+

Task ID: 48
Recorded name: sales.docusign_sequential_signing

Reward: 1

Assertions: 5 / 5

Steps: 16

Tool calls: 25

Model calls: 16

Tokens in / out: 209,689 / 4,386

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedCalendly lead scoring · sales+

Task ID: 50
Recorded name: sales.calendly_lead_scoring

Reward: 1

Assertions: 5 / 5

Steps: 27

Tool calls: 48

Model calls: 27

Tokens in / out: 618,042 / 7,946

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedSocial engagement response · marketing+

Task ID: 51
Recorded name: marketing.social_engagement_response

Reward: 1

Assertions: 7 / 7

Steps: 17

Tool calls: 33

Model calls: 17

Tokens in / out: 225,157 / 5,299

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedConversion tracking · marketing+

Task ID: 56
Recorded name: marketing.conversion_tracking

Reward: 1

Assertions: 5 / 5

Steps: 17

Tool calls: 28

Model calls: 17

Tokens in / out: 304,098 / 5,043

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedInfluencer outreach · marketing+

Task ID: 58
Recorded name: marketing.influencer_outreach

Reward: 1

Assertions: 15 / 15

Steps: 11

Tool calls: 26

Model calls: 11

Tokens in / out: 107,651 / 8,670

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedReferral processing · marketing+

Task ID: 60
Recorded name: marketing.referral_processing

Reward: 1

Assertions: 15 / 15

Steps: 15

Tool calls: 35

Model calls: 15

Tokens in / out: 174,490 / 8,997

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedPartner comarketing · marketing+

Task ID: 61
Recorded name: marketing.partner_comarketing

Reward: 1

Assertions: 16 / 16

Steps: 15

Tool calls: 28

Model calls: 15

Tokens in / out: 172,889 / 7,004

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedSeasonal campaign · marketing+

Task ID: 62
Recorded name: marketing.seasonal_campaign

Reward: 1

Assertions: 3 / 3

Steps: 12

Tool calls: 30

Model calls: 12

Tokens in / out: 179,884 / 3,982

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

Showing 25 of 300 tasks