LocalsOnly · measured2026-09-05

AutomationBench*

Local evaluation · 300 tasks. Task sets and protocols differ. A local sample is not established as a subset of the published public set; scores remain separate.

Qwen 3.8 27B · UD-Q4_K_XL · GB10 · 128GB · llama.cpp

● Tested by LocalsOnly

WALL-CLOCK TIME55.7 h

INPUT TOKENS79,056,645

Recorded effort for this run. Wall time and token counts are not a dollar cost; hardware power and utilization also matter.

Fully completed18.3%55 of 300 tasks fully completed

Mean reward48.8%

Partial credit. Not completion.

Throughput10.1 t/s

Tokens per second · see measurement basis

0 infrastructure outcomes are recorded. Failure counts may overlap with task outcomes.

Reported category outcomes

Each category keeps its own recorded denominator. Some categories exclude unscored trials. Mean reward, when present, is partial credit.

sales

16.0%

8 / 5016.0% passed · mean reward 41.8%

marketing

16.0%

8 / 5016.0% passed · mean reward 60.5%

operations

26.0%

13 / 5026.0% passed · mean reward 55.5%

support

18.0%

9 / 5018.0% passed · mean reward 62.8%

finance

16.0%

8 / 5016.0% passed · mean reward 37.8%

18.0%

9 / 5018.0% passed · mean reward 34.5%

Failures

Where this setup hit a limit

65Context limit exceeded

18Malformed tool-call JSON

Counts from the run’s log summary. These describe execution failures, not an isolated measure of model capability.

Run configuration & provenance +

Context: 32,768
Temperature: Not recorded
Wall time: 3343.1 minutes
Weight file: Qwen3.8-27B-UD-Q4_K_XL.gguf ↗
Revision: 4ca720788d1e01f1bff70c033e0d0028fd02e502
Reasoning: off
Agent version: Not reported
Output token limit: Not reported
Dataset revision: 1.0.6 · 4a8e1061254004d9dac807054eed33fad7d1ff14
Task selection: First 50 tasks in source order per department; six departments, 300 tasks.
Harness host: Not reported
Model host: ASUS GX10 · 128GB
Input tokens: 79,056,645
Output tokens: 1,283,030
Cached input tokens: 77,868,060
Agent turn / step limit: 50
Concurrency: 1
Weight file size: 17.56 GB

Throughput measurement: export summary.total_output_tokens / summary.total_model_time_s

Benchmark-specific agent harness

Run ID: measured-spark-1x-unsloth-ud-q4_k_xl-automationbench-50x6-2026-09-05

Tasks

Task explorer

Passed tasks first, then other outcomes grouped by recorded issue. Use the filters to inspect a particular outcome.

300 task records

55 Passed245 Not passed

All 300 recorded tasks, before filters or pagination. 25 of 300 matching tasks shown below. Recorded issues can also occur on scored tasks.

PassedUpdate contact phone · sales+

Task ID: 1
Recorded name: sales.update_contact_phone

Reward: 1

Assertions: 4 / 4

Steps: 13

Tool calls: 24

Model calls: 13

Tokens in / out: 79,340 / 3,160

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedApply project label · sales+

Task ID: 11
Recorded name: sales.apply_project_label

Reward: 1

Assertions: 7 / 7

Steps: 18

Tool calls: 36

Model calls: 18

Tokens in / out: 138,369 / 3,409

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedRecency selection · sales+

Task ID: 15
Recorded name: sales.recency_selection

Reward: 1

Assertions: 4 / 4

Steps: 8

Tool calls: 19

Model calls: 8

Tokens in / out: 37,781 / 1,816

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedFormat ambiguity · sales+

Task ID: 17
Recorded name: sales.format_ambiguity

Reward: 1

Assertions: 1 / 1

Steps: 13

Tool calls: 15

Model calls: 13

Tokens in / out: 78,623 / 2,003

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedSoft matching · sales+

Task ID: 20
Recorded name: sales.soft_matching

Reward: 1

Assertions: 3 / 3

Steps: 13

Tool calls: 19

Model calls: 13

Tokens in / out: 74,892 / 2,612

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedZoom calendar conflict · sales+

Task ID: 27
Recorded name: sales.zoom_calendar_conflict

Reward: 1

Assertions: 4 / 4

Steps: 17

Tool calls: 26

Model calls: 17

Tokens in / out: 258,567 / 2,749

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedSlack channel for new account · sales+

Task ID: 30
Recorded name: sales.slack_channel_for_new_account

Reward: 1

Assertions: 7 / 7

Steps: 13

Tool calls: 20

Model calls: 13

Tokens in / out: 134,974 / 1,533

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedDocusign sequential signing · sales+

Task ID: 48
Recorded name: sales.docusign_sequential_signing

Reward: 1

Assertions: 5 / 5

Steps: 11

Tool calls: 22

Model calls: 11

Tokens in / out: 98,007 / 2,242

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedSocial engagement response · marketing+

Task ID: 51
Recorded name: marketing.social_engagement_response

Reward: 1

Assertions: 7 / 7

Steps: 20

Tool calls: 31

Model calls: 20

Tokens in / out: 188,935 / 2,561

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedAd platform audit · marketing+

Task ID: 55
Recorded name: marketing.ad_platform_audit

Reward: 1

Assertions: 10 / 10

Steps: 18

Tool calls: 26

Model calls: 18

Tokens in / out: 237,057 / 10,786

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedReferral processing · marketing+

Task ID: 60
Recorded name: marketing.referral_processing

Reward: 1

Assertions: 15 / 15

Steps: 36

Tool calls: 61

Model calls: 36

Tokens in / out: 661,167 / 7,462

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedLead scoring · marketing+

Task ID: 63
Recorded name: marketing.lead_scoring

Reward: 1

Assertions: 11 / 11

Steps: 15

Tool calls: 22

Model calls: 15

Tokens in / out: 155,610 / 6,847

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedContent syndication · marketing+

Task ID: 67
Recorded name: marketing.content_syndication

Reward: 1

Assertions: 6 / 6

Steps: 14

Tool calls: 20

Model calls: 14

Tokens in / out: 102,641 / 3,055

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedCreative asset · marketing+

Task ID: 88
Recorded name: marketing.creative_asset

Reward: 1

Assertions: 7 / 7

Steps: 11

Tool calls: 17

Model calls: 11

Tokens in / out: 89,947 / 1,521

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedAnalyst briefing · marketing+

Task ID: 98
Recorded name: marketing.analyst_briefing

Reward: 1

Assertions: 27 / 27

Steps: 19

Tool calls: 26

Model calls: 19

Tokens in / out: 162,283 / 7,324

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedPress release distribution · marketing+

Task ID: 99
Recorded name: marketing.press_release_distribution

Reward: 1

Assertions: 11 / 11

Steps: 17

Tool calls: 32

Model calls: 17

Tokens in / out: 139,618 / 7,763

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedJira confluence incident · operations+

Task ID: 104
Recorded name: operations.jira_confluence_incident

Reward: 1

Assertions: 3 / 3

Steps: 11

Tool calls: 20

Model calls: 11

Tokens in / out: 91,749 / 1,927

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedConfluence gmail policy notice · operations+

Task ID: 111
Recorded name: operations.confluence_gmail_policy_notice

Reward: 1

Assertions: 5 / 5

Steps: 13

Tool calls: 16

Model calls: 13

Tokens in / out: 93,671 / 2,086

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedCalendar slack training · operations+

Task ID: 118
Recorded name: operations.calendar_slack_training

Reward: 1

Assertions: 5 / 5

Steps: 17

Tool calls: 22

Model calls: 17

Tokens in / out: 201,821 / 1,644

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedSheets asana approved request · operations+

Task ID: 119
Recorded name: operations.sheets_asana_approved_request

Reward: 1

Assertions: 5 / 5

Steps: 21

Tool calls: 32

Model calls: 21

Tokens in / out: 240,917 / 2,687

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedAirtable gmail visitor followup · operations+

Task ID: 122
Recorded name: operations.airtable_gmail_visitor_followup

Reward: 1

Assertions: 3 / 3

Steps: 10

Tool calls: 14

Model calls: 10

Tokens in / out: 41,001 / 1,695

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedZoom change advisory · operations+

Task ID: 130
Recorded name: operations.zoom_change_advisory

Reward: 1

Assertions: 4 / 4

Steps: 12

Tool calls: 21

Model calls: 12

Tokens in / out: 107,011 / 2,244

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedZoom dr drill · operations+

Task ID: 132
Recorded name: operations.zoom_dr_drill

Reward: 1

Assertions: 7 / 7

Steps: 14

Tool calls: 28

Model calls: 14

Tokens in / out: 148,584 / 4,221

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedZoom training setup · operations+

Task ID: 134
Recorded name: operations.zoom_training_setup

Reward: 1

Assertions: 4 / 4

Steps: 7

Tool calls: 18

Model calls: 7

Tokens in / out: 50,405 / 2,025

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedZoom deal review · operations+

Task ID: 137
Recorded name: operations.zoom_deal_review

Reward: 1

Assertions: 4 / 4

Steps: 11

Tool calls: 19

Model calls: 11

Tokens in / out: 67,035 / 1,505

Stop reason: stop

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

Showing 25 of 300 tasks