LocalsOnly · measured2026-09-24

AutomationBench · local 300*

Local evaluation · 300 tasks. Task sets and protocols differ. A local sample is not established as a subset of the published public set; scores remain separate.

DeepSeek V4 Flash · UD-IQ3_XXS · GB10 · 128GB · llama.cpp

● Tested by LocalsOnly

Weight pin not recorded on this receipt. No download URL.

WALL-CLOCK TIME111.9 h

INPUT TOKENS162,578,272

Recorded effort for this run. Wall time and token counts are not a dollar cost; hardware power and utilization also matter.

Fully completed19.3%58 of 300 tasks fully completed

Mean reward42.0%

Partial credit. Not completion.

Throughput15.2 t/s

Tokens per second · see measurement basis

0 infrastructure outcomes are recorded. Failure counts may overlap with task outcomes.

Reported category outcomes

Each category keeps its own recorded denominator. Some categories exclude unscored trials. Mean reward, when present, is partial credit.

sales

20.0%

10 / 5020.0% passed · mean reward 36.5%

marketing

18.0%

9 / 5018.0% passed · mean reward 51.1%

operations

28.0%

14 / 5028.0% passed · mean reward 53.6%

support

20.0%

10 / 5020.0% passed · mean reward 59.4%

finance

20.0%

10 / 5020.0% passed · mean reward 32.4%

10.0%

5 / 5010.0% passed · mean reward 19.2%

Failures

Where this setup hit a limit

37Context limit exceeded

13Malformed tool-call JSON

Counts from the run’s log summary. These describe execution failures, not an isolated measure of model capability.

Run configuration & provenance +

Context: 65,536
Temperature: Not recorded
Wall time: 6712.2 minutes
Weight file: Provider-managed / not reported
Revision: Not reported
Reasoning: Not reported
Agent version: Not reported
Output token limit: Not reported
Dataset revision: 1.0.6 · 4a8e1061254004d9dac807054eed33fad7d1ff14
Task selection: First 50 tasks in source order per department; six departments, 300 tasks.
Harness host: mac
Model host: DGX Spark · 128GB
Input tokens: 162,578,272
Output tokens: 4,849,104
Cached input tokens: 163,722,472
Agent turn / step limit: 50
Concurrency: 1
Weight file size: Not reported

Throughput measurement: (head76+resume191) summary.total_output_tokens / summary.total_model_time_s; held33 checkpoint has no token/time totals

Native AutomationBench toolset=api, max_steps=50, serial (max_concurrent=1), frozen 50×6 Zapier seed (300), benchmark 1.0.6, AutomationBench git 4a8e106. Kris Mac → DGX Spark spark-5c2f llama.cpp OpenAI-compat [local endpoint] (deepseek-v4-flash UD-IQ3_XXS Unsloth DeepSeek-V4-Flash-0731, n_ctx=65536, reasoning_effort=null). Merged from three per-task slices: head76 (20260923a) + held33 (19c ckpt) + resume191 (19c resume). Hardware evidence id dgx-spark = DGX Spark spark-5c2f :8888 (GB10 · 128GB; NOT Asus gx10-e789 / NOT spark-1x). 50 aborted rollouts (ctx overflow / tool-JSON 500); scores retained.

Run ID: measured-dgx-spark-unsloth-ud-iq3_xxs-automationbench-full300-2026-09-24

Tasks

Task explorer

Passed tasks first, then other outcomes grouped by recorded issue. Use the filters to inspect a particular outcome.

300 task records

58 Passed242 Not passed

All 300 recorded tasks, before filters or pagination. 25 of 300 matching tasks shown below. Recorded issues can also occur on scored tasks.

PassedPriority selection · sales+

Task ID: 4
Recorded name: sales.priority_selection

Reward: 1

Assertions: 1 / 1

Steps: 23

Tool calls: 63

Model calls: 23

Tokens in / out: 488,801 / 16,897

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedFormat ambiguity · sales+

Task ID: 5
Recorded name: sales.format_ambiguity

Reward: 1

Assertions: 1 / 1

Steps: 10

Tool calls: 17

Model calls: 10

Tokens in / out: 85,923 / 3,532

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedCalendar crm meeting · sales+

Task ID: 11
Recorded name: sales.calendar_crm_meeting

Reward: 1

Assertions: 3 / 3

Steps: 13

Tool calls: 22

Model calls: 13

Tokens in / out: 146,707 / 4,700

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedSheets reconciliation · sales+

Task ID: 12
Recorded name: sales.sheets_reconciliation

Reward: 1

Assertions: 3 / 3

Steps: 16

Tool calls: 30

Model calls: 16

Tokens in / out: 270,096 / 7,961

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedZoom crm meeting · sales+

Task ID: 14
Recorded name: sales.zoom_crm_meeting

Reward: 1

Assertions: 2 / 2

Steps: 10

Tool calls: 22

Model calls: 10

Tokens in / out: 79,712 / 4,450

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedDocusign compliance template · sales+

Task ID: 39
Recorded name: sales.docusign_compliance_template

Reward: 1

Assertions: 13 / 13

Steps: 20

Tool calls: 55

Model calls: 20

Tokens in / out: 415,446 / 13,171

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedCalendly capacity management · sales+

Task ID: 41
Recorded name: sales.calendly_capacity_management

Reward: 1

Assertions: 4 / 4

Steps: 15

Tool calls: 33

Model calls: 15

Tokens in / out: 254,413 / 9,386

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedZoom attendance pipeline · sales+

Task ID: 43
Recorded name: sales.zoom_attendance_pipeline

Reward: 1

Assertions: 8 / 8

Steps: 20

Tool calls: 61

Model calls: 20

Tokens in / out: 634,763 / 24,629

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedSlack channel for new account · sales+

Task ID: 44
Recorded name: sales.slack_channel_for_new_account

Reward: 1

Assertions: 7 / 7

Steps: 13

Tool calls: 24

Model calls: 13

Tokens in / out: 148,489 / 3,594

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedDocusign void resend · sales+

Task ID: 45
Recorded name: sales.docusign_void_resend

Reward: 1

Assertions: 12 / 12

Steps: 21

Tool calls: 37

Model calls: 21

Tokens in / out: 514,671 / 11,286

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedSocial engagement response · marketing+

Task ID: 51
Recorded name: marketing.social_engagement_response

Reward: 1

Assertions: 7 / 7

Steps: 23

Tool calls: 54

Model calls: 23

Tokens in / out: 728,987 / 10,388

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedInfluencer outreach · marketing+

Task ID: 58
Recorded name: marketing.influencer_outreach

Reward: 1

Assertions: 15 / 15

Steps: 14

Tool calls: 36

Model calls: 14

Tokens in / out: 241,745 / 9,296

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedLead scoring · marketing+

Task ID: 63
Recorded name: marketing.lead_scoring

Reward: 1

Assertions: 11 / 11

Steps: 32

Tool calls: 58

Model calls: 32

Tokens in / out: 798,845 / 19,595

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedEvent registration sync · marketing+

Task ID: 64
Recorded name: marketing.event_registration_sync

Reward: 1

Assertions: 14 / 14

Steps: 19

Tool calls: 45

Model calls: 19

Tokens in / out: 327,832 / 10,303

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedContent syndication · marketing+

Task ID: 68
Recorded name: marketing.content_syndication

Reward: 1

Assertions: 6 / 6

Steps: 11

Tool calls: 22

Model calls: 11

Tokens in / out: 98,434 / 6,232

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedTrending topic · marketing+

Task ID: 72
Recorded name: marketing.trending_topic

Reward: 1

Assertions: 2 / 2

Steps: 24

Tool calls: 47

Model calls: 24

Tokens in / out: 551,703 / 15,672

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedSocial roi · marketing+

Task ID: 82
Recorded name: marketing.social_roi

Reward: 1

Steps: 23

Tool calls: 37

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedCampaign retrospective · marketing+

Task ID: 89
Recorded name: marketing.campaign_retrospective

Reward: 1

Steps: 24

Tool calls: 36

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedLink building · marketing+

Task ID: 95
Recorded name: marketing.link_building

Reward: 1

Steps: 20

Tool calls: 43

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedJira confluence incident · operations+

Task ID: 104
Recorded name: operations.jira_confluence_incident

Reward: 1

Steps: 21

Tool calls: 35

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedCalendar airtable maintenance · operations+

Task ID: 106
Recorded name: operations.calendar_airtable_maintenance

Reward: 1

Steps: 18

Tool calls: 30

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedTrello vendor hold email · operations+

Task ID: 109
Recorded name: operations.trello_vendor_hold_email

Reward: 1

Steps: 12

Tool calls: 29

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedMonday slack inventory · operations+

Task ID: 110
Recorded name: operations.monday_slack_inventory

Reward: 1

Assertions: 4 / 4

Steps: 23

Tool calls: 30

Model calls: 23

Tokens in / out: 316,314 / 7,544

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedPipefy gmail vendor approval · operations+

Task ID: 114
Recorded name: operations.pipefy_gmail_vendor_approval

Reward: 1

Assertions: 10 / 10

Steps: 25

Tool calls: 59

Model calls: 25

Tokens in / out: 686,420 / 22,105

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

PassedCalendar slack training · operations+

Task ID: 118
Recorded name: operations.calendar_slack_training

Reward: 1

Assertions: 5 / 5

Steps: 11

Tool calls: 26

Model calls: 11

Tokens in / out: 128,958 / 10,958

Stop reason: tool_calls

Recorded issue: No failure explanation in this task record.

The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.

Showing 25 of 300 tasks