These records do not establish a matched comparison protocol. A rank requires the same benchmark, scoring metric, protocol, sample, and evidence type within an execution group. Different tests are never averaged into a recipe rating. Hardware, precision, and evaluation settings may differ; comparisons do not isolate quantization effects.
ON THIS SETUP · SOURCE-REPORTED
Setup unspecified · Reported evaluation
Multi-step tasks in a terminal environment.
Version 2.1 · sample size not reported
This is the source’s reported aggregate. Its metric, agent configuration, and evaluation protocol determine how it should be interpreted. Read the original grading definition before comparing completion rates across setups.
Zapier’s public 600-task set. Strict, all-assertions completion at the highest reported effort.
Published public evaluation · 600-task set
A workflow is complete only when every assertion passes. Mean reward gives partial credit for progress; it is not the share of jobs finished. Task sets and evaluation budgets can differ between receipts.
Task traces are not available for this setup.
Context, sampling, harness settings and dataset versions can vary by benchmark. Each evidence link above contains its run-specific configuration.