These records do not establish a matched comparison protocol. A rank requires the same benchmark, scoring metric, protocol, sample, and evidence type within an execution group. Different tests are never averaged into a recipe rating. Hardware, precision, and evaluation settings may differ; comparisons do not isolate quantization effects.
ON THIS SETUP · MEASURED BY LOCALSONLY
GB10 · 128GB · llama.cpp
Solve programming problems with executable tests.
Version 6.0 · 100 tasks
Generated solutions are checked by executable tests. The benchmark version and problem window matter; this is different from maintaining an existing repository.
Resolve issues in real software repositories.
Pro evaluation · 50 tasks
Repository tests verify issue resolution. Task subsets should only be compared when their selection and evaluation protocol match.
Multi-step tasks in a terminal environment.
Version 2.1 · 89 tasks
The task must pass its verification in a terminal environment. Tools, harness, context budget and execution failures all contribute to the result.
Follow verifiable instruction constraints.
Loose prompt grading · 300 tasks
This receipt uses loose prompt-level grading. It is not the strict score, and the local and published protocols are not matched.
Sales, marketing, operations, support, finance and HR workflows.
Local evaluation · 300 tasks
A workflow is complete only when every assertion passes. Mean reward gives partial credit for progress; it is not the share of jobs finished. Task sets and evaluation budgets can differ between receipts.
Mean reward 48.8% · partial credit, not completed jobs.
Graduate-level science questions.
Diamond · version 1.0 · 198 tasks
Credit comes from answering the science question correctly. Multiple-choice accuracy does not establish research or experimental competence.
BEYOND THE HEADLINE
Each block represents one recorded task, in receipt order from left to right, then top to bottom. Fill shows the outcome; work-category colors are not used here.
Tested by LocalsOnly · UD-Q4_K_XL · GB10 · 128GB. Partial progress still requires further work before a task is complete.
Inspect individual task outcomes →Completed the recorded checks.
4 / 4 assertions · 13 steps
Follow this exact task ↗sales.update_contact_phone · ID 1
sales · 4 / 4 assertions · 13 steps / 24 tools · passed
The attempt stopped before completion. The record does not establish why.
0 / 2 assertions · 26 steps
Follow this exact task ↗sales.create_important_draft · ID 9
sales · 0 / 2 assertions · 26 steps / 98 tools · aborted_rollout
Context, sampling, harness settings and dataset versions can vary by benchmark. Each evidence link above contains its run-specific configuration.
Download this setup’s recipe JSON ↓