CodingSoftware engineering
Pro evaluation · 50 tasks. Exact task sets and protocols remain separate.
Can it fix an issue in an existing codebase?
Qwen 3.8 27B · UD-Q4_K_XL · GB10 · 128GB
Inspect the measured run ↗How scoring works ↓THE QUESTION IT ANSWERS
Tests software maintenance: finding the relevant code and producing a change that resolves an issue.
Interpret the score Our local result covers a 50-task subset. It is not interchangeable with a full published evaluation.
BEYOND THE HEADLINE
Each block represents one recorded task, in receipt order from left to right, then top to bottom. Fill shows the outcome; work-category colors are not used here.
Tested by LocalsOnly · UD-Q4_K_XL · GB10 · 128GB. Partial progress still requires further work before a task is complete.
Inspect individual task outcomes →Resolve issues in real software repositories.
The failure analysis distinguishes incorrect solutions, syntax errors, wrong files, instruction following, tool use, context limits and loops. The diagnosis is richer than a single repository-fix score.
SWE-bench Pro paper ↗The LocalsOnly measured receipt records each task as pass or fail.
32 infrastructure outcomes are recorded on the receipt. They are not invented extra grades.
Published rows are vendor or bench cites. They do not share the measured task grid.
Same LocalsOnly receipt on GB10 · 128GB. Qwen 3.8 27B · UD-Q4_K_XL · 2026-09-03. Assertions, steps, and tool calls are on the task record. There is no agent transcript on this page.
passed vs not passed
Completed the recorded checks.
scale-ai/instance_protonmail__webclients-32ff10999a06455cb2147f6873d627456924ae13 · Task ID scale-ai/instance_protonmail__webclients-32ff10999a06455cb2147f6873d627456924ae13
No recorded stop reason
Did not complete the recorded checks. The record does not establish a cause.
scale-ai/instance_flipt-io__flipt-a0cbc0cb65ae601270bdbe3f5313e2dfd49c80e4 · Task ID scale-ai/instance_flipt-io__flipt-a0cbc0cb65ae601270bdbe3f5313e2dfd49c80e4
No recorded stop reason
Place is assigned only among rows that share a comparable score on this board. Evaluation settings can differ across sources.
Results from the exact local setup shown below.
| # | Model / recipe | Reported score ↓ | Runs on | Evidence | |
|---|---|---|---|---|---|
| — | ✳Qwen 3.8 27BUD-Q4_K_XL · llama.cpp | 20.0% | GB10 · 128GB | ● Tested by LocalsOnlyLocalsOnly | ↗ |
Results reported by model developers and benchmark authors.
| # | Model / recipe | Reported score ↓ | Runs on | Evidence | |
|---|---|---|---|---|---|
| 01 | ✺Claude Fable 5Published reference · Reported evaluation | 80.0% | Published reference | ↗ Published resultPublished by OpenAI | ↗ |
| 01 | ✺Claude Fable 5Anthropic system card · Anthropic evaluation | 80.0% | Published reference | ↗ Published resultAnthropic | ↗ |
| 03 | ✺Claude Opus 4.8Published reference · Reported evaluation | 69.2% | Published reference | ↗ Published resultPublished by OpenAI | ↗ |
| 03 | ✺Claude Opus 4.8Anthropic system card · Adaptive thinking · max | 69.2% | Published reference | ↗ Published resultAnthropic | ↗ |
| 05 | ◎GPT-5.6 SolPublished reference · Reported evaluation | 64.6% | Published reference | ↗ Published resultPublished by OpenAI | ↗ |
| 06 | ◎GPT-5.6 TerraPublished reference · Reported evaluation | 63.4% | Published reference | ↗ Published resultPublished by OpenAI | ↗ |
| 07 | ◎GPT-5.6 LunaPublished reference · Reported evaluation | 62.7% | Published reference | ↗ Published resultPublished by OpenAI | ↗ |
| 08 | ✳Qwen 3.8 27BBF16 reference · Reported evaluation | 61.7% | Published reference | ↗ Published resultQwen | ↗ |
| 09 | ✦Gemini 3.1 Pro PreviewPublished reference · Reported evaluation | 54.2% | Published reference | ↗ Published resultPublished by OpenAI | ↗ |
| 09 | ✦Gemini 3.1 Pro PreviewGoogle evaluation · High thinking · Google scaffold | 54.2% | Published reference | ↗ Published resultGoogle DeepMind | ↗ |