PassedPassed on this receipt
Django django 16938
Completed the recorded checks.
Exact task metadata
django__django-16938 · Task ID django__django-16938
No recorded stop reason
Reward1
Assertions—
Steps—
Tool calls—
Model calls—
Tokens in / out—
CodingSoftware engineering
SWE-bench Verified
swe-bench-verified · 100 tasks. Exact task sets and protocols remain separate.
Can it fix an issue in an existing codebase?
60.0%Recorded score · 60 / 100 tasks
GLM-5.3-Flash · NVFP4 (LibertAIDAI) · 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total
Inspect the measured run ↗How scoring works ↓THE QUESTION IT ANSWERS
Can it fix an issue in an existing codebase?
SWE-bench Verified: 500 human-verified repository issues. This is the public-comparable software-engineering headline. Agent and harness belong on the cite.
- 01Repository + issue→
- 02Inspect and change code→
- 03Repository tests
Interpret the score LocalsOnly has run Verified on the first 100 of the 500 tasks for 3 models (GLM-5.3-Flash on two serving engines, vLLM and TensorFold), shown as RECEIPTs; the full 500 has not been run. An empty cell means not run, not zero.
BEYOND THE HEADLINE
What happened across 100 tasks?
Each block represents one recorded task, in receipt order from left to right, then top to bottom. Fill shows the outcome; work-category colors are not used here.
Fully passed60 60.0%
No credit40 40.0%
Tested by LocalsOnly · NVFP4 (LibertAIDAI) · 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total. Partial progress still requires further work before a task is complete.
Inspect individual task outcomes →What this test is
What this benchmark measures
500 human-verified repository issues. Public-comparable headline for repo software engineering. Agent and harness belong on the cite.
SWE-bench Verified is a 500-issue subset that human engineers marked as solvable. A resolved score is a patch that passes the repository tests. Agent, harness, and attempt policy belong with the cite. This is not SWE-bench Pro.
SWE-bench paper ↗How it grades
How a task earns its score
The LocalsOnly measured receipt records each task as pass or fail.
A task can also carry a setup issue recorded on the receipt (for example AgentTimeoutError).
6 infrastructure outcomes are recorded on the receipt. They are not invented extra grades.
Published rows are vendor or bench cites. They do not share the measured task grid.
SuiteSWE-bench Verified
Tasks100 on the measured receipt
PassRecorded pass on the task
FailRecorded fail on the task
Setup issueAgentTimeoutError
Follow one task · this measured recipe
A passed task and a failed one
Same LocalsOnly receipt on 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total. GLM-5.3-Flash · NVFP4 (LibertAIDAI) · 2026-10-22. Assertions, steps, and tool calls are on the task record. There is no agent transcript on this page.
passed vs AgentTimeoutError
Not passedFailed on this receipt
Matplotlib matplotlib 14623
Stopped at the recorded time limit.
Exact task metadata
matplotlib__matplotlib-14623 · Task ID matplotlib__matplotlib-14623
AgentTimeoutError
Reward0
Assertions—
Steps—
Tool calls—
Model calls—
Tokens in / out—
Subject & category breakdown · this recipe
Results by subject or work category
This receipt does not record a subject or work-category breakdown. A suite-wide label cannot support a finer subject comparison. Outcome groups such as “passed” and “timeout” are not subjects.
Ranking
Results & published references
Place is assigned only among rows that share a comparable score on this board. Evaluation settings can differ across sources.
Tested by LocalsOnly
Results from the exact local setup shown below.
| # | Model / recipe | Reported score ↓ | Runs on | Evidence | |
|---|---|---|---|---|---|
| 01 | ✳Qwen 3.8 Flash-Next · llama.cppUD-IQ3_XXS · llama.cpp | 94.0% | GB10 · 128GB | ● Tested by LocalsOnlyLocalsOnly | ↗ |
| 02 | ◎GLM-5.3-Flash · TensorFoldEXL3 TR3 4bpw · TensorFold | 71.0% | 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total | ● Tested by LocalsOnlyLocalsOnly | ↗ |
| 03 | ◎GLM-5.3-Flash · vLLMEXL3 TR3 4bpw · vLLM | 69.0% | 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total | ● Tested by LocalsOnlyLocalsOnly | ↗ |
| 04 | ◎DeepSeek V4 Flash · llama.cppUD-IQ3_XXS · llama.cpp | 64.0% | GB10 · 128GB | ● Tested by LocalsOnlyLocalsOnly | ↗ |
| 05 | ◎GLM-5.3-Flash · SGLangNVFP4 (LibertAIDAI) · SGLang | 60.0% | 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total | ● Tested by LocalsOnlyLocalsOnly | ↗ |
Published reference
Results reported by model developers and benchmark authors.
| # | Model / recipe | Reported score ↓ | Runs on | Evidence | |
|---|---|---|---|---|---|
| 01 | ✺Claude Fable 5Anthropic system card · Anthropic evaluation | 95.0% | Published reference | ↗ Published resultAnthropic | ↗ |
| 02 | ✺Claude Opus 4.8Anthropic system card · Anthropic evaluation | 88.6% | Published reference | ↗ Published resultAnthropic | ↗ |
| 03 | ✦Gemini 3.1 Pro PreviewGoogle evaluation · High thinking · Google scaffold | 80.6% | Published reference | ↗ Published resultGoogle DeepMind | ↗ |
| 04 | ◎DeepSeek V4 FlashPublished reference · Reported evaluation | 79.0% | Published reference | ↗ Published resultDeepSeek | ↗ |
| 05 | ✳MiMo-V2.6-Distill-Qwen-9BBF16 reference · Reported evaluation | 61.1% | Published reference | ↗ Published resultXiaomi MiMo | ↗ |