Published reference2026-08-13

SWE-bench Pro

Pro evaluation · sample size not reported. Compare only matched task sets and protocols.

Qwen 3.8 27B · BF16 reference · Qwen · Reported evaluation

↗ Published result

Qwen

Protocol as described by the publisher

Qwen 3.8 27B official model card ↗

Reported score61.7%

Throughput—

Tokens per second · see measurement basis

Concurrency—

As recorded on this receipt

Published reference. This is a source-reported result, not a test run by this lab. Qwen corrected problematic tasks and reevaluated baselines. Claude Code; temperature 1, top-p 0.95, 256K context. Not the same task set or harness as the local subset.

Run configuration & provenance +

Context: Not recorded
Temperature: Not recorded
Wall time: Not recorded
Weight file: Provider-managed / not reported
Revision: Not reported
Reasoning: Not reported
Agent version: Not reported
Output token limit: Not reported
Dataset revision: Not reported
Harness host: Not reported
Model host: Published reference
Input tokens: Not reported
Output tokens: Not reported
Cached input tokens: Not reported
Agent turn / step limit: Not reported
Concurrency: Not reported
Weight file size: Not reported

Throughput measurement: Measurement method not separately reported. This should not be treated as standardized decode-only speed.

Provider evaluation

Qwen corrected problematic tasks and reevaluated baselines. Claude Code; temperature 1, top-p 0.95, 256K context. Not the same task set or harness as the local subset.

Run ID: research-qwen-3.8-27b-published-reference-swe-bench-pro

Tasks

Task explorer

Passed tasks first, then other outcomes grouped by recorded issue. Use the filters to inspect a particular outcome.

0 task records

↗

No task records in this receipt.

The headline or summary is available above. Individual outcomes have not been provided.

Explore test coverage →