LocalsOnly · measured2026-10-02
SWE-bench Verified
swe-bench-verified · 100 tasks. Compare only matched task sets and protocols.
GLM-5.3-Flash · EXL3 TR3 4bpw · 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total · vllm
Needs two desk boxes: DGX Spark (serve head) + ASUS GX10 (tensor-parallel worker), 128GB each.
● Tested by LocalsOnlyRecorded effort for this run. Wall time and token counts are not a dollar cost; hardware power and utilization also matter.
Tokens per second · see measurement basis
As recorded on this receipt
- Weights are Mia-AiLab EXL3 TR3 4bpw (community quant) with a DFlash2 speculator (k=7), fp8 KV cache; not Z.ai's official weights.
- Server-default sampling (generation_config temp 1.0 / top_p 0.95); the harness passes no sampling overrides. Thinking OFF: enable_thinking=false is passed explicitly (the GLM chat template defaults to ON), matching the Flash-Next 0.94 and DeepSeek V4 Flash 0.64 Verified receipts.
- Context capped at 100,000 tokens (serve max_model_len). Flash-Next ran with 65,536 and DeepSeek V4 Flash with 32,768; the cap is unlikely to have been binding.
- Harbor terminus-2 agentic harness (max_turns=50, k=1, n=1), not Claude Code or an official-card agent: a RECEIPT on the frozen first-100 slice, not an official SWE-bench Verified compare.
- Errors count as 0: 6 of 100 trials failed with RuntimeError (Command timed out after 120 seconds) and are scored 0, so the headline 69/100 is Harbor's mean. All 6 failed in agent setup (a 120 s docker exec while installing tmux), before the model made any call; none has an agent trajectory or verifier result. The model never saw those tasks (django-16612, django-15973, django-16950, django-15037, django-7530, astropy-13579). 69 of the 94 trials that reached the model passed.
- Serve head crash on 2026-10-01 at about 12:07 PM ET (hung worker step, vLLM RPC timeout): 25 trials errored with InternalServerError and 1 was cancelled. Those 26 trial directories were deleted (results backed up) and re-run with harbor jobs resume -f InternalServerError -f CancelledError after the serve was restarted at about 1 PM ET, so every trial that produced a verifier reward ran against a healthy serve. Three of the six RuntimeError trials (django-16950, django-15973, django-15037) fell inside the outage window but failed in agent setup, before any model call, and were kept as recorded results.
- Needs two desk boxes: serve head DGX Spark spark-5c2f (:8888) + tensor-parallel worker ASUS GX10 gx10-e789 (2× GB10, 256GB total). Not a single 128GB desk box.
Run configuration & provenance +
- Context
- 100,000
- Temperature
- Not recorded
- Wall time
- 1153.5 minutes
- Revision
- 25a44fdbf16862a46b7cc9921142c6c81350af2f
- Reasoning
- thinking OFF · enable_thinking=false passed explicitly via chat_template_kwargs (GLM template defaults ON)
- Agent version
- Not reported
- Output token limit
- Not reported
- Dataset revision
- sha256:b934b0cc3dc800fe945eaf9f1623329db97ee3133c706d20644524c7759fb341
- Harness host
- mac
- Model host
- 2× GB10 (DGX Spark head + ASUS GX10 worker) · 256GB total
- Input tokens
- 73,406,984
- Output tokens
- 1,007,105
- Cached input tokens
- 0
- Agent turn / step limit
- 50
- Concurrency
- 1
- Weight file size
- 175.72 GB
Throughput measurement: Measurement method not separately reported. This should not be treated as standardized decode-only speed.
Harbor terminus-2 on Kris Mac mini (docker, DOCKER_DEFAULT_PLATFORM=linux/amd64) -> vLLM OpenAI-compat [local endpoint] (GLM-5.3-Flash-EXL3 4bpw, TP=2 across DGX Spark spark-5c2f head + ASUS GX10 gx10-e789 worker, fp8 KV, DFlash2 k=7, max_model_len 100000, thinking OFF, max_turns=50, k=1 n=1). Job swever-mac-dgx-glm53flash-n100-20261001a. RECEIPT — thin terminus-2 thinking-off for local headline parity with the Flash-Next and DeepSeek V4 Flash Verified RECEIPTs; NOT Claude Code official-compare. Hardware evidence id gb10-2x = DGX Spark spark-5c2f :8888 head + ASUS GX10 gx10-e789 worker (never swap labels).
Run ID: measured-2x-gb10-glm53flash-exl3-4bpw-swe-bench-verified-n100-2026-10-02Task explorer
Passed tasks first, then other outcomes grouped by recorded issue. Use the filters to inspect a particular outcome.
All 100 recorded tasks, before filters or pagination. 25 of 100 matching tasks shown below. Recorded issues can also occur on scored tasks.
PassedAstropy astropy 8707+
Task ID: astropy__astropy-8707
Recorded name: astropy__astropy-8707
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedAstropy astropy 14096+
Task ID: astropy__astropy-14096
Recorded name: astropy__astropy-14096
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedAstropy astropy 14995+
Task ID: astropy__astropy-14995
Recorded name: astropy__astropy-14995
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 10554+
Task ID: django__django-10554
Recorded name: django__django-10554
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 11099+
Task ID: django__django-11099
Recorded name: django__django-11099
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 11179+
Task ID: django__django-11179
Recorded name: django__django-11179
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 11477+
Task ID: django__django-11477
Recorded name: django__django-11477
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 11951+
Task ID: django__django-11951
Recorded name: django__django-11951
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 12125+
Task ID: django__django-12125
Recorded name: django__django-12125
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 12262+
Task ID: django__django-12262
Recorded name: django__django-12262
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 12419+
Task ID: django__django-12419
Recorded name: django__django-12419
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 13033+
Task ID: django__django-13033
Recorded name: django__django-13033
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 13121+
Task ID: django__django-13121
Recorded name: django__django-13121
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 13406+
Task ID: django__django-13406
Recorded name: django__django-13406
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 13417+
Task ID: django__django-13417
Recorded name: django__django-13417
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 13933+
Task ID: django__django-13933
Recorded name: django__django-13933
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 14122+
Task ID: django__django-14122
Recorded name: django__django-14122
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 14140+
Task ID: django__django-14140
Recorded name: django__django-14140
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 14311+
Task ID: django__django-14311
Recorded name: django__django-14311
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 14434+
Task ID: django__django-14434
Recorded name: django__django-14434
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 14999+
Task ID: django__django-14999
Recorded name: django__django-14999
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 15022+
Task ID: django__django-15022
Recorded name: django__django-15022
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 15280+
Task ID: django__django-15280
Recorded name: django__django-15280
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 15561+
Task ID: django__django-15561
Recorded name: django__django-15561
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.PassedDjango django 15563+
Task ID: django__django-15563
Recorded name: django__django-15563
Reward: 1
Recorded issue: No failure explanation in this task record.
The receipt contains outcome metadata. Full prompts and execution transcripts are not included in this view.