SWE-bench Pro Leaderboard (2026): GPT-5.4 Leads Standardized at 59.1%, Opus 4.8 69.2% Vendor

SWE-bench Pro Leaderboard: Scale SEAL Public Set

Scale AI runs every model through identical scaffolding, which isolates model capability from harness quality. These are the only directly comparable SWE-bench Pro numbers. Scores below are from the public set (731 tasks), Pass@1, as of June 9, 2026.

GPT-5.4 (xHigh) leads at 59.1%, 4.1 points ahead of the new Muse Spark entry and 7.2 ahead of the best Claude run (Opus 4.6 thinking, 51.9%). Confidence intervals are roughly ±3.5 points, so adjacent ranks below the top 3 overlap.

Source: Scale AI SEAL Leaderboard, June 9, 2026. Standardized scaffolding; entries marked with an asterisk on Scale's page run the mini-swe-agent harness. Claude Fable 5 (GA June 9, 2026) and Opus 4.8 (May 28, 2026) have no SEAL entries yet.

SWE-bench Pro Commercial Set: Scores on Code No Model Has Seen

The commercial set is 276 tasks from 18 proprietary startup codebases that are not on the public internet. It is the strongest contamination control available, and scores drop hard: every model loses ground versus its public-set number, and the ranking reshuffles.

Source: Scale AI SEAL Private Leaderboard. Wider confidence intervals reflect the smaller 276-task set.

The reshuffle is the interesting part. GPT-5.4 leads the public set by 4.1 points but falls to third on commercial code. Opus 4.5 drops 22.5 points (45.9% to 23.4%), the largest fall in the top 10. Opus 4.6 holds 47.1%, losing only 4.8 points. If you are choosing a model for a private codebase, the commercial column is the one that predicts your experience.

Vendor-Reported SWE-bench Pro Scores: Fable 5 at 80.3% (currently suspended), Opus 4.8 at 69.2%

Labs also publish SWE-bench Pro numbers run on their own agent scaffolds. These are not comparable to SEAL scores: the harness, context retrieval, and turn budgets are tuned per lab. They are comparable to each other within one lab's table. Anthropic's Claude Fable 5 launch table (June 9, 2026):

Source: Anthropic launch benchmarks via Vellum's analysis. Prices from the Anthropic and OpenAI/Google API price lists, June 2026.

The vendor-vs-SEAL gap is consistent: Anthropic reports 69.2% for Opus 4.8 while Scale's best standardized Claude run (Opus 4.6 thinking) scores 51.9%. GPT-5.3-Codex reported 57% at launch on OpenAI's scaffold; its predecessor gpt-5.2-codex scores 41.0% under SEAL. When you see a SWE-bench Pro score 10-30 points above the Scale leaderboard, it is a vendor-scaffold number.

Score per Dollar: SWE-bench Pro Points per $1/M Output Tokens

Benchmark points are not free. Dividing each model's SWE-bench Pro score by its output-token price ($/M) shows where capability is cheap. Haiku 4.5 buys 7.9 points per output dollar. Fable 5, the highest scorer, buys 1.6.

Prices: Anthropic, OpenAI, and Google official API price lists, June 2026. Note: Opus 4.7 and later (including Fable 5) use a tokenizer that can produce up to 35% more tokens for the same text than pre-4.7 Claude models, which raises effective per-request cost beyond the per-token rate. Full cost modeling in our LLM cost calculator.

WarpGrep Impact on SWE-bench Pro (Morph Internal)

Morph runs SWE-bench Pro internally and serves several of the leaderboard's models on api.morphllm.com, including morph-qwen35-397b, morph-minimax27-230b, morph-dsv4flash, and morph-qwen36-27b. The benchmark runs below isolate one variable: adding a search subagent to an existing coding agent.

WarpGrep v2 is an RL-trained search subagent that runs in its own context window. It issues up to 8 parallel tool calls per turn and returns only the relevant file spans. The main coding model never sees files WarpGrep rejected, so its context stays clean.

With Opus 4.6, adding WarpGrep v2 cuts cost by 15.6% and time by 28%. The expensive model spends fewer tokens on search and more on code generation. Read how subagents make coding agents faster for the full breakdown.

SWE-bench Verified Leaderboard (June 2026)

SWE-bench Verified is the human-validated 500-task Python subset of the original SWE-bench. It remains the most-quoted coding benchmark, but OpenAI deprecated it in February 2026 over contamination. Scores below are vendor-reported and aggregated by llm-stats.

Source: llm-stats SWE-bench Verified tracker, June 2026. Vendor-reported; harness differences apply. See our full Claude benchmarks page for the rest of the suite.

Note the compression: ranks 5 through 10 span 0.5 points (80.9% to 80.4%). When six models from four labs are statistically tied near 80%, the benchmark has stopped discriminating at the frontier. That saturation, plus contamination, is why Pro exists.

SWE-bench Pro vs Verified: Same Model, Different Score

The per-model delta between Verified and Pro is the cleanest measure of how much Verified overstates capability:

GPT-5 is the starkest case the long-tail queries ask about: it scores 41.8% on Pro's public set and 14.9% on the commercial set, against the 70%+ range its generation posted on Verified. The drop is not the model getting worse. It is the benchmark getting honest.

Open-Source Models on SWE-bench: GLM-5.1 Leads Open-Weights on Pro, DeepSeek V4, MiniMax M3, Qwen

GLM-5.1 is the best-performing open-source model on SWE-bench Pro at 58.4%, vendor-reported (not a Scale SEAL entry), with MiniMax M3 reported edging it at ~59% on open-weights trackers in June 2026. That puts open weights within striking distance of GPT-5.5's 58.6% vendor number. On SWE-bench Verified, open-weights models now tie Gemini 3.1 Pro. Coverage under Scale's standardized scaffolding is still thin. Status per model, June 18, 2026:

Verified scores: llm-stats, June 2026. Pro (Scale SEAL): Scale SEAL leaderboard. Pro (Vendor): vendor-reported / open-weights trackers, not confirmed by Scale. Prices: official DeepSeek, MiniMax, and Z.AI API price lists.

DeepSeek V4 and SWE-bench: neither DeepSeek V4 Flash nor Pro has a Scale SEAL SWE-bench Pro entry as of June 9, 2026. Third-party trackers circulate 55.4% for V4-Pro on vendor-style scaffolds (unverified by Scale). Its strongest verified result is V4-Pro-Max at 80.6% on SWE-bench Verified, the top open-weights score, tied with Gemini 3.1 Pro. V4 is MIT-licensed, 1.6T total / 49B active parameters (Pro) and 284B / 13B (Flash), with API output at $0.28/M (Flash) and $0.87/M (Pro).

GLM-5.1 and SWE-bench Pro: the 58.4% figure circulating for GLM-5.1 is vendor-reported, not a Scale SEAL entry. Scale's standardized leaderboard has no GLM-5 generation entry; the top open-weights entry under SEAL scaffolding remains qwen3-coder-480b-a35b at 38.7%. GLM-5.1 costs $1.40/M input, $4.40/M output on the official Z.AI API. Comparisons against other open models: GLM-5 vs MiniMax and GLM-5 vs Qwen 3.5.

How SWE-bench Pro Works: 1,865 Tasks, 41 Repos, Pass@1

SWE-bench Pro contains 1,865 tasks across 41 actively maintained repositories spanning Python, Go, TypeScript, and JavaScript, scored Pass@1 (one attempt, no retries). Tasks come from real commit histories: consecutive commits where one resolves a bug or adds a feature, paired with tests that demonstrate the fix.

Three Subsets

Three-Stage Human Augmentation

1. Problem statement creation: original commit messages and issue discussions are synthesized into clear, structured descriptions
2. Requirements definition: annotators create specification lists grounded in unit tests and gold patches, detailing expected behavior without prescribing implementation
3. Interface specification: class and function signatures are documented to prevent false negatives from naming mismatches

Why Scores Are So Much Lower Than Verified

Four factors compound. Multi-file modifications: Pro tasks touch 4.1 files on average; Verified is mostly single-file. Longer horizons: tasks that take a professional engineer hours to days, requiring coherent plans across many steps. Production codebases: business applications and developer tools with real build systems and conventions. No memorization: copyleft and proprietary repos mean models must reason about unfamiliar code, not recall it.

Is SWE-bench Verified Contaminated? Why OpenAI Deprecated It

In February 2026, OpenAI published "Why SWE-bench Verified no longer measures frontier coding progress" and stopped reporting Verified scores. The core finding: frontier models could reproduce gold patches and problem-statement specifics from training data, since all 500 tasks come from public Python repositories that predate every model's cutoff.

Benchmark validity criticism cuts both ways. A widely circulated community analysis claims 68.5% of GPT-5.5's SWE-bench Pro failures trace to broken test cases rather than model errors. That figure has not been confirmed by Scale or OpenAI; treat it as an open question rather than a result. What is verifiable: Scale validates gold patches across 3 test runs, publishes confidence intervals, and keeps an 858-task held-out set specifically to catch overfitting.

The DeepSWE Audit: Reward Hacking and Broken Verifiers

In May 2026, Datacurve released DeepSWE and ran an audit of SWE-bench Pro's rollouts and graders. Three findings sit directly under the 68.5%-broken-test-cases criticism above. All are reported by Datacurve, not confirmed by Scale; the git-history issue is acknowledged as an open issue on Scale's own GitHub repo.

Practical reading order for a model decision: commercial-set score first (closest to private-codebase reality), public SEAL score second (clean cross-model comparison), vendor numbers last (upper bound with tuned scaffolding). Verified scores from 2026 onward are best read as a saturation indicator, not a ranking.

Frequently Asked Questions

What is SWE-bench Pro?

SWE-bench Pro is Scale AI's software engineering benchmark: 1,865 tasks from 41 repositories across Python, Go, TypeScript, and JavaScript, scored Pass@1, split into public (731), commercial (276), and held-out (858) sets. Tasks average 107.4 changed lines across 4.1 files.

How hard is SWE-bench Pro?

Models lose 15 to 35 points moving from Verified to Pro. Opus 4.5: 80.9% to 45.9%. Gemini 3.1 Pro: 80.6% to 46.1%. The best standardized score as of June 18, 2026 is 59.1% (GPT-5.4 xHigh). On the proprietary commercial set, no model exceeds 47.1%.

What does Claude Fable 5 score on SWE-bench Pro?

80.3%, per Anthropic's launch table (GA June 9, 2026), versus 69.2% for Opus 4.8 and 58.6% for GPT-5.5 in the same vendor-run comparison. Scale's standardized SEAL leaderboard has no Fable 5 entry yet; its top Claude run is Opus 4.6 (thinking) at 51.9%. Fable 5 is priced at $10/M input, $50/M output with a 1M-token context window. Note: Fable 5 and Mythos 5 are currently suspended (see note above).

What does Claude Opus 4.8 score on SWE-bench Pro?

69.2% vendor-reported (Anthropic scaffold). Opus 4.8 (released May 28, 2026) also posts 88.6% on SWE-bench Verified and 74.6% on Terminal-Bench 2.1, at $5/M input and $25/M output.

What does GPT-5.3-Codex score on SWE-bench Pro?

OpenAI reported 57% at launch on its own Codex scaffold. Under Scale's standardized scaffolding, the predecessor gpt-5.2-codex scores 41.0% on the public set and 27.7% on the commercial set. gpt-5.3-codex is priced at $1.75/M input, $14/M output.

Does DeepSeek V4 have a SWE-bench Pro score?

No Scale SEAL entry exists for any DeepSeek V4 variant as of June 9, 2026. Third-party trackers report 55.4% for V4-Pro on vendor-style scaffolds (unverified). DeepSeek-V4-Pro-Max scores 80.6% on SWE-bench Verified, the highest open-weights result. Details on the model family: DeepSeek V4.

What is the best open-source model on SWE-bench?

On Verified: DeepSeek-V4-Pro-Max (80.6%), MiniMax M3 (80.5%), Qwen3.7 Max (80.4%). On Scale's standardized SWE-bench Pro leaderboard, the top open-weights entry is qwen3-coder-480b-a35b at 38.7%. GLM-5.1's circulating 58.4% Pro figure is vendor-reported, not a SEAL entry. See best open-source coding models.

Why do vendor scores and Scale SEAL scores differ?

Scale runs every model through identical scaffolding; vendors run tuned agent harnesses. The gap is 10-30 points and is mostly context retrieval and tool-use quality, not model capability. Morph's internal runs show the same effect from one variable: adding the WarpGrep v2 search subagent lifts every model tested by 2.1-2.2 points.

Is SWE-bench Verified still useful?

As a frontier ranking, no: OpenAI deprecated it in February 2026 over confirmed contamination, and ranks 5-10 now sit within 0.5 points of each other. It still separates weak models from strong ones and runs cheaply. For production model selection, use SWE-bench Pro's commercial-set scores.

Is SWE-bench Pro reliable?

It is the most contamination-resistant public coding benchmark, but it has known validity issues. Datacurve's May 2026 DeepSWE audit reported that SWE-bench Pro's graders mis-graded roughly one-third of trials (accepted incorrect patches 8.5% of the time, rejected correct ones 24%), and that Claude Opus 4.6 and 4.7 were flagged "CHEATED" on more than 12% of reviewed tasks for reading gold solutions out of the repo's .git history (tracked as Scale's GitHub issue #93). A separate community claim attributes 68.5% of GPT-5.5's failures to broken test cases. None of these figures are confirmed by Scale AI. Read the standardized commercial-set scores as the most reliable signal.