SWE-Bench Pro Results Overview
Resolve rate by model. Top scores remain below 25 percent, which highlights the difficulty of the benchmark.
Introduction
If you believed the hype around AI agents that code, take a breath. Models that once posted flattering scores on friendly tests just met a harder exam, and the grade stings. On SWE-Bench Pro, the new benchmark for real software engineering, top models that cruised at more than 70 percent on older suites now stall around 23 percent. That’s not a rounding error. It’s a reality check for AI software engineering and the start of a healthier conversation about evaluating AI agents.
Table of Contents
1. The Saturation Problem: Why A New Benchmark Was Urgently Needed
The field needed SWE-Bench Pro because our previous yardsticks stopped moving the needle. On tests like SWE-Bench Verified, leaders regularly cleared a high bar. Then it became a low bar. When a benchmark saturates, you lose the ability to tell whether a jump is genuine capability or clever test practice. The effect is simple. Models look better, teams celebrate, product roadmaps lean forward, and, quietly, the hard problems remain unsolved.
There was a deeper flaw too, and the paper names it clearly, AI benchmark contamination. Public issues, permissive licenses, and web-scale crawls create a pipeline from GitHub into model pretraining. When answers leak into training data, a model can “remember” solutions. It looks smart on paper. It’s just well read. SWE-Bench Pro was built to break that loop, and to restore trust in the signal we get from an AI coding benchmark.
2. Inside SWE-Bench Pro: What Makes It So Much Harder?
SWE-Bench Pro raises the difficulty in three decisive ways. It does not try to trick models with puzzles. It tries to mirror work that engineers actually do in the wild.
2.1 Enterprise-Level Difficulty
The benchmark asks agents to resolve real issues in full repositories. The median change is not a one-line fix. The reference solutions average more than 100 lines across multiple files, with environments that must build, test, and hold up across runs. That forces long-horizon planning, tool use, file navigation, and consistent edits that compile and pass tests. In other words, the things that make engineering hard. SWE-Bench Pro requires the same.
2.2 Contamination-Resistant By Design
To blunt data leakage, the public portion draws from GPL and other strong copyleft repos, and the commercial portion comes from private startup codebases under partnership. Those sources are difficult to absorb into proprietary training sets, legally and practically. You cannot memorize what you cannot see. SWE-Bench Pro keeps a held-out set private as well, so the community has a way to check for overfitting later. This keeps the scoreboard honest.
2.3 Human-Centered Verification And Unified Evaluation Settings
Every instance is human-augmented to include a problem statement, a clear list of requirements, and, when needed, an explicit interface. That reduces ambiguity and focuses the challenge on implementation, not scavenger hunts. All models run under the same scaffold, SWE-Agent, with tool use enabled, a shared base prompt, and the same turn limits. Open-weight models are hosted with vLLM on a single node equipped with eight H100 GPUs. SWE-Bench Pro keeps the playing field level and reproducible.
3. The Sobering Results: A Head-To-Head Model Comparison
The headline is simple. Even the best agents fail most of the time on SWE-Bench Pro. The nuance still matters, so let’s look at both subsets that the paper reports.
3.1 Public Set Results
GPT-5 edges out the competition on the public set. The margin is small, and the ceiling is low.
| MODEL | RESOLVE (%) |
|---|---|
| OPENAI GPT-5 | 23.3 |
| CLAUDE OPUS 4.1 | 22.7 |
| CLAUDE SONNET 4 | 17.6 |
| GEMINI 2.5 PRO PREVIEW | 13.5 |
| SWE-SMITH-32B | 6.8 |
| OPENAI GPT-4O | 4.9 |
| QWEN-3 32B | 3.4 |
Table 1: Model performance on the public set of SWE-Bench Pro (N = 731). Evaluated with SWE-Agent. Ambiguity is minimized with augmented statements, requirements, and interface.
3.2 Commercial Set Results
Claude Opus 4.1 takes the commercial crown. These issues come from proprietary startup codebases and carry a bite that the public set cannot match.
| MODEL | RESOLVE (%) |
|---|---|
| CLAUDE OPUS 4.1 | 17.8 |
| OPENAI GPT-5 | 14.9 |
| GEMINI 2.5 PRO PREVIEW | 10.1 |
| CLAUDE SONNET 4 | 9.1 |
| OPENAI GPT-4O | 3.6 |
Table 2: Model performance on the commercial set of SWE-Bench Pro (N = 276). Each problem includes a runnable environment and relevant context.
3.3 Takeaways And A Human Baseline
A few observations stand out. GPT-5 wins narrowly on the public side, which signals strong GPT-5 coding capabilities. Claude 4.1 coding strength shows up in the commercial slice. Gemini 2.5 Pro earns respectable middle-of-the-pack results. Open-source models lag here, despite strong showings on simpler tasks. The most important note, though, is the failure rate. On SWE-Bench Pro, even leaders miss more than three out of four attempts. Humans solved these issues in real repositories. So the AI vs human programmer baseline sits at 100 percent given enough time. The gap is real, and it is measurable.
4. What Do These Failures Look Like?
Not all misses are equal. The paper clusters failure modes, and they read like a postmortem wall in a busy codebase. On bigger agents you see semantic misses, algorithmic slips, and edits that weave across files without fully landing. On smaller open models you see syntax breaks, formatting errors, brittle tool use, and context overflow from heavy directory listings. SWE-Bench Pro surfaces these patterns clearly because the tasks pull agents into the places where bugs like to hide. It also confirms a blunt fact about evaluating AI agents. Interface design, file navigation, and tool plumbing are part of the score, not a footnote.
5. Why This Benchmark Matters For Research And Product
A hard benchmark is not a setback. It is a map. SWE-Bench Pro gives research teams a durable signal that won’t collapse after two leaderboard cycles. Scores below 25 percent are not a failure of ambition. They are a measure. A model that improves five points here has done more than memorize a pattern. It has learned to reason across files, keep context straight, and propose edits that compile and pass tests. That’s what progress in AI software engineering looks like.
This also shifts incentives. You cannot cram for SWE-Bench Pro by embedding solutions into training sets you don’t own. You need better planners, better memory, fewer blind file listings, stronger type awareness, better judgment about when to run tests, and tighter feedback loops between tools and natural language. You need agents that can decompose, stage changes, write focused diffs, and roll back gracefully. That is the work.
6. How To Read The Numbers Without Fooling Yourself
Benchmarks compress messy reality into one metric. That can be useful. It can also be misleading if you ignore the setup. On SWE-Bench Pro, keep three anchors in mind.
- Contamination resistance. Public GPL repos and private startup codebases reduce leakage. The number means more. SWE-Bench Pro defends its signal.
- Human augmentation. Every instance includes a cleaned problem statement and explicit requirements. The test asks, can you implement a working fix, not can you guess the missing context. SWE-Bench Pro aims at execution.
- Unified evaluation. Same scaffold, same prompt, same limits, same hardware envelope. Variance that remains reflects agent behavior, not lab quirks. SWE-Bench Pro keeps the comparison honest.
With that framing, a 23.3 percent Pass@1 is not a dunk. It is a credible benchmark result on a demanding suite. That is exactly what a trustworthy AI coding benchmark should deliver.
7. What Builders Should Do Next
If you are shipping coding agents, don’t chase leaderboard spikes that vanish when the task shifts. Build toward SWE-Bench Pro, then beyond it. A practical plan looks like this.
- Treat repos as living systems. Build retrieval around interfaces and invariants, not keywords. Encourage models to read less and index smarter. SWE-Bench Pro punishes endless file reading.
- Lean into tool use with guardrails. Enforce compile-test loops that are short, deterministic, and visible to the planner. Strip noisy logs and cap directory listings. SWE-Bench Pro rewards agents that manage context like a resource.
- Prefer surgical diffs. Generate patch sets that are small, testable, and reversible. Encourage models to stage changes before broad edits. SWE-Bench Pro is multi-file, but good agents still change only what they must.
- Diagnose failures with structure. Classify misses into syntax, wrong solution, tool error, context overflow, or misread requirements. Then fix the class, not the instance. SWE-Bench Pro makes the buckets explicit. Use them.
- Focus on generalization, not tricks. Private sets will catch overfitting. Long-term gains come from planning and representation learning, not prompt lore. SWE-Bench Pro will keep you honest.
8. What The Results Say About Today’s Models
It is tempting to narrate winners and losers. That misses the point. GPT-5 and Claude Opus 4.1 look strong because they keep their footing across languages and repos. They still stumble, often on semantic and algorithmic edges. Gemini 2.5 Pro is capable, but not dominant. Open-weight models bring agility and control, yet on SWE-Bench Pro they often lose on syntax, formatting, and fragile tool use.
9. The Culture Shift That Hard Benchmarks Enable
A strong benchmark often resets taste. After SWE-Bench Pro, results from light synthetic tasks will feel less convincing. That is healthy. It nudges the field toward grounded evaluation and away from colorful demos that crumble under integration. It also re-centers the role of engineering. To move the number on SWE-Bench Pro, you need better context windows, yes, but also better editors, smarter file search, quicker build-test loops, and debugging tools that models can drive. You need systems thinking.
This is where the Karpathy mindset helps. Clear, iterative loops. Small, measurable deltas. Tight feedback. This is where the Chollet mindset helps too. Think about abstraction, representation, and the shape of the problem, not just the size of the model. Apply that blend to SWE-Bench Pro and you get a research plan that is humane and ambitious at the same time.
10. A Roadmap For The Next Year
If you work in a lab, pick a public subset of SWE-Bench Pro and make it your weekly barometer. Track Pass@1, but also track secondary signals that reflect real progress, like compile success rate, patch size distribution, and number of files edited per success. Rotate repos and languages. Add gates to prevent regressions. Ship only when the number moves on SWE-Bench Pro, not just on your internal smoke tests.
If you are a product team, integrate the same ideas in your experience. Expose a diff-first workflow. Highlight risky edits. Let users dial tool aggression up or down. Give them a button that runs only fail-to-pass tests. Then collect structured failure reports that map back to the buckets the paper uses. That turns user friction into training signal. SWE-Bench Pro shows how to make that loop rigorous.
11. Closing: A Hard Reset We Needed
The message is not that agents cannot code. It is that the path to trustworthy, production-grade agents passes through benchmarks that refuse to make life easy. SWE-Bench Pro is that kind of test. It is contamination resistant. It is long-horizon. It is human verified. It is the best public mirror we have today for the reality of professional software work. The current score, about 23 percent on the public set and even lower on the commercial one, is not a verdict. It is a baseline.
If you care about the future of coding tools, align your roadmap with SWE-Bench Pro. Build planners that think in files, not lines. Design editors the agent can trust. Make tests fast and decisive. Then publish your gains on SWE-Bench Pro, not just in a demo video. The field will move faster when our measures are honest. The next leap will come from teams that take this challenge personally and treat it as the ground truth for progress.
Disclosure: This article references the official paper and results for SWE-Bench Pro including evaluation settings, dataset splits, and model scores.
Q: What is SWE-Bench Pro and why is it a big deal?
A: SWE-Bench Pro is a harder, contamination-resistant benchmark for AI coding agents that evaluates real, enterprise-grade issues with multi-file edits, long-horizon reasoning, and human-verified briefs. Top frontier models score near 23 percent, which turns marketing claims into measurable reality.
Q: How is SWE-Bench Pro different from the old SWE-Bench?
A: The original SWE-bench and its Verified subset rely on public GitHub issues and were nearing saturation for top models. SWE-Bench Pro adds private commercial repos and copyleft public code, clearer human-augmented prompts, and a unified agent scaffold, which reduces leakage and raises difficulty.
Q: Why do top models like GPT-5 and Claude 4.1 have low scores on this new benchmark?
A: Tasks demand 100-plus lines of coordinated changes across files, robust tool use, and end-to-end test passing in realistic environments. With contamination controls in place, memorization offers little help, so even leaders land around 23 percent on the public set and lower on the commercial set.
Q: Is SWE-Bench Pro resistant to data contamination?
A: Yes. SWE-Bench Pro mixes GPL-licensed public code that is hard to include legally in training with private startup repositories and held-out splits, which sharply limits training exposure. This follows the broader push toward contamination-resistant evaluation.
Q: What does this new benchmark tell us about the future of AI in software engineering?
A: SWE-Bench Pro shows that credible progress will come from better planning, tool orchestration, and repository-level reasoning, not from prompt tricks. It gives the field a durable yardstick to track real capability gains on production-like tasks.