SWE-bench Verified is the industry-standard benchmark for evaluating whether AI models can do real software engineering — not just write code snippets, but navigate complex codebases, diagnose bugs from issue descriptions, and produce patches that actually pass test suites. Created by Carlos E. Jimenez, John Yang, and colleagues at Princeton University in 2023, SWE-bench draws its tasks from genuine GitHub issues in widely-used open-source Python repositories including Django, Flask, scikit-learn, sympy, matplotlib, and requests. The “Verified” subset, introduced in 2024, uses human-validated test cases to ensure evaluation reliability and has become the standard reporting variant.
What SWE-bench Verified Measures
SWE-bench tests the end-to-end software engineering pipeline: reading a natural-language issue description, understanding the relevant codebase, identifying which files and functions need modification, generating a correct patch, and ensuring that patch passes the project’s test suite without breaking existing functionality. This is fundamentally different from code generation benchmarks like HumanEval, which test whether a model can write a short function from a specification.
Each task presents the model with a snapshot of a real GitHub repository at the commit just before the fix was merged, along with the original issue description. The model must produce a git-format patch. Success is binary: either the patch passes all tests associated with the issue, or it does not. There is no partial credit for patches that are “close” or that fix part of the problem.
The Verified subset was created after researchers discovered that roughly 20% of the original SWE-bench test cases had unreliable tests — cases where the ground-truth patch might not be the only valid solution, or where the test suite itself had issues. Human annotators reviewed and validated the test cases in the Verified subset, making scores on this variant more trustworthy as a measure of actual capability.
Current Leaderboard Analysis
The SWE-bench Verified leaderboard shows the widest spread of any major benchmark, revealing stark differences in real-world coding capability. Claude Opus 4 dominates at 72.5%, holding a 7.2-point lead over the second-place Claude Sonnet 4 at 65.3%. Anthropic’s models occupy the top two positions, followed by Gemini 2.5 Pro at 63.8% and GPT-4.1 at 54.6%.
The gap between the leaders and the rest is striking. DeepSeek V3 at 42.0% and GPT-4o at 38.4% trail the leaders by 30+ points, underscoring how dramatically agentic coding capability has improved across model generations. GPT-4.1’s 54.6% is notably lower than both Gemini 2.5 Pro and Claude Sonnet 4 despite GPT-4.1 leading on several other benchmarks, suggesting that SWE-bench captures a distinct dimension of capability — one where understanding large codebases and producing precise patches matters more than general intelligence.
The Anthropic advantage here is not marginal. Claude Opus 4’s 72.5% means it resolves nearly three out of four real-world software engineering tasks — a capability that was considered implausible just 18 months ago.
Methodology Deep Dive
SWE-bench Verified contains 500 human-validated task instances drawn from 12 popular Python repositories. The original full SWE-bench contains 2,294 tasks, but the Verified subset is now the standard evaluation target due to its higher reliability.
Each task instance includes: the repository at a specific commit, the GitHub issue text (including any comments), and a test suite that the fix must pass. Models typically operate within an agentic framework — they can browse the repository, read files, run tests, and iteratively refine their patches. This agentic setup is crucial because most SWE-bench tasks require understanding code that spans multiple files and directories.
Scoring is pass/fail per task, reported as the percentage of tasks where the model’s patch causes all associated tests to pass. Importantly, the patch does not need to match the original developer’s solution — any valid fix counts. This means multiple solution paths are acceptable, which better reflects real software engineering where there is rarely a single correct implementation.
The benchmark is typically run with scaffolding systems that provide the model with tools for file navigation, code search, and test execution. Different scaffolding approaches (e.g., SWE-Agent, Agentless, OpenHands) can significantly affect scores, making it important to note the evaluation framework when comparing results. Most reported scores use SWE-Agent or similar agentic harnesses.
Why This Benchmark Matters
SWE-bench matters because it is the closest any benchmark comes to measuring whether an AI model can replace or augment a professional software engineer. The tasks are not synthetic exercises — they are real bugs that real developers filed and fixed in production codebases. Solving them requires the same skills a junior-to-mid-level engineer uses daily: reading unfamiliar code, understanding project conventions, tracing execution paths, and writing patches that integrate cleanly with existing architecture.
The benchmark also captures the compound nature of software engineering in a way that isolated code generation tests cannot. A single SWE-bench task might require understanding an ORM’s query compilation pipeline, identifying where a specific SQL dialect is mishandled, modifying the relevant code generator, and ensuring the fix does not break query compilation for other databases. This multi-step, multi-file reasoning is where the gap between “can write code” and “can do engineering” becomes visible.
For AI-assisted development tools, SWE-bench scores serve as a direct predictor of real-world usefulness. A model scoring 72.5% can be expected to resolve a significant majority of straightforward bug fixes and feature requests with minimal human intervention, fundamentally changing the economics of software maintenance.
Known Limitations and Criticisms
SWE-bench is limited to Python repositories, which means it does not capture performance on languages with stricter type systems (Rust, Go), different paradigms (Haskell, Clojure), or systems-level concerns (C, C++). The Aider Polyglot benchmark partially fills this gap, but a true polyglot version of SWE-bench does not yet exist.
The benchmark’s reliance on agentic scaffolding makes apples-to-apples comparison difficult. A model evaluated with a sophisticated scaffolding system that includes intelligent file search, error recovery, and multi-attempt strategies will score higher than the same model with a simpler harness. Published scores do not always specify the scaffolding used, or they may use proprietary scaffolding that is not available for independent replication.
Task difficulty varies considerably. Some SWE-bench issues involve one-line fixes to obvious bugs, while others require understanding complex architectural patterns. The aggregate score does not distinguish between these, meaning a model that solves all the easy tasks but fails on every hard one could score similarly to a model that handles a broader difficulty range.
There are also concerns about memorization. Since the repositories and their git histories are publicly available, models may have encountered the actual fixes during training. The Verified subset mitigates this somewhat by focusing on tasks where the test validation is most rigorous, but the underlying contamination concern remains.
How Scores Have Changed Over Time
| Year | Top Model | Score | Key Insight |
|---|---|---|---|
| 2023 | GPT-4 (non-agentic) | 1.7% | Initial launch; models struggled without scaffolding |
| 2023 | SWE-Agent + GPT-4 | 12.5% | Agentic scaffolding dramatically improved results |
| 2024 | Claude 3.5 Sonnet | 49.0% | First model to resolve nearly half of all tasks |
| 2024 | GPT-4o | 38.4% | Strong but lagged behind Claude on agentic coding |
| 2025 | Claude Opus 4 | 72.5% | Resolved nearly three-quarters of verified tasks |
The progression from 1.7% to 72.5% in under two years is one of the most dramatic capability gains in any AI benchmark. The critical inflection point was the introduction of agentic scaffolding in late 2023, which transformed SWE-bench from a near-impossible task into one where models could systematically explore codebases and iterate on solutions.
SWE-bench vs Other Benchmarks
| Benchmark | What It Tests | Language Coverage | Task Type | Difficulty |
|---|---|---|---|---|
| SWE-bench Verified | Real-world engineering | Python only | Bug fixes, features in real repos | Hard |
| HumanEval | Function-level coding | Python only | Write single functions | Easy (saturated) |
| Aider Polyglot | Code editing | 12+ languages | Edit existing code | Medium |
| MBPP | Simple coding | Python only | Short programming tasks | Easy |
| CodeContests | Competition coding | Multiple | Algorithmic problems | Hard (different axis) |
SWE-bench and HumanEval are often cited together but measure fundamentally different things. HumanEval tests whether a model can write a correct function given a specification; SWE-bench tests whether it can operate as a software engineer within a complex codebase. A model scoring 93% on HumanEval but 40% on SWE-bench can write code but cannot yet engineer software.
Aider Polyglot complements SWE-bench by testing code editing across multiple languages, while SWE-bench provides depth in Python-only real-world engineering scenarios. Together, they give the most complete picture of a model’s practical coding capability.
Practical Implications
For engineering teams evaluating AI coding assistants, SWE-bench scores are the most predictive benchmark for real-world utility. A model scoring above 65% on SWE-bench Verified can be expected to autonomously resolve a substantial fraction of straightforward bug reports and feature requests, making it suitable for integration into CI/CD pipelines for automated issue resolution.
The 72.5% achieved by Claude Opus 4 represents a threshold where autonomous coding agents become economically viable for software maintenance. At this level, the majority of routine engineering tasks — fixing type errors, handling edge cases, updating API integrations — can be delegated to the model with reasonable confidence that the generated patches will be correct.
For individual developers, the practical difference between a 54.6% model and a 72.5% model is the difference between a tool that occasionally produces useful patches and one that reliably resolves most coding tasks on the first attempt. This gap is large enough to affect workflow decisions — engineers using higher-scoring models can delegate more work to AI assistants with less review overhead.
The Python-only limitation means SWE-bench scores do not directly predict performance for teams working primarily in other languages. Teams working in TypeScript, Rust, or Go should weight Aider Polyglot scores more heavily in their model selection.
Frequently Asked Questions
What is the difference between SWE-bench and SWE-bench Verified?
The original SWE-bench contains 2,294 tasks, but researchers discovered that roughly 20% had unreliable test cases — ambiguous acceptance criteria or tests that could pass with incorrect patches. SWE-bench Verified is a 500-task subset where human annotators have validated that the tests reliably distinguish correct fixes from incorrect ones. Verified scores are lower but more trustworthy, and it is now the standard reporting variant.
Why do Anthropic models dominate SWE-bench?
Claude models have consistently led on SWE-bench, which likely reflects training emphasis on code comprehension, agentic tool use, and instruction following for structured output formats like diffs. SWE-bench rewards models that can read and navigate large codebases systematically rather than models that are strongest at generating code from scratch. The specific strengths required — long-context understanding, precise edit generation, and multi-step planning — appear to align with Anthropic’s training priorities.
How much does the scaffolding system affect SWE-bench scores?
Scaffolding can swing scores by 15-30 percentage points for the same underlying model. A model evaluated with a sophisticated agent framework that includes intelligent code search, error recovery, and multi-attempt strategies will significantly outperform the same model with a bare-bones prompt-and-respond setup. When comparing scores, always check which scaffolding system was used.
Can SWE-bench scores predict how useful a model will be for my coding work?
SWE-bench is the best available predictor for Python-centric software engineering tasks — bug fixes, refactoring, and feature implementation in real codebases. It is less predictive for algorithmic problem-solving (use CodeContests for that), multi-language work (use Aider Polyglot), or green-field code generation from specifications (use HumanEval). For most professional software engineering, SWE-bench is the single most informative benchmark.
Is SWE-bench too easy now that models score above 70%?
At 72.5%, the benchmark still has meaningful headroom — nearly 28% of tasks remain unsolved even by the best model. The unsolved tasks tend to be the most complex ones, requiring deep architectural understanding, multi-file refactoring, or nuanced behavioral changes. SWE-bench is likely to remain informative until models consistently exceed 85-90%, which may still be years away given the difficulty of the remaining tasks.