Blog
SWE-bench leaderboard: what AI coding benchmarks measure
How to read SWE-bench and other coding benchmark scores, and what bug rates in real pull requests show that a test pass rate can't.
Alex Mercer
The SWE-bench leaderboard ranks AI coding systems by the percentage of real GitHub issues they resolve. A task counts as resolved when the system's patch makes the issue's failing tests pass without breaking tests that passed before. The score tells you how often a model, inside a particular agent setup, fixes a well-specified issue that tests can check.
How many bugs that model's code carries in real pull requests is a separate question. Bug Bench ranks models by the bugs we find when we review the code teams wrote with them, and our guide to the best LLM for coding covers how to pick one.
What SWE-bench measures
Each SWE-bench task pairs a GitHub issue with the pull request that fixed it. The system under test gets the issue text and the repository as it was before the fix, and it has to write a patch. The evaluation then runs two groups of tests from that pull request: fail-to-pass tests, which failed before the fix and should pass after it, and pass-to-pass tests, which check that nothing else broke. A task is resolved only if the patch applies and every one of those tests passes. The score, % Resolved, is the share of tasks resolved, with no partial credit (SWE-bench paper).
The original set, released in October 2023, has 2,294 tasks from 12 popular Python repositories. Every task has at least one fail-to-pass test, and a median of 51 more tests check existing behavior.
The post introducing SWE-bench Multilingual says SWE-bench's collection method looks for well-defined problems with unambiguous unit tests, which favors pull requests that change only a few lines. In the Multilingual dataset the reference fix changes 10 lines at the median and 110 at the 95th percentile, and the post notes that real software work often needs much larger changes (SWE-bench Multilingual).
SWE-bench variants on the leaderboard
The official leaderboards cover five datasets plus a Bash Only view of Verified, and every board reports % Resolved.
Leaderboard | Tasks | What it tests |
|---|---|---|
SWE-bench (full) | 2,294 | Real issues from 12 Python repositories |
300 | A cheaper subset of self-contained fixes: one file edited, no images or links in the issue | |
500 | Tasks that human annotators checked for a clear problem description, correct tests and enough information to solve, built with OpenAI | |
Bash Only | 500 | The Verified tasks, with every model run in the same minimal agent (mini-SWE-agent) and only a bash shell |
300 | Tasks from 42 repositories in C, C++, Go, Java, JavaScript, TypeScript, PHP, Ruby and Rust | |
480 | Issues that include screenshots, design mockups or diagrams; version 2 dropped tasks with flaky or ungradeable tests |
Since November 18, 2025, Verified and Multilingual accept new submissions only from teams that publish an arXiv preprint or technical report and have at least one author at an academic institution or established research lab. New results from commercial agents without a paper won't appear on those boards, while anyone can still submit to Multimodal (submission rules).
SWE-bench Pro, which many leaderboard sites list, comes from Scale AI and isn't one of the SWE-bench team's datasets. The team's newer benchmarks, CodeClash and ProgramBench, have their own sites. CodeClash evaluates models as goal-oriented developers, and ProgramBench tests whether models can build meaningful software from scratch.
How to read the SWE-bench leaderboard
A resolved percentage is a test pass rate
The score doesn't grade the code beyond the selected tests, and testing is rarely exhaustive. A study of plausible patches from three issue-solving tools on SWE-bench Verified found that 29.6% behaved differently from the developers' own fix, and that 28.6% of those divergent patches were certainly incorrect. It also found that 7.8% of all patches counted as correct while failing the developer-written test suite, and estimated that these weaknesses inflate reported resolution rates by 6.2 percentage points (Are "Solved Issues" in SWE-bench Really Solved Correctly?). In a real repository, a patch like that can pass CI and leave the bug for a reviewer or a user to find.
Models run inside agents
Each entry pairs a model with the scaffold around it, which can add its own tools, prompts, step limits and retries. The SWE-bench Verified leaderboard includes everything from simple agent loops to retrieval-augmented systems and setups that run several attempts and review them. Scaffolds get tuned to the benchmark too. The SWE-bench Multilingual post says existing agent frameworks often rely on Python-specific tooling, "effectively overfitting to SWE-bench Verified".
To compare models on equal terms, use the Bash Only view, which runs every model in mini-SWE-agent with only a bash shell and one shared configuration. Results from mini-SWE-agent 1.x and 2.x aren't necessarily comparable, because 2.x invokes actions through tool calls. The board also marks open-weights models and the runs the SWE-bench team performed or checked itself, and its compare view plots each entry's resolved rate against cost and step limits.
Contamination
SWE-bench tasks come from public repositories, so the issues and their fixes may be in a model's training data. In one study, models given only the issue text, without access to the repository's structure, named the file that needed fixing with up to 76% accuracy on SWE-bench Verified, against up to 53% on repositories outside SWE-bench. The authors read the gap as a sign of possible contamination or memorization (The SWE-Bench Illusion).
The SWE-bench team has measured how often submitted patches contain the reference fix verbatim. Across Verified submissions the average was 6.7% of tasks, excluding one outlier, and the team said it planned to run the check on every submission and ask for clarification from any above 20% (SWE-bench blog).
In February 2026, OpenAI, which built Verified with the SWE-bench team, published "Why we no longer evaluate SWE-bench Verified". It said the set is increasingly contaminated and mismeasures frontier coding progress, citing flawed tests and training leakage. It recommended SWE-bench Pro instead.
Saturation
A benchmark saturates when top systems solve nearly every task and its scores stop separating them. Maintainers respond by pruning or replacing tasks. Terminal-Bench 4.0 removed tasks that every model in the latest generation solved five times out of five, along with tasks that had public solutions. Aider moved its leaderboard to a harder polyglot set after new leaders on its original Python benchmark were beating the record by only one or two problems (Aider).
Other coding benchmarks you'll see quoted
Model announcements and leaderboard sites quote several other coding benchmarks. They differ in what the model has to produce and in how the answer gets checked.
Benchmark | What the model does | Score | Notes |
|---|---|---|---|
Fixes bugs and adds features in 41 professional repositories; reference fixes average 107.4 lines across 4.1 files | Resolve rate, with fail-to-pass and pass-to-pass tests | Public tasks come from copyleft (GPL-style) repositories, and a private set from startups' proprietary code is scored on a separate board | |
Completes hand-made tasks in a terminal inside a Docker container | Resolution rate, checked by each task's tests | Version 4.0 changed time, CPU and memory limits and the task set, so its scores aren't comparable with earlier versions | |
Solves contest problems from LeetCode, AtCoder and Codeforces, plus self-repair, code execution and test output prediction | Pass@1, the share of problems where the generated program passes all tests | Each problem has a release date, so you can score a model only on problems published after its training cutoff | |
Edits code to solve the 225 hardest Exercism exercises in C++, Go, Java, JavaScript, Python and Rust | Percent of exercises completed correctly, plus how often the model used the requested edit format | The leaderboard page was last updated November 20, 2025 |
LiveCodeBench and Aider's polyglot set grade self-contained programs, which puts them closest to puzzles. SWE-bench, SWE-bench Pro and Terminal-Bench work inside real repositories or full environments, but each task still ends in a fixed check written in advance.
What Bug Bench measures
Bug Bench ranks AI coding models by relative bug rate, which it defines as "counted bugs per AI-attributed line in code reviewed by cubic, compared with the rate across all AI-attributed lines" (Bug Bench). A rate of 1.00 is average, and lower is better. A counted bug is "a cubic code review finding on AI-written code that a developer upvoted or an AI coding agent addressed". Two models are statistically tied when their 95% confidence intervals overlap, so neither is clearly ahead. A model is ranked once enough of its code has been reviewed across many organizations (methodology).
In the Jun 22 to Oct 11, 2026 ranking window, GPT-5.6 Sol had the lowest relative bug rate of the 7 ranked models, at 0.65 (35% fewer counted bugs than average). No other model was statistically tied with it. Claude Fable 5.1 was second at 1.09, statistically tied with GPT-6 Astra (1.13), GPT-6.1 Sol (1.23) and Claude Sonnet 5.5 (1.27), so none of those four is clearly ahead of the others. Each rate covers only the weeks since that model was first seen. For Claude Fable 5.1 and GPT-6 Astra that is the last six weeks of the window, and for GPT-6.1 Sol and Claude Sonnet 5.5 the last two.
Bug Bench also rates models inside coding agents against the same overall average, which is the closest it gets to SWE-bench's model-versus-agent question. GPT-5.6 Sol rates 0.62 in Codex and 1.52 in Cursor, where it is statistically tied with Claude Opus 5. Bug Bench is observational, not causal: teams choose their models, so a rate reflects the code each model was used for as well as the model. The difference between the two rates doesn't isolate the agent's effect either, because each rate also reflects the teams and tasks behind that agent.
Bug Bench publishes its rankings as Markdown, JSON and CSV under CC BY 4.0, so you can pull them into your own analysis.
SWE-bench vs Bug Bench
SWE-bench and other task benchmarks | Bug Bench | |
|---|---|---|
What is measured | Whether a model or agent completes a fixed task, judged by tests | Counted bugs per AI-attributed line, relative to the average across all AI-written code in the window |
Data source | Curated tasks, mostly from public repositories or contest sites, frozen in a dataset | Pull requests we reviewed, written by teams with the models and coding agents they chose |
Who runs the task | The benchmark team or a submitter, in a sandbox with a chosen agent setup | Developers and their coding agents, in their own repositories |
What a score means | The share of tasks passed; higher is better | 1.00 is average and lower is better; 0.65 means 35% fewer counted bugs than average |
Blind spots | Contamination, weak tests, saturated tasks, small well-specified changes, agent setups that differ | Observational, not causal; only bugs our review found and someone acted on; only models with enough reviewed code; only code we review |
What Bug Bench does not measure
Bug Bench doesn't replace SWE-bench or any other task benchmark.
With no fixed tasks or tests, it can't say whether a model finishes a task.
Bugs our review didn't find aren't counted, and neither are findings that no developer upvoted and no agent addressed.
It isn't a controlled comparison. Models don't get the same tasks, so the work each one was used for shows up in its rate.
Only models with enough reviewed code get a rank. A new model is listed without a rank until it has that, and two models that led in earlier weeks are no longer ranked because they now write under 0.5% of AI-attributed lines. All 10 models Bug Bench lists come from Anthropic or OpenAI.
Its figures, including each model's share of AI-written code, describe only the teams whose pull requests we reviewed.
It doesn't time tasks or measure what one costs. Its price page compares each model's bug rate with its list price per million tokens.
The main ranking pools all languages. For one language, type-safe Java with AI looks at what Bug Bench can and can't say about Java, and at build checks that catch type errors before the code runs.
How to use coding benchmarks to pick a model
Shortlist with task benchmarks, and read the fine print: the dataset, the agent setup, the number of attempts, and whether the benchmark team ran or checked the result. For SWE-bench, compare models in the Bash Only view.
Prefer task sets built to resist contamination, such as LiveCodeBench problems published after the model's training cutoff or SWE-bench Pro's copyleft and private sets.
Check how each candidate rates inside the coding agent your team uses, on Bug Bench's coding agent pages, and use its compare page to put two ranked models side by side.
Run your top two candidates on your own repository before you switch.
Keep reviewing the code. Every ranked model had counted bugs in the window, GPT-5.6 Sol included. On GitHub, cubic reviews each pull request with context from your repository, docs and tickets, and Bug Bench is built from those reviews. Our AI code review page explains how the review works, and how to review AI-generated code covers what to check by hand.
FAQ
Why is SWE-bench Verified no longer used?
It still is. The SWE-bench team maintains the Verified board, and Bash Only is its default view. OpenAI dropped Verified from its own evaluations and recommended SWE-bench Pro. In July 2026 it published an analysis, "Separating signal from noise in coding evaluations", that reports issues in SWE-bench Pro and raises concerns about its reliability and accuracy.
What is Terminal-Bench vs SWE-bench?
SWE-bench gives an agent a GitHub issue and grades its patch with the repository's own tests. Terminal-Bench gives an agent a job to do in a terminal inside a container, and its tasks have included building a Linux kernel from source and fitting peaks in a Raman spectrum, each checked by tests written for that task (Terminal-Bench). Its leaderboard ranks model and agent pairs and shows cost and tokens next to each resolution rate.
Who has the highest SWE-bench Verified score?
The leader changes as new entries arrive and depends on the view, so check the official board on the day you need the answer. The SWE-bench team built the Bash Only view as an apples-to-apples comparison of language models and says it doesn't tune that setup to reach higher scores. For other entries, look for the checkmark, which a submitter earns by sending run instructions so the team can rerun the system on a random subset of tasks and verify the results.
What is a benchmark for an LLM?
An LLM benchmark is a fixed set of tasks with an automatic way to score the answers, so different models can be run the same way and compared. Coding benchmarks usually score with tests. HumanEval, from the 2021 OpenAI paper "Evaluating Large Language Models Trained on Code", has 164 hand-written Python problems and reports pass@k, where a problem counts as solved if any of k generated samples passes its unit tests (HumanEval paper). The authors wrote the problems by hand because their models were trained on a large fraction of GitHub, which already contains solutions to problems from many sources.
How do you benchmark LLM output on your own code?
Keep the coding agent fixed, give two candidate models the same kind of tickets in your main repository for a few weeks, and review every pull request the same way. Count the review findings your team acts on per 1,000 lines of AI-written code, which is close to how Bug Bench counts bugs, and track test failures and reverts alongside. Put a 95% confidence interval on each model's rate, and if the two intervals overlap, call it a tie, as Bug Bench does.
