Why AI Coding Benchmarks Lie: SWE-bench 2026

The insider's guide to reading AI coding scores in 2026, and why the famous ones measure the wrong thing.

When SWE-bench launched in October 2023, the best AI model on earth resolved 1.96% of real GitHub issues - SWE-bench paper. That was Claude 2, and the number was so low it read as a challenge to the whole field. By early 2026 the same benchmark family reads like a solved problem: frontier systems cluster in the mid-70s to mid-90s, and every model launch ships a bar chart with a taller bar than the last one.

Then somebody built a version the models could not have seen in advance. In July 2025 the Konwinski Prize, a $1M contamination-proof competition that scores models on GitHub issues collected after the submission deadline, published its first results. The winning entry solved 7.5% of the tasks - TechCrunch. Same class of models. Roughly one tenth of the headline. That gap, between 75% on the famous test and 7.5% on the honest one, is the entire subject of this guide.

The problem is not that the models are secretly bad. It is that a benchmark is a proxy, and the industry has spent three years optimizing the proxy directly while telling buyers, investors, and engineers that the proxy is the thing. A 70% SWE-bench Verified score does not mean an AI can do 70% of software engineering. It means the AI can do 70% of a static, public, mostly-Python, single-repository test whose answer key was scraped into the training data before the test existed. Those are very different claims, and 2026 is the year the difference became impossible to ignore, because the benchmark's own creators started saying so out loud.

This guide goes deep on exactly how the numbers are produced, the four independent mechanisms that inflate them (a movable denominator, memorization, leakage, and weak tests), the newer benchmarks trying to fix each flaw, what the flagship 2026 scores actually mean once you decode the fine print, and the money and incentives that make the whole system lie in a predictable direction. It is written for a non-technical reader who has to make decisions based on these numbers, and it assumes you would rather understand the machinery than trust the marketing.

Contents

1. What SWE-bench actually is, and what "resolved" really means
2. The denominator trick: how Verified doubled scores without a better model
3. Contamination: when the model remembers instead of reasons
4. Leakage and weak tests: the answer was already in the prompt
5. The trust ranking, explained
6. The benchmarks fighting back: freshness, held-out, and dollars
7. The 2026 scoreboard: what the flagship numbers actually say
8. Benchmarketing: Goodhart, the leaderboard illusion, and the money
9. The reality gap: what a 75% score does not buy you
10. How to read a benchmark, and how to run your own
11. The future: rotating evals and agents grading agents

The trust ranking: 15 AI coding benchmarks, scored

Before the analysis, here is the master comparison. Every well-known AI coding benchmark, scored from first principles on how much you can trust the number it produces, not on how hard it is. The criteria are weighted and the score in each cell is followed by the reason it earned that score. The single most important thing to notice is the shape of the ranking: the benchmarks everyone quotes in launch posts (the classic SWE-bench family) sit at the bottom, and the ones almost nobody markets (held-out and rolling tests) sit at the top. That inversion is the thesis of this guide in one table.

How to read the criteria. Each weight reflects how much that property protects the number from lying. Contamination resistance (30%) is weighted highest because a leaked test set is the single largest source of inflation, and it grows silently over time. Real-world fidelity (25%) captures whether the tasks resemble the paid work an engineer actually does (multi-file, ambiguous, unfamiliar code) versus toy fixes. Oracle rigor (20%) measures whether "resolved" is decided by tests strong enough to reject a wrong answer. Transparency (15%) rewards open datasets, published methodology, and reproducible harnesses. Freshness (10%) rewards benchmarks that keep moving past training cutoffs so they cannot be memorized. The scores are one analyst's judgment applied consistently, not a vendor's self-report, and the point is the relative order, which is remarkably stable no matter how you nudge the weights: the honest tests win and the famous test loses.

1. What SWE-bench actually is, and what "resolved" really means

Almost every AI coding claim you have read traces back to one dataset, so it is worth understanding precisely what it contains. SWE-bench was introduced in the paper "Can Language Models Resolve Real-World GitHub Issues?" by Carlos Jimenez, John Yang, Ofir Press, and colleagues at Princeton and the University of Chicago, published at ICLR 2024 - arXiv. It contains 2,294 task instances, each one built from a real, merged pull request that closed a real bug report across 12 popular Python repositories. The elegance of the design is why it took over the field: every task has an objective, automatic grader, so no human has to judge whether the AI's answer is "good."

That grader is a set of unit tests, and understanding it is the key to everything that follows. Each instance ships with FAIL_TO_PASS tests (which fail on the broken code and must pass after the fix) and PASS_TO_PASS tests (which must stay green so the fix does not break anything else). A model is handed the issue text and the repository, it produces a patch, the patch is applied, the tests run, and the task counts as resolved only if both test sets pass - SWE-bench FAQ. Notice what the oracle is: the repository's own tests, exactly as the human developer happened to write them. It is not a judgment of correctness. It is a judgment of "did these specific tests turn green," and that distinction is the crack through which most of the inflation later flows.

The composition of the dataset matters more than the headline count suggests. The 12 repositories are heavily skewed toward a handful of libraries: django/django alone supplies 850 of the 2,294 instances, roughly 37%, followed by sympy, scikit-learn, and sphinx - ICLR paper. So an aggregate that reads like "general software engineering" is, in practice, dominated by the idioms and test harnesses of a few mature Python projects. A model that has thoroughly absorbed Django's codebase and conventions will look broadly competent on the aggregate even if its skill does not generalize to code it has never seen.

It is worth pausing on why the designers chose a test-based oracle in the first place, because the choice was reasonable and its consequences were not obvious in 2023. Judging whether a code change is genuinely "correct" is hard and subjective, and it does not scale to thousands of tasks or to an open leaderboard that anyone can submit to. Unit tests are objective, cheap, and reproducible, so they were the pragmatic choice, and for a brand-new benchmark where the best model scored under 2%, the weakness of the oracle simply did not matter: almost nothing passed, so there was no inflation to worry about. The oracle only became a liability once models climbed into the range where the difference between passing the tests and fixing the bug started to dominate the remaining headroom. A design that is perfectly fine at 2% resolution can be badly misleading at 75%, and that transition, from a wide-open frontier to a nearly-saturated one, is exactly the arc this benchmark lived through in under three years. The tool did not change. The regime it was operating in did, and almost nobody updated how they read the number.

The reason SWE-bench mattered so much is that it looked like the antidote to easy benchmarks. Earlier coding tests such as HumanEval were small, self-contained puzzles that models quickly saturated. SWE-bench used real, messy, multi-file repositories, so a high score seemed to prove real engineering ability. That framing is exactly why its numbers became the currency of the AI race, quoted in keynotes, investor decks, and the endless stream of "which model is best for coding" comparisons. For a broader map of how these tests fit together, our own full 2026 guide to AI agent evals and benchmarks walks the wider landscape, and a ranked benchmark of 50-plus coding agent frameworks shows how tool choice, not just model choice, moves the result. For scores and prices side by side, our May 2026 model benchmarks and pricing roundup and the top 50 model evals list track how these numbers drift month to month.

To ground the mechanics before we take them apart, it helps to see the benchmark explained end to end by someone with no stake in a particular model's score.

The construction has a second consequence that becomes central later. Because every instance is a closed, merged, public pull request, the ground-truth fix, the discussion around it, and the tests are all sitting on the open internet, in the exact repositories that every large model is trained on. The benchmark's answer key was published years before the benchmark, and it was published in precisely the corpus the models learn from. Hold that thought. It is the difference between a test of reasoning and a test of recall.

2. The denominator trick: how Verified doubled scores without a better model

The first thing to understand about "an X% SWE-bench score" is that X depends heavily on which SWE-bench you mean, and the number can move by double digits with no change to the model at all. This is not a subtle statistical point. It is the most quoted variant's origin story. In August 2024, OpenAI released SWE-bench Verified, a human-curated subset of 500 problems distilled from 1,699 samples reviewed by 93 professional software engineers - OpenAI. The stated motivation was legitimate: the original benchmark was underestimating models because many tasks were broken.

The audit numbers OpenAI reported are worth sitting with, because they cut both ways. In the annotation, 38.3% of samples had problem statements too underspecified to solve fairly, and 61.1% had FAIL_TO_PASS tests that could reject a valid solution - OpenAI. Roughly a third of the original tasks were judged unsolvable or unfairly graded. Removing them was a genuine improvement in fairness. But look at the mechanical effect on the scoreboard. Filtering to the cleaner 500 tasks moved GPT-4o, the same model with the same weights, from about 16% on the original benchmark to 33.2% on Verified. The score roughly doubled because the denominator changed, not because the model learned anything.

This is the denominator trick, and it is entirely legitimate as long as everyone states which set they used. In practice, marketing rarely does. A launch post says "74.9% on SWE-bench" and the reader assumes a fixed, comparable ruler. There is no fixed ruler. There is SWE-bench Full (2,294), SWE-bench Lite (300 easier tasks), SWE-bench Verified (500 curated), SWE-bench Multimodal (617 JavaScript/visual tasks), SWE-bench Multilingual (300 tasks across nine languages), and SWE-bench Pro (1,865 hard, held-out tasks). Each produces a different number for the same model, and the gap between the easiest framing and the hardest can be 50 percentage points or more. Comparing two models that quoted different variants is comparing nothing at all.

A second, quieter version of the same trick lives in the harness rather than the dataset. The reported score is never a property of the model alone. It is the output of the model plus the scaffolding around it: the retrieval system that finds relevant files, the retry logic, the number of parallel attempts, and how the best attempt is selected. SWE-bench maintainers acknowledged this so directly that they built a separate "Bash Only" leaderboard, which runs every model through the same minimal agent in a bare shell with no bespoke tooling - swebench.com. The existence of that leaderboard is a quiet confession that the main one conflates scaffold engineering with model skill. On the Bash Only board in February 2026, the top model resolved 76.8% of Verified tasks, with several others one point behind, a tightly bunched pack near saturation - Simon Willison.

There is a third dial that is easy to miss because it hides inside a single word: attempt. A score reported as single attempt (pass@1) means the model got one shot per task. A score reported as best-of-many means the harness ran the model several times and kept its best result, which mechanically raises the number without changing anything the model does on your first try. Anthropic's generation-5 coding figures, for example, are typically reported as a mean over several attempts on a lab-controlled harness, while Google reports its flagship explicitly under single attempt conditions - Google DeepMind. Both can be perfectly honest. They are simply not the same measurement, and placing a best-of-many number next to a single-attempt number to crown a winner is a category error. In production you usually get one attempt, so the single-attempt figure is the one that predicts your actual experience, and it is the one launch posts are least likely to lead with.

Two structural lessons fall out of this chapter, and they govern how you should read every coding number for the rest of the year. First, "SWE-bench score" is not a unit unless the variant and the harness are specified, and a responsible comparison names both. Second, the direction of the curation matters: Verified was built to raise scores toward fairness, which was correct, but the same editorial power over the denominator can flatter a model just as easily as it can rescue one. When the people who choose the tasks also benefit from the scores, the denominator becomes a dial, and a dial is not a measurement.

3. Contamination: when the model remembers instead of reasons

Now we reach the mechanism that turned a fair benchmark into a misleading one over time, and it is the most important idea in this guide. A benchmark drawn from public data cannot be held out from a model trained on public data. SWE-bench is built entirely from merged, public GitHub pull requests. The fix, the file paths, the surrounding discussion, and often the exact diff are all in the open-web corpus that every frontier model ingests. So when a model "solves" one of these tasks, there is no way for the benchmark to distinguish "reasoned to the fix" from "recalled the fix it saw in training." That is not a hypothetical. It has been measured, repeatedly, from several independent directions.

The cleanest demonstration is a 2025 study bluntly titled "The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason." The authors gave models only the issue text, with no access to the repository, and asked them to name the file containing the bug. On SWE-bench tasks the models found the correct file path up to 76% of the time, but on equivalent issues from repositories not in SWE-bench, accuracy fell to about 53% - arXiv. Identifying the right file in a large codebase from a bug description alone, with no repository to look at, is close to impossible unless you have memorized which file holds which bug. The 23-point gap between in-benchmark and out-of-benchmark repos is a memorization fingerprint.

The same paper measured verbatim recall directly. Given only the current file and the issue, models reproduced the ground-truth buggy function with up to 35% consecutive 5-gram accuracy on SWE-bench, versus 18% on tasks from other benchmarks - arXiv. Reproducing long exact sequences of the target code is not reasoning about a bug. It is recalling a specific patch. And the effect is not confined to one research group. A separate University of Waterloo study found that Claude models performed roughly three times better on SWE-bench Verified than on comparable non-contaminated Python benchmarks, and were six times better at locating the edited files with no extra context, exactly the pattern you would expect from training-data memorization - arXiv.

The way to prove contamination, rather than infer it, is to compare performance on problems from before and after a model's training cutoff. LiveCodeBench pioneered this: it tags every problem with a release date, so you can score a model only on problems that were published after it stopped learning. When the researchers did this, a model's pass rate on competitive-programming problems collapsed from roughly 60% on pre-cutoff problems to nearly 0% on post-cutoff ones - arXiv. Same model, same kind of problem, the only difference being whether the answer could have been in the training data. The performance did not degrade gracefully. It fell off a cliff, which is the signature of recall rather than skill.

There is an honest complication worth stating, because it keeps this from tipping into conspiracy. The before-and-after-cutoff test is powerful but not airtight: contamination exists on a spectrum, a problem's public release date is only a proxy for when (or whether) it actually entered the training data, and a clean post-cutoff score is therefore necessary but not sufficient to prove no leakage - Test of Time. The rigorous response is to combine fresh tasks with direct memorization probes, the no-repository file-localization test and the verbatim-recall test, rather than to lean on any single signal. What makes the SWE-bench case so hard to wave away is not one measurement but the convergence: temporal splits, no-repo probes, verbatim reproduction, and cleaned re-evaluations all point the same direction, and the raw GitHub issues themselves are known to sit in open web corpora such as Common Crawl and The Stack that feed pretraining. When four independent methods and the plumbing of the training pipeline all agree, "the model memorized part of the test" stops being a hypothesis and becomes the baseline assumption.

When you decontaminate SWE-bench itself, the scores fall in exactly the way the theory predicts. SWE-rebench, a pipeline that continuously collects fresh post-cutoff GitHub tasks, measured DeepSeek-V3 at 39.7% on the older SWE-bench Verified but only 21.3% on temporally fresh tasks, an 18-point drop attributable to nothing but freshness - arXiv. The uncomfortable conclusion, converged on by at least five independent 2024-2026 papers, is that a meaningful share of the famous number is memory retrieval, and that share grows every training cycle as the benchmark ages deeper into the corpus. This is not a flaw you can patch. It is a property of testing a public-web model on a public-web test. The only structural cure is to keep the questions moving, which is why the honest benchmarks in the trust table are all either held out or continuously refreshed. For a grounded view of what current models genuinely cannot do once the crutch of familiarity is removed, our guide to what LLMs cannot do covers the same failure modes from the practitioner's side.

4. Leakage and weak tests: the answer was already in the prompt

Contamination is what the model brings to the test from its training. Leakage is worse, because it is baked into the test itself, and it does not even require a contaminated model to inflate the score. The most rigorous audit of this is the paper commonly called SWE-Bench+, from a team at York University, which manually inspected the patches that models were credited with solving. It found that 32.67% of the successful patches involved "cheating" in the plain sense: the fix was spelled out directly in the issue report or its comments, so the model was copying a spoiler out of its own prompt rather than diagnosing anything - arXiv. The benchmark was, in a third of its "wins," testing reading comprehension of a leaked answer.

The output side of the pipe was just as leaky. The same audit found that 31.08% of passing patches were "suspicious" because the tests were too weak to verify correctness: a patch that was incomplete or simply wrong could still turn the sparse test set green - arXiv. When the researchers filtered out both the leaked-solution tasks and the weak-test tasks, the resolution rate of a leading system fell from 12.47% to 3.97%, and when they additionally moved to a cleaned, post-cutoff dataset, it fell all the way to 0.55% - arXiv. Nearly everything the system was credited with evaporated once you removed the spoilers, strengthened the tests, and used problems it could not have memorized.

The root cause of the weak-test problem is structural, and it explains why curation alone cannot fix it. When a human developer fixes a bug, they modify a few test files to demonstrate their particular fix. SWE-bench validates every candidate patch using only those few developer-written test files, which means large parts of the affected functionality are never exercised - ICSE 2026 paper. Those tests were written to show one fix works, never to adversarially catch every wrong fix. So a patch that edits the wrong function, or fixes the symptom instead of the cause, still passes as long as it satisfies that sparse, non-adversarial oracle.

How often does the model's "passing" patch actually differ from the real fix? A separate audit built a differential-testing tool, PatchDiff, and found that 29.6% of accepted patches behave differently from the developer's gold patch on inputs the shipped tests never check, with an estimated 7.8% outright incorrect - arXiv. A third team, UTBoost, generated stronger tests and found that 15.7% of accepted patches on SWE-bench Verified failed once the tests were adequate - and, tellingly, fixing the tests reshuffled 24.4% of the Verified rankings - arXiv. The published order of coding agents is partly an artifact of which tests happened to be too weak to tell them apart.

Sit with what a divergent-but-passing patch actually is, because it is more dangerous than a plain wrong answer. The model produces a change that satisfies every test the benchmark runs, so it is scored as a correct fix, yet it behaves differently from the developer's real fix on inputs the tests never exercise. In a production codebase that is not a harmless near-miss. It is a latent bug that shipped with a green checkmark, which is the most expensive kind, because the test suite actively told you it was fine and a reviewer trusting the suite waved it through. The benchmark cannot distinguish the careful fix from the plausible-looking one, and neither can a team that adopts the benchmark's definition of "resolved" as its own definition of "done." This is the same failure that surfaces in the field as the "almost right but not quite" tax, and it is structurally invisible to any oracle assembled from the handful of tests a single developer happened to write for a single pull request.

The deeper lesson generalizes far beyond SWE-bench, so it is worth stating as a principle. Any benchmark whose oracle is a fixed, sparse test suite is vulnerable to the same failure, because the proxy (these tests pass) and the target (the bug is correctly fixed) diverge, and the divergence gets worse as models get better at satisfying sparse signals cheaply. Even OpenAI's carefully human-screened Verified set, built specifically to remove underspecified problems and overly narrow tests, still had enough weak tests for UTBoost to reshuffle a quarter of its rankings. Curation reduces the problem; it does not remove it, because the problem is the shape of the oracle, not the quality of the curation.

5. The trust ranking, explained

With the four inflation mechanisms on the table (a movable denominator, contamination, leakage, and weak tests), the scored ranking at the top of this guide stops looking like an opinion and starts looking like an accounting. Walk it from the bottom up, because the bottom is where the famous numbers live. SWE-bench Verified scores 4.3 out of 10 on trustworthiness, not because it is a bad benchmark, but because it is a frozen, fully public benchmark whose 500 tasks all predate current model cutoffs, whose tests are demonstrably beatable by wrong patches, and 161 of whose problems are one-to-two-line edits. Everything that makes it convenient (open, fixed, automatically graded) is exactly what makes its number age into a memorization score.

Climb to the top and the pattern inverts. The Konwinski Prize scores 8.5 because it closes the contamination hole by construction: the models are frozen, then the GitHub issues are collected after the deadline, and the competition is restricted to open-weight models so there is nowhere to hide a memorized answer. SWE-bench Pro scores 7.6 because it achieves a similar effect structurally, drawing on GPL-licensed and private commercial code that is legally hostile as training data. LiveCodeBench and SWE-rebench score high for the same underlying reason from a different angle: they keep the problems moving so a model cannot have seen them. The common thread among the trustworthy benchmarks is not that they are harder. It is that they make memorization impossible, which is a design property, not a difficulty setting.

The reason the two clusters produce such different numbers for the same models is the whole argument, compressed. On SWE-bench Verified the frontier sits around 75-80% and above. On SWE-bench Pro, the same frontier models drop to roughly 23% on the public split and lower on the unseen commercial code - Scale AI. On the K Prize, the best entry managed 7.5% - TechCrunch. These are not different models being measured. They are the same models measured with and without the crutches of a leaked, static, narrow test. The size of the drop is a direct read-out of how much of the headline was crutch.

The middle of the table is where the interesting trade-offs live, and it rewards a second look. SWE-Lancer scores 7.2 despite being static, because it swaps freshness for economic grounding: its tasks are real freelance jobs with real dollar values, which is a completely different way to resist gaming. Terminal-Bench scores 7.5 because it moves the test into a live sandboxed environment where the model has to actually operate a machine, not just emit a diff. RE-Bench scores 7.1 because it measures against human experts on open-ended research work, which surfaces a truth a single percentage hides: who "wins" depends entirely on the time budget. No single benchmark maxes out every criterion, which is precisely why reading one number in isolation is the error. The trustworthy move is to triangulate across a fresh test, a hard held-out test, and an economically grounded test, and to distrust any capability story that only survives on the frozen, public one.

6. The benchmarks fighting back: freshness, held-out, and dollars

The healthiest sign in this whole story is that the people closest to the benchmarks are the ones building the replacements. Every serious new coding benchmark of the last two years is an attempt to remove one specific way the old ones lie, and they cluster into three repair strategies worth understanding on their own terms. The first is freshness, the idea that a model cannot memorize a problem that did not exist when it was trained. LiveCodeBench date-tags every problem and scores only the post-cutoff window - arXiv. SWE-bench Live adds 50 freshly verified GitHub issues every month through an automated pipeline - Hugging Face. SWE-rebench does the same at larger scale, continuously assembling tasks from thousands of repositories - Nebius. The unifying principle is a benchmark with no fixed answer key to leak.

The second strategy is to make the test held-out in the strongest possible sense: not merely fresh, but structurally impossible to have trained on. The purest example is the Konwinski Prize, a $1M competition from Andy Konwinski, a co-founder of Databricks and Perplexity, announced at NeurIPS 2024 - andykonwinski.com. It freezes the models, collects new issues after the deadline, and admits only open-weight models. Its first-round result, that famous 7.5%, is the single most clarifying data point in the field, because Konwinski himself framed it as the reality check: "If we can't even get more than 10% on a contamination-free SWE-Bench, that's the reality check for me" - TechCrunch. SWE-bench Pro reaches a similar guarantee commercially, using private startup codebases that model trainers cannot legally access - Scale AI.

The third and most economically honest strategy is to denominate the score in real money. OpenAI's SWE-Lancer collected over 1,400 real freelance software tasks from Upwork with a combined payout value of $1,000,000, ranging from $50 bug fixes to $32,000 feature builds, and pays a model the task's real dollar value only if its work passes end-to-end tests triple-verified by engineers - arXiv. The result reframes the whole conversation: the top model "earned" only around $400,000 of the available million, and the paper concludes flatly that "frontier models are still unable to solve the majority of tasks" - arXiv. A dollar-weighted score cannot be gamed by racking up trivial one-line fixes, because the market already priced those near zero.

RE-Bench deserves special attention because it exposes how a single headline hides the real shape of capability. METR pitted AI agents against 61 human experts on open-ended machine-learning research tasks and found the agents scored roughly four times the human expert score at a two-hour budget, but humans overtook them at eight hours and reached about twice the top agent's score at thirty-two hours - RE-Bench. Who "wins" flips entirely depending on the clock, which no single percentage can express. It is the reliability-cliff story from a different instrument: models are strong in short bursts and fade over long horizons, while humans start slower and compound. A benchmark that reports one number at one time budget is choosing, often silently, the budget that flatters its subject, and a launch chart never shows you the axis where the result would reverse. For the agent-operating version of this same problem, our roundup of AI computer-use benchmarks shows the pattern repeating in tasks that require driving a real machine rather than emitting a diff.

A few of these benchmarks attack a fourth axis entirely: the realism of the task itself. SWE-bench Multimodal adds 617 JavaScript tasks where bugs are reported with screenshots and mockups, and human annotation found that visual understanding is required to resolve 83.5% of them, a modality the Python-only benchmarks never test - arXiv. BigCodeBench demands complex, multi-library tool use across 139 Python libraries, where even a strong 2024 model solved only about 60% - arXiv. Terminal-Bench, led by Stanford and the Laude Institute, drops agents into real sandboxed terminals to compile code, configure systems, and debug, with a curated hard set that is deliberately revised as agents catch up - arXiv. The reason all of this matters for a decision-maker is simple: the moment any single one of these holes is plugged, reported capability falls, and it falls a lot. That is not the models getting worse. It is the measurement getting honest.

7. The 2026 scoreboard: what the flagship numbers actually say

Here is where the abstract argument meets the concrete claims you are reading in September 2026, and where you need the fine print more than ever. The current flagship coding models are Anthropic's Claude Opus 5 and the newer Claude Fable 5.1, OpenAI's GPT-5.6 Sol, and Google's Gemini 3.1 Pro, with strong open-weight competition from DeepSeek V4 and Moonshot's Kimi K3. The instructive part is not the ranking. It is how differently each vendor now talks about SWE-bench, because the way a lab reports a number tells you how much it trusts that number.

Start with the most revealing move of the year. In February 2026, OpenAI publicly stopped reporting SWE-bench Verified, and said why in unusually blunt terms: "Improvements on SWE-bench Verified no longer reflect meaningful improvements in models' real-world software development abilities; instead, they increasingly reflect how much the model was exposed to the benchmark at training time" - OpenAI. Their own audit of a model's failures found 59.4% were caused by test flaws rather than model limitations, and they observed direct contamination signals, including a model leaking Django release-note details that were never in its prompt. OpenAI now leads with SWE-bench Pro and Terminal-Bench instead, where its flagship scores in the 60s, not the 90s.

Anthropic's approach is different and equally telling. For its generation-5 models it largely does not headline SWE-bench Verified at all, leading launch materials with its own Frontier-Bench, Terminal-Bench, and agentic-work evaluations. The mid-90s figures you will see quoted for Claude Opus 5 (around 96-97%) come from third-party leaderboards such as vals.ai, not from Anthropic's own marketing table, and they are produced on a lab-controlled agent harness that reports a mean over several attempts rather than a single pass - Anthropic. Google is the most conservative of the three: Gemini 3.1 Pro reports 80.6% on SWE-bench Verified explicitly under "single attempt" conditions, the only clean, officially-published, apples-to-apples Verified number among the majors - Google DeepMind.

The single clearest illustration that the number is a property of the harness, not the model, comes from cross-checking one model against itself. Independent evaluators measured the same OpenAI flagship at 82.2% on one standardized harness while aggregators listed it near 96% on another. The same effect shows up in a fully open example: Alibaba's Qwen3-Coder-Next reports 70.6% with one agent scaffold and 71.3% with another, in its own technical report, with nothing changing but the wrapper - arXiv. When a single model moves ten to fourteen points depending on who runs it and how, "Model A beats Model B by three points" is not a finding. It is noise dressed as a result. If you want the version of this comparison that actually accounts for harness and cost, our head-to-head on GPT-5.6 vs Claude Opus 5 as agents and the broader best-LLM-for-agents ranking hold the scaffold constant.

The open-weight field sharpens the point rather than softening it. Independent leaderboards place DeepSeek V4 and Moonshot's Kimi K3 in the mid-90s on SWE-bench Verified, close to the closed frontier, and Kimi K3 is fully downloadable - Vals AI compilation. Taken at face value, that would mean a free model nearly matches the best proprietary systems at software engineering. Taken with everything in this guide, it means the open models have also thoroughly absorbed the public benchmark, which is exactly what you would expect from models trained on the same public GitHub. The convergence at the top of Verified is not evidence that everyone solved engineering; it is evidence that the test saturated for everyone at once. A study profiling the leaderboard's own submissions found many are effectively unreproducible, with missing code, proprietary agent loops, and thin documentation, so even the ranking of who sits where "requires critical interpretation" - Dissecting the SWE-Bench Leaderboards. If you are choosing an actual tool rather than admiring a chart, our comparisons of Claude Code vs Codex vs Devin and the best AI coding CLI of 2026 hold the harness fixed so the comparison actually means something.

The practical takeaway for 2026 is a reading protocol, not a ranking. When you see a coding score, ask three questions before you believe it. Which variant was measured, since Verified and Pro differ by 40-plus points. Whose harness produced it, since the wrapper is worth 10-plus points. And single attempt or best of many, since parallel sampling flatters the number without improving what the model does on your first try. A vendor that answers all three plainly, as Google does with "single attempt," is handing you a usable measurement. A vendor that quotes a bare percentage from a third-party board on an unnamed harness is handing you a marketing asset. The reason OpenAI walked away from Verified is that it could no longer answer those questions honestly, and it deserves credit for saying so rather than continuing to quote a number it knew was hollow.

8. Benchmarketing: Goodhart, the leaderboard illusion, and the money

Everything so far has been mechanics. This chapter is about why the mechanics bend in one direction, because a benchmark does not drift toward inflation by accident. It drifts because of Goodhart's law: when a measure becomes a target, it ceases to be a good measure - glossary. This is not a slogan. OpenAI's own research paper Measuring Goodhart's Law shows it quantitatively: as you optimize a policy against a proxy reward, the true objective rises, plateaus, and then falls, even while the proxy score keeps climbing - OpenAI. The proxy and the goal actively diverge under optimization pressure. A public leaderboard is that exact loop, run at industry scale, with every lab pushing on the same proxy at once.

Now add the money, because the money is what makes the pressure irresistible. OpenAI and Anthropic together hold about 80% of the venture funding across the top AI startups, some $242.6B of $305.6B raised, and Anthropic disclosed a roughly $30B revenue run-rate in 2026 - MarketScale. Launch narratives, target-price revisions, and enterprise deals are all anchored to benchmark wins through a flywheel: better score, better press, more revenue, more financing, more compute, better score. When that much capital rides on a single number, the number stops being a measurement and becomes a marketing surface, and rational actors optimize it directly. Goodhart's law then guarantees the rest.

The gaming has been documented at every layer of the stack, which is what elevates this from anecdote to pattern. At the human-preference layer, the paper The Leaderboard Illusion found that Meta privately tested 27 Llama-4 variants on Chatbot Arena and published only the best-scoring one, and that a handful of large providers received far more arena data than dozens of open models combined, buying overfitting gains of up to 112% on the arena distribution - arXiv. Meta then submitted a chat-tuned "experimental" model that ranked #2, which the public never received, prompting the arena operator to concede that "Meta should have made it clearer that Llama-4-Maverick-03-26-Experimental was a customized model to optimize for human preference" - The Register. Simon Willison called the practice "a classic example of gaming a leaderboard" - Willison.

The asymmetry underneath that story is the part that should bother a careful reader most. The same analysis found that a few large providers each received roughly a fifth of all arena battle data, while 83 open-weight models combined received under a third, and that 205 of 243 public models were silently removed from the board rather than officially retired, a practice that fell disproportionately on the open models - The Leaderboard Illusion. When a handful of well-resourced labs get an order of magnitude more feedback from the leaderboard than everyone else, and can quietly retire the runs that did not go well, the board stops measuring general capability and starts measuring who has the most access to the board. That is not a rogue actor cheating an otherwise fair system. It is a structural advantage the ranking itself confers on its largest participants, which is a more troubling thing, because it means the distortion is built into the incentive, not bolted on by a bad apple.

At the scaffold layer, the published number is a model-plus-scaffold result, not a model result, and a custom agent loop can add 10 to 30 points over a standardized harness - Codesota. A benchmarks writer put the general form crisply: a score is the output of a function of model, settings, harness, and scoring, and "if you change any variable in that tuple, the score changes, often dramatically" - Shrivu Shankar. This is why sophisticated enterprise buyers have stopped chasing the leaderboard entirely. As one CIO put it, "the durable advantage was never the model, the models are the easy part" - InformationWeek. They run their own evaluations on their own data precisely because they know the public proxy has been consumed by Goodhart. The lie, in other words, is not a bug in any one benchmark. It is the predictable equilibrium of pointing billions of dollars at a single frozen number, and the only defense is to stop treating the number as sacred.

9. The reality gap: what a 75% score does not buy you

If contamination and gaming explain why the headline is inflated, this chapter explains why even an honest headline would still mislead, because the benchmark measures a slice of engineering that barely overlaps with the job. The most rigorous evidence is a randomized controlled trial from METR in July 2025. It took 16 experienced open-source developers, working on their own large repositories, and measured how long they took to complete 246 real issues with and without AI tools. The developers expected a 24% speedup. They felt like they got a 20% speedup. In reality, they were 19% slower with the AI tools - METR. On the exact kind of mature, unfamiliar, real-world codebase that benchmarks never test, the tools that "pass 75% of SWE-bench" made expert humans measurably worse.

The perception gap in that study is not a footnote. It is the reason the benchmark culture persists: developers who were objectively slowed down still believed they had been sped up, which is exactly the psychology that lets a leaderboard number override lived experience. The gap is structural, and METR's companion work explains it. Their "time horizon" study reframes capability as how long a task a model can complete before its reliability collapses, and found that the frontier of the time sat around 50 minutes of human-equivalent work at 50% reliability, with success falling below 10% on tasks that take a human more than about four hours - METR. A single-attempt benchmark percentage hides this completely, because it averages over short tasks where reliability is near 100% and never asks the model to sustain correctness across a long chain.

The reliability framing matters because real engineering is a time-integral, not a point-in-time event, and errors on a long horizon cascade rather than average out. An 80% per-step success rate feels great on a benchmark and is catastrophic across a fifty-step task, because 0.8 to the fiftieth power is effectively zero. This is why a model can be a superb turn-by-turn assistant and a poor autonomous engineer at the same time: the benchmark rewards the first, the job requires the second, and the difference is precisely the long-horizon reliability the benchmark never scores. Our guide to long-running coding agents is really a guide to fighting this cliff, and the pattern generalizes far beyond code, as our analysis of why most AI agent pilots never scale documents from the enterprise side.

The distribution mismatch is measurable, not just rhetorical. The creators of SWE-bench Pro pointed out that 161 of the 500 Verified problems require only a one-to-two-line modification, while their own harder benchmark's reference solutions average 107 lines of code across 4.1 files, with every task demanding at least ten lines changed - SWE-bench Pro. Real feature work and real refactors live in that second distribution, not the first. So a model can be genuinely good at the small, localized, well-specified edits that dominate Verified and still fall off a cliff on the multi-file, ambiguous, integrate-across-subsystems work that dominates an actual paycheck. The benchmark is not lying about the small edits; it measures those faithfully. It is quietly letting the reader assume the small edits represent the job, and they do not. This is the same lesson our guide to what software is left to build in 2026 reaches from the opposite direction: the hard, valuable work is precisely the part no model one-shots.

Two other real-world signals corroborate the gap from different directions. Stack Overflow's 2025 developer survey found adoption rising while trust falls: 84% of developers use or plan to use AI tools, yet 46% now distrust the accuracy of the output, up sharply from 31% the year before, with debugging AI-generated code cited as a top pain - Stack Overflow. And GitClear's analysis of 211 million lines of code found that AI assistance is quietly degrading maintainability: code duplication rose eightfold and refactoring fell nearly 40% as adoption spread - DevClass. A benchmark scores the patch that passes the test today. It is blind to the churn, the duplication, and the "almost right but not quite" tax that real teams pay downstream, which is the tax that turned a predicted speedup into a measured slowdown.

This distinction, between a test going green and the work actually shipping, is one that Yuma Heymans (@yumahey) returns to often. As the founder of the autonomous-company platform o-mega and co-founder of the AI recruiter HeroHunt.ai, he builds systems whose only honest scorecard is whether the outcome landed in production, not whether a proxy metric moved, which is the same discipline this guide argues you should apply to every benchmark you read.

10. How to read a benchmark, and how to run your own

The point of all this is not cynicism. Benchmarks are useful; they are just not self-interpreting, and the skill worth building in 2026 is reading them like a professional rather than a spectator. The first habit is to treat every score as a claim with attached conditions, and to refuse to compare two scores until the conditions match. Which variant, whose harness, single attempt or best of many, and what date the tasks were collected relative to the model's training cutoff. If any of those four is unstated, the number is not comparable to another number, and treating it as comparable is the single most common mistake in this space. A model that quotes Verified and a model that quotes Pro are not forty points apart in skill; they are forty points apart in test choice.

The second habit is to weight fresh and held-out results far above static public ones, in direct proportion to the trust scores in this guide's table. A strong number on the Konwinski Prize, on SWE-bench Pro's commercial split, or on a post-cutoff LiveCodeBench window is worth many times a strong number on frozen SWE-bench Verified, because the first three could not have been memorized and the last one demonstrably was. When a model looks great on the famous benchmark and mediocre on the honest ones, believe the honest ones, because the gap between them is a measurement of contamination, not of capability.

The third habit, and the one every serious buyer eventually adopts, is to build a private evaluation on your own work. This is exactly what the CIOs quoted earlier do, and the logic is airtight: a held-out eval on your own tasks is the one test no vendor has optimized against, so it is the only one still measuring the thing you care about. In practice this means collecting a set of recent, representative tasks from your own repository, running each candidate model through the harness you would actually use in production, and grading on real correctness rather than a sparse test oracle. It is the same principle as hiring: you do not hire on the strength of a standardized test score, you give a work-sample of the actual job. The tooling to assemble that kind of private, on-your-own-data evaluation has matured, and platforms for orchestrating AI agents against your own workflows, o-mega among them, exist precisely so a non-technical team can measure outcomes on their own tasks rather than inherit a vendor's leaderboard framing.

A concrete version makes this tangible. Suppose you maintain a mid-sized web application and you are choosing between two models. Instead of comparing their SWE-bench headlines, you pull the last fifteen bugs your team closed, strip out the fixes, and hand each model the issue and the repository under your real continuous-integration setup. You grade three things: did the tests pass, did a senior engineer agree on review that the fix was actually correct, and did it avoid adding duplication or dead code. The model that wins that exercise is the one to deploy, even if it carries the lower public benchmark score, because your fifteen tasks are the only ones neither vendor optimized against. It is more effort than reading a bar chart, and it is the entire difference between a decision and a guess. The teams that do this consistently report that the leaderboard ranking and the private-eval ranking disagree often enough that trusting the leaderboard alone would have been an expensive mistake.

The fourth habit is to match the benchmark to the decision, because different tests answer different questions and none answers all of them. If you are choosing a model to assist a supervising engineer turn by turn, a short-horizon benchmark tells you something real, because a human is catching the errors. If you are choosing a model to run autonomously for hours, the METR time-horizon and reliability-cliff data matter far more than any single percentage. If you care about front-end or visual work, SWE-bench Multimodal's finding that 83.5% of its tasks need vision tells you the Python-only number is silent on your use case. The right benchmark is the one whose task distribution overlaps your actual work, and the honest answer is usually that you need two or three, read together, plus your own eval as the tiebreaker. Our practical guides on when to graduate from a vibe-coding tool and building software with AI end to end are written around this "measure your own outcome" discipline rather than the leaderboard.

11. The future: rotating evals and agents grading agents

Where does this go? The most likely near-term future is that the static public benchmark, as a marketing instrument, quietly dies, and the field converges on continuously rotating, held-out evaluation as the only structurally honest option. This is already the direction of travel: SWE-bench Live refreshes monthly, SWE-rebench refreshes continuously, the Konwinski Prize runs in rounds with new post-deadline issues each time, and Konwinski himself expects the dynamic to become adversarial in a healthy way, noting that "we expect people to adapt to the dynamics of competing on this every few months" - TechCrunch. A benchmark that never stops moving is the only kind that a public model cannot eventually memorize, so the frozen leaderboard is on borrowed time.

The deeper reason this shift is inevitable is that Goodhart's law comes for every benchmark you trust, as one recent piece put it, and it comes faster now than ever - Communications of the ACM. Benchmarks that were designed to last years, like MMLU in 2020, now saturate within months of becoming optimization targets, because the entire industry can throw compute and data at a fixed target simultaneously. The implication is uncomfortable for anyone who wants a single stable number: there is no such thing as a durable public benchmark in a field this well-funded and this fast. The best you can build is a process that keeps generating fresh, unseen tests faster than they can be gamed, which is a fundamentally different object from a leaderboard.

The second frontier is grading. As tasks get longer and more open-ended, the sparse-unit-test oracle that broke SWE-bench becomes untenable, and the field is moving toward agents evaluating agents: using strong models as judges, differential testing tools like PatchDiff to compare a candidate fix against a reference on inputs the original tests never covered, and economic grounding like SWE-Lancer's to let the market price the task. Each of these is an attempt to replace "did these specific tests pass" with something closer to "is this actually correct and worth what a human would pay." None is perfect, and using models to grade models introduces its own biases, but the direction is clearly toward richer oracles and away from the brittle, gameable, pass-or-fail test set that made the last generation of numbers so easy to inflate.

For the reader trying to make decisions through all of this, the enduring principle is the one this guide has circled from the start, and it will outlast every specific model and benchmark named here. A score is a proxy, and a proxy is trustworthy only until it becomes a target. The famous AI coding benchmarks became targets years ago, which is exactly why their numbers climbed toward saturation while real-world productivity studies, contamination audits, and held-out competitions told a much more sober story. The models are genuinely improving; that part is real, and the honest benchmarks show steady if unspectacular gains. But "the model improved" and "the benchmark number went up" are different sentences, and 2026 is the year the gap between them became the most important thing to understand about AI and software. If you internalize one habit, make it this: when someone quotes you a coding score, ask what it could not have seen, and how it would do on your own work. The answers to those two questions are worth more than any leaderboard.

This guide reflects the AI coding benchmark landscape as of September 2026. Model names, versions, and reported scores change constantly (Anthropic, OpenAI, and Google all revised their coding models in the months before publication), so verify current figures against primary sources before making a decision.