Contamination in Coding Benchmarks: Why SWE-bench Scores Lie

The Benchmark Everyone Trusts Is Quietly Broken

If you've been picking your AI coding tool based on SWE-bench Verified scores, you're choosing on a leaderboard that its own creators no longer trust. In February 2026, OpenAI announced it would stop reporting SWE-bench Verified scores entirely — and the announcement came from Mia Glaese, a coauthor of the original benchmark, who described it as "effectively saturated and also highly contaminated."

That's not a minor methodological quibble. It's the de facto industry standard for measuring coding ability being retired by the people who built it. Glaese and Olivia Watkins stated plainly that the eval is "not really measuring coding performance improvements well anymore."

The contamination is structural, not accidental. SWE-bench problems are sourced from open-source repositories like Django — the same repositories prominently featured in the training corpora of major LLMs. Because the data comes from GitHub, no canary string could ever be embedded to detect leakage. The result, according to OpenAI's own disclosure, is that all frontier models tested — GPT-5.2-Chat, Claude Opus 4.5, and Gemini 3 Flash Preview — could reproduce gold patches or verbatim problem details for certain tasks. In one Django task, GPT-5.2 reproduced the exact human-written fix, down to a specific early-return condition, on a problem OpenAI classified as "almost impossible to solve" legitimately.

Independent academic work reaches the same conclusion from a different angle. A Purdue and Microsoft study found that state-of-the-art models can identify the buggy file in a SWE-bench task with up to 76% accuracy using only the issue description and no repository access — a feat the researchers call logically impossible without prior exposure. On repositories outside the benchmark, accuracy drops to at most 53%.

And contamination is only half the problem. The tests themselves are broken:

• An audit of 138 problems found 59.4% had material flaws in test design or problem description.
• An estimated 24% of the top-50 leaderboard positions are incorrect because unit tests miss edge cases.
• Agents can pass evaluations without producing correct patches on 5.2% of SWE-bench Verified tasks.

OpenAI likens the situation to "sharing problems and solutions for an upcoming test with students before the test." Scores that look like capability gains increasingly reflect training-time exposure instead. Frontier models now cluster around 80% on the benchmark, with state-of-the-art progress crawling from 74.9% to 80.9% over six months — the signature of a saturated metric, not a stalled field.

This is exactly why benchmark design matters more than benchmark scores. It's the principle behind SkillzBench, AIQ Labs' independent testing lab for agent skills: controlled conditions, the same model tested with and without a skill, and results published as a transparent Lift in percentage points rather than a leaderboard position. When the industry's most trusted coding benchmark can be quietly memorized, independent, contamination-resistant testing isn't a nice-to-have — it's the only signal left worth trusting.

Want the honest numbers as they emerge? Join the SkillzBench waitlist — skills, proven.

Contamination Isn't a Bug — It's Structural

If a model can find the buggy file in a codebase it has never seen, using nothing but the issue description, something is deeply wrong with the benchmark. On SWE-bench Verified, that logically impossible task is routinely solved — which tells you the models aren't solving it at all. They're remembering it.

The contamination is structural, not accidental. SWE-bench tasks are drawn from open-source repositories like Django, the same repos that dominate the training corpora of major LLMs. As Mia Glaese, coauthor of SWE-bench Verified, explained, there is no way to embed a canary string in data sourced from GitHub, so model developers have no practical mechanism to exclude it. The test set and the training set are, in effect, the same corpus.

The smoking-gun numbers converge on the same conclusion:

• State-of-the-art models identify buggy file paths from issue text alone with up to 76% accuracy — with no repository access. On tasks from repos not included in SWE-bench, accuracy drops to at most 53%.
• Claude models performed roughly 3x better on file localization for SWE-bench Verified than on comparable benchmarks, and about 6x better at finding edited files when given only the issue text.
• Performance collapses by up to 47 percentage points when evaluating on repositories written in a different language (C#) — a drop that makes no sense if the skill being measured is general debugging ability.
• 94% of SWE-bench issues predate LLM training cutoffs, and post-cutoff variants show significantly lower resolution rates.

The most damning evidence comes from OpenAI itself. In its February 2026 announcement retiring the benchmark, OpenAI reported that all frontier models tested — GPT-5.2-Chat, Claude Opus 4.5, and Gemini 3 Flash Preview — could reproduce original human-written gold patches or verbatim problem details for certain tasks. In one Django task, GPT-5.2 reproduced the exact gold patch, down to the specific early-return condition. OpenAI likens this to handing students the problems and solutions before the exam.

This is why SkillzBench treats contamination resistance as a design requirement, not an afterthought. When benchmarking agent skills, we're building controlled comparisons — the same model, the same sandbox, tested with and without the skill — so the Lift we publish reflects what the skill actually contributes, not what the model memorized. Task provenance and freshness are part of the methodology we plan to document openly.

A benchmark score is only as trustworthy as the firewall between the test and the training data. SWE-bench never had one, and its own authors have conceded the point.

Even Without Contamination, the Tests Themselves Fail

Here's the uncomfortable truth: even if you scrubbed every trace of contamination from a coding benchmark, the scores would still lie — in both directions. Flawed test design means agents get credit they didn't earn, and lose credit they deserved.

OpenAI's own audit found that 59.4% of the 138 problems it examined had material flaws in test design or problem description, with 35.5% relying on tests so narrow that wrong implementations could pass. That finding is part of why OpenAI stopped reporting SWE-bench Verified scores in February 2026.

The downstream consequence is measurable. An analysis of leaderboard integrity estimated that 24% of the top-50 SWE-bench Verified positions are incorrect because unit tests miss edge cases, according to the Agentic Benchmark Checklist (ABC) study from a multi-institution team including Percy Liang and Matei Zaharia. Worse, agents can pass evaluations without generating a correct patch on 5.2% of SWE-bench Verified tasks — the test says "pass," the code says "broken."

Mia Glaese, a coauthor of SWE-bench Verified, put it plainly: just because an agent didn't pass a bad test doesn't mean its implementation wasn't a good one. Scores lie in both directions.

And this isn't a SWE-bench problem. The ABC framework audited 10 popular agent benchmarks — including OSWorld, KernelBench, and τ-bench — and found:

• 7 of 10 fail outcome validity, meaning the evaluation doesn't truly indicate success
• 7 of 10 contain shortcuts or impossible tasks that distort results
• A "do-nothing" agent that sends empty responses scores correct on 38% of τ-bench airline tasks — beating a GPT-4o-based agent at 35%
• KernelBench overestimates agent capability by 31 percentage points due to incomprehensive fuzz testing

The lesson generalizes: any benchmark that rewards passing tests rather than solving problems can be gamed by doing nothing, and any benchmark with weak tests can reward broken code. That's why SkillzBench is building its skill benchmarking around controlled, side-by-side testing — the same model, the same sandbox, with and without a given skill — and publishing the Lift in percentage points rather than an absolute score that flawed tests could inflate or deflate. Methodology transparency, not leaderboard position, is the credibility signal that matters now.

If the industry's most-watched benchmark can't be trusted to measure what it claims, independent verification isn't a nice-to-have. It's the only honest option left.

What Contamination-Resistant Evaluation Actually Requires

If SWE-bench Verified scores can't be trusted, what would a benchmark that can be trusted actually look like? The research points to a handful of design principles — and they're demanding.

The first requirement is fresh, documented tasks. The contamination evidence is overwhelming: 94% of SWE-bench issues predate LLM training cutoffs, and when researchers built post-cutoff alternatives, resolution rates dropped significantly. A trustworthy benchmark must either use tasks created after model cutoffs or purpose-built tasks with provenance you can verify — not problems scraped from GitHub repositories that every major training corpus includes.

The second is controlled, same-model conditions. Absolute scores invite memorization and gaming; relative measurement doesn't. When OpenAI probed frontier models, all three could reproduce gold patches or verbatim problem details for certain tasks — proof that absolute numbers on reused tasks measure exposure, not ability. Testing under identical conditions, with everything held constant except the variable being measured, is the honest alternative.

The third is validity checking. The Agentic Benchmark Checklist (ABC) framework defines two criteria that most benchmarks fail: Task Validity (the task is solvable if and only if the agent has the target capability) and Outcome Validity (the evaluation genuinely indicates success). These aren't academic niceties. In a review of ten popular agent benchmarks, seven contained shortcuts or impossible tasks, and seven failed outcome validity — with misestimation reaching 100% in some cases, according to research from a multi-institution team including Percy Liang and Matei Zaharia.

The fourth is radical methodology transparency. OpenAI's announcement earned credibility precisely because it published its methods: an automated red-teaming setup where GPT-5 probed target models over 15 turns, judge models labeling contamination severity, multi-expert audits of 138 problems, and manual review. Vague claims of "rigorous testing" mean nothing; reproducible methods mean everything.

This is the standard SkillzBench is being built against. Rather than chasing absolute leaderboard scores, it's designing its Benchmark pillar around controlled relative measurement: every skill is tested in a sandbox with the same model, with and without the skill, and the result is published as a Lift in percentage points. Lift answers a different question than SWE-bench does — not "how good is this model," but "does this skill actually help, under identical conditions?" Because both runs share the same model and task set, memorization inflates both sides equally and cancels out.

The same transparency principle extends to the rest of the pipeline: pinning every skill version to a commit SHA, running security scans, and keeping commercial influence out of scores entirely. As the industry abandons benchmarks its own authors call "effectively saturated and also highly contaminated," per the benchmark's original creators, controlled relative measurement isn't just more honest — it's the only measurement left standing.

If you want to know which skills actually help your agents before you install them, join the SkillzBench waitlist. Skills, proven.

How to Evaluate Skills When You Can't Trust the Numbers

If a quarter of a leaderboard can be wrong, what exactly are you buying when you trust a benchmark score? Researchers found that an estimated 24% of the top-50 SWE-bench Verified leaderboard positions are incorrect, largely because unit tests miss edge cases. And that was before the benchmark's own authors declared it "effectively saturated and also highly contaminated."

So how do you evaluate anything — a model, an agent, or a skill — when the headline numbers can't be trusted? You change what you ask for.

Treat vendor claims with skepticism, especially absolute scores. OpenAI's own analysis found that all frontier models tested could reproduce gold patches or verbatim problem details for certain SWE-bench tasks, and one model solved 31 tasks flagged as "almost impossible to solve" without prior exposure. When the people selling you a capability are also the ones reporting the score, and the score can be inflated by memorization, the absolute number tells you more about training data than ability.

Ask about task provenance and test audits. Contamination is structural: SWE-bench problems come from open-source repositories that appear prominently in LLM training corpora, and canary strings can't be embedded in GitHub-sourced data. The strongest evaluations now document where tasks came from and whether they postdate training cutoffs — 94% of SWE-bench issues predate LLM cutoffs, and post-cutoff variants showed significantly lower resolution rates. If a benchmark won't tell you how fresh its tasks are, assume the worst.

Favor relative effect sizes over absolute scores. A controlled comparison — same model, same conditions, with and without the thing being tested — is far harder to game than a leaderboard position. This is the logic behind SkillzBench's Lift metric: every skill is benchmarked in a sandbox with the same model, with and without the skill, and the result is published as Lift in percentage points. A relative measurement under controlled conditions measures what the skill actually changes, not what the model memorized.

Demand published methodology. OpenAI's deprecation announcement earned credibility precisely because it showed its work — automated red-teaming over multiple turns, expert audits, manual review. The broader benchmark literature agrees transparency matters: researchers found severe validity issues in 8 of 10 popular agent benchmarks, sometimes causing up to 100% misestimation of capabilities, and most fail to disclose known problems. Methodology you can read is the minimum bar.

This is the standard SkillzBench is building against. Every skill gets pinned to a commit SHA, security-scanned, and quality-judged before it's ever benchmarked. Trust scores combine security, quality, license, maintenance, and community signals — and Lift is always displayed separately from ratings, with money never touching scores. The waitlist is open now; the benchmarking infrastructure is actively being built, and early readers will see the methodology before the results.

Skills, proven. Join the waitlist at skillzbench.com and evaluate your next skill on evidence, not vibes.