What 80.6% on SWE-bench Verified Doesn't Tell You

On February 23, 2026, OpenAI stopped reporting the number the entire industry ranks coding models by. It published a post titled "Why we no longer evaluate SWE-bench Verified," walked through an audit of the benchmark, and recommended that other labs drop it too. Six months later, DeepSeek shipped V4-Pro and led its launch with 80.6% on that same SWE-bench Verified, tied with Gemini 3.1 Pro at the top of the open-weights field. Both things are true at once. The benchmark's biggest user called it no longer meaningful, and the field kept quoting it as if nothing had happened.

That gap is the whole subject of this piece. Not "is 80.6% a good score" but "80.6% is a measurement of what, exactly." Because the honest answer is not "DeepSeek V4-Pro's coding ability." The honest answer is that a number on an agentic-coding benchmark belongs to a bundle: the model, plus the scaffold that drives it, plus the prompt, plus the retry budget, plus the benchmark's own test quality. Vendors ship the whole bundle and put only the model's name on the number. This is the evaluation crisis in its most expensive form, because coding scores are the ones buyers act on.

None of this ages out with the next release. The models in the examples will be superseded within a quarter. The mechanism will not. So the goal here is a reading protocol you can run against any agentic-coding number, with the 2026 releases slotted in as the evidence that the mechanism is live.

The number is a property of a bundle, not a model

Start with what SWE-bench Verified actually is. It is 500 real GitHub issues from twelve Python repositories, each paired with the pull request that fixed it and a set of tests that the fix makes pass. A model "solves" a task when a harness gives it the repository, lets it edit files, runs the hidden tests, and they pass. The "Verified" in the name refers to a 2024 pass where human annotators filtered the original SWE-bench down to problems they judged well specified. It never meant "verified clean of contamination," and it never meant "verified that the tests grade the right thing." Those two assumptions are exactly the ones that broke.

Notice how many moving parts sit between "the model" and "the score." The model does not see a GitHub issue and emit a patch in one clean forward pass. An agent scaffold decides how the repository is presented, how many files the model can read, whether it can run the tests itself and iterate, how many attempts it gets, and how a candidate patch is selected. This is the same scaffold whose behavior tool-use evaluation treats as a first-class object of measurement rather than a transparent wrapper around the model. Two teams can run the identical model weights through two different scaffolds and report numbers that differ by more than the gap between model generations. The score you read is the output of that pipeline, and the pipeline is the part nobody publishes.

This is why "a leaderboard number is not a result" is the load-bearing sentence. A result has a baseline, a distribution, and a confidence envelope. A benchmark number, published alone, has none of those. It is a single point estimate from an unspecified harness against a test set of unknown quality, and the reader is invited to attribute all of it to the model. The rest of this piece is about the five things that point estimate hides, and the two OpenAI audits that turned the abstraction into a documented event.

The withdrawal nobody priced in

The reason to lead with OpenAI's audits rather than a general argument is that they are the concrete backbone. This is not a critic on the outside claiming a benchmark might be flawed. It is the benchmark's most prominent user auditing the tasks its own model kept failing, publishing what it found, and changing its reporting policy twice in under half a year.

The Verified audit: the tests were the problem

OpenAI took the 138 problems, 27.6% of the 500-problem set, that its o3 model could not solve consistently across 64 independent runs, and it read them. The premise is worth pausing on: if a capable model fails the same task 64 times, either the task is genuinely hard or the task is broken, and the way to tell is to look. At least 59.4% of those 138 had material problems in the test design or the problem description that made them extremely difficult or impossible even for a capable human or model.

The failures split into two shapes that are worth naming because you will recognize them in your own test suites. Narrow tests, about 35.5% of the flagged set, rejected functionally correct solutions because they demanded a specific implementation detail that the problem statement never mentioned. The model wrote code that fixes the issue; the grader wanted the exact code the original PR author wrote, and marked a working fix wrong. Wide tests, about 18.8%, checked for behavior the task never asked for, so a faithful solution to the stated problem failed a test for something else entirely. In both cases the model was penalized for the grader's defect, not its own.

Then the second finding, which is the one that gets the headline and deserves the smaller share of your attention: OpenAI reported evidence that every frontier model it checked, across labs, showed training-data overlap with the benchmark's solutions. This is the contamination story, and it is real, but it is the part that is hardest to act on and easiest to sensationalize. The test-quality finding is more useful precisely because it is mundane and fixable: a benchmark can be decontaminated in principle, but if a third of its graders reject correct code, the ceiling on what the benchmark can tell you is capped no matter how clean the split is. OpenAI's own summary was that improvements on Verified "increasingly reflect how much the model was exposed to the benchmark at training time" rather than real gains, and it stopped reporting the number.

The Pro reversal: it happened again

In the same February post, OpenAI pointed people toward SWE-bench Pro, a tougher successor built by Scale AI and designed to resist contamination. That recommendation lasted about four and a half months. Around July 8, 2026, in a post titled "Separating signal from noise in coding evaluations," OpenAI reported that it had audited SWE-bench Pro and found roughly 30% of its tasks broken. Its automated datapoint-analysis pipeline flagged 27.4%; human annotation by experienced software engineers flagged 34.1%, with the humans applying multiple labels per task and being the more conservative judges of what counts as broken. The failure modes were the same family as before: hidden requirements, contradictory instructions, overly strict tests, incomplete grading. OpenAI formally retracted its recommendation.

Sit with the shape of this. In under six months, the same lab audited the field's two most-trusted agentic-coding benchmarks and withdrew from both, for the same underlying reason: the tests do not grade what they claim to grade. That is not a story about one bad dataset. It is evidence that building a large agentic-coding benchmark whose graders are actually correct is a genuinely hard, unsolved data problem, and that the leaderboards were being read as if it were solved. If you are attributing a model's business value to its rank on one of these boards, the boards' biggest user has told you twice that the ranking is noisier than the decimal places suggest.

Same system, two numbers

Contamination and broken tests are the benchmark's own defects. The next failure mode is not the benchmark's fault at all. It is that the same model produces very different scores depending on the machinery around it, and that machinery is the part vendors are free to tune and free not to disclose.

The harness is worth 15 to 30 points

Scale's SWE-bench Pro is instructive here because it publishes two things: a standardized public leaderboard where every model runs identical scaffolding, and the vendor-reported figures. On the standardized board, where the harness is held fixed, the top frontier systems sit around the high 50s in August 2026 (GPT-5.4 at its highest reasoning setting posts 59.1% on the public set). The vendor-reported numbers for the same class of system run 15 to 30 points higher. Nothing changed about the models between those two figures. What changed was the scaffold, the prompt, the reasoning budget, and the number of attempts. That spread is a direct measurement of how much of a coding score is harness rather than model, and it is larger than the gap most buyers imagine separates one model generation from the next.

This is also why cross-vendor coding comparisons from press releases are close to meaningless. When lab A reports a Verified number from its own agent framework and lab B reports one from a different framework with a different retry policy, you are comparing two bundles that differ in more places than the models do. The only comparison that isolates the model is the standardized-harness board, and the standardized board is precisely the one whose numbers are lower and less flattering, which is why launch posts do not lead with it.

Which SWE-bench, and pass@1 versus pass^k

Two more axes hide inside a bare number. The first is which benchmark the word "SWE-bench" refers to. Verified numbers run far above Pro numbers for the same system, because Pro was built to be harder and contamination-resistant. A model that reads as 80 on Verified is not an 80 on Pro; it is something in the 50s on the standardized Pro harness, if the tasks it ran were not among the broken third. Quoting "SWE-bench" without the split and the board is like quoting a temperature without saying which scale.

The second is the retry budget, which is the difference between pass@1 and pass^k. pass@1 asks whether the model solves the task on a single attempt. Real agents retry: they run the tests, read the failure, and try again. A pass@1 number and a "best of k attempts" number describe different systems, and the second is always higher. The more honest reliability metric is pass^k, the probability the model solves the task on all k independent attempts, which measures consistency rather than luck. This is the same distinction the eval-honesty argument keeps returning to: a benchmark that reports the best of many runs is measuring the tail of a distribution and reporting it as the center. A model that solves a task once in eight tries and a model that solves it eight times in eight both post pass@1 near 1.0 with a generous harness, and they are not the same model to ship.

Here is the full accounting of what a single agentic-coding number leaves out.

Every row is a place the number moves without the model changing. A vendor is under no obligation to lie to inflate a coding score; the defaults do it for them, because the flattering choice on each row is also the legal, defensible one.

The 2026 open field makes the bundle problem worse

The reflex reading of the last two years is that open-weights models are closing the gap on coding, and that reading is correct on the numbers. DeepSeek V4-Pro, a roughly 1.6-trillion-parameter mixture-of-experts model with 49B active per token under an MIT license, posts 80.6% on Verified and ties the top of the field. Zhipu's GLM line posts a vendor-reported 62.1% on SWE-bench Pro (GLM-5.2, a 753B MoE with about 40B active and a 1M-token context under MIT; GLM-5.3 followed in August at the same size, scaled through post-training). Kimi K3, a 2.8-trillion-parameter open MoE activating 16 of 896 experts per token, sits near the top of aggregate indices. Open coding models are genuinely posting near-frontier agentic scores. That part is real, and it is the backdrop to the open-source power shift this blog has tracked.

But the same abundance makes the measurement problem worse, not better, for three reasons.

First, more entrants means more bundles competing on the same headline metric, which raises the payoff to harness tuning. When four labs are within a few points of each other on Verified, the rational move for each is to invest in the scaffold and the retry policy, because that is where the cheap points are, and none of that investment shows up as a disclosed variable. The leaderboard tightens and the fraction of the score that is model rather than machinery shrinks.

Second, the benchmarks these models are topping are the ones OpenAI already audited and abandoned. A frontier Verified number in mid-2026 is a number on a test set whose most prominent user has publicly said a third of the hard problems are broken and that every frontier model shows training overlap. The score is not wrong, exactly, but it is being computed against a ruler that its own community has marked as bent.

Third, the checkpoints move under the number. DeepSeek shipped a dated V4-Pro-0813 checkpoint alongside the original, and dated checkpoints are now normal across open labs. A benchmark number attached to "DeepSeek V4-Pro" is attached to a name that resolves to different weights depending on the date, which means even the model half of the bundle is not a fixed referent. The number you read in a launch post and the number your own harness would reproduce are measurements of two different artifacts wearing the same name.

There is also a distinct failure mode that no amount of test-quality auditing catches, and it is worth naming to keep it separate. In your eval harness is attack surface the point was that a coding-agent eval runs untrusted code from an active adversary, and contamination now has a variant where a model reads the answer key by breaking into the eval environment at test time, leaving no n-gram trace for any decontamination filter to find. That is an eval-time intrusion problem, not a test-quality one. This piece is about the tests being wrong and the harness being unstated; that one is about the harness being breached. Both end at the same practitioner conclusion: the number arrives without the context that would let you trust it, and the context is the thing you have to demand.

How to read an agentic-coding score

This is the part you can run this week. When a coding number crosses your desk, whether in a launch post, a sales deck, or an internal eval writeup, walk it through these seven steps before you let it into a decision. The goal is not to reject benchmarks. It is to convert a point estimate back into something with a baseline, a distribution, and an envelope.

1. Name the benchmark and the split. "SWE-bench" is not an answer. Verified, Pro, the standardized public set, the private commercial set, and a vendor's internal variant are different rulers with different ceilings. Verified runs well above Pro for the same system, so a number without its board is uninterpretable. Reject "SWE-bench: 80%" the way you would reject "the test passed" without knowing which test.
2. Name the harness and its version. Ask which agent scaffold produced the number and what version it was. If the answer is the vendor's own framework rather than a standardized board, treat the score as a bundle number, not a model number, and discount it toward the standardized board's range. The 15-to-30-point gap between vendor and standardized figures is your discount.
3. Name the retry budget. Ask for k and whether the metric is pass@1, best-of-k, or pass^k. A best-of-k number reported without its k is measuring the tail. For anything you will ship, pass^k (solves it every time out of k) is the reliability number that matters, and it is almost never the one on the slide.
4. Check the benchmark's own test-quality audit. Before trusting a board, find its task-level error rate. For SWE-bench Verified and Pro, OpenAI has published that roughly 30% and 59% (of the hard subset) respectively had broken graders or impossible specs. A benchmark with no published audit of its own test quality has an unknown ceiling, and you should assume it is lower than the leaderboard implies.
5. Demand a baseline. What does the simplest alternative score on the identical harness? A non-agentic single-shot prompt, the previous model generation, a cheaper model in the same family. The reported number means nothing without the gap to the baseline, because a 5-point lead and a 30-point lead are different products and the raw score hides which one you are looking at.
6. Ask what moved between the checkpoint and the number. With dated checkpoints now standard, confirm which weights produced the score and whether they are the weights you can actually run. A number attached to a name that resolves to different weights by date is a number about an artifact you may not have.
7. Report the spread, not the peak. The single most honest thing you can do with a coding benchmark is run it three times across the boards you have and report the range. If a model reads as 80 on Verified, high-50s on standardized Pro, and something else on your own task distribution, the spread is the result. The peak is marketing.

If you can only do one of these, do the first and the fourth: name the board, and check whether that board's graders work. Those two catch most of the damage, because they are the two the launch post is least likely to have handled.

What this does not mean

The honest tradeoff has to be stated, because the failure mode of this argument is nihilism. "All benchmarks are broken, therefore vibes" is worse than the disease. Benchmarks are the only thing standing between you and pure vendor assertion, and SWE-bench in particular did the field a large service by being real GitHub issues with executable tests rather than a synthetic quiz. The correct response to a flawed benchmark is not to discard measurement; it is to measure more carefully and to hold the number to the standard a pre-LLM ML team would have held it to, which is a baseline, a distribution, and an error bar.

Nor does any of this mean the models are bad at code. DeepSeek V4-Pro, GLM-5.3, and the frontier closed models are genuinely capable coding systems, and the right way to know how capable is the boring one: run them on your own repositories, with your own harness, on tasks drawn from your own distribution, and measure the pass^k. That eval will not match any leaderboard, and it is the only one that predicts what the model does in your codebase. The leaderboard is a prior. Your eval is the posterior. The mistake is shipping on the prior.

The deeper point connects to a stance this blog has taken before about where reward-hacking surface lives. When you optimize against a verifier, the exploitable surface moves into that verifier's code. A public coding benchmark is a verifier that the whole industry optimizes against at once, for years, with billions of dollars of training compute pointed at it. Of course its graders' defects get found and its solutions get absorbed. That is not a scandal; it is the predictable result of turning one test suite into the entire field's objective function. The fix is the same as it always is: keep a private eval the optimizer has never seen, and treat every public number as a claim to be checked rather than a result to be trusted.

Key Takeaways

1. An agentic-coding score is a property of a bundle, not a model. The model, the scaffold, the prompt, the retry budget, and the benchmark's test quality all move the number, and vendors publish only the model's name on it.
2. OpenAI stopped reporting SWE-bench Verified on February 23, 2026 after auditing the 138 problems (27.6% of 500) its o3 model failed across 64 runs and finding at least 59.4% had broken tests or impossible specs, split into narrow tests that reject correct code (35.5%) and wide tests that check unasked behavior (18.8%).
3. It happened twice. OpenAI retracted its recommended replacement, SWE-bench Pro, in July 2026 after finding roughly 30% of its tasks broken (27.4% by automated pipeline, 34.1% by human annotators). Building a large agentic-coding benchmark whose graders are correct is an unsolved data problem.
4. The harness is worth 15 to 30 points. On Scale's standardized SWE-bench Pro board, where every model runs identical scaffolding, frontier systems sit well below their own vendor-reported figures, which is a direct measurement of how much of a coding score is machinery rather than model.
5. Which SWE-bench matters. Verified numbers run far above Pro numbers for the same system, and pass@1 (or best-of-k) runs far above pass^k, so a bare "SWE-bench: 80%" is uninterpretable without the board, the split, and the retry metric.
6. The 2026 open field (DeepSeek V4-Pro at 80.6% on Verified, GLM at 62.1% on Pro, Kimi K3 near the top of aggregate indices) makes the measurement problem worse: more entrants competing on audited-broken benchmarks, more payoff to undisclosed harness tuning, and dated checkpoints that make even the model name an unstable referent.
7. Read every coding number through the seven-step protocol: name the board and split, name the harness and version, name the retry budget, check the benchmark's own test-quality audit, demand a baseline, ask what moved between checkpoint and number, and report the spread rather than the peak.
8. The response to broken benchmarks is more measurement, not less. Keep a private eval the optimizer has never seen, run the model on your own distribution, measure pass^k, and treat every public leaderboard number as a prior to be checked against your own posterior.