{"rewrite":{"id":"r_2e76fa560faa2680cdd1f5eb","clusterId":"c_de932257a7c373f4a78694e6","slug":"openai-finds-about-30-of-swe-bench-pro-tasks-are-flawed","model":"deepseek-v4-flash","headline":"OpenAI Finds About 30% of SWE-Bench Pro Tasks Are Flawed","summary":"OpenAI analyzed the SWE-Bench Pro coding benchmark and found that approximately 30% of its tasks have evaluation problems. Issues include overly strict tests, insufficient explanations, narrow test coverage, and misleading prompts. The company withdrew its earlier recommendation to use SWE-Bench Pro and called for new benchmarks designed by experienced software engineers.","whyItMatters":"The finding undermines the reliability of reported AI coding performance gains and questions the validity of popular benchmarks used to measure AI progress.","webCardHtml":"\u003cp\u003eOpenAI used a pipeline combining automated filters, a Codex-based AI agent, researcher judgment, and five experienced software engineers. They first identified 286 suspicious tasks, then manually reviewed them. The automated pipeline flagged 27.4% of tasks as broken, while human reviewers found 34.1% problematic. The company estimates roughly 30% of SWE-Bench Pro tasks are flawed. OpenAI noted that GitHub issues and pull requests, which form the basis of the benchmark, are designed for human collaboration, not for clean AI evaluation. The company has retracted its previous recommendation to adopt SWE-Bench Pro and now says new benchmarks must be designed by experienced developers specifically for AI evaluation.\u003c/p\u003e","blueskyPost":"OpenAI found that about 30% of tasks in the SWE-Bench Pro coding benchmark are broken. The company retracted its recommendation to use it and called for better benchmarks.","twitterPost":"OpenAI's analysis shows ~30% of SWE-Bench Pro tasks are flawed. The company withdrew its earlier recommendation and wants new benchmarks from experienced devs.","threadsPost":null,"newsletterBlurb":"OpenAI has reported that the SWE-Bench Pro benchmark, which it previously recommended, has about 30% of tasks with evaluation issues. The company used a combination of automated and human review to reach this conclusion. It now calls for a new generation of benchmarks designed by experienced software engineers.","attributionJson":"[{\"source\":\"GIGAZINE\",\"url\":\"https://gigazine.net/news/20260709-openai-coding-evaluations/\",\"title\":\"OpenAIが「SWE-Bench Proの約30％が壊れている」と報告、AIのコーディング能力を測るベンチマークに大量の不備\"}]","lintFlagsJson":null,"lintHits":0,"costUsd":0,"inputTokens":4870,"outputTokens":1827,"status":"published","repairAttempts":0,"nextRepairAt":null,"factsAttemptedAt":1783582288,"createdAt":"2026-07-09T07:23:56.000Z","publishedAt":"2026-07-09T07:27:28.000Z","updatedAt":"2026-07-09T07:23:56.000Z"},"cluster":{"id":"c_de932257a7c373f4a78694e6","canonicalTitle":"OpenAIが「SWE-Bench Proの約30％が壊れている」と報告、AIのコーディング能力を測るベンチマークに大量の不備","representativeArticleId":"a_f275612a80091ac4c0cdb18d","sourceCount":1,"writtenSourceCount":1,"writeAttempts":0,"isSolo":true,"entitiesJson":"{\"anime_titles\":[],\"manga_titles\":[],\"work_titles\":[],\"studios\":[],\"people\":[],\"type\":\"news\",\"domain\":\"other\",\"is_roundup\":false}","contentType":"news","status":"published","firstSeenAt":"2026-07-09T03:40:00.000Z","lastSeenAt":"2026-07-09T03:40:00.000Z","updatedAt":"2026-07-09T07:27:29.000Z"},"attribution":[{"source":"GIGAZINE","url":"https://gigazine.net/news/20260709-openai-coding-evaluations/","title":"OpenAIが「SWE-Bench Proの約30％が壊れている」と報告、AIのコーディング能力を測るベンチマークに大量の不備"}],"entities":{"anime_titles":[],"manga_titles":[],"work_titles":[],"studios":[],"people":[],"type":"news","domain":"other","is_roundup":false},"keyFacts":null}