AI Vulnerability Patching Has a Grading Problem
Help Net Security, Monday, October 12th, 2026
Northwestern researchers find AI patches that stop a crash often fail the maintainers' own tests for the real fix.
Researchers at Northwestern University tested AI coding agents on 112 real security bugs from 84 open-source C/C++, Go and Rust projects.
Many benchmarks only check whether the proof-of-concept input still crashes the program, a test a patch can pass by blocking the path while leaving the flaw in place.
Running AI patches against the tests maintainers wrote alongside the real fixes, the team found that about a third of Claude Opus 4.7 patches that stopped the crash still failed.
The authors call such fixes worse than no fix, because they signal the problem is solved when it is not.