LLM Code Editors Have a Deletion Problem

Lead

Modern coding models can generate fixes, pass tests, and navigate large repositories. But this paper, “To Add Is Machine, To Delete Is Human,” highlights a quieter weakness: LLMs often struggle to remove the right code and then stop. The authors call this behavior deletion avoidance, a systematic tendency to preserve code that the intended edit requires deleting.

Key points

• Passing tests can hide messy patches. Across five leading submissions on the official SWE-bench Verified leaderboard, deletion recall against the developer patch reached at most 71.7%, even on tasks all five models solved. Models reached the correct file for more than 92% of required deletions, yet removed the exact line in under 52% of cases.
• The common workaround is “Guard-and-Go.” Instead of deleting obsolete logic, models often wrap it in a guard, fallback, or conditional branch. The study reports that 29.0% of passing patches showed this pattern. Such patches may satisfy existing tests while leaving unwanted behavior and maintenance debt in the codebase.
• Existing tests rarely verify absence. When the authors retrofitted 34 deletion-heavy SWE-bench Verified tasks with tests that fail if the targeted code remains, four frontier models dropped from 63.2% to 41.9% resolution. In other words, models were often rewarded for routing around old code rather than removing it.
• CanItDelete isolates the skill. Because real repairs usually mix additions and removals, the team built CanItDelete: 200 real-commit tasks where the required edit is deletion only. Even with the addition work removed, the best model still failed about one in five tasks, while smaller open models performed much worse.
• Localization is not the whole answer. Giving models the exact lines to delete reduced incomplete deletion, but it exposed another failure mode: some models over-deleted beyond the intended span or added new code anyway.

Why it matters

This research reframes code generation quality around maintainability, not just test success. A human engineer knows that deleting dead or obsolete logic is part of a clean repair. An AI agent that merely guards old code can create patches that look acceptable to a benchmark but remain risky for production.

The optimistic finding is that deletion can be improved. A pilot study suggests that adding a small amount of deletion-focused post-training data reduced incomplete deletion and improved SWE-bench Verified performance. However, over-deletion increased, which means “finish the removal” and “respect the boundary” are separate capabilities.

For AI coding agents to become reliable collaborators, benchmarks and review systems need to check whether code that should disappear is actually gone. The next stage of coding AI evaluation may need to measure not only what models can write, but also what they can safely remove.

Source: Hugging Face Daily Papers