Passing Tests Is Not Enough: Why Code Models Over-Edit

Introduction

When a language model repairs code, producing a passing program is only the beginning. In a production repository, engineers also need patches that are small, easy to review, and faithful to the existing implementation. A repair that happens to pass tests can still be costly if it rewrites unrelated logic, obscures the actual fix, or creates new review and regression risks.

A study featured by Hugging Face Daily Papers examines this behavior, which the authors call “over-editing”: changing more of a codebase than is required to correct the injected defect.

Measuring more than correctness

The researchers started with 400 BigCodeBench problems and introduced controlled corruptions at the abstract syntax tree level. Because the corruption process is known, the evaluation can identify the minimal patch needed to restore the reference solution. Model outputs can then be judged not only by whether they work, but also by how far they depart from that minimal repair.

The framework adds several dimensions beyond Pass@1:

• whether unrelated code was changed;
• the distance between the generated patch and the minimal patch;
• whether the repair increases cognitive complexity;
• whether the original implementation is preserved while correctness is restored.

The results suggest that over-editing is not limited to weak systems. Frontier models, including GPT-5.5, can achieve high Pass@1 while making unnecessarily broad changes. This separates two qualities that are often conflated: functional correctness and edit fidelity.

A preservation instruction helps

The study also tested a simple behavioral constraint: explicitly instructing the model to preserve the existing implementation and make only necessary changes. Under this instruction, average excess Levenshtein distance fell from 0.195 to 0.131. Added cognitive complexity dropped by 26.6%, while Pass@1 increased by 2.3 percentage points.

The result is important because it does not support the assumption that more reasoning or a larger model automatically produces more disciplined edits. A model may be better at rewriting code without being better at identifying the boundary of the requested change. In repair workflows, restraint has to be specified or learned rather than assumed.

What post-training changes

The researchers then examined whether edit fidelity could be learned during post-training. Supervised fine-tuning performed well on corruption patterns represented in its training data, but showed signs of overfitting when the patterns changed. Reinforcement learning produced the strongest trade-off between out-of-distribution edit fidelity and retention of repair performance.

For code-agent builders, the message is practical. Test success should remain essential, but it should not be the only optimization target. Training and evaluation should also account for patch size, unnecessary modifications, cognitive complexity, and the cost of human review. For users, a request such as “make the smallest possible fix and do not refactor unrelated code” may be more valuable than simply allocating a larger reasoning budget.

The broader contribution of this work is to establish minimal editing as a distinct axis of code-repair quality. It does not argue against large refactors when they are requested. Instead, it shows that for localized bug fixes, knowing when not to change code is itself a measurable capability.

Source: Hugging Face Daily Papers