SWE-Touch Tests Coding Agents When Users Edit the Same Codebase

Introduction

Most repository-level coding-agent benchmarks assume a relatively clean setup: an agent receives a task, works on a static codebase, and is judged by whether its final patch resolves the issue. Real software development is messier. Users inspect files, add quick fixes, revert lines, or modify nearby logic while an agent is still planning and editing.

SWE-Touch focuses on this gap. Instead of asking only whether an agent can solve a bug in isolation, it asks whether the agent can keep track of a shared workspace whose contents may change during the task. The motivation is practical: the authors report that 59% of SWE-chat sessions contain user-authored repository changes, suggesting that interactive code editing is not a rare edge case.

Core ideas

• A benchmark for shared workspaces: SWE-Touch evaluates coding agents under conditions where the user can touch task-relevant code during an ongoing repair trajectory.
• Counter-Edits as stress tests: The framework injects small, plausible code changes that conflict with successful task completion.
• Task-critical placement: SWE-Touch mines important code regions from multiple repair trajectories, then uses a separate User Patch Generator to construct the edits.
• Contextual injection: The conflicting edit is introduced when the agent reaches the relevant area, together with a contextual user message.
• Measurable performance drop: On SWE-bench Verified, the average resolve rate across nine coding models falls by 7.7 percentage points.

What the results show

The most important lesson is that autonomous benchmark strength is not the same as collaborative robustness. Many agents appear to carry forward an outdated view of the repository after the user has changed it. In failed runs, 63.3% simply retained the user’s conflicting code without resolving the conflict.

The paper also reports that the performance degradation persists on longer-horizon tasks from SWE-Bench Pro and DeepSWE. This matters because longer tasks usually require more navigation, more file reads, and more opportunities for workspace state to drift away from what the agent last observed.

Ablation results further indicate that the code edit itself, rather than the accompanying user message, is what drives the drop. In other words, the challenge is not merely conversational understanding. The agent must re-inspect the repository, reason about whether the user change conflicts with the target fix, and verify the affected behavior with appropriate tests.

The submitted summary also notes that Claude Opus 4.8 and GPT 5.5 show stronger resilience, while some open-source models that are competitive on autonomous benchmarks degrade substantially, with drops of up to 16.5 points. This reshuffling of rankings is an important signal: static leaderboards can hide weaknesses that appear only in collaborative settings.

Why it matters

SWE-Touch points to a more realistic evaluation direction for coding AI. If agents are expected to work inside IDEs, terminals, and repositories shared with humans, they need more than patch generation. They need continuous workspace awareness, change detection, conflict reconciliation, and targeted validation.

For product builders, this may imply stronger file-watch mechanisms, explicit handling of user edits, better refresh strategies for repository context, and test-driven checks after potential conflicts. For researchers, SWE-Touch offers a concrete way to evaluate whether a coding model can operate in a dynamic development environment rather than a frozen benchmark snapshot.

Source: Hugging Face Daily Papers