CAPA Tests Whether Coding Assistants Can Remember User-Specific Ambiguity

Introduction

Modern coding assistants are increasingly asked to turn informal, incomplete requests into executable software. In practice, many requests are ambiguous: “fix the error handling,” “clean up the output,” or “optimize this function” can mean different things depending on the user, project, and prior conventions. Most existing disambiguation approaches treat each request as an isolated problem inside the current session, often by asking the user for clarification.

This paper shifts the focus to a more realistic long-term scenario. If the same user has already resolved similar ambiguities in previous sessions, can a newly opened coding assistant use that history as memory? The authors define this as personalized ambiguity adaptation: given past resolved sessions from a user and a new ambiguous coding request, the assistant should identify the recurring pattern, produce the intended executable solution, and minimize clarification.

Key Points

• A new evaluation task: The work treats ambiguity not only as a local dialogue issue, but as a cross-session personalization problem.
• The CAPA benchmark: CAPA characterizes personalized coding ambiguity through six mechanisms and injects them into otherwise unambiguous executable tasks using a controlled three-stage generation pipeline.
• Dataset structure: The benchmark includes 600 coding sessions across 60 balanced user–ambiguity cells, with 300 held-out sessions for evaluation.
• Evaluation setup: The authors test 12 recent LLMs under two conditions: no available history and same-user history.
• Metrics: The study measures executable success, first-turn success, and turns-to-completion, making it possible to assess both correctness and interaction cost.
• Memory management: The paper also analyzes task difficulty, user identity, and how history is used, and proposes a lightweight same-user history gating method at inference time.

Why It Matters

The contribution is not just another benchmark. CAPA makes a common frustration measurable: coding agents often fail to remember how a particular user tends to phrase recurring requests. A useful assistant should not need to be told repeatedly that “fix error handling” usually means adding retries rather than changing exception types, if that preference has already been resolved before.

At the same time, long-term memory is risky if applied blindly. User preferences change, codebases evolve, and stale history can be worse than no history. The discussion around the paper highlights the need for forgetting, re-ranking, or gating mechanisms that decide when past behavior is still relevant.

Overall, CAPA provides a concrete foundation for building coding assistants that are more aligned with individual users over time. It suggests that better code generation may depend not only on larger models, but also on better memory infrastructure and evaluation methods for using that memory safely.

Source: Hugging Face Daily Papers