Why LLMs Lose Track When User Intent Keeps Changing

Lead

As large language models move from one-shot assistants to collaborative agents, the way users express goals becomes much messier. People rarely state a fully specified objective at the start. They add constraints, clarify priorities, revise earlier assumptions, and sometimes redirect the task altogether. The paper LLMs Get Lost in Evolving User Intent focuses on this gap between benchmark conditions and real interaction: can current LLMs reliably act on an intent that changes over time?

Key points

• The failure is deeper than forgetting context. The issue is not simply that a model runs out of context or loses earlier messages. A model may keep producing locally coherent responses while drifting away from the user’s actual objective. It can over-anchor on the first request, or over-correct toward the latest utterance and drop constraints that still matter.
• Single-turn evaluation hides the problem. Most benchmarks present models with a complete instruction in a single prompt. That setup is useful for measuring static task performance, but it does not test whether a model can integrate new information, handle revisions, or distinguish a clarification from a change in direction.
• The proposed framework reuses existing benchmarks. The authors transform static, single-turn tasks into dynamic conversations where intent is incrementally revealed, revised, or redirected across turns. Crucially, they preserve each task’s original evaluation protocol, which allows existing benchmarks to serve as controlled testbeds without new annotation.
• Strong static performance does not transfer cleanly. Across multiple tasks, the paper reports a consistent drop when models are evaluated under evolving-intent conditions. The finding appears across model families, suggesting a broad limitation rather than a narrow failure of one system.

Why it matters

This work gives structure to a failure mode many builders of coding agents and long-running assistants already observe in practice. A user discovers what they want through the conversation, but the agent may continue optimizing for an outdated version of the task, or may treat the newest sentence as if it erased the entire history. In tool-use settings, that kind of misalignment is not just an incorrect answer; it can mean wasted API calls, poor edits, or harmful actions.

The implication is that future evaluation should ask more than whether a model can follow a clear instruction. It should test whether the model maintains an evolving representation of user intent, including which constraints remain active and which goals have changed. Progress may require explicit intent tracking, inspectable agent state, objective-preservation mechanisms, and training setups that reflect interactive work rather than isolated prompts. A useful collaborative agent must listen across turns, not merely pattern-match the last message.

Source: Hugging Face Daily Papers