UserArena Tests Shopping Agents Against Long-Horizon User Behavior

Recommendation systems are increasingly being asked to do more than rank a single page of products. A shopping agent may need to interpret a user’s goal, react to clicks or skips, revise its recommendations, and continue making decisions across an entire session. This creates a difficult evaluation problem: can a simulated shopper behave like a real person after many dependent interactions, rather than merely predict the next recorded action?

A benchmark built around interaction

UserArena addresses this problem with a real-world e-commerce benchmark containing 64,729 session trajectories from 11,010 users. The data covers 566,102 items and 13 action types. Rather than treating the logs as a static prediction dataset, the benchmark pairs them with an interactive environment in which recommendation agents can make sequential decisions and receive simulated user responses.

The distinction matters because a locally plausible action can still lead to an unrealistic long-term session. UserArena therefore evaluates systems at five levels:

• Output validity: whether the agent produces an action accepted by the environment;
• Local decisions: whether individual recommendations or responses resemble user behavior;
• Sequential behavior: whether transitions across multiple steps remain realistic;
• Personalization: whether the simulator preserves differences between users;
• Outcomes: whether the resulting long-horizon results are consistent with real interactions.

The benchmark also supports offline validation against outcomes from a controlled production A/B test. This gives researchers a stronger test than behavioral imitation alone. A simulator should not only look realistic; it should also help predict how alternative recommendation policies compare when deployed in practice.

Distilling intent during model rollouts

The paper further proposes On-Policy Trajectory Distillation, or OPTD. Standard supervision is often tied to actions observed in logged trajectories. However, once a student simulator makes a different choice, its subsequent state may no longer match the historical data. The model can then accumulate errors over the remainder of the session.

OPTD addresses this mismatch by using intent expressed at the trajectory level to construct denser teacher targets on rollouts generated by the student itself. In effect, supervision follows the states the model actually visits, while still reflecting the direction of the original user trajectory. This is intended to help the simulator preserve coherent behavior over longer interactions instead of optimizing isolated next-step predictions.

Under UserArena’s evaluation protocol, OPTD achieves the strongest overall fidelity among the evaluated simulators and most closely reproduces the production policy effect in offline simulation. The supplied material does not report detailed metric values, so the result should be read as a comparative finding within the benchmark rather than a universal guarantee across platforms or shopping domains.

Why it matters

UserArena shifts the focus of user simulation from next-action accuracy to trajectory-level world modeling. For agentic recommendation, the central question is not simply whether one recommendation is plausible. It is whether an agent can maintain a consistent understanding of a shopper’s evolving goal and whether a simulated environment can support reliable policy comparisons.

The combination of real trajectories, an interactive setting, and production-level validation makes the benchmark useful for research on long-horizon recommendation agents. Its scope also sets a clear boundary: performance is measured within a particular e-commerce behavior distribution, and high fidelity there does not automatically imply transfer to other platforms, product categories, or user populations. Broader coverage and stronger robustness tests will be important next steps.

Source: Hugging Face Daily Papers