Image Editing Assistants Need to See the Image Before Suggesting What Comes Next

Introduction

Image generation is increasingly becoming an iterative process. Users rarely stop after a single prompt: they ask for a different style, a new background, an added object, a changed composition, or a more polished final result. Yet many conversational assistants still treat follow-up suggestions as a text-only recommendation problem, proposing generic next steps based mainly on dialogue history.

That approach breaks down in image editing. A useful suggestion must be grounded in the current visual state. If the image contains no person, recommending a clothing change may be irrelevant; if the scene is already set at night, suggesting “make it a night scene” adds little value. The paper “What to Edit Next” focuses on this overlooked problem: how can a conversational image-creation system recommend the next edit in a way that reflects the user’s intent, remains diverse, and is executable on the actual image?

Key ideas

• Real-world evidence for visual grounding: The authors analyze 100,000 real multi-turn image-creation conversations from Qwen App. They find that 80.1% of follow-up interactions are image-dependent, showing that text-only recommendation is insufficient for this setting.
• Human-reviewed intent construction: In the first stage, the team uses online data to build a table of appropriate follow-up editing intents, reviewed by humans. These intents become supervised fine-tuning targets for a multimodal policy, helping the system learn suggestions that reflect realistic editing needs rather than generic prompts.
• Optimization with user click feedback: In the second stage, the framework uses click behavior to align rule-guided suggestions with actual user choices. Multi-objective reinforcement learning is applied so the policy can optimize for practical user preference signals, not just linguistic plausibility.
• A visual verifier for consistency: In the third stage, a visual verifier provides additional supervision by penalizing suggestions that conflict with the current image. This directly addresses a common failure mode: a recommendation may sound fluent but still be visually impossible or inconsistent.

Why it matters

The strongest evidence comes from the live randomized A/B test involving millions of users. The final framework reduces visual inconsistency from 3.7% to 0.9%. It also improves recommendation CTR by 32.70%, image take-away rate by 16.32%, and average conversation turns by 39.90%.

These results suggest that visually aligned follow-up suggestions are not a minor interface feature. They can make image-creation tools easier to continue using, increase engagement, and help users discover editing directions they might not have articulated themselves. More broadly, the work points toward a new role for multimodal assistants: not just generating images on command, but actively collaborating with users by understanding the visual context and proposing feasible next steps.

For AI drawing apps, creative design tools, and multimodal agents, this kind of recommendation layer could become an important product capability. The next competitive edge may not only be better image quality, but better guidance through the creative process.

Source: Hugging Face Daily Papers