WebHarbor Builds Reproducible Websites for GUI Agents

Introduction

A GUI web agent must do more than identify text on a page. It has to navigate, click, enter information, authenticate, and sometimes leave the website in a changed state. Evaluating such behavior directly on the live web is attractive, but it is difficult to reproduce: pages change, bot defenses block requests, and many useful account-based workflows are unavailable. Synthetic websites offer control, yet they often sacrifice the visual detail or functional depth that makes real websites challenging.

WebHarbor is presented as an attempt to close that gap. The project provides executable and Dockerized reconstructions of real-world web services, allowing researchers to run similar tasks under deterministic control while retaining substantial interface and workflow complexity.

Key points

• 153 executable environments: The suite recreates website interfaces, content, and interactions in deployable containers. The reported environments average 37.3 pages and approximately 1,319.9 images each, indicating an emphasis on both visual coverage and content richness.
• Human-agent construction loop: Environments grow as new tasks reveal missing functionality. Coding agents implement changes, while human contributors review visual fidelity, functional correctness, and whether the proposed tasks are valid. The material reports 57 contributors from 26 affiliations working over five months.
• More than information lookup: The 3,518 tasks include multi-step and post-authentication workflows involving accounts, bookings, and transactions. Verifiers check not only an answer but also relevant state changes. Fast resets make the environments suitable for repeated reinforcement-learning trials.
• Training evidence: A Qwen 4B-level agent trained with reinforcement learning only in WebHarbor obtained a 75.8 average across WebVoyager and Online-Mind2Web in the reported comparison. It exceeded the cited live-web-trained OpenWebRL-4B and the shared supervised-fine-tuning checkpoint. The 15 WebVoyager websites were excluded from training, providing a test of generalization to unseen sites.

Why it matters

WebHarbor treats the environment as a first-class part of GUI-agent research rather than as a disposable benchmark wrapper. A fixed, resettable site can reduce noise caused by live content, access restrictions, and service outages. At the same time, state-based verification makes it possible to distinguish between an agent that merely reaches a page and one that completes the intended operation. This is especially important for tasks that involve authentication, reservations, or transactions.

The project does not eliminate the gap between a Dockerized reconstruction and the open web. Researchers still need to examine how faithfully the simulated services represent real-world variation, and how well skills learned in them transfer to new sites. Nevertheless, the released code, data, model, and container images make the setup easier to reproduce and extend. The broader lesson is that better GUI agents require not only stronger multimodal reasoning, but also stable interactive worlds in which their actions can be trained, checked, and reset.

Source: Hugging Face Daily Papers