Teaching Browser Agents What Web Elements Are, Not Just What They Say

Introduction

Browser agents increasingly act on behalf of users: they search, fill in forms, navigate account pages, and complete multi-step tasks. Most current systems expose a language model to a screenshot or a flattened stream of page text, then let the model decide what to click or type. That interface creates a fundamental security problem. A malicious sentence embedded in an otherwise ordinary page can look like an instruction, and the model may use the user’s authority for a third party that has no such authority.

The paper What It Is, Not What It Says presents Intact, a system designed around a different premise. An element’s authority should depend on what the element is and where it comes from—not on the words displayed inside or around it. Instead of asking the model to solve every security question through intent recognition, Intact preserves browser semantics and enforces key boundaries outside the model.

Key ideas

• GenTree preserves provenance and capability. Intact runs with an unmodified browser and constructs GenTree, a representation of the page that records where elements originate, how they relate to one another, and what kinds of interactions they support. A button, input field, or link is therefore more than a piece of text in a shared context.
• The model receives one element at a time. Intact passes elements from GenTree to the language model individually. This limits the ability of text attached to one element to persuade the model to operate on another element. The model still makes decisions, but its decision context is deliberately narrowed.
• The harness enforces the final boundary. The execution layer checks whether a proposed step is allowed by the selected element’s capabilities. When an action would operate on third-party content, the system asks the user before proceeding. A model that follows hostile instructions therefore cannot automatically turn an untrusted element into a broader permission.
• The reported results are substantial. On 1,270 real webpages, GenTree attributed 2.3 times as many elements to third parties as the browser’s existing attribution, which misattributed 58.7% of them to the first party. On 1,091 pages containing injected attacks, 73.1% of attacks entered the plans of a conventional planner, while only 12.8% entered Intact’s plans. In every such case, the attack could reach no further than a request for user consent. Isolation disrupted functionality on 3.7% of tasks, and the authors note that parallel scheduling could remove much of the added latency.

Why it matters

Intact shifts the focus of agent security from making models perfectly recognize prompt injection to constraining what a model can cause the browser to do. Detection remains useful, but it should not be the only defense. If the system knows an element’s origin and capability, a mistaken model decision does not have to become an authorized transaction.

The work also exposes the security cost of screenshot and plain-text interfaces. They are convenient for general reasoning, but they erase provenance and interaction semantics that the browser already understands. Future browser agents may need structured information about origin, element type, and permission boundaries as part of their interface to the web, while keeping enforcement outside the model.

Intact does not eliminate confirmation prompts or guarantee that every attack is filtered before planning. Its contribution is more practical: it demonstrates that a browser agent can remain useful while treating the model as an interpreter rather than the final authority. The browser and its surrounding harness can preserve the distinction between trusted actions and untrusted content.

Source: Hugging Face Daily Papers