EarthRecord Tests Whether Multimodal Agents Can Read Real Land Change

Introduction

Multimodal agents are increasingly able to write code, inspect images, run command-line tools, and assemble the results into an analysis. Earth observation is a demanding test of that capability. The real task rarely begins with a clean image and a carefully defined label. Instead, analysts work with long, uneven archives collected by different instruments and must decide whether a visible signal represents an actual change on the ground. EarthRecord is designed to measure that gap.

What the benchmark tests

• Rawer inputs: The record covers 56 sites over about four decades and combines Landsat, MODIS, VIIRS, and Sentinel-2 observations. Cloud masking and cross-sensor harmonization are left to the agent rather than supplied as finished inputs.
• Broader analysis tasks: It contains 238 questions across ten land-change tasks. Scores are compared with withheld references from global products, agreement among methods, and visual interpretation.
• Two diagnostic controls: A closed-book setting removes the imagery record, helping separate prior knowledge from evidence-based reasoning. Instrument-pair tests alter one satellite’s data while leaving the underlying ground situation unchanged, exposing dependence on a particular sensor.
• A gap between execution and interpretation: Twelve multimodal agents can complete much of the workflow end to end, but performance declines when the answer requires mapping image evidence to a concrete change. When one satellite’s data are altered, about half of the correct answers follow the alteration.

Why it matters

EarthRecord’s contribution is not simply a larger remote-sensing dataset. It changes what is being measured. Tool use and procedural completion are useful, but they do not prove that an agent has identified a real-world event. The closed-book control tests whether an answer comes from the record at all, while the sensor perturbation tests whether the system can resist a misleading or incomplete source.

The findings matter for any future system intended to automate environmental monitoring. An agent that can download imagery, run a script, and produce a chart may still misread cloud contamination, sensor differences, or a transient anomaly. Reliable systems will need to compare multiple sources, inspect data quality, explain conflicts, and express uncertainty instead of following the most salient signal.

The benchmark therefore presents a mixed picture. Multimodal agents are becoming capable research operators, but the final step from pixels to defensible land-change claims remains fragile. By releasing the dataset and code, EarthRecord offers a common setting for improving temporal reasoning, cross-sensor validation, and evidence-grounded analysis.

Hugging Face Daily Papers