UrbanGround Tests Whether Multimodal Agents Can Navigate a Real-Scale City

Introduction

Recognizing a street scene is only the first step toward acting in a city. An embodied agent must preserve useful spatial information as its viewpoint changes, choose actions under incomplete instructions, and recover when a route or an earlier assumption turns out to be wrong. UrbanGround is designed to make this gap between perception and agency measurable.

A city-scale interactive testbed

UrbanGround builds a physically constrained, real-scale replica of Hong Kong from territory-wide 3D geospatial data. Agents enter the environment through a first-person view, explore the city, and use an interactive map for navigation. The setup emphasizes a closed loop: an agent observes its surroundings, selects an action, moves through the environment, and then has to interpret the new evidence before continuing.

This design differs from a static image benchmark or a fixed collection of short routes. The platform can connect multimodal large language model agents to tasks involving visual grounding, spatial question answering, navigation, exploration, planning, and adaptation. It also supports environmental variation, including different times of day, weather conditions, road closures, and moving pedestrians. Web and native builds, together with evaluation code and tasks, are intended to make the sandbox useful for further embodied-urban research.

Key findings

• Local competence does not guarantee global competence. Current MLLMs can often recognize urban elements and perform useful short-range spatial reasoning.
• Longer routes expose accumulated error. As destinations become more distant or less explicit, orientation and route maintenance become less reliable.
• Correction remains a weak link. Agents may continue from an incorrect assumption instead of using new observations to identify and repair the mistake.
• Dynamic conditions increase difficulty. Changes in route availability and pedestrian motion challenge both planning and pedestrian-aware movement.

Why it matters

UrbanGround is valuable because it treats spatial agency as a progression rather than a single score. An agent must first ground a local scene, then use that grounding for nearby actions, and eventually sustain a goal across a changing city. This structure helps researchers separate visual recognition from localization, map use, long-term state tracking, planning, and control.

The results also point to a broader limitation in current multimodal agents. Strong single-step perception is not enough if the agent lacks a stable orientation representation, persistent state updates, uncertainty-aware planning, and a mechanism for active correction. A system can make several individually sensible decisions and still fail at the larger task because those decisions do not compose over time.

For embodied AI, this suggests that progress should be measured not only by whether an agent can answer what is visible now, but also by whether it can maintain a useful spatial model while moving. UrbanGround provides a realistic setting for that investigation and a foundation for new city-scale tasks.

Source: Hugging Face Daily Papers