What they did

The authors constructed UrbanGround, the first sandbox to evaluate MLLM agents in a physically constrained replica of an entire real city (Hong Kong) built from territory-wide 3D geospatial data. The sandbox supports closed-loop interaction from a first-person view and provides an interactive map for navigation. Agents can directly enter the 3D city, explore, and act based on local visual input. The analysis was structured around three research questions: whether agents can ground a local scene after active observation, whether that grounding supports navigation as destinations become farther and less explicit, and whether behavior survives changes in route availability and pedestrian motion.

Key findings

• Agents show useful atomic abilities in visual recognition and short-range spatial reasoning, but orientation and pedestrian-aware movement remain unreliable.
• The central failure mode is that local abilities do not compose into sustained goal-directed behavior over extended exploration; errors accumulate without effective correction.
• Performance degrades significantly as destinations become more distant and less explicit, and when route availability or pedestrian motion changes.

Why it matters

The work highlights a critical gap between current MLLM capabilities and the demands of real-world urban navigation, where sustained reasoning, error correction, and adaptation to dynamic obstacles are essential. UrbanGround provides a reproducible testbed to probe these limitations, potentially guiding the development of more robust spatial agency in embodied AI systems.

Caveats

UrbanGround is a simulated environment; real-world urban navigation involves additional sensory modalities (e.g., audio, haptics) and physical constraints not captured here. The evaluation focuses on a single city (Hong Kong), and generality to other urban layouts is untested. The study does not propose new methods or training techniques, only diagnostic tasks.