OVEarth-Bench is a unified zero-shot benchmark for open-vocabulary Earth observation across broad categories and diverse queries, with findings that guide the design of future open-vocabulary EO methods.
OVEarth-Bench tests whether a model can localize broad geospatial concepts expressed as category phrases, spatial references, or functional descriptions, while also rejecting plausible concepts that are absent.
The benchmark separates the number and diversity of concepts from the linguistic form used to specify them, then evaluates masks, horizontal boxes, and oriented boxes under one zero-shot protocol.
Seven top-level EO domains span common land cover, fine-grained infrastructure, transportation, industry, terrain, water systems, and special facilities.
Queries progress from explicit category phrases to spatial descriptions and implicit functional reasoning, with negative vocabulary testing absence discrimination.
Localize a present concept from an open noun phrase and return an empty prediction for a plausible but absent concept.
Resolve compositional spatial relations and identify the complete target region without relying on a short category label alone.
Infer the target from its role or function when the category name is deliberately omitted from the query.
All images are newly collected rather than copied from existing remote-sensing benchmarks. Masks, language queries, and box annotations pass explicit quality-control stages.
National standards, 200+ datasets, and OSM tags define 172 categories.
POI-guided retrieval samples six continents across varied scales and seasons.
Annotators draw or correct polygons and managers review boundaries and omissions.
LLM-assisted generation is followed by automated checks and expert review.
Instance polygons yield reviewed horizontal and minimum-area oriented boxes.
Forty-nine model variants are evaluated zero-shot. Even the best systems stay below 42% macro IoU for segmentation and 36% micro F1 at 0.5 IoU for box localization.
They account for nine of the ten best vocabulary segmentation results and all top-ten referring and reasoning results.
Domain-specific methods improve some matched baselines, but rarely reach the strongest general-purpose models.
Localization and absence rejection are nearly decoupled: vocabulary ma-IoU and MCC have Spearman rho = 0.107.
Evaluating more categories makes model comparisons more reliable: median rank correlation rises from 0.84 with five categories to 0.98 with fifty.
| Query | Best seg. ma-IoU | Best det. mi-F1@0.5 |
|---|---|---|
| Vocabulary | 38.75% | 24.53% |
| Referring | 41.32% | 35.56% |
| Reasoning | 37.96% | 26.12% |
Best segmentation scores are achieved by Rex-Omni + SAM. Scores are reported under the unified zero-shot protocol.
Zero-shot evaluation on OVEarth-Bench. Localization metrics are percentages; MCC is unitless in [-1, 1]. Tables are ordered by the primary localization metric.
The benchmark spans both ubiquitous and uncommon EO concepts, including fine-grained facilities and the same locations observed under temporal and seasonal change.
The released package supports masks, HBBs, and OBBs with macro and micro localization metrics plus presence/absence evaluation.
@article{li2026ovearth,
title = {OVEarth-Bench: Evaluating Category Breadth and
Query Diversity for Open-Vocabulary Earth Observation},
author = {Li, Kaiyu and Xin, Zepeng and Jiang, Zixuan
and Fu, Jing and Xue, Lanxuan and Zhang, Lingyu
and Cao, Xiangyong},
year = {2026}
}