This repository contains the evaluation code for the paper: OpenEarthSensing: A Large-Scale Fine-Grained Benchmark for Open-World Remote Sensing
Submitted to NeurIPS 2026

Dataset Preparation

Download the dataset from the official website OES.

For the Far OOD dataset SUN, please download following the official MOS repository: MOS.

Environment Setup

The OES benchmark evaluation covers multiple tasks and adopts different frameworks. Please install the environment separately according to each folder.

• For the OpenOOD folder: follow the official repo OpenOOD.
• For the OOD-VLMs folder: follow the official repo Dassl.pytorch.

Evaluation

Closed-set Classification

We provide shell scripts for closed-set classification under the OpenOOD/scripts/OES/ directory.
For example, to evaluate the performance of ResNet-50 on the OES RGB domain1 split, you can run the following command:

python main.py \
    --config configs/datasets/Benchmark/RGB_all.yml \
    configs/preprocessors/base_preprocessor.yml \
    configs/networks/resnet50-0676ba61.yml \
    configs/pipelines/train/baseline_lr.0.01.yml \
    --seed 0

Zero-shot Classification

We provide zero-shot classification evaluation scripts under the OOD-VLMs/scripts/ZSCLIP/ID directory.
For example, to evaluate the performance of GeoRSCLIP_b32 CLIP on the OES RGB domain1, run the following command:

python main.py \
    --config configs/datasets/Benchmark/RGB_all.yml \
    configs/preprocessors/base_preprocessor.yml \
    configs/networks/resnet50-0676ba61.yml \
    configs/pipelines/train/baseline_lr0.01.yml \
    --seed 0

OOD Detection

1.Post hoc methods (OpenOOD) All unimodal OOD detection scripts are under OpenOOD/scripts/OES/. Step 1: Train the base model (post-hoc methods) First, train a ResNet-50 on the in-distribution set:

python main.py \
    --config configs/datasets/Benchmark/RGB_id.yml \
    configs/preprocessors/base_preprocessor.yml \
    configs/networks/resnet50-19c8e357.yml \
    configs/pipelines/train/baseline_lr0.01.yml \
    --seed 0

Step 2: Run post-hoc OOD evaluation After training, run post-hoc methods such as MSP on the standard OOD split:

python main.py \
    --config configs/datasets/Benchmark/RGB_id.yml \
    configs/datasets/Benchmark/RGB_testood.yml \
    configs/networks/resnet50-0676ba61.yml \
    configs/pipelines/test/test_ood.yml \
    configs/preprocessors/base_preprocessor.yml \
    configs/postprocessors/msp.yml \
    --num_workers 8 \
    --network.checkpoint '/home/data/xz2002/OpenOOD/results/Benchmark_RGB_ID_resnet50-0676ba61_base_e100_lr0.01_default/s0/best.ckpt' \
    --mark 0 \
    --merge_option merge

2. Training-required methods (OpenOOD) For training-required OOD methods, use the corresponding scripts in OpenOOD/scripts/OES/

3. VLM-based OOD Detection VLM-based OOD detection scripts are provided under OOD-VLMs/scripts/LoCoOp/OOD/. For example, to evaluate the LoCoOp method with GeoRSCLIP (ViT-B/32) on the OES RGB domain1:

python CoOp_main.py \
    --trainer LoCoOp \
    --config-file './configs/trainer/Benchmark_GeoRSCLIP_b32.yaml' \
    --output_dir "./output/LoCoOp/Benchmark_RGB_RGB/GeoRSCLIP_b32" \
    --dataset-config-file "./configs/datasets/Benchmark_RGB_RGB.yaml" \
    --model_path "./checkpoint/RS5M_ViT-B-32.pt" \
    --train --load --test --ood

Additional Supported Frameworks

In addition to the frameworks mentioned above, OpenHaiv also supports evaluation on the OES benchmark.

Acknowledgements

We would like to express our sincere gratitude to the authors and contributors of the open-source projects that have supported the development of this benchmark. Specifically, we acknowledge the valuable contributions of OpenOOD and Dassl.pytorch, whose codebases and frameworks have laid an important foundation for the implementation of our evaluation pipelines. We also thank all other open-source works that have provided inspiration and technical support for this research.

Citation

@article{xiang2025openearthsensing,
  title={Openearthsensing: Large-scale fine-grained benchmark for open-world remote sensing},
  author={Xiang, Xiang and Xu, Zhuo and Deng, Yao and Zhou, Qinhao and Liang, Yifan and Chen, Ke and Zheng, Qingfang and Wang, Yaowei and Chen, Xilin and Gao, Wen},
  journal={arXiv preprint arXiv:2502.20668},
  year={2025}
}

@article{xiang2025openhaiv,
  title={OpenHAIV: A Framework Towards Practical Open-World Learning},
  author={Xiang, Xiang and Zhou, Qinhao and Xu, Zhuo and Ma, Jing and Dai, Jiaxin and Liang, Yifan and Li, Hanlin},
  journal={arXiv preprint arXiv:2508.07270},
  year={2025}
}