OSMa-Bench Dataset

Project Page | Paper | Code

OSMa-Bench (Open Semantic Mapping Benchmark) dataset is a fully automatically generated dataset for evaluating the robustness of open semantic mapping and segmentation systems under varying indoor lighting conditions and robot movement dynamics. This dataset is part of OSMa-Bench pipeline.

Dataset Summary

This dataset provides simulated RGB-D and semantically annotated posed sequences for evaluation of semantic mapping and segmentation, with a particular focus on handling dynamic lighting—a critical but often overlooked factor in existing benchmarks. It also includes a collection of automatically generated question–answer pairs across multiple categories to support the evaluation of scene-graph–based reasoning, offering a task-driven measure of how well a system’s reconstructed scene captures semantic relationships between objects.

The data is built upon two base datasets:

• ReplicaCAD: 22 scenes with 4 lighting configurations and a velocity modifier.
• Habitat Matterport 3D (HM3D): 8 scenes with 2 lighting configurations and a velocity modifier.

Installation

We offer two versions of the dataset: one with separate files and one as a single compressed archive. Use the following command to download the separated files (may be slow):

git xet install
git clone https://huggingface.co/datasets/warmhammer/OSMa-Bench_dataset -b main

and this command to download the compressed version (faster one):

git xet install
git clone https://huggingface.co/datasets/warmhammer/OSMa-Bench_dataset -b compressed
unzip data.zip

Data Configurations

The dataset includes the following configurations for the ReplicaCAD and HM3D scenes:

Data Structure

The dataset provides structured data for each scene, suitable for tasks like 3D scene understanding, visual question answering, and robotics. Each scene contains the following components:

VQA Question Categories

The dataset includes a automatically generated answer-question pairs with the following question types:

1. Binary General – Yes/No questions about the presence of objects and general scene characteristics
   Example: Is there a blue sofa?

2. Binary Existence-Based – Yes/No questions designed to track false positives by querying non-existent objects
   Example: Is there a piano?

3. Binary Logical – Yes/No questions with logical operators such as AND/OR
   Example: Is there a chair AND a table?

4. Measurement – Questions requiring numerical answers related to object counts or scene attributes
   Example: How many windows are present?

5. Object Attributes – Queries about object properties, including color, shape, and material
   Example: What color is the door?

6. Object Relations (Functional) – Questions about functional relationships between objects
   Example: Which object supports the table?

7. Object Relations (Spatial) – Queries about spatial placement of objects within the scene
   Example: What is in front of the staircase?

8. Comparison – Questions that compare object properties such as size, color, and position
   Example: Which is taller: the bookshelf or the lamp?

Citation

Using OSMa-Bench dataset in your research? Please cite following paper: OSMa-Bench arxiv.

@inproceedings{popov2025osmabench,
    title     = {OSMa-Bench: Evaluating Open Semantic Mapping Under Varying Lighting Conditions},
    author    = {Popov, Maxim and Kurkova, Regina and Iumanov, Mikhail and Mahmoud, Jaafar and Kolyubin, Sergey},
    booktitle = {2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
    year      = {2025}
}