license: mit
task_categories:

• image-to-3d
  tags:
• autonomous-driving
• occupancy-prediction
• occupancy-forecasting
• 3d
• 3d-occupancy
• flow-estimation
• benchmark
• multi-modal
• nuScenes
• Waymo
• CARLA
• OpenCOOD

UniOcc: A Unified Benchmark for Occupancy Forecasting and Prediction in Autonomous Driving

Paper | Project Page | Code

Autonomous Driving researchers, have you ever been bothered by the fact that popular datasets all have their different
formats, and standardizing them is a pain? Have you ever been frustrated by the difficulty of just understanding
the file semantics? This challenge is even worse in the occupancy domain. But, UniOcc is here to help.

UniOcc is a unified framework for occupancy forecasting, single-frame occupancy prediction, and occupancy flow
estimation in autonomous driving. By integrating multiple real-world (nuScenes, Waymo) and
synthetic (CARLA, OpenCOOD) datasets, UniOcc enables multi-domain training, seamless cross-dataset evaluation,
and robust benchmarking across diverse driving environments.

Yuping Wang^1,2*,
Xiangyu Huang³*,
Xiaokang Sun¹*,
Mingxuan Yan¹,
Shuo Xing⁴,
Zhengzhong Tu⁴,
Jiachen Li¹

¹University of California, Riverside; ²University of Michigan; ³University of Wisconsin-Madison; ⁴Texas A&M University

Supported Tasks

• Occupancy Forecasting: Predict future 3D occupancy grids over time given historical occupancies or camera inputs.
• Occupancy Prediction: Generate detailed 3D occupancy grids from camera inputs.
• Flow Estimation: Provides forward and backward voxel-level flow fields for more accurate motion modeling and object tracking.
• Multi-Domain Dataset Integration: Supports major autonomous driving datasets (nuScenes, Waymo, CARLA, etc.) with consistent annotation and evaluation pipelines.
• Ground-Truth-Free Metrics: Beyond standard IoU, introduces shape and dimension plausibility checks for generative or multi-modal tasks.
• Cooperative Autonomous Driving: Enables multi-agent occupancy fusion and forecasting, leveraging viewpoint diversity from multiple vehicles.

Pre-requisites

We simplify our benchmark so you only need:

• Python 3.9 or higher

  pip install torch torchvision pillow tqdm numpy open3d shapely matplotlib scikit-learn pickle
  

• Huggingface

  pip install "huggingface_hub[cli]"
  

You do not need:

• nuscenes-devkit
• waymo-open-dataset
• tensorflow

Dataset Download

The UniOcc dataset is available on HuggingFace. The size of each dataset is as follows:

To download each dataset, use the following command (recommend you to download only the folders you need):

huggingface-cli download tasl-lab/uniocc --include "NuScenes-via-Occ3D-2Hz-mini*" --repo-type dataset --local-dir ./datasets
huggingface-cli download tasl-lab/uniocc --include "Carla-2Hz-train*" --repo-type dataset --local-dir ./datasets
...

Contents

Inside each dataset, you will find the following files:

datasets
├── NuScenes-via-Occ3D-2Hz-mini
│   ├── scene_infos.pkl
│   ├── scene_001           <-- Scene Name
│   │   ├── 1.npz           <-- Time Step
│   │   ├── 2.npz
│   │   ├── ...
│   ├── scene_002
│   ...
├── OpenCOOD-via-OpV2V-10Hz-val
│   ├── scene_infos.pkl
│   ├── scene_001           <-- Scene Name
│   │   ├── 1061            <-- CAV ID
│   │   │   │   ├── 1.npz   <-- Time Step
│   │   │   │   ├── 2.npz
│   │   │   │   ├── ...
│   │   │   ├── scene_002
│   ...

• scene_infos.pkl: A list of dictionaries, each containing the scene name, start and end frame, and other metadata.
• scene_XXX: A directory containing the data for a single scenario.
• YYY.npz: A NumPy file containing the following data for a single time step.
  • occ_label: A 3D occupancy grid (L x W x H) with semantic labels.
  • occ_mask_camera: A 3D grid (L x W x H) with binary values with 1 indicating the voxel is in the camera FOV and 0 otherwise.
  • occ_flow_forward: A 3D flow field (L x W x H x 3) with voxel flow vectors pointing to each voxel's next frame coordinate. In the last frame, flow is 0. The unit of the flow is num_voxels.
  • occ_flow_backward: A 3D flow field (L x W x H x 3) with voxel flow vectors pointing to each voxel's previous frame coordinate. In the first frame, flow is 0. The unit of the flow is num_voxels.
  • ego_to_world_transformation: A 4x4 transformation matrix from the ego vehicle to the world coordinate system.
  • cameras: A list of camera objects with intrinsic and extrinsic parameters.
    • name: The camera name (i.e. CAM_FRONT in nuScenes).
    • filename: The relative path to the camera image from the original datasource (i.e. nuScenes).
    • intrinsics: A 3x3 intrinsic matrix.
    • extrinsics: A 4x4 extrinsic matrix from the camera to the ego vehicle's LiDAR.
  • annotations: A list of objects with bounding boxes and class labels.
    • token: The object token, consistent with their original datasource.
    • agent_to_ego: A 4x4 transformation matrix from the object to the ego vehicle.
    • agent_to_world: A 4x4 transformation matrix from the object to the world coordinate system.
    • size: The size of the agent's bounding box in meters. (Length, Width, Height)
    • category_id: The object category (i.e. 1 for car, 4 for pedestrian, etc.)

Note: we provide the flow annotation to both dynamic voxels (agents) and static voxels (environments) in the scene.

Visualizing the Dataset

You can visualize the dataset using the provided viz.py script. For example:

python uniocc_viz.py --file_path datasets/NuScenes-via-Occ3D-2Hz-mini/scene-0061/0.npz

In this script, we also provide the API to visualize any 3D occupancy grid, with or without a flow field.

Usage

Without Camera Images

If you only need the occupancy data, you can use the provided uniocc_dataset.py script to load the dataset.

from uniocc_dataset import UniOcc

dataset_carla_mini = UniOcc(
    data_root="datasets/Carla-2Hz-mini",
    obs_len=8,
    fut_len=12
)
dataset_nusc_mini = UniOcc(
    data_root="datasets/NuScenes-via-Occ3D-2Hz-mini",
    obs_len=8,
    fut_len=12
)
dataset = torch.utils.data.ConcatDataset([dataset_carla_mini, dataset_nusc_mini])

With Camera Images

If you want to use the camera images from nuScenes, Waymo or OpV2V, it is necessary to download them from the original dataset.

• nuScenes
• Waymo Open Dataset v1
  • Convert to KITTI format using this tool
• OpV2V

You can then provide the root directory to the dataloader to load the camera images.

from uniocc_dataset import UniOcc

dataset_carla_mini = UniOcc(
    data_root="datasets/Carla-2Hz-mini",
    obs_len=8,
    fut_len=12,
    datasource_root="datasets/Carla-2Hz-mini"
)
dataset_nusc_mini = UniOcc(
    data_root="datasets/NuScenes-via-Occ3D-2Hz-mini",
    obs_len=8,
    fut_len=12,
    datasource_root="<YOUR_NUSCENES_ROOT>"  # e.g. <YOUR_NUSCENES_ROOT>/sweeps/CAM_FRONT
)
dataset_waymo_mini = UniOcc(
    data_root="datasets/Waymo-via-Occ3D-2Hz-mini",
    obs_len=8,
    fut_len=12,
    datasource_root="<YOUR_KITTI_WAYMO_ROOT>"  # e.g. <YOUR_KITTI_WAYMO_ROOT>/training/image_0
)
dataset = torch.utils.data.ConcatDataset([dataset_carla_mini, dataset_nusc_mini, dataset_waymo_mini])

🚘 Occupancy Space Localization, Segmentation, Voxel Alignment, Tracking

In uniocc_utils.py, we provide a set of utility functions for occupancy space localization, segmentation, voxel alignment, and tracking. These functions are designed to work with the voxelized occupancy grids and can be used for various tasks such as object tracking, segmentation, and motion estimation.

👀 Visualization API

In uniocc_viz.py, we provide a set of visualization functions to render occupancy grids, flow fields, and camera images in 3D using Open3D. These functions can be used to visualize occupancy grids, flow vectors, and the ego vehicle model in a 3D scene.

🌀 Voxel Flow Computation

uniocc_flow_gen.py provides utility functions for computing voxel-level flow fields from occupancy grids and object annotations.

Run the script directly to compute and verify flow fields on sample data from the UniOcc dataset:

python uniocc_flow_gen.py

📏 Evaluation

Demo (needs a sample dataset in datasets/):

python uniocc_eval.py

We provide these evaluation APIs, as described in our paper.

Checklist

• Release non-cooperative datasets
• Release cooperative dataset
• Release the dataset API
• Release the visualization script
• Release the evaluation scripts
• Release the occupancy segmentation, localization, tracking scripts
• Release data generation scripts

Change History

• 2025-07-17: Fix bug in nuScenes datasets where token is incorrectly set to annotation token but should be instance token. We changed the token to the instance token in the annotations field of the occupancy grid. This change is backward compatible, as we also provide the original annotation token in the annotation_token field.

Citation

If you find this work useful, please consider citing our paper:

@inproceedings{wang2025uniocc,
  title={UniOcc: A Unified Benchmark for Occupancy Forecasting and Prediction in Autonomous Driving},
  author={Wang, Yuping and Huang, Xiangyu and Sun, Xiaokang and Yan, Mingxuan and Xing, Shuo and Tu, Zhengzhong and Li, Jiachen},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  year={2025},
  publisher={IEEE}
}