PhysicalAI-Autonomous-Vehicle-Cosmos-Drive-Dreams

Paper | Paper Website | GitHub

Download

We provide a download script to download our dataset. If you have enough space, you can use git to download a dataset from huggingface.

usage: download.py [-h] --odir ODIR
                                       [--file_types {hdmap,lidar,synthetic}[,…]]
                                       [--workers N] [--clean_cache]
required arguments:
  --odir ODIR            Output directory where files are stored.
optional arguments:
  -h, --help             Show this help message and exit.
  --file_types {hdmap,lidar,synthetic}[,…]
                  Comma-separated list of data groups to fetch.
                  • hdmap     → common folders + 3d_* HD-map layers  
                  • lidar     → common folders + lidar_raw  
                  • synthetic → common folders + cosmos_synthetic  
                  Default: hdmap,lidar,synthetic (all groups).
  --workers N            Parallel download threads (default: 1).
                         Increase on fast networks; reduce if you hit
                         rate limits or disk bottlenecks.
  --clean_cache          Delete the temporary HuggingFace cache after
                         each run to reclaim disk space.
common folders (always downloaded, regardless of --file_types):
  all_object_info, captions, car_mask_coarse, ftheta_intrinsic,
  pinhole_intrinsic, pose, vehicle_pose

Here are some examples:

<h1 class="text-2xl font-bold mt-8 mb-4 text-gray-900 dark:text-white">Use this to download the download.py script </h1>
wget https://raw.githubusercontent.com/nv-tlabs/Cosmos-Drive-Dreams/main/scripts/download.py
<h1 class="text-2xl font-bold mt-8 mb-4 text-gray-900 dark:text-white">download all (about 3TB)</h1>
python download.py --odir YOUR_DATASET_PATH --workers YOUR_WORKER_NUMBER
<h1 class="text-2xl font-bold mt-8 mb-4 text-gray-900 dark:text-white">download hdmap only</h1>
python download.py --odir YOUR_DATASET_PATH --file_types hdmap --workers YOUR_WORKER_NUMBER
<h1 class="text-2xl font-bold mt-8 mb-4 text-gray-900 dark:text-white">download lidar only</h1>
python download.py --odir YOUR_DATASET_PATH --file_types lidar --workers YOUR_WORKER_NUMBER
<h1 class="text-2xl font-bold mt-8 mb-4 text-gray-900 dark:text-white">download synthetic video only (about 700GB)</h1>
python download.py --odir YOUR_DATASET_PATH --file_types synthetic --workers YOUR_WORKER_NUMBER

Dataset Description

Dataset Owner(s)

Dataset Creation Date

License/Terms of Use

Intended Usage

Dataset Characterization

Data Collection Method
* Synthetic

Labeling Method
* Automatic/Sensors - Machine-derived

Dataset Format

* Modality: HDMap Annotations, LiDAR, Cosmos-generated Videos, Camera Intrinsics and Extrinsics * Format Classification: Structured data (LiDAR Point Clouds, HDMap Annotations), Cosmos-generated Videos

You will see the following folder structure, including several attributes (HDMap / LiDAR data, etc).

Synthetic Videos

We store synthetic videos in the subfolder of cosmos_synthetic/ with the following structure

.
└── cosmos_synthetic
    └── single_view
        ├── caption.tar.gz
        ├── generation.tar.gz.part-*
        └── hdmap.tar.gz

• caption.tar.gz stores the prompt we used to generate synthetic videos, which is in the format of {clip_id}_{chunk_id}_{weather}.txt.
• generation.tar.gz.part-* stores the synthetic videos generated by Cosmos-Drive-Dreams, which is in the format of {clip_id}_{chunk_id}_{weather}.mp4.
• hdmap.tar.gz stores the HDMap condition videos, which is in the format of {clip_id}_{chunk_id}.mp4.

Sensor Data

| Folder | File Format | Description | Explanation | | ----- | ----- | ----- | ----- | | lidar_raw | .tar | Motion-compensated LiDAR point clouds (10 FPS) | keys include 000000.lidar_raw.npz, 000003.lidar_raw.npz, ..., etc. The 000000.lidar_raw.npz includes xyz, intensity, row, column, starting_timestamp, lidar_to_world | | vehicle_pose | .tar | Vehicle poses (30 FPS) in FLU convention | keys include 000000.vehicle_pose.npy, 000001.vehicle_pose.npy, etc. | | pose | .tar | Camera poses derived from vehicle pose (30 FPS) in OpenCV convention | keys include 000000.pose.{camera_name}.npy, 000001.pose.{camera_name}.npy, etc. | | ftheta_intrinsic | .tar | Camera intrinsic parameters for each view | keys include ftheta_intrinsic.{camera_name}.npy. The npy file stores a vector [cx, cy, w, h, poly, is_bw_poly, linear_cde]. poly includes 6 polynomial parameters for f-theta camera, is_bw_poly indicates if it is a backward polynomial, linear_cde includes 3 parameters for f-theta camera | | pinhole_intrinsic | .tar | Pinhole camera intrinsic parameters for each view (for rectification) | keys include pinhole_intrinsic.{camera_name}.npy. The npy file stores a vector [fx, fy, cx, cy, w, h]. | | car_mask_coarse | .png | A coarse mask for the vehicle hood | pixel value > 0 means hood area.|

{camera_name} includes

• camera_front_wide_120fov
• camera_cross_left_120fov
• camera_cross_right_120fov
• camera_rear_left_70fov
• camera_rear_right_70fov
• camera_rear_tele_30fov
• camera_front_tele_30fov

HDMap Annotations

| Folder | File Format | Description | Explanation | | ----- | ----- | ----- | ----- | | 3d_lanes | .tar | 3D lane boundaries (left and right), polyline format | keys include lanes.json. You can access the left and right boundaries via ['lanes.josn']['labels'][0/1/2/…]['labelData']['shape3d']['polylines3d']['polylines']. Here ['lanes.json']['labels'] is a list, includes many left-right lane pairs. | | 3d_lanelines | .tar | 3D lane centerlines, polyline format | keys include lanelines.json. Laneline is the center of left and right lanes. You can access the vertices via ['lanelines.json']['labels'][0/1/2/…]['labelData']['shape3d']['polyline3d']['vertices'] | | 3d_road_boundaries | .tar | Road boundary annotations, polyline format | keys include road_boundaries.json. You can access the vertices via ['road_boundaries.json']['labels'][0/1/2/…]['labelData']['shape3d']['polyline3d']['vertices'] | | 3d_wait_lines | .tar | Waiting lines at intersections, polyline format | keys include wait_lines.json. You can access the vertices via ['wait_lines.json']['labels'][0/1/2/…]['labelData']['shape3d']['polyline3d']['vertices'] | | 3d_crosswalks | .tar | Crosswalk annotations, polygon format | keys include crosswalks.json. You can access the vertices via ['crosswalks.json']['labels'][0/1/2/…]['labelData']['shape3d']['surface']['vertices'] | | 3d_road_markings | .tar | Road surface markings (turning arrows, stop lines, etc.), polygon format | keys include road_markings.json. You can access the vertices via ['road_markings.json']['labels'][0/1/2/…]['labelData']['shape3d']['surface']['vertices'] | | 3d_poles | .tar | Traffic poles, polyline format | keys include poles.json. You can access the vertices via ['poles.json']['labels'][0/1/2/…]['labelData']['shape3d']['polyline3d']['vertices'] | | 3d_traffic_lights | .tar | Traffic lights, 3D cuboid format | keys include 3d_traffic_lights.json. You can access 8 corner vertices via ['3d_traffic_lights.json']['labels'][0/1/2/…]['labelData']['shape3d']['cuboid3d']['vertices'] | | 3d_traffic_signs | .tar | Traffic signs, 3D cuboid format | keys include 3d_traffic_signs.json. You can access 8 corner vertices via ['3d_traffic_signs.json']['labels'][0/1/2/…]['labelData']['shape3d']['cuboid3d']['vertices'] |

Dynamic Object Annotations

| Folder | File Format | Description | Explanation | | ----- | ----- | ----- | ----- | | all_object_info | .tar | 4D object tracking (position, dimensions, movement state) | keys include 000000.all_object_info.json, 000003.all_object_info.json, etc. For 000000.all_object_info.json, they store {tracking_id :{'object_to_world': 4x4 transformation matrix, 'object_lwh': [length, width, height], 'object_is_moving': True or False, 'object_type': str }} |

Object type includes

• Automobile
• Heavy_truck
• Bus
• Train_or_tram_car
• Trolley_bus
• Other_vehicle
• Trailer
• Person
• Stroller
• Rider
• Animal
• Protruding_object

Real-world Videos

We will release multi-view real-world videos for 5,843 clips soon.

Synthetic Videos Ground-Truth Label Correspondence

One clip will have 2 chunks and 7 weather variants, resulting in 14 synthetic video samples.

The first chunk (chunk_id=0) corresponds to 000000 to 000120 indices in the label.

The second chunk (chunk_id=1) corresponds to 000121 to 000241 indices in the label.

Camera and LiDAR Synchronization

* Camera Frame Rate: 30 FPS * LiDAR Frame Rate: 10 FPS * Synchronization: Each LiDAR frame corresponds to 3 consecutive camera frames. * Pose Interpolation: Camera poses are interpolated at the starting timestamp of each image frame.

!image/png

Please visit https://github.com/nv-tlabs/Cosmos-Drive-Dreams/cosmos-drive-dreams-toolkits to learn about how to use this dataset example.

Dataset Quantification

* Record Count: 81,802 sample data points * Feature Count: HDMap control data, LiDAR data, Cosmos-generated Videos * Measurement of Total Data Storage: 3TB

Reference(s):

Public Repo Location: https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicle-Cosmos-Drive-Dreams

Ethical Considerations:

Please report security vulnerabilities or NVIDIA AI Concerns here.

Citation

@misc{nvidia2025cosmosdrivedreams,
  title     = {Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Models},
  author = {Ren, Xuanchi and Lu, Yifan and Cao, Tianshi and Gao, Ruiyuan and
          Huang, Shengyu and Sabour, Amirmojtaba and Shen, Tianchang and
          Pfaff, Tobias and Wu, Jay Zhangjie and Chen, Runjian and
          Kim, Seung Wook and Gao, Jun and Leal-Taixe, Laura and
          Chen, Mike and Fidler, Sanja and Ling, Huan}
  year      = {2025},
  url       = {https://arxiv.org/abs/2506.09042}
}