Opens2v 5m

OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generation

If you like our project, please give us a star ⭐ on GitHub for the latest update.

We release the high-quality subset. It’s not just 0.3M samples — we applied filtering across the entire 5M data. You can click here for more details, and click here to download.

Regarding how to use OpenS2V-5M during the training phase, we provide a demo dataloader here. Alternatively, you can refer to the BindWeave code, which uses OpenS2V-5M for training.

✨ Summary

We create the first open-source large-scale S2V generation dataset OpenS2V-5M, which consists of five million high-quality 720P subject-text-video triples. To ensure subject-information diversity in our dataset by, we (1) segmenting subjects and building pairing information via cross-video associations and (2) prompting GPT-Image on raw frames to synthesize multi-view representations. The dataset supports both Subject-to-Video and Text-to-Video generation tasks.

This dataset is presented in the paper: OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generation

📣 Usage

We provide some resources that may help you: Tutorial 1, Tutorial 2, Demo Regular-Pairs Json, Demo Cross-Pairs Json, Demo Dataloader.

Due to the large quantity, we divide the subject-text-video triples into 13 parts. AestheticScore, GmeScore, and other scores for all videos/images are recorded in the JSON file, you can adjust thresholds to balance quality and quantity.

Get Caption and Metadata

We store the video caption, width, height, fps, aesthetic score, motion score, and other details in JSON files at the following path:

cd Jsons

Get Mask and Bbox

Due to the slower I/O speed for small files, we store the subject mask and subject bbox in RLE format within JSON files instead of extracting them as separate image files. To use them, simply unzip each total_part.json.zip file (e.g., using the unzip command). For usage instructions, please refer here.

cd Jsons/mask_and_bbox
unzip total_part1.json.zip
unzip total_part2.json.zip
...

OpenS2V-5M only extracts the human head without human face as subject image. For higher-quality human face subject images, we recommend using the following script to dynamically extract them during training.

!pip install diffusers==0.33.1
from diffusers.pipelines.consisid.consisid_utils import prepare_face_models, process_face_embeddings_infer
from huggingface_hub import snapshot_download

snapshot_download(repo_id="BestWishYsh/ConsisID-preview", local_dir="BestWishYsh/ConsisID-preview")
face_helper_1, face_helper_2, face_clip_model, face_main_model, eva_transform_mean, eva_transform_std = (
    prepare_face_models("BestWishYsh/ConsisID-preview", device="cuda", dtype=torch.bfloat16)
)

image = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/consisid/consisid_input.png?download=true"
_, _, face_image, _ = process_face_embeddings_infer(
    face_helper_1,
    face_clip_model,
    face_helper_2,
    eva_transform_mean,
    eva_transform_std,
    face_main_model,
    "cuda",
    torch.bfloat16,
    image,
    is_align_face=True,
)

Get Background

We store the pure background image corresponding to each training sample in the following path:

cd Background

Get Cross-Frame Pairs

We identify 0.35M clustering centers from the regular data, each containing an average of 10.13 samples, meaning we could theoretically create far more than 0.35M × 10.13 pairs. In addition to using processed pairs, you can use this script online during training to build pairing information.

cd Jsons/cross_video_pairs

Get GPT-Frame Pairs

We are unable to open-source this data for the time being, but you can build it yourself using this script. Some examples here:

cd Jsons/gpt_video_pairs

Get Videos

We split some large files (> 50G) into multiple small files, you can recover these files by cat, for instance:

cd Videos
cat total_part1.tar.split* > total_part1.tar
tar -xvf total_part1.tar

To save space, we initially excluded audio during video collection (Part 1–Part 9). Later, we realized this was actually quite important, so only Parts 10 through 13 include audio. To obtain the video without jump cuts or watermarks, extract it using the crop and face_cut specified in the JSON file：

import decord

# load original video
vr = decord.VideoReader(video_path, num_threads=1)

# crop video
s_x, e_x, s_y, e_y = ele["crop"]

# cut video
video_start, video_end = ele["face_cut"]
idx = list(range(video_start, video_end))

# get final video
video = vr.get_batch(idx).asnumpy()
video = torch.tensor(video).permute(0, 3, 1, 2)  # Convert to TCHW format
video = video[:, :, s_y:e_y, s_x:e_x]

⚙️ Process Your Own Videos

For how to process your own data like OpenS2V-5M in the OpenS2V-Nexus paper, please refer to here.

💡 Description

• Repository: Code, Page, Dataset, Benchmark
• Paper: https://huggingface.co/papers/2505.20292
• Point of Contact: Shenghai Yuan
• License: CC-BY-4.0

✏️ Citation

If you find our paper and code useful in your research, please consider giving a star and citation.

@article{yuan2025opens2v,
  title={OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generation},
  author={Yuan, Shenghai and He, Xianyi and Deng, Yufan and Ye, Yang and Huang, Jinfa and Lin, Bin and Luo, Jiebo and Yuan, Li},
  journal={arXiv preprint arXiv:2505.20292},
  year={2025}
}