EvoStruggle Dataset

Data of video recordings of manual activities of various people performing manual tasks from a first-person perspective. Activities include origami, card shuffling, tangrams and knot tying.

Please see the paper "EvoStruggle: A Dataset Capturing the Evolution of Struggle across Activities and Skill Levels" for more details. The EvoStruggle Dataset Promo Video is available on YouTube.

A related talk was given by Prof Walterio Mayol - Keynote: From Skill to Struggle at the ICCV 2025 SAUAFG Workshop.

Introduction of the Struggle Determination

Our Definition of Struggle

“To struggle, as defined in the dictionary (verb), is to experience difficulty and make a great effort in order to do something.”

In this work, struggle is defined as observable difficulty in completing a given activity. It may be characterized by one or more of the following indicators:

• Motor hesitation of the hands
• Repeated attempts
• Prolonged actions
• Body-gesture signs of frustration (e.g., hand and/or head movements)
• Disruptive errors and pauses

Struggle examples in each of the activities:

What's new in this dataset?

• Over 60 hours video recordings, 2,793 videos, and 5,385 annotated temporal struggle segments from 76 participants.

• Evolution of Skill: Five Attempts/Repetitions Each Task.

• Diversity: 18 Tasks Grouped into Four activities--Tying Knots, Origami, Tangram Puzzles, and Shuffling Cards.

Activities and Tasks

Usage of the Data

This section describes the annotation format, video naming convention, and data splits used in the Struggle Temporal Action Localization (Struggle TAL) task.

1. Annotations

The annotation files provide metadata for struggle moments in each video, including:

• Start time of a struggle segment
• End time of a struggle segment

These timestamps indicate when observable struggle occurs during task execution.

2. Video Naming Convention

Each video follows the naming format: , where

• participant_id: two-digit participant identifier (e.g. 01)
• task_index: two-digit index of the task within the corresponding activity (as listed above)
• attempt_id: repetition number of the task, ranging from 01 to 05

Example:
01_03_04 denotes participant 01 performing task 03 (e.g. Helmet, Ribbon Spread and Wave, Cyclist, or Carrick Bend, depending on the activity) on the fourth attempt.

3. Code Release

The official code release for the Struggle Temporal Action Localization task is available at:

• GitHub Repository: StruggleTAL

This repository can be used to reproduce the experimental results reported in the paper.

4. Data Splits

We provide three types of data splits for different training and evaluation settings (see Figure 6 in the paper for a visual overview).

4.1 Activity-Level Generalization (Cross-Domain)

Directory: splits/crossdomain_generalization

Description:
These splits are used for Activity-Level Generalization experiments across the four activities.

Files:

• <activity_name>_crossdomain.json
• <activity_name>_crossdomain_testonvalonly.json (recommended)

Notes:

• JSON files are used to run the experiments
• CSV files provide lists of video metadata
• The *_testonvalonly.json file contains only test samples from the validation split of the unseen activity

4.2 Task-Level Generalization (In-Domain)

Directory: splits/indomain_generalization

Description:
These splits support Task-Level Generalization experiments within each activity.

Files: <activity_name>_subactivity_data.json. Use these files to load the corresponding training and testing data.

4.3 Within-Activity and Separate-Attempts Evaluation

Directory: splits/separate_attempts

Description:

• Within-Activity Evaluation:
  Provides baseline Struggle TAL performance within the same activity (vanilla setting).
• Separate Attempts Evaluation:
  Investigates the effect of multiple attempts on Struggle TAL performance.

File: <activity_name>_sepattempt.json. Use this file to run both evaluation settings.

How to Download

Option 1: Hugging Face Dataset (Recommended)

Hugging Face Dataset - Contains 360p resized videos

Please refer to Downloading Datasets documents on Hugging Face to find the suitable command to download the dataset.

Option 2: Baidu NetDisk / 百度网盘

Choose between the full 1080p version or the compressed 360p version:

• EvoStruggle_Dataset - Full download with original 1080p video recordings (1.18 TB)
• new_struggle_dataset.tar.gz - Compressed version with 360p resized videos (41.81 GB)

Note: The Baidu NetDisk option provides higher resolution videos compared to the Hugging Face version.

Contributors

• Shijia Feng
• Michael Wray
• Walterio Mayol-Cuevas

Citation to this work

@misc{feng2025evostruggledatasetcapturingevolution,
      title={EvoStruggle: A Dataset Capturing the Evolution of Struggle across Activities and Skill Levels}, 
      author={Shijia Feng and Michael Wray and Walterio Mayol-Cuevas},
      year={2025},
      eprint={2510.01362},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2510.01362}, 
}