📚 FineWeb-Edu

1.3 trillion tokens of the finest educational data the 🌐 web has to offer

Paper: https://arxiv.org/abs/2406.17557

What is it?

📚 FineWeb-Edu dataset consists of 1.3T tokens and 5.4T tokens (FineWeb-Edu-score-2) of educational web pages filtered from 🍷 FineWeb dataset. This is the 1.3 trillion version.

To enhance FineWeb's quality, we developed an educational quality classifier using annotations generated by LLama3-70B-Instruct. We then used this classifier to retain only the most educational web pages. FineWeb-Edu outperforms FineWeb on popular benchmarks and shows the power of classifiers trained on synthetic data.

The Dataset Curation section details the process for creating the dataset.

You can find a deduplicated version of FineWeb-edu in SmolLM-Corpus. We find that the deduplication of this dataset doesn't have any impact on model performance in our ablation setup (1.8B trained on 350B tokens).

What is being released?

Along with the dataset, which includes all filtered CommonCrawl dumps since 2013, we also release the educational classifier used for the filtering as well as the code for training it and running inference at: https://github.com/huggingface/cosmopedia/tree/main/classification

Changelog

Previous versions remain available in the branch version name.

• v1.4.0 (11-07-2025): Added 6 new snapshots: CC-MAIN-2025-05, CC-MAIN-2025-08, CC-MAIN-2025-13, CC-MAIN-2025-18, CC-MAIN-2025-21, and CC-MAIN-2025-26 (January to June 2025)
• v1.3.0 (31-01-2025): Fixed an issue with some dumps where some documents hadn't been processed: CC-MAIN-2024-10, CC-MAIN-2024-18, CC-MAIN-2024-22, CC-MAIN-2024-26, CC-MAIN-2024-30, CC-MAIN-2024-33, CC-MAIN-2024-38, CC-MAIN-2024-42, CC-MAIN-2024-46 -- they now contain more data (~35B additional tokens).
• v1.2.0 (03-01-2025): Added 9 new snapshots: CC-MAIN-2024-18, CC-MAIN-2024-22, CC-MAIN-2024-26, CC-MAIN-2024-30, CC-MAIN-2024-33, CC-MAIN-2024-38, CC-MAIN-2024-42, CC-MAIN-2024-46, CC-MAIN-2024-51, covering April to December 2024.
• v1.0.0 (02-06-2024): Initial version

How to load the dataset

Similarily to FineWeb, You can load the full dataset or a specific crawl/dump. Dumps have the format CC-MAIN-(year)-(week number).

(Smaller) sample versions

Along with config default (all the data), and the configs for each individual dump, you can also download the following configs:

• sample-350BT: a subset randomly sampled from the whole dataset of around 350B gpt2 tokens
• sample-100BT: a subset randomly sampled from the whole dataset of around 100B gpt2 tokens
• sample-10BT: a subset randomly sampled from the whole dataset of around 10B gpt2 tokens

sample-10BT was sampled from sample-100BT which in turn was sampled from sample-350BT.

Using 🏭 [`datatrove`](https://github.com/huggingface/datatrove/)

from datatrove.pipeline.readers import ParquetReader

# limit determines how many documents will be streamed (remove for all)
data_reader = ParquetReader("hf://datasets/HuggingFaceFW/fineweb-edu", glob_pattern="data/*/*.parquet", limit=1000)
# or to fetch a specific dump CC-MAIN-2024-10,  eplace "CC-MAIN-2024-10" with "sample/100BT" to use the 100BT sample
data_reader = ParquetReader("hf://datasets/HuggingFaceFW/fineweb-edu/CC-MAIN-2024-10", limit=1000) 
for document in data_reader():
    # do something with document
    print(document)

###############################    
# OR for a processing pipeline:
###############################

from datatrove.executor import LocalPipelineExecutor
from datatrove.pipeline.readers import ParquetReader
from datatrove.pipeline.filters import LambdaFilter
from datatrove.pipeline.writers import JsonlWriter

pipeline_exec = LocalPipelineExecutor(
    pipeline=[
        # replace "CC-MAIN-2024-10" with "sample/100BT" to use the 100BT sample
        ParquetReader("hf://datasets/HuggingFaceFW/fineweb-edu/CC-MAIN-2024-10", limit=1000),
        LambdaFilter(lambda doc: "hugging" in doc.text),
        JsonlWriter("some-output-path")
    ],
    tasks=10
)
pipeline_exec.run()

Using `datasets`

from datasets import load_dataset
# use name="sample-10BT" to use the 10BT sample
fw = load_dataset("HuggingFaceFW/fineweb-edu", name="CC-MAIN-2024-10", split="train", streaming=True)

Dataset curation

A new approach has recently emerged for filtering LLM training datasets: using synthetic data to develop classifiers for identifying educational content. This technique was used in the trainings of LLama3 and Phi3, but its large-scale impact on web data filtering hasn't been fully explored or published.

The highly popular Phi3 models were trained on 3.3 and 4.8 trillion tokens, with the paper stating: “Our training data consists of heavily filtered publicly available web data (according to the 'educational level') from various open internet sources, as well as synthetic LLM-generated data". Similarly, the LLama3 blog post notes: “We found that previous generations of Llama are good at identifying high-quality data, so we used Llama 2 to help build the text-quality classifiers that are powering Llama 3.” However these classifiers and filtered datasets are not publicly available. To enhance FineWeb's quality, we developed an educational quality classifier using annotations generated by LLama3-70B-Instruct to create FineWeb-Edu.

Annotation

We used Llama3-70B-Instruct to score 500k FineWeb samples for their educational quality on a scale from 0 to 5.

We explored various prompts and found that the additive scale by Yuan et al. worked best. To avoid the LLM favoring highly technical pages like arXiv abstracts and submissions, we focused on grade-school and middle-school level knowledge. By setting a threshold of 3 (on a scale of 0 to 5) during the filtering process, we were able to also retain some high-level educational pages. The final prompt can be found here.

We also experimented with different LLMs: Llama3-70B-Instruct, Mixtral-8x-7B-Instruct, and Mixtral-8x22B-Instruct. Llama 3 and Mixtral-8x22B produced similar scores, while Mixtral-8x7B tended to be more generous, not fully adhering to the score scale. Verga et al. suggest using multiple LLMs as juries. We tried averaging the scores from the three models, but this shifted the distribution to the right due to the higher scores from Mixtral-8x7B. Training on a dataset filtered with a classifier using jury annotations performed worse than using a classifier based on Llama3 annotations. We hypothesize that the jury-based approach retains more low-quality samples.

Classifier training

We fine-tuned a Bert-like regression model using these annotations, based on Snowflake-arctic-embed. When converted to a binary classification using a score of 3 as a threshold for keeping and removing files, the model achieved an F1 score of 82%. The classification of FineWeb 15T tokens took 6k H100 GPU hours.

The classifier is available at: HuggingFaceFW/fineweb-edu-classifier/

Filtering and results

Note: You can find more details about the ablations and results in the FineWeb blog post.

We investigated the impact of using different thresholds for the filtering and found that threshold 3 gave the best overall results. Although using a threshold higher than 3 improves performance on knowledge and reasoning intensive benchmarks, it significantly degrades performance on HellaSwag and PIQA.

We then built 📚 FineWeb-Edu by filtering out samples with scores lower than 3. This removed 92% of the dataset, leaving us with 1.3T educational tokens. Our ablation demonstrated that this refined dataset surpasses 🍷 FineWeb and all other open web datasets, with remarkable improvements on educational benchmarks such as MMLU, ARC, and OpenBookQA. The plot below compares FineWeb-Edu to other web datasets:

To retain more tokens, we also experimented with a less strict threshold of 2 instead of 3. While being less performant than using threshold 3, it still outperformed FineWeb and it preserved 5.4T tokens. We release these two dataset as FineWeb-Edu and FineWeb-Edu-score-2 along with the classifier.

You will find all the ablation models in this collection. The FineWeb-Edu ablation model (trained on 350B tokens) is available at https://huggingface.co/HuggingFaceFW/ablation-model-fineweb-edu.

Considerations for Using the Data

This section is copied from the parent dataset: FineWeb.

Social Impact of Dataset

With the release of this dataset we aim to make model training more accessible to the machine learning community at large.

While multiple open-weights models with strong performance have been publicly released in the past, more often than not these releases are not accompanied by the corresponding training dataset. This is unfortunate as the dataset specificities and characteristics have been demonstrated to have a very large impact and role in the performances of the models. As the creation of a high quality training dataset is a fundamental requirement to training an LLM capable of excelling at downstream tasks, with 🍷 FineWeb we (a) not only make the dataset creation process more transparent, by sharing our entire processing setup including the codebase used, we also (b) help alleviate the costs of dataset curation, both in time and in compute, for model creators by publicly releasing our dataset with the community.

Discussion of Biases

Efforts were made to minimize the amount of NSFW and toxic content present in the dataset by employing filtering on the URL level. However, there are still a significant number of documents present in the final dataset that could be considered toxic or contain harmful content. As 🍷 FineWeb was sourced from the web as a whole, any harmful biases typically present in it may be reproduced on our dataset.

We deliberately avoided using machine learning filtering methods that define text quality based on the similarity to a “gold” source such as wikipedia or toxicity classifiers as these methods have been known to disproportionately remove content in specific dialects and overclassify as toxic text related to specific social identities, respectively.

Other Known Limitations

As a consequence of some of the filtering steps applied, it is likely that code content is not prevalent in our dataset. If you are training a model that should also perform code tasks, we recommend you use 🍷 FineWeb with a code dataset, such as The Stack v2. You should also probably consider complementing 🍷 FineWeb with specialized curated sources (such as Wikipedia, for example) as they will likely have better formatting than the wikipedia content included in 🍷 FineWeb (we did not tailor the processing to individual websites).

Additional Information

Licensing Information

The dataset is released under the Open Data Commons Attribution License (ODC-By) v1.0 license. The use of this dataset is also subject to CommonCrawl's Terms of Use.

Future work

We plan to work on better educational classifier to improve the quality of FineWeb-Edu.

Citation Information

You can cite our paper https://arxiv.org/abs/2406.17557 or this dataset:

@misc{lozhkov2024fineweb-edu,
    author       = { Lozhkov, Anton and Ben Allal, Loubna and von Werra, Leandro and Wolf, Thomas },  
    title        = { FineWeb-Edu: the Finest Collection of Educational Content }, 
    year         = 2024,  
    url          = { https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu },  
    doi          = { 10.57967/hf/2497 },
    publisher    = { Hugging Face }
}

📊 Structured Schema (Zero-Fabrication)

Estimated Rows: 1,525,223,056