license: odc-by
task_categories:

• translation
• text-generation
  language:
• bos
• bul
• cat
• ces
• dan
• deu
• ell
• eng
• est
• eus
• fin
• fra
• gle
• glg
• hrv
• hun
• isl
• ita
• kat
• lav
• lit
• mkd
• mlt
• nld
• nno
• nob
• pol
• por
• ron
• slk
• slv
• spa
• sqi
• srp
• swe
• tur
• ukr
  configs:
• config_name: bos
  data_files:
  • split: train
    path: bos/fineweb-edu*
• config_name: bul
  data_files:
  • split: train
    path: bul/fineweb-edu*
• config_name: cat
  data_files:
  • split: train
    path: cat/fineweb-edu*
• config_name: ces
  data_files:
  • split: train
    path: ces/fineweb-edu*
• config_name: dan
  data_files:
  • split: train
    path: dan/fineweb-edu*
• config_name: deu
  data_files:
  • split: train
    path: deu/fineweb-edu*
• config_name: ell
  data_files:
  • split: train
    path: ell/fineweb-edu*
• config_name: eng
  data_files:
  • split: train
    path: eng/fineweb-edu*
• config_name: est
  data_files:
  • split: train
    path: est/fineweb-edu*
• config_name: eus
  data_files:
  • split: train
    path: eus/fineweb-edu*
• config_name: fin
  data_files:
  • split: train
    path: fin/fineweb-edu*
• config_name: fra
  data_files:
  • split: train
    path: fra/fineweb-edu*
• config_name: gle
  data_files:
  • split: train
    path: gle/fineweb-edu*
• config_name: glg
  data_files:
  • split: train
    path: glg/fineweb-edu*
• config_name: hrv
  data_files:
  • split: train
    path: hrv/fineweb-edu*
• config_name: hun
  data_files:
  • split: train
    path: hun/fineweb-edu*
• config_name: isl
  data_files:
  • split: train
    path: isl/fineweb-edu*
• config_name: ita
  data_files:
  • split: train
    path: ita/fineweb-edu*
• config_name: kat
  data_files:
  • split: train
    path: kat/fineweb-edu*
• config_name: lav
  data_files:
  • split: train
    path: lav/fineweb-edu*
• config_name: lit
  data_files:
  • split: train
    path: lit/fineweb-edu*
• config_name: mkd
  data_files:
  • split: train
    path: mkd/fineweb-edu*
• config_name: mlt
  data_files:
  • split: train
    path: mlt/fineweb-edu*
• config_name: nld
  data_files:
  • split: train
    path: nld/fineweb-edu*
• config_name: nno
  data_files:
  • split: train
    path: nno/fineweb-edu*
• config_name: nob
  data_files:
  • split: train
    path: nob/fineweb-edu*
• config_name: pol
  data_files:
  • split: train
    path: pol/fineweb-edu*
• config_name: por
  data_files:
  • split: train
    path: por/fineweb-edu*
• config_name: ron
  data_files:
  • split: train
    path: ron/fineweb-edu*
• config_name: slk
  data_files:
  • split: train
    path: slk/fineweb-edu*
• config_name: slv
  data_files:
  • split: train
    path: slv/fineweb-edu*
• config_name: spa
  data_files:
  • split: train
    path: spa/fineweb-edu*
• config_name: sqi
  data_files:
  • split: train
    path: sqi/fineweb-edu*
• config_name: srp_Cyrl
  data_files:
  • split: train
    path: srp_Cyrl/fineweb-edu*
• config_name: swe
  data_files:
  • split: train
    path: swe/fineweb-edu*
• config_name: tur
  data_files:
  • split: train
    path: tur/fineweb-edu*
• config_name: ukr
  data_files:
  • split: train
    path: ukr/fineweb-edu*

Helsinki-NLP/fineweb-edu-translated

fineweb-edu-tanslated is a collection of automatically translated documents from fineweb-edu.
Translations are based on OPUS-MT and HPLT-MT models.
The data covers 36,704,000 documents with over 28 billion space-searated tokens of English data translated into 36 languages.
The total data set is incudes of over 960 billion tokens and the translated documents are aligned across all languages.

More information about how the data has been produced can be found on https://github.com/Helsinki-NLP/translate-fineweb
The corpus is also available with aligned sentences in synOPUS: https://opus.nlpl.eu/synthetic/transweb-edu.php

Supported Languages

Citation Information

Please acknowledge the source when using the data and, please,
cite the following article if you use any part of this corpus in your own work:

@article{tiedemann2023democratizing,
  title={Democratizing neural machine translation with {OPUS-MT}},
  author={Tiedemann, J{\"o}rg and Aulamo, Mikko and Bakshandaeva, Daria and Boggia, Michele and Gr{\"o}nroos, Stig-Arne and Nieminen, Tommi and Raganato, Alessandro and Scherrer, Yves and Vazquez, Raul and Virpioja, Sami},
  journal={Language Resources and Evaluation},
  number={58},
  pages={713--755},
  year={2023},
  publisher={Springer Nature},
  issn={1574-0218},
  doi={10.1007/s10579-023-09704-w}
}

Translation Models

The following translation models have been used for creating the data:

Acknowledgements

None of this would be possible without the enormous work done by Common Crawl providing the essential data that most open datasets for language modeling are based on.
Furthermore, we are also grateful for the data preparation work done by Hugging Face and the community on top of the crawled data from Common Crawl published under the label of fineweb-edu.
Important for this work is also the availability of parallel data through OPUS and the public translation models based on that data.
Furthermore, this project was supported by the European Union's Horizon Europe research and innovation programme through the HPLT project under grant agreement No 101070350.
Finally, we also want to acknowledge the computational resource made available from the Finnish national allocation for the LUMI supercomputer (https://www.lumi-supercomputer.eu)
through the extreme scale project MaMuLaM: Massively Multilingual Language Models. None of the translated data would exist without this infrastructure and the compute.