language:

• en
• fr
• de
• it
• es
• la
• nl
• pl

Common Corpus

Full data paper

Common Corpus is the largest open and permissible licensed text dataset, comprising 2 trillion tokens (1,998,647,168,282 tokens). It is a diverse dataset, consisting of books, newspapers, scientific articles, government and legal documents, code, and more. Common Corpus has been created by Pleias in association with several partners.

Common Corpus differs from existing open datasets in that it is:

• Truly Open: contains only data that is either uncopyrighted or permissively licensed
• Traceable: each individual document is associated with documented contextual information, including licensed use or lack of copyright.
• Multilingual: mostly representing English and French data, but contains data for 8 languages with more than 10 billion tokens (German, Spanish, Italian, Polish, Greek, Latin) and 33 languages with more than 1 billion tokens.
• Diverse: consisting of scientific articles, government and legal documents, code, and cultural heritage data, including books and newspapers
• Extensively Curated: spelling and formatting has been corrected from digitized texts, harmful and toxic content has been removed, and content with low educational content has also been removed.

The dataset in its entirety meets the requirements of the Code of Conduct of the AI Act and goes further than the current requirements for data transparency. It aims to set a new standard of openness in AI, showing that detailed provenance at a granular document level is a realistic objective, even at the scale of 2 trillion tokens.

Common Corpus makes it possible to train model compatible with the Open Source Initiative’s definition of open-source AI, which includes openness of use, meaning use is permitted for “any purpose and without having to ask for permission". Based on the available licensing information Common Corpus can be filtered to only include public domain works or a subset of free licenses (like attribution only).

About Common Corpus

Common Corpus is made of six carefully curated collections:

• OpenCulture: our largest collection at 885,982,490,090 tokens, featuring public domain books, newspapers from cultural heritage repositories and open projets like Wikisource ad Gutenberg. We're developing innovative tools of OCR correction based on Pleias Models to correct historical digitization errors, while implementing advanced toxicity filtering to ensure content meets modern ethical standards.
• OpenGovernment: 406,581,454,455 tokens of financial and legal documents, including Finance Commons (from sources like SEC and WTO) and Legal Commons (including Europarl and Caselaw Access Project), providing enterprise-grade training data from regulatory bodies and administrative sources.
• OpenSource: 283,227,402,898 tokens of high-quality code in open source from GitHub, filtered using ArmoRM to ensure only the top 80% of submissions by quality rating are included.
• OpenScience: 281,193,563,789 tokens of academic content from Open Alex and other open science reposiories, processed using vision-language models to preserve crucial document structure and formatting.
• OpenWeb: 73,217,485,489 tokens from Wikipedia (official releases from the Wikimedia Foundation on Huggingface), YouTube Commons and Stack-Exchange.
• Open Semantic: 67,958,671,827 tokens from Wikidata (official releases from the Wikimedia Foundation on Huggingface). The data has been reprocessed thanks to support and help of Wikidata and Wikimedia Germany. It includes the transcriptions of all the semantic triplets into natural language statements in over 300 languages.

The first version of Common Corpus was released in November of 2024. The second version includes the following features:

• Additional datasets, including OpenSemantic thanks to a collaboration with Wikidata, extended scraps from Wikisource and the Gutenberg project, as well large open science datasets with permissive licences.
• Detailed document-level information, including licensing and other core metadata whenever available.
• Direct link to original publication for a significant share of documents (especially, from Github, Wikipedia, Wikidata…)

The dataset release is accompanied by a comprehensive technical report detailing our methodologies and data sources will accompany the release, ensuring full transparency and reproducibility.

Dataset Structure

Provenance

The provenance of the datasets that make up Refined Common Corpus is detailed in the technical report [link]. Additionally, the original source URL is available in the metadata for each document for most of the dataset.

How to Use

Considerations for Using the Data

All data in Common Corpus are permissibly licensed and may be used for both commercial and non-commercial purposes.

The dataset is multilingual. The language text is included in the metadata, so data can be filtered by language. Additionally, some of the text data are historical. The year each text is written is included in the metadata, therefore it is possible to construct a dataset with a custom date cutoff if desired.

Discussion of Bias

Some of the dataset sources contain biased and toxic content, such as stereotypes about certain minoritized groups. We have removed texts which had high toxicity scores according to our toxicity classifier, Celadon, or which contain offensive terms and slurs. See our preprint for more details.

Personal and Sensitive Information

We have attempted to remove personally identifiable information (PII). We primarily use Microsoft Presidio, but make additional modifications to account for language- and country-specific considerations, such as European phone number formats.

Some small parts of the French administrative common crawl have been entirely dropped using our unreleased small reasoning model for GDPR-filtering, due to the heightened risk of transmitting identifiable indirect personal information.

Using Common Corpus

from datasets import load_dataset
data = load_dataset('PleIAs/common_corpus')

Acknowledgements

The Corpus was built up with the support and concerted efforts of the AI Alliance, the French Ministry of Culture and DINUM, as part of the prefiguration of the service offering of the Alliance for Language technologies EDIC (ALT-EDIC).

This dataset was also made in partnership with Wikimedia Enterprise and Wikidata/Wikimedia Germany.

The corpus was stored and processed with the generous support of the AI Alliance, Jean Zay (Eviden, Idris), Tracto AI, Mozilla. Generation of OCR correction at scale were performed using HPC resources from two GENCI–IDRIS grants: 2023-AD011014736 and GC011015451.

The collection of the corpus has been largely facilitated thanks to the open science LLM community insights, cooperation and support (Eleuther AI, Allen AI, HuggingFace…).