💻 EAI-Taxonomy Code w/ DCLM

🏆 Website | 🖥️ Code | 📖 Paper

A 564 billion token dataset of high-quality code curated from web data using taxonomy-based filtering.

🎯 Dataset Overview

This dataset is part of the Essential-Web project, which introduces a new paradigm for dataset curation using expressive metadata and simple semantic filters. Unlike traditional code datasets that require complex domain-specific pipelines, our approach leverages a 12-category taxonomy to efficiently identify and extract high-quality code data.

💡 EAI-Taxonomy Code w/ DCLM (564B tokens): Documents targeting code that exhibit intermediate to advanced reasoning, combined with the DCLM classifier to filter for instruction-dense documents. Also includes mathematics content (51 - Mathematics) to match the scope of existing code datasets.

🏆 Performance

Our taxonomy-based approach achieves competitive results with significantly less curation effort:

Results show competitive code generation performance with a +46.8% improvement in computer science knowledge (MMLU-CS) compared to baseline.

🔍 Key Findings

• Code Generation: All datasets perform within statistical error on single-function generation benchmarks (HumanEval+, MBPP+)
• Code Knowledge: Clear impact on general computer science knowledge when using taxonomy-curated data
• Efficiency: Achieves strong performance without complex domain-specific curation pipelines

Dataset Schema Documentation

Overview

This dataset contains web-crawled text data with comprehensive metadata, quality signals, and taxonomic classifications. Each record represents a document extracted from web archives with detailed provenance tracking and quality assessment metrics.

Core Fields

EAI Taxonomy Classification

Comprehensive hierarchical classification system with primary and secondary labels - the most important feature of this dataset. The taxonomy is designed to provide detailed subject categorization, document type identification, content quality assessment, and extraction quality indicators.

How to Load the Dataset

This section provides examples of how to load the EssentialAI/eai-taxonomy-code-w-dclm dataset using different Python libraries and frameworks.

Using Hugging Face Datasets (Standard Method)

The simplest way to load the dataset is using the Hugging Face datasets library:

from datasets import load_dataset

# Load the entire dataset
dataset = load_dataset("EssentialAI/eai-taxonomy-code-w-dclm")

# View dataset structure
print(dataset)
print(f"Number of examples: {len(dataset['train'])}")

You can also load the dataset in streaming mode to avoid downloading the entire dataset at once:

from datasets import load_dataset

# Load in streaming mode
dataset = load_dataset("EssentialAI/eai-taxonomy-code-w-dclm", streaming=True)
data_stream = dataset["train"]

# Iterate through examples
for example in data_stream.take(5):
    print(example)

Using PySpark

For large-scale distributed processing, you can load the dataset using PySpark with the pyspark_huggingface library:

# First install the required library:
# pip install pyspark_huggingface

import pyspark_huggingface
from pyspark.sql import SparkSession

# Initialize Spark session
spark = SparkSession.builder.appName("EAI-Taxonomy-Code-w-DCLM").getOrCreate()

# Load the dataset using the "huggingface" data source
df = spark.read.format("huggingface").load("EssentialAI/eai-taxonomy-code-w-dclm")

# Basic dataset exploration
print(f"Dataset shape: {df.count()} rows, {len(df.columns)} columns")
df.show(10)
df.printSchema()

# Load only specific columns for efficiency
df_subset = (
    spark.read.format("huggingface")
    .option("columns", '["column1", "column2"]')  # Replace with actual column names
    .load("EssentialAI/eai-taxonomy-code-w-dclm")
)

# Run SQL queries on the dataset
df.createOrReplaceTempView("eai_taxonomy_code_w_dclm_dataset")
result = spark.sql("""
    SELECT COUNT(*) as total_examples
    FROM eai_taxonomy_code_w_dclm_dataset
""")
result.show()

Using Daft

Daft provides a modern DataFrame library optimized for machine learning workloads. You can load the dataset directly from Hugging Face:

import daft

# Load the entire dataset
df = daft.read_parquet("hf://datasets/EssentialAI/eai-taxonomy-code-w-dclm")

# Basic exploration
print("Dataset schema:")
df.schema()

print("First 5 rows:")
df.show(5)

If you need to access private datasets or use authentication:

import daft
from daft.io import IOConfig, HTTPConfig

io_config = IOConfig(http=HTTPConfig(bearer_token="your_token"))
df = daft.read_parquet("hf://datasets/EssentialAI/eai-taxonomy-code-w-dclm", io_config=io_config)

Installation Requirements

Make sure you have the required libraries installed:

# For Hugging Face datasets
pip install datasets

# For PySpark with Hugging Face integration
pip install pyspark_huggingface

# For Daft
pip install daft

📜 License

Essential-Web-v1.0 contributions are made available under the ODC attribution license; however, users should also abide by the Common Crawl - Terms of Use. We do not alter the license of any of the underlying data.

📝 Citation

@misc{ai2025essentialwebv1024ttokens,
      title={Essential-Web v1.0: 24T tokens of organized web data}, 
      author={Essential AI and : and Andrew Hojel and Michael Pust and Tim Romanski and Yash Vanjani and Ritvik Kapila and Mohit Parmar and Adarsh Chaluvaraju and Alok Tripathy and Anil Thomas and Ashish Tanwer and Darsh J Shah and Ishaan Shah and Karl Stratos and Khoi Nguyen and Kurt Smith and Michael Callahan and Peter Rushton and Philip Monk and Platon Mazarakis and Saad Jamal and Saurabh Srivastava and Somanshu Singla and Ashish Vaswani},
      year={2025},
      eprint={2506.14111},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2506.14111}, 
}

📊 Structured Schema (Zero-Fabrication)

Estimated Rows: 273,847,736