task_categories:

• text-generation
• question-answering
  language:
• en
• zh
  tags:
• LLM
• pretraining
• finetuning
• midtraining
• reasoning
• STEM
• math
  size_categories:
• 10B<n<100B
  configs:
• config_name: automathtext-v2-ultra
  data_files:
  • split: train
    path:
    • nemotron_cc_high/80-90/*.parquet
    • nemotron_cc_high/90-100/*.parquet
    • nemotron_cc_medium_high/80-90/*.parquet
    • nemotron_cc_medium_high/90-100/*.parquet
    • dclm/80-90/*.parquet
    • dclm/90-100/*.parquet
    • fineweb_edu/80-90/*.parquet
    • fineweb_edu/90-100/*.parquet
    • math_web/80-90/*.parquet
    • math_web/90-100/*.parquet
    • reasoning_qa/80-90/*.parquet
    • reasoning_qa/90-100/*.parquet
      default: true
• config_name: automathtext-v2-high
  data_files:
  • split: train
    path:
    • nemotron_cc_high/50-60/*.parquet
    • nemotron_cc_high/60-70/*.parquet
    • nemotron_cc_high/70-80/*.parquet
    • nemotron_cc_high/80-90/*.parquet
    • nemotron_cc_high/90-100/*.parquet
    • nemotron_cc_medium_high/50-60/*.parquet
    • nemotron_cc_medium_high/60-70/*.parquet
    • nemotron_cc_medium_high/70-80/*.parquet
    • nemotron_cc_medium_high/80-90/*.parquet
    • nemotron_cc_medium_high/90-100/*.parquet
    • dclm/50-60/*.parquet
    • dclm/60-70/*.parquet
    • dclm/70-80/*.parquet
    • dclm/80-90/*.parquet
    • dclm/90-100/*.parquet
    • fineweb_edu/50-60/*.parquet
    • fineweb_edu/60-70/*.parquet
    • fineweb_edu/70-80/*.parquet
    • fineweb_edu/80-90/*.parquet
    • fineweb_edu/90-100/*.parquet
    • math_web/50-60/*.parquet
    • math_web/60-70/*.parquet
    • math_web/70-80/*.parquet
    • math_web/80-90/*.parquet
    • math_web/90-100/*.parquet
    • reasoning_qa/50-60/*.parquet
    • reasoning_qa/60-70/*.parquet
    • reasoning_qa/70-80/*.parquet
    • reasoning_qa/80-90/*.parquet
    • reasoning_qa/90-100/*.parquet
• config_name: automathtext-v2-medium-high
  data_files:
  • split: train
    path:
    • nemotron_cc_high/20-30/*.parquet
    • nemotron_cc_high/30-40/*.parquet
    • nemotron_cc_high/40-50/*.parquet
    • nemotron_cc_high/50-60/*.parquet
    • nemotron_cc_high/60-70/*.parquet
    • nemotron_cc_high/70-80/*.parquet
    • nemotron_cc_high/80-90/*.parquet
    • nemotron_cc_high/90-100/*.parquet
    • nemotron_cc_medium_high/20-30/*.parquet
    • nemotron_cc_medium_high/30-40/*.parquet
    • nemotron_cc_medium_high/40-50/*.parquet
    • nemotron_cc_medium_high/50-60/*.parquet
    • nemotron_cc_medium_high/60-70/*.parquet
    • nemotron_cc_medium_high/70-80/*.parquet
    • nemotron_cc_medium_high/80-90/*.parquet
    • nemotron_cc_medium_high/90-100/*.parquet
    • dclm/20-30/*.parquet
    • dclm/30-40/*.parquet
    • dclm/40-50/*.parquet
    • dclm/50-60/*.parquet
    • dclm/60-70/*.parquet
    • dclm/70-80/*.parquet
    • dclm/80-90/*.parquet
    • dclm/90-100/*.parquet
    • fineweb_edu/20-30/*.parquet
    • fineweb_edu/30-40/*.parquet
    • fineweb_edu/40-50/*.parquet
    • fineweb_edu/50-60/*.parquet
    • fineweb_edu/60-70/*.parquet
    • fineweb_edu/70-80/*.parquet
    • fineweb_edu/80-90/*.parquet
    • fineweb_edu/90-100/*.parquet
    • math_web/20-30/*.parquet
    • math_web/30-40/*.parquet
    • math_web/40-50/*.parquet
    • math_web/50-60/*.parquet
    • math_web/60-70/*.parquet
    • math_web/70-80/*.parquet
    • math_web/80-90/*.parquet
    • math_web/90-100/*.parquet
    • reasoning_qa/20-30/*.parquet
    • reasoning_qa/30-40/*.parquet
    • reasoning_qa/40-50/*.parquet
    • reasoning_qa/50-60/*.parquet
    • reasoning_qa/60-70/*.parquet
    • reasoning_qa/70-80/*.parquet
    • reasoning_qa/80-90/*.parquet
    • reasoning_qa/90-100/*.parquet
• config_name: automathtext-v2-low-medium-high
  data_files:
  • split: train
    path:
    • nemotron_cc_high//.parquet
    • nemotron_cc_medium_high//.parquet
    • dclm//.parquet
    • fineweb_edu//.parquet
    • math_web//.parquet
    • reasoning_qa//.parquet
• config_name: nemotron_cc_high
  data_files:
  • split: train
    path:
    • nemotron_cc_high//.parquet
• config_name: nemotron_cc_medium_high
  data_files:
  • split: train
    path:
    • nemotron_cc_medium_high//.parquet
• config_name: dclm
  data_files:
  • split: train
    path:
    • dclm//.parquet
• config_name: fineweb_edu
  data_files:
  • split: train
    path:
    • fineweb_edu//.parquet
• config_name: math_web
  data_files:
  • split: train
    path:
    • math_web//.parquet
• config_name: megamath
  data_files:
  • split: train
    path:
    • megamath//.parquet
• config_name: reasoning_qa
  data_files:
  • split: train
    path:
    • reasoning_qa//.parquet
• config_name: refinecode
  data_files:
  • split: train
    path:
    • refinecode//.parquet

🚀 AutoMathText-V2: A 2.46 Trillion Token AI-Curated STEM Pretraining Dataset

📊 AutoMathText-V2 consists of 2.46 trillion tokens of high-quality, deduplicated text spanning web content, mathematics, code, reasoning, and bilingual data. This dataset was meticulously curated using a three-tier deduplication pipeline and AI-powered quality assessment to provide superior training data for large language models.

Our dataset combines 50+ premium data sources with advanced processing techniques, including semantic deduplication, contamination detection, and intelligent text cleaning to deliver exceptional model performance across diverse domains.

🎯 What makes AutoMathText-V2 special?

• 🔢 STEM Concentration: Specially optimized for STEM content (especially Math)
• 🔍 Triple Deduplication: Exact → Fuzzy (MinHash+LSH) → Semantic (GTE embeddings)
• 🤖 AI Quality Assessment: Qwen2-based classifier with multi-source score fusion
• 🧹 Advanced Text Cleaning: All text data was processed using Ultimate Data Cleaner v7.5, which provides robust, high-performance cleaning tailored for web-scraped and scientific data.
• 🛡️ Contamination Prevention: Automatic test set leak detection and removal

📚 Dataset Composition

Token Distribution by Domain

🔥 Complete Data Sources by Domain (52 Premium Datasets)

📍 DCLM Domain

📚 FineWeb Edu Domain

🌏 FineWeb Edu Chinese Domain

🔢 Math Web Domain

🏆 Nemotron CC Domain

💻 RefineCode Domain

🧠 Reasoning QA Domain

🔄 Translation Domain

📊 MegaMath Domain

Total: 52 Premium Data Sources with official HuggingFace dataset links covering web content, mathematics, code, reasoning, formal proofs, and bilingual data.

🛠️ Processing Pipeline

1. Data Extraction & Standardization

{
    "domain_prefix": "lbty.org",
    "id": "117b6a7d-5126-41fe-9bc2-d276e98632e6",
    "meta": "{\"domain\": \"dclm\", \"ori_score\": 0.043276190757751465, \"source\": \"dclm_baseline\"}",
    "text": "Sabine Expedition\n\nThe Sabine Expedition was an expedition approved by the United States Congress in 1806...",
    "tokens": 145,  # Token count using Qwen2.5 tokenizer
    "url": "[https://lbty.org/american-indian-battles/sabine-expedition/](https://lbty.org/american-indian-battles/sabine-expedition/)",
    "score": 0.19072403013706207
}

2. Three-Tier Deduplication

🎯 Exact Deduplication

• SHA256 content hashing
• Priority-based duplicate resolution
• Result: ~30% exact duplicates removed

🔄 Fuzzy Deduplication

• MinHash Locality Sensitive Hashing (LSH)
• Jaccard similarity threshold: 0.9
• Connected components clustering
• Result: ~20% near-duplicates removed

🧠 Semantic Deduplication

• Alibaba-NLP/gte-multilingual-base embeddings
• K-means clustering (k=100,000)  
• Cosine similarity threshold: 0.007
• Result: ~10% semantic duplicates removed

3. 🤖 AI Quality Assessment

Qwen2-Based Classifier Architecture:

• Fine-tuned regression head for quality scoring
• Multi-source score normalization and fusion
• MSE loss with sigmoid activation

4. 🧹 Advanced Text Cleaning

All text data was processed using Ultimate Data Cleaner v7.5.0.5, which provides robust, high-performance cleaning tailored for web-scraped and scientific data.

Key Features Used:

• Advanced LaTeX & Code Protection: protect complex nested LaTeX environments (\begin{}...\end{}), inline math ($...$), commands, and markdown code fences.
• Quality Heuristics: Removes corrupted samples with excessive repetition, severe bracket imbalances, etc.

5. 🛡️ Contamination Detection

Test Set Protection:

• Math dataset test questions
• GSM8K evaluation problems  
• Exact string matching with preprocessing
• Automatic filtering during data extraction

🚀 How to Use

Loading with Datasets

from datasets import load_dataset

Load the high-quality subset: medium-high < high < ultra
dataset = load_dataset("OpenSQZ/AutoMathText-V2", name="medium-high", streaming=True)
dataset = load_dataset("OpenSQZ/AutoMathText-V2", name="high", streaming=True)
dataset = load_dataset("OpenSQZ/AutoMathText-V2", name="ultra", streaming=True)
Load specific domain
math_data = load_dataset("OpenSQZ/AutoMathText-V2", name="math_web", streaming=True)

💻 RefineCode Content Download

Important: For the RefineCode domain, only metadata is included in the dataset. The actual code content was removed to reduce storage requirements. To access the full code content, use the blob_id field from the metadata to download from AWS S3:

import os
import json
import boto3
from smart_open import open
from datasets import load_dataset

Setup AWS credentials
session = boto3.Session(
    aws_access_key_id=os.environ["AWS_ACCESS_KEY_ID"],
    aws_secret_access_key=os.environ["AWS_SECRET_ACCESS_KEY"]
)
s3 = session.client("s3")
def download_code_content(blob_id, src_encoding):
    """Download code content from AWS S3 using blob_id"""
    s3_url = f"s3://softwareheritage/content/{blob_id}"
    
    try:
        with open(s3_url, "rb", compression=".gz", transport_params={"client": s3}) as fin:
            content = fin.read().decode(src_encoding)
        return {"content": content}
    except Exception as e:
        return {"content": None, "error": str(e)}
Load RefineCode domain
refinecode_data = load_dataset("OpenSQZ/AutoMathText-V2", name="refinecode", streaming=True)
Process each sample to download content
for sample in refinecode_data:
    # Parse metadata to extract blob_id and encoding
    meta = json.loads(sample["meta"])
    blob_id = meta.get("blob_id")
    src_encoding = meta.get("src_encoding", "utf-8")
    
    if blob_id:
        # Download the actual code content
        code_data = download_code_content(blob_id, src_encoding)
        
        # Combine metadata with downloaded content
        full_sample = {
            **sample,
            "code_content": code_data["content"]
        }
        
        print(f"Downloaded content for {sample['id']}")
        print(f"Content length: {len(code_data['content']) if code_data['content'] else 0}")
        break

Requirements:

• AWS credentials with access to Software Heritage S3 bucket
• smart_open library: pip install smart_open[s3]
• boto3 library: pip install boto3

Note: This download method is required only for the RefineCode domain. All other domains contain the full text content directly in the dataset.

🌐 Dataset Structure & Configurations

Directory Structure

The dataset is organized by domain with quality-based token splits:

AutoMathText-V2/
├── dclm/                  # DCLM baseline web content
│   ├── 0-10/             # Bottom 10% quality tokens (score-based)
│   ├── 10-20/            # 10-20% quality tokens
│   ├── 20-30/            # 20-30% quality tokens
│   ├── ...               # Additional percentile ranges
│   └── 90-100/           # Top 10% highest quality tokens
├── fineweb_edu/           # FineWeb educational content
│   ├── 0-10/             # Bottom 10% quality tokens
│   ├── 10-20/            # 10-20% quality tokens
│   ├── ...               # Additional percentile ranges
│   └── 90-100/           # Top 10% highest quality tokens
├── fineweb_edu_chinese/   # Chinese educational content
│   ├── 0-10/             # Bottom 10% quality tokens
│   ├── ...               # Additional percentile ranges
│   └── 90-100/           # Top 10% highest quality tokens
├── math_web/              # Mathematics and scientific content
│   ├── 0-10/  .          # Bottom 10% quality tokens
│   ├── ...               # Additional percentile ranges
│   └── 90-100/           # Top 10% highest quality tokens
├── megamath/              # Specialized math collections
│   ├── 0-10/             # Bottom 10% quality tokens
│   ├── ...               # Additional percentile ranges
│   └── 90-100/           # Top 10% highest quality tokens
├── nemotron_cc_high/      # High quality Nemotron CommonCrawl
│   ├── 0-10/             # Bottom 10% quality tokens
│   ├── ...               # Additional percentile ranges
│   └── 90-100/           # Top 10% highest quality tokens
├── nemotron_cc_medium_high/ # Medium-high quality Nemotron CommonCrawl
│   ├── 0-10/            . # Bottom 10% quality tokens
│   ├── ...               # Additional percentile ranges
│   └── 90-100/           # Top 10% highest quality tokens
├── reasoning_qa/          # Instruction and reasoning data
│   ├── 0-10/             # Bottom 10% quality tokens
│   ├── ...               # Additional percentile ranges
│   └── 90-100/           # Top 10% highest quality tokens
├── refinecode/            # GitHub code repositories (Academic Use Only)
│   ├── 0-10/             # Bottom 10% quality tokens
│   ├── ...               # Additional percentile ranges
│   └── 90-100/           # Top 10% highest quality tokens
└── translation/           # English-Chinese translation pairs
    ├── 0-10/             # Bottom 10% quality tokens
    ├── ...               # Additional percentile ranges
    └── 90-100/           # Top 10% highest quality tokens

Quality-Based Token Distribution

Each domain is divided into 10 quality percentiles (0-10, 10-20, ..., 90-100) based on:

• Token count: Equal number of tokens per percentile bucket
• Quality scores: AI classifier scores from Qwen2-based quality assessment
• Percentile ranking: Higher percentiles contain higher quality content

Available Configurations

• Domain-specific configs: Load individual domains (dclm, fineweb_edu, math_web, reasoning_qa, etc.)
• Quality-filtered configs: Load specific quality ranges (e.g., dclm/90-100 for top quality DCLM content)
• Nemotron variants: Choose between nemotron_cc_high and nemotron_cc_medium_high based on quality needs
• Combined configs: Mix domains and quality levels based on training requirements
• Custom sampling: Select percentile ranges across multiple domains for balanced training

Language Distribution

• English: ~95% of content
• Chinese: ~5% of content

🔬 Technical Deep Dive

For detailed technical documentation, including:

• Complete processing pipeline specifications  
• Deduplication algorithm details
• Quality classifier training procedures
• Contamination detection methodology

Please refer to our Technical Documentation and GitHub Repository.

🤝 Contributing

We welcome contributions to improve dataset quality and processing techniques:

• 🐛 Bug Reports: Issues with data quality or processing
• 💡 Feature Requests: New data sources or processing improvements  
• 📚 Documentation: Help improve our guides and examples
• 🔬 Research: Collaborate on quality assessment and deduplication methods

📜 Licensing & Citation

License

Released under AutoMathText Data Agreement for Model Training (See LICENSE). 

Citation

@misc{automathtext_v2_2025,
  title={AutoMathText-V2: A 2.46 Trillion Token AI-Curated STEM Pretraining Dataset},
  author={Li, Chao and Zhang, Yifan and Yao, Andrew C},
  year={2025},
  publisher={Hugging Face},
  url={https://huggingface.co/datasets/OpenSQZ/AutoMathText-V2},
  note={A 2.46T token multi-domain dataset with fine-grained deduplication and AI-powered quality assessment.}
}

@article{zhang2025autonomous,
  title={Autonomous Data Selection with Zero-shot Generative Classifiers for Mathematical Texts},
  author={Zhang, Yifan and Luo, Yifan and Yuan, Yang and Yao, Andrew C},
  journal={Findings of the Association for Computational Linguistics: ACL 2025},
  year={2025}
}