license: mit
task_categories:

• fill-mask
  tags:
• pretraining
• encoder
• multilingual

mmBERT Mid-training Data

Phase 2 of 3: High-quality mid-training data mixture (600B tokens) with context extension to 8192 tokens.

This dataset contains the mid-training phase data used to train all mmBERT encoder models. This phase focuses on higher quality data sources and extends the context length from 1024 to 8192 tokens. The data is provided in MDS format ready for use with Composer and the ModernBERT training repository.

📊 Data Composition

🌍 Language Coverage

This phase covers 110 languages plus code, with inverse temperature sampling at τ=0.5. Expands from the initial 60 languages to include:

• Additional mid-resource languages: Uzbek, Bosnian, Catalan, Albanian, and 46 others
• Enhanced quality: Uses filtered FineWeb2-HQ and higher quality DCLM
• Longer contexts: Optimized for 8192 token sequences

⚙️ Key Features

• Context Extension: RoPE base frequency adjusted to 160k for 8192 token support
• Quality Upgrade: Switches to filtered, higher-quality versions of datasets
• Reduced Masking: Mask rate lowered to 15% (from 30% in pre-training)
• Language Expansion: Adds 50 new languages while maintaining data quality

🚀 Usage

For mid-training, see the ModernBERT repo: https://github.com/AnswerDotAI/ModernBERT

Direct Access

Use the script at this link to load any section of the dataset on the fly. This will fail if you try to access too many samples though, due to HF rate-limiting. To download the full dataset, use HF Hub's Snapshot Download.

🔗 Related Resources

• Models: mmBERT Model Suite
• Phase 1: Pre-training Data (2.3T tokens)
• Phase 3: Decay Phase Data (100B tokens)
• Checkpoints: Training Checkpoints
• Paper: Arxiv link
• Code: GitHub Repository

Citation

@misc{marone2025mmbertmodernmultilingualencoder,
      title={mmBERT: A Modern Multilingual Encoder with Annealed Language Learning}, 
      author={Marc Marone and Orion Weller and William Fleshman and Eugene Yang and Dawn Lawrie and Benjamin Van Durme},
      year={2025},
      eprint={2509.06888},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2509.06888}, 
}