Repetition over Diversity: High-Signal Data Filtering for Sample-Efficient German Language Modeling

Repetition over Diversity: High-Signal Data Filtering for Sample-Efficient German Language Modeling

   Title: 
   

  Content selection saved. Describe the issue below:

   Description: 
    
 

License: CC BY 4.0

arXiv:2604.28075v2 [cs.CL] 01 May 2026

Repetition over Diversity: High-Signal Data Filtering for Sample-Efficient German Language Modeling

Ansar Aynetdinov &Patrick Haller

Humboldt-Universität zu Berlin

{aynetdia , patrick.haller.1 , alan.akbik}@hu-berlin.de &Alan Akbik

Abstract

Recent research has shown that filtering massive English web corpora into high-quality subsets significantly improves training efficiency. However, for high-resource non-English languages like German, French, or Japanese, aggressive filtering creates a strategic dilemma: should practitioners prioritize diversity by training once on large amounts of lightly filtered web data, or prioritize quality by strictly filtering for a high-quality core and repeating it over multiple epochs? We investigate this trade-off for German by constructing hierarchical quality filters applied to 500M web documents, comparing multi-epoch training on the filtered subsets against single-pass training on a diverse corpus. Our experiments across multiple model scales and token budgets show that repeating high-quality data consistently outperforms single-pass training on larger, less filtered sets. Notably, the performance gap persists even after 7 epochs. Our findings suggest that for non-English LLMs, semantic concentration through quality filtering offers a more viable path to efficient language modeling than simply maximizing unique data volume. We release our German language models (called Boldt ), as well as our cleaned evaluation benchmarks to the research community. Our experiments indicate that they achieve state-of-the-art results despite training on 10-360x fewer tokens than comparable models.

Repetition over Diversity: High-Signal Data Filtering for Sample-Efficient German Language Modeling

Ansar Aynetdinov

Patrick Haller

Humboldt-Universität zu Berlin

{aynetdia , patrick.haller.1 , alan.akbik}@hu-berlin.de

Alan Akbik

1

Introduction

The prevailing scaling laws for large language models (LLMs) emphasize a simple triad: more parameters, more compute, and more data Kaplan et al. ( 2020 ); Hoffmann et al. ( 2022 ) . However, as the field moves towards more sample-efficient training, the "more is better" paradigm has been increasingly challenged by "quality-first" approaches. Previous works demonstrated that training LLMs on web corpora filtered for high-quality content can yield significantly stronger performance than training on massive, unfiltered datasets Du et al. ( 2022 ); PaLM Team ( 2022 ); Gunasekar et al. ( 2023 ); Lozhkov et al. ( 2024 ) .

While this shift is well-documented for English, it presents a unique challenge for non-English high-resource languages such as German, French, Japanese, and Chinese. These languages possess substantial web corpora containing hundreds of billions of tokens, but they lack the multi-trillion-token abundance of English. In these data-constrained scenarios, strict filtering creates a strategic dilemma: should practitioners prioritize diversity by applying only light filters to maintain large token pools (e.g. 100B tokens), or should semantic density , i.e. expected training signal per token, be prioritized by applying strict quality filters to a smaller subset (e.g. 25B tokens) and repeating it over multiple epochs?

This Study. We investigate this trade-off using German as a representative case study for non-English high-resource languages. We filter raw web corpora through three hierarchical qualitative tiers: Coherence , which removes structural noise to ensure syntactic flow; Information Value , which retains only content-rich and fact-bearing documents; and Educational Quality , which selects for pedagogical clarity and explanatory depth. We further derive a Dense Core subset – the intersection of these criteria – to represent the upper bound of semantic density available in German web data.

By training language models from scratch on these filtered subsets, we isolate the impact of each semantic property on downstream performance and test whether the cautious approach to multi-epoch training prevalent in the literature Muennighoff et al. ( 2023 ); Faysse et al. ( 2025 ); Luukkonen et al. ( 2023 ) remains warranted when data selection is optimized for knowledge density. It also enables us to explore training curricula that transition from diverse data pools to high-density subsets, testing whether data quality staging provides benefits over static data mixtures.

Contributions. This work makes the following contributions:

•
We demonstrate that repeated training on high-quality German data outperforms maximizing unique web document coverage under a fixed pre-training budget. We specifically analyze the distinct impacts of our three semantic tiers: Coherence , Information Value , and Educational Quality .

•
We show that multi-epoch pre-training on filtered data positively impacts instruction-tuning behavior, leading to higher correctness in assistant tasks.

•
We identify and correct significant noise in existing German translation of ARC-Challenge, Hellaswag, Lambada, and OpenBookQA benchmarks. We release these corrected and filtered benchmarks to ensure more reliable assessment of LLM performance in the German NLP community.

•
We release a series of German Small Language Models (SLMs) up to 1B parameters. Despite being trained on an order of magnitude fewer tokens than competitive baselines, our models achieve state-of-the-art results for their size, underscoring the importance of high-quality data selection in resource-constrained settings.

2

Related Work

Data Filtering for LLMs. Since GPT-2 Radford et al. ( 2019 ) and T5 Raffel et al. ( 2023 ) , filtered CommonCrawl Common Crawl Foundation ( 2024 ) has been central to LLM pre-training. Datasets like C4 Raffel et al. ( 2020 ) , The Pile Gao et al. ( 2020 ) , OSCAR Ortiz Suárez et al. ( 2020 ) , and RefinedWeb Penedo et al. ( 2023 ) employed deduplication, language filtering, and heuristics to remove noisy data. GPT-3 GPT-3 Team ( 2020 ) and Gopher Team ( 2022 ) further demonstrated that classifier-based quality filtering significantly improves outcomes.

Recent work has further refined model-based filtering: the Phi models Gunasekar et al. ( 2023 ); Li et al. ( 2023 ) showed that "textbook-quality" data dramatically improves sample efficiency, while Penedo et al. ( 2024 ) further scaled scaled this with the creation of FineWeb and its more restrictive subset, FineWeb-Edu Lozhkov et al. ( 2024 ) , which uses a classifier to identify documents with high educational value. Unfortunately, these techniques remain less explored for non-English languages where strict filtering may critically reduce the available token budget.

Subset
N Docs
Yield
Token Count
Doc Length
Tokenizer Fertility

(Millions) (%) (Tokens) ( μ ± σ \mu\pm\sigma ) Train Test Benchmarks

FW2-DE (Full Pool) 496.0 100.0

-

-

Hierarchical Tiers

Random (Baseline) 128.9* 26.0* 100B* 786 ± \pm 1725 1.49 1.48 1.57

Coherence
300.6 (138.3*) 60.6 (27.8*) 100B* 730 ± \pm 1540 1.48 1.50 1.56

Information Value
43.5 8.8 65B 1494 ± \pm 2561 1.36 1.42 1.42

Educational Quality
30.2 6.1 33B 1087 ± \pm 2103 1.33 1.40
1.38

Target Core

Dense Core (Intersection) 24.5
5.1
28B
1150 ± \pm 2193
1.32
1.40
1.38

External Baselines

FW HQ Messmer et al. ( 2025 )

43.2 8.7 35B 823 ± \pm 1895 1.33 1.40
1.38

AA High Burns et al. ( 2025 )

70.4 14.2 21B 296 ± \pm 368 1.35 1.40
1.43

Table 1: Dataset statistics for the German split of FineWeb-2 (FW2-DE) and derived subsets. Yield represents the percentage of documents retained from the original pool. Doc Length is expressed in tokens. Coherence reports document statistics for both the full subset and the 100B sample (denoted with *) used in further experiments.

Overcoming the data bottleneck. As high-quality data demand outpaced the growth of available corpora, practitioners incorporated code, social media, books, and scientific papers into pre-training mixtures Gao et al. ( 2020 ); Soldaini et al. ( 2024 ) , and explored synthetic data generation for pre-training data enrichment Gunasekar et al. ( 2023 ); Li et al. ( 2023 ); Kang et al. ( 2025 ) , despite the dangers of model collapse Shumailov et al. ( 2024 ) .

Muennighoff et al. ( 2023 ) investigated scaling laws in data-constrained regimes, concluding that the returns from repeated exposure to the same dataset diminish after 4 epochs. Fang et al. ( 2026 ) extend this by showing that repeatedly training on an aggressively filtered dataset for up to 10 epochs can outperform a single pass over a 10x larger unfiltered superset. Our work builds upon these findings by investigating whether strict quality filtering presents a practical obstacle in non-English settings where the unfiltered data pool is substantially smaller, using German as a case study.

Non-English Language Modeling. While English dominates the LLM landscape, several efforts have targeted monolingual or highly specialized multilingual models for high-resource languages like Japanese LLM-jp Team ( 2024 ) and German Pfister et al. ( 2025 ) . A lot of the progress in non-English language modeling, however, comes from either continued-pre-training of English-centric models Zheng et al. ( 2024 ); Hoffmann et al. ( 2025 ); Kuulmets et al. ( 2024 ) or pre-training on massive multilingual dataset mixtures BigScience Workshop and Bloom Team ( 2023 ); Gemma 3 Team ( 2025 ); Llama 3 Team ( 2024 ); Qwen3 Team ( 2025 ) .

In the context of German, two recent projects have proposed specialized filtering strategies for the German subset of the multilingual FineWeb-2 dataset Penedo et al. ( 2025 ) . Burns et al. ( 2025 ) developed Aleph-Alpha-GermanWeb (AA-High), using a multi-dimensional classifier to bucket documents by grammar and style. Separately, Messmer et al. ( 2025 ) introduced FW2-MKC, which uses an embedding-based approach to select web documents that align with established knowledge benchmarks and instruction-tuning sets. In our experiments, we use these two datasets as external benchmarks to compare against our own hierarchical qualitative filters.

3

Data and Filtering

Our data pipeline is built upon the German split of the FineWeb-2 (FW2-DE) dataset Penedo et al. ( 2025 ) . While FW2-DE provides a foundation of deduplicated web content, its quality-based filtering is rather basic. To enable our study of semantic density, we define a hierarchical filtering framework that moves from structural surface features to deep pedagogical value.

3.1

Hierarchical Qualitative Filters

We implement three document-level classifiers to score the FW2-DE pool. These filters are not mutually exclusive but represent increasing tiers of selection strictness. Details pertaining to the classifier training pipeline are provided in Appendix  A .

Coherence . This filter targets basic linguistic and structural integrity. It is designed to remove "word-salad" documents, truncated HTML exports, and fragmentary snippets. A high coherence score indicates a document with natural syntactic flow, regardless of whether the content is informative.

Information Value . This tier selects for "signal density." It identifies documents that are fact-bearing and content-rich, like e.g. technical reports, news articles, or specialized documentation, while filtering out generic web prose, SEO-heavy landing pages, and repetitive boilerplate.

Educational Quality . This is our most restrictive tier, modeled after the criteria used in FineWeb-Edu Penedo et al. ( 2024 ) . It prioritizes textbook-like clarity and pedagogical value, selecting for documents that explain concepts or provide structured knowledge suitable for a curriculum.

Dense Core . By taking the intersection of these three filters, we derive Dense Core . This subset contains documents that are simultaneously coherent, information-dense, and educational. As shown in Table  1 , this subset represents a significantly "denser" version of the German web, which we use as our primary high-quality core for multi-epoch training.

3.2

Dataset Statistics and Tokenization

To ensure fair comparison across data regimes, we train dedicated BPE tokenizers (vocabulary size 32,000) for each subset, optimizing for their specific lexical distributions.

Analysis. Table  1 shows that while the Coherence filter is relatively permissive and retains the majority of the documents from the original FW2-DE pool, the Educational Quality filter is substantially more selective. As selection strictness increases, average document length rises markedly – the Dense-Core exhibits nearly 50% longer documents than Random , reflecting substantive long-form content rather than fragmented web snippets.

As for the tokenizer efficiency, we see a clear trend indicating that discarding lower-quality data from the tokenizer training data improves the resulting tokenizer fertility both on the respective train splits, as well as the shared holdout FW2-DE test set. We also report average tokenizer fertility on question prompts from the benchmarks in the evaluation suite used in our experiments, and observe the same trend there.

 EN (Original) : […] Both Olaf and the boy heard it and walked towards  Bob  

DE (Old) : […] Sowohl Olaf als auch der Junge hörten es und gingen auf Bob zu
DE (Fixed) : […] Sowohl Olaf als auch der Junge hörten es und gingen auf Bob zu

Figure 1: Example of a problematic instance from the German translation of the LAMBADA benchmark in EleutherAI’s Evaluation Harness Gao et al. ( 2024 ) . We highlight correct and incorrect target labels.

4

Evaluation Datasets

A significant challenge in assessing LLMs in non-English languages is the lack of broad-range evaluation benchmarks. While many seminal benchmarks were devised in English by human experts, their non-English counterparts are often simply machine-translated without any attention to language-specific rules, often produced by older, less capable models Lai et al. ( 2023 ); Bellagente et al. ( 2024 ) . As a result, these versions often contain non-idiomatic phrasing, grammatical errors, and, more critically, structural artifacts that fundamentally alter the difficulty of the original task. As a part of this work, we contribute a modernized and cleaned suite of German benchmarks designed to provide a more reliable signal for model performance.

4.1

Addressing Translation Artifacts

In our analysis of existing German benchmark variants, we identified the main issue with completion-based tasks, often used for evaluation of pre-trained LLMs: discrepancies in the word order between English and German.

In completion tasks like HellaSwag and LAMBADA, German’s verb-final structures and flexible word order frequently move the original "completion target" away from sentence-final position, disrupting the intended prediction task. Refer to Figure  1 for an example of this issue: instead of tracking the entity "Bob" across sentences, the task becomes predicting a verb ending two words away.

To address this issue, we re-translated the suite using the state-of-the-art machine translation Tower+ 72B model Rei et al. ( 2025 ) . Instead of translating benchmarks instance components separately, we provide full instances as inputs to the model. We discarded the rare instances (

Benchmark
Original
Removed
Modernized

Global MMLU 14,042 0 0

ARC-Challenge 1,168 0 1,168

ARC-Easy 2,376 6 2,370

OpenBookQA 500 5 495

HellaSwag 10,042 47 9,995

LAMBADA 5,153 3 5,150

Table 2: Statistics for our German evaluation suite. All benchmarks except Global MMLU were re-translated from their original English version. Removed items indicate instances where translation itself failed or broke the task integrity.

Subset Tokens MMLU ARC-C ARC-E H-Swag LAMBADA OBQA Avg.

Baseline

Random
100B (1.0x) 27.13 26.15 41.10 37.18 33.52 41.01 34.35

Single Filters

Coherence
100B (1.0x) 27.06 27.65 42.74 40.45 38.33 42.63 36.48

Information Value
65B (1.5x) 28.29 30.46 46.20 40.71 38.52 44.04 38.04

Educational Quality
33B (3.0x) 28.64 31.49
50.91
40.57 36.49 43.64 38.62

Filter Combinations

Dense-Core
28B (3.6x) 28.97
31.40 50.55 41.10
37.55 45.86
39.24

MKC Messmer et al. ( 2025 )

35B (2.9x) 28.00 27.37 46.37 39.49 40.43
42.02 37.28

AA High Burns et al. ( 2025 )

21B (4.8x) 26.55 25.31 39.83 37.39 29.75 40.00 33.14

Curriculum-Based (100B Budget)

Sorted
100B (1.0x) 28.00 30.18 46.25 40.25 35.98 42.63 37.22

Phased
78B (1.3x) 28.52 29.34 47.89 40.35 36.06 43.67 37.64

Reference Models (Total Tokens Trained)

LLäMmlein-120M Pfister et al. ( 2025 )

1T 26.41 24.74 37.13 23.02 26.97 43.03 33.33

Gemma-3-270M Gemma 3 Team ( 2025 )

6T ∗

26.09 24.93 34.68 31.60 29.85 37.37 32.07

Qwen-3-0.6B-Base Qwen3 Team ( 2025 )

36T ∗

29.87 32.90 41.90 38.13 39.57 41.01 37.23

Table 3: Benchmark results for 350M models. Tokens refers to unique tokens in the subset, while the bracketed value indicates the number of epochs to reach the 100B budget. Token counts of multilingual data mixtures are denoted with ∗ .

4.2

Final Evaluation Suite

The resulting evaluation suite provides a holistic perspective on model performance across factual knowledge, commonsense reasoning, and linguistic context tracking. Factual knowledge is assessed via the German subset of Global MMLU Singh et al. ( 2025 ) , which spans 57 subjects across STEM, the social sciences, and the humanities. Reasoning capabilities are measured using ARC-Easy, ARC-Challenge Clark et al. ( 2018 ) , and OpenBookQA Mihaylov et al. ( 2018 ) , which require models to perform grade-school level science inference. Finally, we evaluate commonsense narrative continuation and discourse-level context tracking through HellaSwag Zellers et al. ( 2019 ) and the OpenAI version of LAMBADA Paperno et al. ( 2016 ) . For all evaluations in this paper, we rely on length-normalized conditional log-likelihoods of continuation options to choose the correct answer in multiple choice benchmarks.

Table  2 summarizes the composition of the modernized suite and the impact of our manual cleaning process. Notably, the exclusions required to maintain task integrity were minimal, with the largest adjustment occurring in HellaSwag due to its larger size in general. We release these cleaned benchmarks to the community to facilitate more rigorous and standardized assessment in German NLP.

5

Investigating the Quality-Quantity Trade-off

In this section, we present a series of experiments designed to isolate the impact of data density on model performance. We move away from the traditional "single-pass" pre-training paradigm to investigate whether high-quality repetition can compensate for a lack of unique token diversity. Our investigation is divided into four parts: a study of token allocation strategies under a fixed 100B token budget, a scaling analysis in the 1B parameter regime, exploration of the repetition ceiling, and an evaluation of the downstream impact on instruction tuning.

5.1

Experiment I: Token Allocation Strategies (100B Budget)

The primary objective of this experiment is to determine the optimal way to distribute a fixed training budget of 100B tokens when faced with a choice between broad, unique web data and narrow, high-density educational content.

5.1.1

Experimental Setup

We utilize a decoder-only transformer architecture following the Llama model family Llama 2 Team ( 2023 ) , with a primary model size of 350M non-embedding parameters. To ensure a fair comparison, all models in this study are restricted to a total exposure of 100B tokens.

Details on hyperparameter and optimizer choice are provided in Appendix  B .

To test the interaction between diversity and density, we define three training strategies:

Uniform Baselines: Single-pass training on unique documents from the Random or Coherence pools. For both baselines we randomly sample documents until we reach the 100B limit from the full FW2-DE and Coherence pools of documents respectively.

Dense Repetition: Multi-epoch training on the remaining, higher-quality subsets. For instance, for the 28B-token Dense Core core, the 100B budget results in approximately 3.6 training epochs. We also consider the quality-filtered MKC and AA High subsets of FW2-DE devised independently from this work by Messmer et al. ( 2025 ) and Burns et al. ( 2025 ) respectively.

Hybrid Curricula: "Coarse-to-fine" schedules designed to test whether finishing pre-training on higher-quality data can leverage both diversity and quality. The motivation for these curricula stems from the intuition that models might benefit from broad initial exposure to diverse linguistic patterns available in lower-quality data before refining on concentrated knowledge. We implement two variants: a Phased 50/50 split (50B tokens of Random followed by 50B tokens of Dense Core ) and a Sorted schedule, where 100B Coherence tokens are presented in ascending order of their Educational score.

5.1.2

Results and Analysis

Table  3 demonstrates significant gains from prioritizing semantic density over document diversity. Dense-Core , outperforms the Random baseline by 4.89 points on average while training for approximately 3.6 epochs over 28B unique tokens, suggesting that for information-dense German text, high signal-to-noise ratio of the curated content outweigh potential multi-epoch repetition risks.

Training dynamics and checkpoint analysis. Figure  2 shows that Dense-Core maintains a consistent advantage throughout the entire 100B token trajectory rather than only at convergence. This indicates that the benefit of high-quality repetition is not confined to the final stages of training, but rather provides a steeper learning curve from the onset. The poor performance of the AA High subset (repeated 4.8 × \times ) indicates that multi-epoch training only benefits from sufficiently high-quality data. We attribute its poor downstream impact to usage of smaller and by proxy weaker models as annotators and classifiers for its filtering.

The Phased and Sorted curricula on the other hand show a marked inflection point: their performance improves markedly in the second half of training, coinciding with the transition to higher-quality data. While these curricula eventually reach high performance, they consistently trail the pure Dense-Core trajectory. This suggests that the inclusion of low-signal web tokens, even as an initial "warm-up" phase, may dilute the overall information density of a fixed 100B token budget.

Comparison to state-of-the-art models. We note that our 350M Dense-Core model outperforms LLäMmlein-120M Pfister et al. ( 2025 ) , Gemma-3-270M Gemma 3 Team ( 2025 ) , and Qwen-3-0.6B-Base Qwen3 Team ( 2025 ) on our evaluation suite, despite these models being trained on 10 × 10\times , 60 × 60\times , and 360 × 360\times more tokens, respectively. This suggests that targeted high-signal dataset cores enable performance typically requiring far larger compute and data budgets.

Figure 2: Zero-shot evaluation of 350M models trained on 100B tokens of different FW2-DE subsets over pre-training checkpoints.

5.2

Experiment II: Parameter Scaling and Efficiency

To verify that the performance gains observed in the 350M regime generalize as model capacity increases, we scale our primary experiments to 1B parameter models (see Appendix B for details).

5.2.1

Experimental Setup

In this experiment, we focus on the two most distinct strategies: the Random baseline (representing high-coverage, single-pass training) and the Dense-Core (representing high-density repetition). Both models are trained on the same 100B token budget, using the same hyperparemeters as in Experiment I.

5.2.2

Results and Analysis

Figure  3 shows that performance gap between the Random and Dense-Core not only persists but widens as parameter count increases. While the 350M Dense-Core outperformed its baseline by 4.89 points on average, the 1B Dense-Core achieves a 5.14-point lead over the 1B Random baseline.

Furthermore, despite being trained on an order of magnitude less tokens in total, the 1B Dense-Core model achieves parity with or exceeds the performance of multilingual Gemma-3-1B and Llama-3.2-1B , and outperforms monolingual LLäMmlein-1B .

These findings suggest that as model capacity increases, the transformer architecture becomes more adept at incorporating information from high-density sources. In this regime, a 100B-token budget is not a limiting factor for reaching state-of-the-art performance for 1B models, provided the training signal is sufficiently concentrated. This indicates that for non-English high-resource languages like German, the path to high-performance small models lies in maximizing token utility rather than volume or cross-lingual transfer.

Figure 3: Performance gain when scaling the model size from 350M to 1B parameters. Table 8 breaks down the performance of the models depicted in the graph on each benchmark.

5.3

Experiment III: Exploring Repetition Limits (200B Budget)

While Experiment I established that 3.6 epochs of repetition on high-density data is superior to a single pass on diverse data, it remains unclear where the point of diminishing returns lies for various data regimes. In this experiment, we explore the limits of multi-epoch training on high-density subsets by using an extended 200B token budget, examining at what point the benefits of data density are offset by the lack of novelty, and comparing this trajectory to the scaling behavior of lower-density sets over the same budget.

5.3.1

Experimental Setup

We extend the training of three key configurations from Experiment I to a total of 200B tokens: Random , Phased , and Dense-Core . For the Random baseline this represents a transition from a 100B single-pass to a 200B single-pass (adding new documents into the mix). For the Dense-Core , the 200B budget results in approximately 7.2 training epochs over the 28B-token subset. The Phased curriculum consists of the 100B Random subset folllowed by 100B tokens of continued exposure to the Dense-Core data. We maintain the same hyperparameters as in Experiment 1 to ensure comparable learning dynamics across all runs.

Figure 4: Performance of 350M models trained on 100B and 200B tokens of Random and Edu High subsets. Table 7 breaks down the performance of each model’s final checkpoint on each benchmark.

5.3.2

Results and Analysis

Figure  4 illustrates that the benefits of multi-epoch training on Dense-Core subset persist well beyond the four-epoch threshold identified by Muennighoff et al. ( 2023 ) – extending the findings of Fang et al. ( 2026 ) to a non-English setting where the unfiltered data pool is severely more constrained.

We do not observe a loss in generalization on our benchmark suite even when doubling the number of epochs on the same dataset. Instead, the Dense-Core model maintains a significant lead over the Random model, even though the latter is still seeing entirely new unique data. While the Phased curriculum continues to scale and narrow the gap to the pure Dense-Core model effectively, it never quite overtakes it, suggesting that training on a more diverse sample in the first stage holds back optimization on a more dense subset.

To investigate whether these trends hold at larger model scales, we conduct an additional pre-training run of a 1B model on 200B tokens of Dense-Core . Table  8 shows that this model achieves an uplift of 2.08 points on average on our evaluation suite, compared to pre-training on 100B tokens. Notably, this gain is more than twice the corresponding improvement observed for the 350M model. These results indicate that the effectiveness of repeated exposure to high-quality data depends on model size and may extend to longer training horizons as parameter count increases if the underlying corpus is sufficiently information-dense. In this regime, larger models can continue to extract additional value from repeated passes over curated data, before the requirement for larger pools of unique tokens ultimately becomes the limiting factor.

5.4

Experiment IV: Generalization to Instruction Tuning

Finally, we evaluate whether the advantages of a high-density pre-training "core" translate into improved instruction-following capabilities.

5.4.1

Experimental Setup

We fine-tune the final checkpoints of our models on the German subset of the SmolTalk2 instruction-tuning dataset Bakouch et al. ( 2025 ) . Both the 350M and 1B parameter variants are tuned using identical hyperparameters (see Appendix B ). To evaluate the models, we utilize an LLM-as-a-judge protocol (see Figures 7 and 8 for the prompt templates) using Llama-3.3-70B-Instruct Llama 3 Team ( 2024 ) on 1,000 held-out prompts.

Table  4 reports the average Likert-scale score (1-10) and the total number of correct answers (binary accuracy) across the test set.

5.4.2

Results and Analysis

Table  4 confirms that the "density advantage" is preserved through the SFT process. For the 350M models at the 100B token budget, the Dense-Core configuration achieves the highest count of correct responses (253/1,000), closely followed by Information Value (249/1,000), significantly exceeding Random baseline (178/1,000).

Interestingly, the MKC subset, which was filtered for documents similar to instruction-tuning datasets like Aya Singh et al. ( 2024 ) , fails to outperform our manually defined Dense-Core . This suggests that the fundamental reasoning capabilities and factual grounding provided by high-quality, information-rich data are more important for SFT than simply matching the formatting or distribution of typical instruction sets. The poor performance of the AA High subset (199 correct answers) further reinforces that repetition only benefits sufficiently high-quality data.

Scaling in Parameters and Tokens. The benefits of density scale effectively across both parameter counts and token budgets. Our 1B Dense-Core model achieves a score of 6.13 score with 338 correct answers. Most strikingly, the 350M Dense-Core model trained for 200B tokens (7.2 epochs) produces 278 correct answers – nearly matching the 1B Random model (293 correct) despite having 3x fewer parameters.

Subset
Tokens
Score
Correct

350M params @ 100B tokens

Random
1.0x 5.25 178

Coherence
1.0x 5.62 226

Info. Value
1.5x 5.69 249

Edu. Quality
3.2x 5.75
241

Dense-Core
3.6x 5.74 253

MKC
1.0x 5.63 219

AA High
4.8x 5.41 199

Sorted
1.0x 5.61 225

Phased
1.3x 5.66 231

1B params @ 100B tokens

Random 1B
1.0x 5.87 293

Dense-Core 1B
3.6x 6.13
338

350M params @ 200B tokens

Random
1.0x 5.45 198

Dense-Core
7.2x 5.96
278

Phased
2.6x 5.78 251

Table 4: LLM-as-a-Judge (Llama-3.3-70B) results. Tokens indicates the repetition factor over the unique subset. Score is the 1–10 Likert average; Correct is the binary accuracy count out of 1,000 prompts.

Model Tokens MMLU ARC-C ARC-E H-Swag LAMBADA OBQA Avg.

Ours

Boldt-DC-350M
200B 29.29 32.24 52.87 43.21 37.48 45.86 40.16

Boldt-DC-1B
200B 31.06 35.99
57.30
48.69
42.80 48.48
44.05

Boldt-1B
230B 31.42
34.11
55.78
48.77
44.70
52.32
44.52

Reference models - 1B

LLäMmlein-1B Pfister et al. ( 2025 )

1T 29.26 30.27 48.19 44.80 44.89
47.27 40.78

Gemma-3-1B Gemma 3 Team ( 2025 )

2T ∗

30.01 30.55 47.89 43.43 41.71
45.05 39.77

Llama-3.2-1B Llama 3 Team ( 2024 )

9T ∗

28.58 29.90 40.51 40.07 44.31 44.04 37.90

Reference models - >1B

EuroLLM-1.7B Martins et al. ( 2024 )

4T ∗

31.04 31.58 54.68 45.30 44.52 50.50 42.94

Qwen3-1.7B-Base Qwen3 Team ( 2025 )

36T ∗

34.17 37.49 57.00 45.20 49.81 45.66 44.89

BübleLM-2B Delobelle et al. ( 2024 )

2T ∗

29.68 32.62 53.63 46.57 43.55 49.70 42.63

Gemma-2-2B Gemma 2 Team ( 2024 )

2T ∗

33.99 37.11 57.47 49.62 52.64 48.89 46.62

Table 5: Benchmark results of Boldt models compared to other pre-trained models of the similar size, as well as larger reference models.

6

Model and Benchmark Release

We release our trained models, called Boldt , as well as the updated benchmarks from our evaluation suite to the research community 1 1 1 https://huggingface.co/Boldt . The initial release consists of three models:

Boldt-DC-350M & Boldt-DC-1B . 350M and 1B parameter models trained as part of our repetition limits experiments (see Section  5.3 ). In total, they were trained on a 200B token budget using the Dense Core subset of FineWeb-2. These models are released for the purpose of reproducibility of this paper’s results.

Boldt-1B . A 1B parameter model trained on a combination of Dense Core and a corpus of 6B tokens of news data in German obtained using the Fundus library  Dallabetta et al. ( 2024 ) . Our Fundus crawlers were used continuously since 2022, so the main body of news articles stems from the time period between 2022 and February 2026 (with smaller numbers of articles going back to 1994). Boldt-1B was trained for multiple epochs on the combined corpus until reaching the effective training token count of 230B. The context window size is also extended from 2048 to 4096 compared to Boldt-DC-1B .

We compare our models against other similarly-sized pre-trained models on our evaluation suite. The results are shown in Table  5 . We find that Boldt-1B improves upon Boldt-DC-1B in all benchmarks except for ARC-Challenge and ARC-Easy. Despite being trained on a substantially smaller training corpus compared to relevant similarly-sized LLMs capable of German, our 1B models are competitive even with larger-sized (around 2B) multilingual models.

In addition to our base models, we also release a preview of an instruction-tuned version of Boldt-1B . Our instruction tuning dataset consists of a combination of real and synthetic instruction-output pairs from diverse sources. Please refer to the HF model card for further details.

7

Conclusion

This work addresses a practical question for non-English LLM development: is aggressive quality filtering worthwhile when total available text is limited, or does it discard too much useful diversity for pre-training?

Our experiments suggest that quality filtering remains beneficial despite the smaller pool of available web data. Across the model sizes and token budgets we considered, models trained for multiple epochs on small, high-quality subsets consistently outperform those trained on larger, less strongly filtered mixtures. Within our setups, we do not observe an early saturation point where repeating high-quality data ceases to help. Additional passes over the filtered corpus continue to yield improvements, whereas adding more unfiltered data brings only limited gains. This indicates that careful filtering and multi-epoch training are a viable and effective strategy rather than a risky trade-off for high-resource non-English languages like German.

Our results also show continuity between quality-first pre-training and instruction-tuning. Instruction-tuned assistants that were pre-trained on curated high-quality subsets outperform those pre-trained on more diverse, less filtered data in both correctness and helpfulness. Introducing lower-quality documents into pre-training mixtures decreased instruction-tuned model output quality.

In practice, our findings suggest that extensive quality filtering relying on clearly-defined rules and strong annotator models provides a viable path towards sample-efficient pre-training in non-English high-resource languages like German.

Limitations

Language scope. Our investigation focuses exclusively on German as a representative high-resource non-English language. While German’s web corpus characteristics (hundreds of billions of tokens but not trillions) likely generalize to languages like French, Japanese, and Chinese, languages with smaller corpora or different linguistic structures may exhibit different quality-quantity trade-offs. Future work should validate these findings across diverse language families.

Model and compute scale. Our experiments are limited to models up to 1B parameters trained on at most 200B tokens. While our findings demonstrate clear efficiency gains at this scale, it remains unclear whether these quality-quantity trade-offs hold at substantially larger, industry-level, scales where both compute budgets and available high-quality data increase dramatically.

Architecture selection. We focus exclusively on dense transformer architectures. Mixture-of-experts models might exhibit different behaviors with respect to data quality and repetition. Similarly, we do not explore architectural innovations like alternative attention mechanisms that might exhibit different behavior during pre-training.

Toxicity and bias assessment. We do not evaluate our models for toxic content generation, demographic biases, or harmful stereotypes. While our quality filters prioritize educational and informative content, which may reduce certain risks compared to unfiltered web data, we cannot guarantee that aggressive filtering eliminates problematic content or prevents biased model behavior.

High-quality educational content can still encode societal biases, and repeated exposure during multi-epoch training might amplify rather than mitigate such issues. Future work should conduct comprehensive bias audits across demographic dimensions and toxicity benchmarks to understand how quality filtering strategies interact with model safety. Additionally, our LLM-as-judge evaluation for instruction-tuned models does not assess safety, appropriateness, or potential for harmful outputs.

Despite these limitations, our work provides valuable evidence that quality filtering and multi-epoch training offer a practical path toward efficient pre-training for high-resource non-English languages, challenging the notion that maximizing unique token exposure should be the primary objective in data-constrained settings.

Acknowledgments

The authors are supported by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Emmy Noether grant “Eidetic Representations of Natural Language” (project number 448414230). Further, Alan Akbik is supported by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy "Science of Intelligence" (EXC 2002/1, project number 390523135).

The authors gratefully acknowledge the scientific support and HPC resources provided by the Erlangen National High Performance Computing Center (NHR@FAU) of the Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU) under the NHR project c106fa. NHR funding is provided by federal and Bavarian state authorities. NHR@FAU hardware is partially funded by the German Research Foundation (DFG) – 440719683.

References

E. Bakouch, L. Ben Allal, A. Lozhkov, N. Tazi, L. Tunstall, C. M. Patiño, E. Beeching, A. Roucher, A. J. Reedi, Q. Gallouédec, K. Rasul, N. Habib, C. Fourrier, H. Kydlicek, G. Penedo, H. Larcher, M. Morlon, V. Srivastav, J. Lochner, X. Nguyen, C. Raffel, L. von Werra, and T. Wolf (2025) SmolLM3: smol, multilingual, long-context reasoner .

Note: https://huggingface.co/blog/smollm3

Cited by: §5.4.1 .

M. Bellagente, J. Tow, D. Mahan, D. Phung, M. Zhuravinskyi, R. Adithyan, J. Baicoianu, B. Brooks, N. Cooper, A. Datta, M. Lee, E. Mostaque, M. Pieler, N. Pinnaparju, P. Rocha, H. Saini, H. Teufel, N. Zanichelli, and C. Riquelme (2024) Stable lm 2 1.6b technical report .

External Links: 2402.17834 , Link

Cited by: §4 .

BigScience Workshop and Bloom Team (2023) BLOOM: a 176b-parameter open-access multilingual language model .

External Links: 2211.05100 , Link

Cited by: §2 .

T. F. Burns, L. Parcalabescu, S. Wäldchen, M. Barlow, G. Ziegltrum, V. Stampa, B. Harren, and B. Deiseroth (2025) Aleph-alpha-germanweb: improving german-language llm pre-training with model-based data curation and synthetic data generation .

External Links: 2505.00022 , Link

Cited by: Table 1 , §2 , Table 3 , item 2 .

P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord (2018) Think you have solved question answering? try arc, the ai2 reasoning challenge .

arXiv:1803.05457v1 .

Cited by: §4.2 .

Common Crawl Foundation (2024) Common crawl dataset .

Common Crawl Foundation .

Note: Web dataset

External Links: Link

Cited by: §2 .

M. Dallabetta, C. Dobberstein, A. Breiding, and A. Akbik (2024) Fundus: a simple-to-use news scraper optimized for high quality extractions .

In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) , Y. Cao, Y. Feng, and D. Xiong (Eds.) ,

Bangkok, Thailand , pp. 305–314 .

External Links: Link , Document

Cited by: §6 .

P. Delobelle, A. Akbik, et al. (2024) BübleLM: a small german lm .

Cited by: Table 5 .

N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, B. Zoph, L. Fedus, M. Bosma, Z. Zhou, T. Wang, Y. E. Wang, K. Webster, M. Pellat, K. Robinson, K. Meier-Hellstern, T. Duke, L. Dixon, K. Zhang, Q. V. Le, Y. Wu, Z. Chen, and C. Cui (2022) GLaM: efficient scaling of language models with mixture-of-experts .

External Links: 2112.06905 , Link

Cited by: §1 .

A. Fang, H. Pouransari, M. Jordan, A. T. Toshev, V. Shankar, L. Schmidt, and T. Gunter (2026) Datasets, documents, and repetitions: the practicalities of unequal data quality .

In The Thirty-ninth Annual Conference on Neural Information Processing Systems ,

External Links: Link

Cited by: §2 , §5.3.2 .

M. Faysse, P. Fernandes, N. M. Guerreiro, A. Loison, D. M. Alves, C. Corro, N. Boizard, J. Alves, R. Rei, P. H. Martins, A. B. Casademunt, F. Yvon, A. Martins, G. Viaud, C. HUDELOT, and P. Colombo (2025) CroissantLLM: a truly bilingual french-english language model .

Transactions on Machine Learning Research .

Note:

External Links: ISSN 2835-8856 , Link

Cited by: §1 .

L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy (2020) The pile: an 800gb dataset of diverse text for language modeling .

External Links: 2101.00027 , Link

Cited by: §2 , §2 .

L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac’h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou (2024) The language model evaluation harness .

Zenodo .

External Links: Document , Link

Cited by: Figure 1 .

Gemma 2 Team (2024) Gemma 2: improving open language models at a practical size .

External Links: 2408.00118 , Link

Cited by: Table 5 .

Gemma 3 Team (2025) Gemma 3 technical report .

External Links: 2503.19786 , Link

Cited by: Appendix A , §2 , Table 3 , §5.1.2 , Table 5 .

GPT-3 Team (2020) Language models are few-shot learners .

In Advances in Neural Information Processing Systems , H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.) ,

Vol. 33 , pp. 1877–1901 .

External Links: Link

Cited by: §2 .

S. Gunasekar, Y. Zhang, J. Aneja, C. C. T. Mendes, A. D. Giorno, S. Gopi, M. Javaheripi, P. Kauffmann, G. de Rosa, O. Saarikivi, A. Salim, S. Shah, H. S. Behl, X. Wang, S. Bubeck, R. Eldan, A. T. Kalai, Y. T. Lee, and Y. Li (2023) Textbooks are all you need .

External Links: 2306.11644 , Link

Cited by: §1 , §2 , §2 .

J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, O. Vinyals, J. Rae, and L. Sifre (2022) An empirical analysis of compute-optimal large language model training .

In Advances in Neural Information Processing Systems , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.) ,

Vol. 35 , pp. 30016–30030 .

External Links: Link

Cited by: §1 .

M. Hoffmann, J. John, S. Schweter, G. Ramakrishnan, H. Mak, A. Zhang, D. Gaynullin, and N. J. Hammer (2025) Llama-genba-10b: a trilingual large language model for german, english and bavarian .

External Links: 2509.05668 , Link

Cited by: §2 .

F. Kang, N. Ardalani, M. Kuchnik, Y. Emad, M. Elhoushi, S. Sengupta, S. Li, R. Raghavendra, R. Jia, and C. Wu (2025) Demystifying synthetic data in LLM pre-training: a systematic study of scaling laws, benefits, and pitfalls .

In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.) ,

Suzhou, China , pp. 10739–10758 .

External Links: Link , Document , ISBN 979-8-89176-332-6

Cited by: §2 .

J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei (2020) Scaling laws for neural language models .

External Links: 2001.08361 , Link

Cited by: §1 .

H. Kuulmets, T. Purason, A. Luhtaru, and M. Fishel (2024) Teaching llama a new language through cross-lingual knowledge transfer .

In Findings of the Association for Computational Linguistics: NAACL 2024 , K. Duh, H. Gomez, and S. Bethard (Eds.) ,

Mexico City, Mexico , pp. 3309–3325 .

External Links: Link , Document

Cited by: §2 .

V. D. Lai, C. V. Nguyen, N. T. Ngo, T. Nguyen, F. Dernoncourt, R. A. Rossi, and T. H. Nguyen (2023) Okapi: instruction-tuned large language models in multiple languages with reinforcement learning from human feedback .

External Links: 2307.16039 , Link

Cited by: §4 .

Y. Li, S. Bubeck, R. Eldan, A. D. Giorno, S. Gunasekar, and Y. T. Lee (2023) Textbooks are all you need ii: phi-1.5 technical report .

External Links: 2309.05463 , Link

Cited by: §2 , §2 .

Llama 2 Team (2023) Llama 2: open foundation and fine-tuned chat models .

External Links: 2307.09288 , Link

Cited by: §5.1.1 .

Llama 3 Team (2024) The llama 3 herd of models .

External Links: 2407.21783 , Link

Cited by: Appendix A , §2 , §5.4.1 , Table 5 .

LLM-jp Team (2024) LLM-jp: a cross-organizational project for the research and development of fully open japanese llms .

External Links: 2407.03963 , Link

Cited by: §2 .

I. Loshchilov and F. Hutter (2017) SGDR: stochastic gradient descent with warm restarts .

In International Conference on Learning Representations ,

External Links: Link

Cited by: Appendix B .

I. Loshchilov and F. Hutter (2019) Decoupled weight decay regularization .

In International Conference on Learning Representations ,

External Links: Link

Cited by: Appendix B .

A. Lozhkov, L. Ben Allal, L. von Werra, and T. Wolf (2024) FineWeb-edu: the finest collection of educational content .

Hugging Face .

External Links: Link , Document

Cited by: Appendix A , §1 , §2 .

R. Luukkonen, V. Komulainen, J. Luoma, A. Eskelinen, J. Kanerva, H. Kupari, F. Ginter, V. Laippala, N. Muennighoff, A. Piktus, T. Wang, N. Tazi, T. Scao, T. Wolf, O. Suominen, S. Sairanen, M. Merioksa, J. Heinonen, A. Vahtola, S. Antao, and S. Pyysalo (2023) FinGPT: large generative models for a small language .

In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , H. Bouamor, J. Pino, and K. Bali (Eds.) ,

Singapore , pp. 2710–2726 .

External Links: Link , Document

Cited by: §1 .

P. H. Martins, P. Fernandes, J. Alves, N. M. Guerreiro, R. Rei, D. M. Alves, J. Pombal, A. Farajian, M. Faysse, M. Klimaszewski, P. Colombo, B. Haddow, J. G. C. de Souza, A. Birch, and A. F. T. Martins (2024) EuroLLM: multilingual language models for europe .

External Links: 2409.16235 , Link

Cited by: Table 5 .

B. Messmer, V. Sabolčec, and M. Jaggi (2025) Enhancing multilingual LLM pretraining with model-based data selection .

In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track ,

External Links: Link

Cited by: Table 1 , §2 , Table 3 , item 2 .

T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal (2018) Can a suit of armor conduct electricity? a new dataset for open book question answering .

In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii (Eds.) ,

Brussels, Belgium , pp. 2381–2391 .

External Links: Link , Document

Cited by: §4.2 .

N. Muennighoff, A. Rush, B. Barak, T. Le Scao, N. Tazi, A. Piktus, S. Pyysalo, T. Wolf, and C. A. Raffel (2023) Scaling data-constrained language models .

In Advances in Neural Information Processing Systems , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.) ,

Vol. 36 , pp. 50358–50376 .

External Links: Link

Cited by: §1 , §2 , §5.3.2 .

P. J. Ortiz Suárez, L. Romary, and B. Sagot (2020) A monolingual approach to contextualized word embeddings for mid-resource languages .

In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics ,

Online , pp. 1703–1714 .

External Links: Link

Cited by: §2 .

PaLM Team (2022) PaLM: scaling language modeling with pathways .

External Links: 2204.02311 , Link

Cited by: §1 .

D. Paperno, G. Kruszewski, A. Lazaridou, Q. N. Pham, R. Bernardi, S. Pezzelle, M. Baroni, G. Boleda, and R. Fernández (2016) The lambada dataset .

Zenodo .

External Links: Document

Cited by: §4.2 .

G. Penedo, H. Kydlíček, L. B. allal, A. Lozhkov, M. Mitchell, C. Raffel, L. Von Werra, and T. Wolf (2024) The fineweb datasets: decanting the web for the finest text data at scale .

In Advances in Neural Information Processing Systems , A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.) ,

Vol. 37 , pp. 30811–30849 .

External Links: Document , Link

Cited by: §2 , §3.1 .

G. Penedo, H. Kydlíček, V. Sabolčec, B. Messmer, N. Foroutan, A. H. Kargaran, C. Raffel, M. Jaggi, L. V. Werra, and T. Wolf (2025) FineWeb2: one pipeline to scale them all — adapting pre-training data processing to every language .

In Second Conference on Language Modeling ,

External Links: Link

Cited by: §2 , §3 .

G. Penedo, Q. Malartic, D. Hesslow, R. Cojocaru, H. Alobeidli, A. Cappelli, B. Pannier, E. Almazrouei, and J. Launay (2023) The refinedweb dataset for falcon llm: outperforming curated corpora with web data only .

In Proceedings of the 37th International Conference on Neural Information Processing Systems ,

NIPS ’23 , Red Hook, NY, USA .

Cited by: §2 .

J. Pfister, J. Wunderle, and A. Hotho (2025) LLäMmlein: transparent, compact and competitive German-only language models from scratch .

In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.) ,

Vienna, Austria , pp. 2227–2246 .

External Links: Link , Document , ISBN 979-8-89176-251-0

Cited by: §2 , Table 3 , §5.1.2 , Table 5 .

Qwen3 Team (2025) Qwen3 technical report .

External Links: 2505.09388 , Link

Cited by: Appendix A , §2 , Table 3 , §5.1.2 , Table 5 .

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever (2019) Language models are unsupervised multitask learners .

External Links: Link

Cited by: §2 .

C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu (2020) Exploring the limits of transfer learning with a unified text-to-text transformer .

Journal of Machine Learning Research 21 ( 140 ), pp. 1–67 .

External Links: Link

Cited by: §2 .

C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu (2023) Exploring the limits of transfer learning with a unified text-to-text transformer .

External Links: 1910.10683 , Link

Cited by: §2 .

R. Rei, N. M. Guerreiro, J. Pombal, J. Alves, P. Teixeirinha, A. Farajian, and A. F. T. Martins (2025) Tower+: bridging generality and translation specialization in multilingual llms .

External Links: 2506.17080 , Link

Cited by: §4.1 .

I. Shumailov, Z. Shumaylov, Y. Zhao, N. Papernot, R. Anderson, and Y. Gal (2024) AI models collapse when trained on recursively generated data .

Nature 631 , pp. 755 – 759 .

External Links: Link

Cited by: §2 .

S. Singh, A. Romanou, C. Fourrier, D. I. Adelani, J. G. Ngui, D. Vila-Suero, P. Limkonchotiwat, K. Marchisio, W. Q. Leong, Y. Susanto, R. Ng, S. Longpre, S. Ruder, W. Ko, A. Bosselut, A. Oh, A. Martins, L. Choshen, D. Ippolito, E. Ferrante, M. Fadaee, B. Ermis, and S. Hooker (2025) Global MMLU: understanding and addressing cultural and linguistic biases in multilingual evaluation .

In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.) ,

Vienna, Austria , pp. 18761–18799 .

External Links: Link , Document , ISBN 979-8-89176-251-0

Cited by: §4.2 .

S. Singh, F. Vargus, D. Dsouza, B. F. Karlsson, A. Mahendiran, W. Ko, H. Shandilya, J. Patel, D. Mataciunas, L. OMahony, M. Zhang, R. Hettiarachchi, J. Wilson, M. Machado, L. S. Moura, D. Krzemiński, H. Fadaei, I. Ergün, I. Okoh, A. Alaagib, O. Mudannayake, Z. Alyafeai, V. M. Chien, S. Ruder, S. Guthikonda, E. A. Alghamdi, S. Gehrmann, N. Muennighoff, M. Bartolo, J. Kreutzer, A. Üstün, M. Fadaee, and S. Hooker (2024) Aya dataset: an open-access collection for multilingual instruction tuning .

External Links: 2402.06619 , Link

Cited by: §5.4.2 .

L. Soldaini, R. Kinney, A. Bhagia, D. Schwenk, D. Atkinson, R. Authur, B. Bogin, K. Chandu, J. Dumas, Y. Elazar, V. Hofmann, A. H. Jha, S. Kumar, L. Lucy, X. Lyu, N. Lambert, I. Magnusson, J. Morrison, N. Muennighoff, A. Naik, C. Nam, M. E. Peters, A. Ravichander, K. Richardson, Z. Shen, E. Strubell, N. Subramani, O. Tafjord, P. Walsh, L. Zettlemoyer, N. A. Smith, H. Hajishirzi, I. Beltagy, D. Groeneveld, J. Dodge, and K. Lo (2024) Dolma: an open corpus of three trillion tokens for language model pretraining research .

External Links: 2402.00159 , Link

Cited by: §2 .

G. Team (2022) Scaling language models: methods, analysis and insights from training gopher .

External Links: 2112.11446 , Link

Cited by: §2 .

P. Yu, L. Merrick, G. Nuti, and D. Campos (2024) Arctic-embed 2.0: multilingual retrieval without compromise .

External Links: 2412.04506 , Link

Cited by: Appendix A .

R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi (2019) HellaSwag: can a machine really finish your sentence? .

In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , A. Korhonen, D. Traum, and L. Màrquez (Eds.) ,

Florence, Italy , pp. 4791–4800 .

External Links: Link , Document

Cited by: §4.2 .

L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica (2023) Judging llm-as-a-judge with mt-bench and chatbot arena .

External Links: 2306.05685 , Link

Cited by: Figure 7 .

W. Zheng, W. Pan, X. Xu, L. Qin, L. Yue, and M. Zhou (2024) Breaking language barriers: cross-lingual continual pre-training at scale .

In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.) ,

Miami, Florida, USA , pp. 7725–7738 .

External Links: Link , Document

Cited by: §2 .

Appendix A

FW2-DE Annotation Details

We closely follow the annotation pipeline outlined by Lozhkov et al. ( 2024 ) in creation of the English Fineweb-Edu. We first used an LLM to score 500k randomly sampled FW2-DE documents for their Coherence, Information Value, and Educational Quality. For Educational Quality, we use the prompt defined by Lozhkov et al. ( 2024 ) , while for Coherence and Information Value we define our own prompt, provided in Figure 6 . The resulting label distributions are shown in Figure 5 . We truncate input documents to a max length of 4096 tokens.

The LLM that we use for initial annotation is Llama-3.3-70B-Instruct Llama 3 Team ( 2024 ) . We also considered using Qwen3-32B Qwen3 Team ( 2025 ) and Gemma-3-27B Gemma 3 Team ( 2025 ) due to their extensive multilingual support, however we found their performance to be on average weaker compared to a more capable 70B model, as we show in Table 6 , and the resulting distributions to be inferior to the one yielded by the 70B model.

We then fine-tuned three respective snowflake-arctic-embed-m-v2.0 Yu et al. ( 2024 ) regression models on the annotated 500k sample, each responsible for scoring one of the document’s quality aspects. A hyperparameter sweep showed the linear decay of a max learning rate of 1e-4 to 0 and a batch size of 64 to be optimal over 10 epochs. The best epoch is chosen at the end based on the holdout validation set accuracy. We truncate the web documents to a max length of 1024 tokens.

The fine-tuned classifiers were then used annotate the full 500M document German subset of FineWeb-2.

The Coherence and Information Value subsets are derived by filtering out the documents receiving a score below the respective maximum scores, i.e. 3 and 4, while the Educational Quality subset is derived by applying a threshold of 3. The intersection of these subsets, Dense-Core , is derived by applying all 3 score thresholds at the same time.

Appendix B

LLM Training Details

We parametrize our 350M models with 24 layers, each with hidden layer size of 1024 and FFN size of 4096, while our 1B models consist of 16 layers, each with hidden layer size of 2048 and FFN size of 8192. All models operate on the context window size of 2048.

In all pre-training runs we use the cosine decay learning rate schedule Loshchilov and Hutter ( 2017 ) with a warmup lasting 1% of total training steps and decay to 1% of the peak learning rate, which we empirically determined to be optimal at 5 ​ e − 4 5e-4 for all models. We use the AdamW optimizer Loshchilov and Hutter ( 2019 ) with β 1 = 0.9 \beta_{1}=0.9 , β 2 = 0.95 \beta_{2}=0.95 and decoupled L 2 L_{2} weight decay coefficient of 0.1 0.1 . The effective batch size is 0.5M tokens and the gradients are clipped to a maximal Euclidean norm of 1.0 1.0 .

For instruction tuning we use a similar set of hyperparameters with the main differences in the learning warmup lasting 5% of training steps, maximum learning rate being equal to 5 ​ e − 5 5e-5 , the batch size being 32, and AdamW β 2 = 0.99 \beta_{2}=0.99 .

We rely on Huggingface’s Nanotron, Datatrove, and TRL libraries to handle the pre-training and instruction-tuning infrastructure. All pre-trianing runs were conducted on 8xA100 Nvidia GPUs, and all instruction-tuning runs on 1xA100 Nvidia GPU. All models were trained with bfloat16 precision.

For output generation with instruction-tuned variants of our models, we use the same random seed and rely on top-p sampling with p = 0.9 p=0.9 and temperature t = 0.6 t=0.6 . For LLM-as-a-judge output generation, we force greedy decoding to ensure consistency of evaluation. All open-ended output generation was conducted using the vLLM library.

Figure 5: Distribution of Coherence, Information Value, and Educational Quality scores in FW2-DE yielded by Llama-3.3-70B-Instruct annotation. We add the Education Quality score distribution from the original English Fineweb-Edu for reference.

Model
MMLU (DE)
MGSM (DE)
IFEval
Include (DE)
MLQA (DE_EN)
Avg.

(Acc.) (Acc.) (Acc.) (Acc.) (F1)

Gemma-3-27B-IT 59.09 69.60 86.21 56.18 20.70 58.36

Qwen3-32B 73.59
48.40 88.49 60.93 18.79 58.04

Llama-3.3-70B-Instruct 73.13 74.80
93.17
64.07
50.44
71.12

Table 6: IT-LLM Annotator Evaluation. Model precision: bfloat16. Each benchmark evaluation was done with and without the provided prompt template, and the best result is reported. The Include (DE) results reflect the average over only the STEM and Social Science subsets. IFEval results reflect the "strict" version of the task.

Subset Token Count MMLU ARC-Challenge ARC-Easy Hellaswag LAMBADA OBQA Avg.

Baseline

Random
100B (1.0x) 27.13 26.15 41.10 37.18 33.52 41.01 34.35

Random (197B) 200B (1.0x) 26.82 26.24 42.11 38.39 36.16 38.79 34.75

Filter Combinations

Dense-Core
28B (3.6x) 28.97 31.40 50.55 41.10 37.55
45.86 39.24

Dense-Core 200B 28B (7.2x) 29.29
32.24
52.87
43.21
37.48 45.86 40.16

Curriculum-Based

Phased
78B (1.3x) 28.24 29.05 48.35 40.38 36.45 45.05 37.92

Phased 200B 128B (1.6x) 28.54 30.65 51.27 40.93 36.14 46.87
39.07

Table 7: Benchmark results of 350M models trained on 200B tokens of different FW2-DE subsets compared to models trained on 100B tokens of (conceptually) the same subsets. For the model trained on 200B tokens from the Random subset, we report the performance of the 197B checkpoint, as it performs better than the 200B checkpoint on average.

Subset Token Count MMLU ARC-Challenge ARC-Easy Hellaswag LAMBADA OBQA Avg.

Baseline

Random 350M 100B (1.0x) 27.13 26.15 41.10 37.18 33.52 41.01 34.35

Random 1B 100B (1.0x) 27.28 27.46 43.97 41.60 38.84 41.82 36.83

Filter Combinations

Dense-Core 350M 28B (3.6x) 28.97 31.40 50.55 41.10 37.55 45.86 39.24

Dense-Core 1B 28B (3.6x) 29.88 32.99 54.64 45.51 40.29 48.49 41.97

Dense-Core 1B 28B (7.2x)

31.06
35.99
57.30
48.69
42.80
48.48
44.05

Reference models

LLäMmlein-1B 1T 29.26 30.27 48.19 44.80 44.89 47.27 40.78

Gemma-3-1B 2T ∗

30.01 30.55 47.89 43.43 41.71 45.05 39.77

Llama-3.2-1B 9T ∗

28.58 29.90 40.51 40.07 44.31 44.04 37.90

Table 8: Benchmark results of 1B models trained on 100B and 200B tokens of the Dense-Core subset compared to other similar-sized pre-trained models.

Below is an extract from a web page. Evaluate the quality of the content based on its coherence and whether it provides valuable information on any subject matter that a reader may take away after reading it. Valuable information refers to information that is unbiased, non-promotional, and either well-reasoned or well-put. On the other hand, overly promotional materials, personal opinions or experiences without solid foundations, and spam/adult content are not considered valuable information. Assign two scores based on the scoring system described below. Coherence score (range: 1-3) Indicates whether the extract is logically structured, easy to follow, and clearly articulated.

• 1 point: the extract is mostly incoherent. It may contain rambling or non-linear trains of thought; it could include a lot of interruptions or advertisements.
• 2 points: the extract is somewhat coherent, but contains some distracting asides.
• 3 points: the extract is mostly or fully coherent. It consists of complete sentences and logical paragraphs, the ideas are well-put and well-argued, and has minimal to no irrelevant interruptions. Information value score (range: 1-4) Measures the amount of unbiased and useful information contained in the extract.
• 1 point: the extract has little to no information value. It contains only promotional information, biased information, or personal views and preferences that are not well argued.
• 2 points: the extract has some information value, but also some clearly biased or promotional information.
• 3 points: the extract has good information value. The content is clearly formulated, well-reasoned, unbiased, and non-promotional.
• 4 points: the extract has exceptional information value. It provides in-depth insights into its subject matter, clearly enhancing a reader’s understanding of the topic. The extract: {extract} After examining the extract:
• Briefly justify your scores, up to 100 words.
• Conclude with the scores using the format: "Coherence score: . Information value score: "

Figure 6: Annotation prompt used to assign Information Value and Coherence scores.

Please act as an impartial judge and evaluate the quality of the response provided by a German AI assistant to the user question displayed below. Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of the response. Begin your evaluation by providing a short explanation. Be as objective as possible. After providing your brief explanation, please rate the response on a scale of 1 to 10 by strictly following this format: "[[rating]]", for example: "Rating: [[5]]". Your evaluation must be performed in English - this includes your explanation, reasoning, and the final verdict. Do not use German at any point. [User Question] {question} [The Start of Assistant’s Answer] {answer} [The End of Assistant’s Answer]

Figure 7: Prompt template used for standalone Likert-scale evaluation of outputs generated by instruction-tuned models. We closely follow the prompt design provided by Zheng et al. ( 2023 ) .

Please act as an impartial judge and evaluate the quality of the response provided by a German AI assistant to the user question displayed below. Your evaluation should consider correctness and helpfulness. You will be given a reference answer in addition to the assistant’s answer. Your job is to evaluate whether the assistant’s answer is correct. Begin your evaluation by comparing the assistant’s answer with the reference answer. Identify and correct any mistakes. Do not allow the length of the responses to influence your evaluation. Be as objective as possible. After providing your brief explanation, output your final verdict by strictly following this format: "[[1]]" if the assistant’s answer is correct, and "[[0]]" if assistant’s answer is incorrect. Your evaluation must be performed in English - this includes your explanation, reasoning, and the final verdict. Do not use German at any point. [User Question] {question} [The Start of Reference Answer] {answer} [The End of Reference Answer]
[The Start of Assistant’s Answer] {answer} [The End of Assistant’s Answer]

Figure 8: Prompt template used for evaluation of binary correctness of outputs generated by instruction-tuned models..

BETA