tags:

• code
• math
• reasoning
• llm
  license: other
  language:
• en
  source_datasets:
• HuggingFaceTB/smollm-corpus
• jon-tow/starcoderdata-python-edu
• ubaada/booksum-complete-cleaned
• euirim/goodwiki
• togethercomputer/RedPajama-Data-1T
• allenai/dolma
• bigcode/the-stack-v2-train-smol-ids
• bigcode/starcoderdata
• m-a-p/Matrix
• cerebras/SlimPajama-627B
• open-phi/textbooks
• open-phi/textbooks_grounded
• open-phi/programming_books_llama
• nampdn-ai/tiny-strange-textbooks
• nampdn-ai/tiny-textbooks
• nampdn-ai/tiny-code-textbooks
• nampdn-ai/tiny-orca-textbooks
• SciPhi/textbooks-are-all-you-need-lite
• vikp/textbook_quality_programming
• EleutherAI/proof-pile-2
• open-web-math/open-web-math
• biglam/blbooks-parquet
• storytracer/LoC-PD-Books
• GAIR/MathPile
• tomg-group-umd/CLRS-Text-train
• math-ai/AutoMathText
• bigcode/commitpackft
• bigcode/stack-dedup-python-fns
• vikp/python_code_instructions_filtered
• mlabonne/chessllm
• Waterhorse/chess_data
• EleutherAI/lichess-puzzles
• chargoddard/WebInstructSub-prometheus
• Locutusque/hercules-v5.0
• nvidia/OpenMathInstruct-1
• meta-math/MetaMathQA
• m-a-p/CodeFeedback-Filtered-Instruction
• nvidia/Daring-Anteater
• nvidia/sft_datablend_v1
• BAAI/Infinity-Instruct
• anthracite-org/Stheno-Data-Filtered
• Nopm/Opus_WritingStruct
• xinlai/Math-Step-DPO-10K
• bigcode/self-oss-instruct-sc2-exec-filter-50k
• HuggingFaceTB/everyday-conversations
• hkust-nlp/gsm8k-fix
• HuggingFaceH4/no_robots
• THUDM/LongWriter-6k
• THUDM/webglm-qa
• AlgorithmicResearchGroup/ArXivDLInstruct
• allenai/tulu-v2-sft-mixture-olmo-4096
• bigscience/P3
• Gryphe/Sonnet3.5-SlimOrcaDedupCleaned
• Gryphe/Opus-WritingPrompts
• nothingiisreal/Reddit-Dirty-And-WritingPrompts
• nothingiisreal/Kalomaze-Opus-Instruct-25k-filtered
• internlm/Lean-Github
• pkuAI4M/LeanWorkbook
• casey-martin/multilingual-mathematical-autoformalization
• AI4M/leandojo-informalized
• casey-martin/oa_cpp_annotate_gen
• l3lab/ntp-mathlib-instruct-st
• ajibawa-2023/Maths-College
• ajibawa-2023/Maths-Grade-School
• ajibawa-2023/General-Stories-Collection
• XinyaoHu/AMPS_mathematica
• XinyaoHu/AMPS_khan
• Magpie-Align/Magpie-Pro-MT-300K-v0.1
• Magpie-Align/Magpie-Reasoning-150K
• gair-prox/FineWeb-pro
• gair-prox/c4-pro
• gair-prox/RedPajama-pro
• gair-prox/open-web-math-pro
• togethercomputer/Long-Data-Collections
• emozilla/pg19
• MathGenie/MathCode-Pile
• KingNish/reasoning-base-20k
• nvidia/OpenMathInstruct-2
• LLM360/TxT360
• neuralwork/arxiver
  task_categories:
• text-generation
  pretty_name: Huginn Dataset
  size_categories:
• n>1T

The Huginn Dataset

This is a record of the dataset collection used to train the huginn-0125 model. The data is provided in a semi-prepared format. We provide 4096 parquet files for train and val each which contain the exact rows used for training and validation (on the 4096 accelerators the model was trained on).
Each row is 4097 tokens long, which includes formatting tokens. The tokenizer here is the same as the model, https://huggingface.co/tomg-group-umd/huginn-0125. However, we also provide a text field that is human-readable.

The original model was trained data-parallel, so the first step of training corresponds to the first row of each parquet file, the 2nd row to the 2nd step and so forth.

If you are interested in using this data to train another model, there is no need to split the parquet files in the same way. However, note that the prepared data includes chat templating that spells out the name of this model. You can replace this by regexing for <|begin_header|>Huginn<|end_header|>, or [65506, 72, 3125, 20359, 65507].

Data Mix

Overall, this repo contains a bit more than 1 trillion tokens of data (274,349,409 * 4097 = 1.124T to be precise) from a large number of source datasets, also available on Hugging Face, mixed with the aim of pretraining a generic reasoning model.
Due to compute limitations, this mix was not ablated, so some choices that we have made may be suboptimal.

The data is processed to include instruction data (based on Allen-Zhu and Li (2024)), and document ends longer than the maximal context length are discarded in all domains except math.

The full dataset has the following proportions

and consists of data from these sources:

Pretraining Data

Standard Sources

Synthetic Data Sources

Specialized Sources

Instruction Data

Additional Sources

A machine-readable formatting of this data is provided in sources.yaml, here.

Licensing

We cannot provide a blanket license for this aggregated dataset, each subcomponent operates under a separate license, which we all list above. By using this dataset, you are also bound any license agreements and terms of use of the original data sources.

Full processing details

Given access to only enough compute
for a single large scale model run, we opted for a dataset
mixture that maximized the potential for emergent reasoning behaviors, not necessarily for optimal benchmark performance.
Our final mixture is heavily skewed towards
code and mathematical reasoning data with (hopefully) just
enough general webtext to allow the model to acquire standard language modeling abilities. All sources are publicly
available. We provide an overview in Figure 4. Following
Allen-Zhu and Li (2024), we directly mix relevant instruction data into the pretraining data. However, due to compute
and time constraints, we were not able to ablate this mixture.
We expect that a more careful data preparation could further
improve the model’s performance. We list all data sources
in Appendix C.

We construct a vocabulary of 65536 tokens via BPE (Sennrich et al., 2016),
using the implementation of Dagan (2024). In comparison to conventional tokenizer training, we construct our tokenizer directly on the instruction data split of our pretraining
corpus, to maximize tokenization efficiency on the target
domain. We also substantially modify the pre-tokenization
regex (e.g. of Dagan et al. (2024)) to better support code,
contractions and LaTeX. We include a <|begin_text|>
token at the start of every document. After tokenizing our
pretraining corpus, we pack our tokenized documents into
sequences of length 4096. When packing, we discard document ends that would otherwise lack previous context, to
fix an issue described as the “grounding problem” in Ding
et al. (2024), aside from several long-document sources of
mathematical content, which we preserve in their entirety.

Citation

@article{geiping_scaling_2025,
  title = {Scaling up {{Test-Time Compute}} with {{Latent Reasoning}}: {{A Recurrent Depth Approach}}},
  shorttitle = {Scaling up {{Test-Time Compute}} with {{Latent Reasoning}}},
  author = {Geiping, Jonas and McLeish, Sean and Jain, Neel and Kirchenbauer, John and Singh, Siddharth and Bartoldson, Brian R. and Kailkhura, Bhavya and Bhatele, Abhinav and Goldstein, Tom},
  year = {2025},
  month = feb,
  eprint = {2502.05171},
  primaryclass = {cs},
  publisher = {arXiv},
  doi = {10.48550/arXiv.2502.05171},
  url = {http://arxiv.org/abs/2502.05171},
  urldate = {2025-02-10},
  archiveprefix = {arXiv},
  keywords = {Computer Science - Computation and Language,Computer Science - Machine Learning},
  journal = {arxiv:2502.05171[cs]}
}
You can also find the paper at https://huggingface.co/papers/2502.05171.

Contact

Please, feel free to contact us with any questions, or open a discussion thread on Hugging Face.