arXiv Papers by Subject

A reorganised version of the nick007x/arxiv-papers dataset, partitioned by subject code, year, and month for efficient selective access.

Dataset Description

This dataset contains metadata for over 2.5 million arXiv papers, organised into a hierarchical directory structure that allows users to download only the specific subjects and time periods they need, rather than the entire dataset.

Motivation

The original nick007x/arxiv-papers dataset is an excellent resource containing comprehensive arXiv paper metadata. However, its monolithic structure requires downloading the entire dataset even when only a subset of papers is needed.

This derived dataset addresses that limitation by partitioning the data into small, focused parquet files organised by:

1. Subject code (e.g., cs.AI, astro-ph.CO, math.NA)
2. Year (1989–2025)
3. Month (01–12)

This structure enables:

• Downloading only specific research domains
• Fetching data for particular time ranges
• Incremental updates as new papers are published
• Efficient caching and lazy loading

Dataset Structure

data/
├── astro-ph.CO/
│   ├── 2009/
│   │   ├── 01/
│   │   │   └── 00000000.parquet
│   │   ├── 02/
│   │   │   └── 00000000.parquet
│   │   └── ...
│   └── ...
├── cs.AI/
│   ├── 1993/
│   │   └── ...
│   └── 2025/
│       └── ...
├── cs.LG/
│   └── ...
└── ...

Subject Categories

The dataset includes 148 arXiv subject categories spanning:

Data Fields

Each parquet file contains the following fields (inherited from the source dataset):

• Note that the ZIP files in file_path point to nick007x/arxiv-papers !

Usage

Loading Specific Subjects and Time Periods

from huggingface_hub import hf_hub_download

# Download a specific subject/year/month
local_path = hf_hub_download(
    repo_id="permutans/arxiv-papers-by-subject",
    repo_type="dataset",
    filename="data/cs.LG/2024/06/00000000.parquet"
)

import polars as pl
df = pl.read_parquet(local_path)

Loading Multiple Files with Glob Patterns

from huggingface_hub import snapshot_download

# Download all cs.LG papers from 2024
snapshot_download(
    repo_id="permutans/arxiv-papers-by-subject",
    repo_type="dataset",
    allow_patterns="data/cs.LG/2024/*/*.parquet",
    local_dir="./arxiv_data"
)

Using with Polars LazyFrames

import polars as pl

# Scan multiple files lazily
lf = pl.scan_parquet("arxiv_data/data/cs.*/2024/*/*.parquet")

# Filter and collect only what you need
recent_ml = lf.filter(
    pl.col("primary_subject").str.contains("Machine Learning")
).collect()

Dataset Statistics

• Total papers: ~2.55 million
• Subject categories: 167
• Year range: 1998–2025
• File format: Parquet (compressed)

Source Attribution

This dataset is derived from nick007x/arxiv-papers, which provides the complete arXiv scientific papers archive. The original dataset contains both metadata and PDFs; this derived dataset includes only the metadata, reorganised for efficient partial access.

The underlying paper content originates from arXiv.org, operated by Cornell University.

License

This dataset follows the licensing structure of the source:

• Dataset packaging and organisation: MIT License, as for nick007x/arxiv-papers
• Individual paper content: Subject to each paper's license as specified by arXiv and the respective authors

Citation

If you use this dataset, please cite both this reorganized version and the original source:

@dataset{arxiv_papers_by_subject_2025,
  title = {arXiv Papers by Subject},
  author = {permutans},
  year = {2025},
  publisher = {Hugging Face},
  url = {https://huggingface.co/datasets/permutans/arxiv-papers-by-subject}
}

@dataset{arxiv_papers_2025,
  title = {arXiv Papers Dataset},
  author = {nick007x},
  year = {2025},
  publisher = {Hugging Face},
  url = {https://huggingface.co/datasets/nick007x/arxiv-papers}
}