Github Code 2025 Language Split

source_datasets:

• nick007x/github-code-2025
  license:
• other
  configs:
• config_name: assembly
  data_files:
  • split: train
    path: Assembly/*.parquet
• config_name: batch
  data_files:
  • split: train
    path: Batch/*.parquet
• config_name: c
  data_files:
  • split: train
    path: C/*.parquet
• config_name: csharp
  data_files:
  • split: train
    path: C#/*.parquet
• config_name: cpp
  data_files:
  • split: train
    path: C++/*.parquet
• config_name: css
  data_files:
  • split: train
    path: CSS/*.parquet
• config_name: clojure
  data_files:
  • split: train
    path: Clojure/*.parquet
• config_name: crystal
  data_files:
  • split: train
    path: Crystal/*.parquet
• config_name: dart
  data_files:
  • split: train
    path: Dart/*.parquet
• config_name: dockerfile
  data_files:
  • split: train
    path: Dockerfile/*.parquet
• config_name: elixir
  data_files:
  • split: train
    path: Elixir/*.parquet
• config_name: erlang
  data_files:
  • split: train
    path: Erlang/*.parquet
• config_name: fsharp
  data_files:
  • split: train
    path: F#/*.parquet
• config_name: go
  data_files:
  • split: train
    path: Go/*.parquet
• config_name: gradle
  data_files:
  • split: train
    path: Gradle/*.parquet
• config_name: groovy
  data_files:
  • split: train
    path: Groovy/*.parquet
• config_name: html
  data_files:
  • split: train
    path: HTML/*.parquet
• config_name: haskell
  data_files:
  • split: train
    path: Haskell/*.parquet
• config_name: json
  data_files:
  • split: train
    path: JSON/*.parquet
• config_name: java
  data_files:
  • split: train
    path: Java/*.parquet
• config_name: javascript
  data_files:
  • split: train
    path: JavaScript/*.parquet
• config_name: julia
  data_files:
  • split: train
    path: Julia/*.parquet
• config_name: kotlin
  data_files:
  • split: train
    path: Kotlin/*.parquet
• config_name: lua
  data_files:
  • split: train
    path: Lua/*.parquet
• config_name: makefile
  data_files:
  • split: train
    path: Makefile/*.parquet
• config_name: markdown
  data_files:
  • split: train
    path: Markdown/*.parquet
• config_name: nim
  data_files:
  • split: train
    path: Nim/*.parquet
• config_name: ocaml
  data_files:
  • split: train
    path: OCaml/*.parquet
• config_name: objective_c
  data_files:
  • split: train
    path: Objective-C/*.parquet
• config_name: objective_cpp
  data_files:
  • split: train
    path: Objective-C++/*.parquet
• config_name: php
  data_files:
  • split: train
    path: PHP/*.parquet
• config_name: perl
  data_files:
  • split: train
    path: Perl/*.parquet
• config_name: powershell
  data_files:
  • split: train
    path: PowerShell/*.parquet
• config_name: python
  data_files:
  • split: train
    path: Python/*.parquet
• config_name: r
  data_files:
  • split: train
    path: R/*.parquet
• config_name: ruby
  data_files:
  • split: train
    path: Ruby/*.parquet
• config_name: rust
  data_files:
  • split: train
    path: Rust/*.parquet
• config_name: sql
  data_files:
  • split: train
    path: SQL/*.parquet
• config_name: scala
  data_files:
  • split: train
    path: Scala/*.parquet
• config_name: shell
  data_files:
  • split: train
    path: Shell/*.parquet
• config_name: solidity
  data_files:
  • split: train
    path: Solidity/*.parquet
• config_name: svelte
  data_files:
  • split: train
    path: Svelte/*.parquet
• config_name: swift
  data_files:
  • split: train
    path: Swift/*.parquet
• config_name: systemverilog
  data_files:
  • split: train
    path: SystemVerilog/*.parquet
• config_name: toml
  data_files:
  • split: train
    path: TOML/*.parquet
• config_name: text
  data_files:
  • split: train
    path: Text/*.parquet
• config_name: typescript
  data_files:
  • split: train
    path: TypeScript/*.parquet
• config_name: unknown
  data_files:
  • split: train
    path: Unknown/*.parquet
• config_name: vhdl
  data_files:
  • split: train
    path: VHDL/*.parquet
• config_name: verilog
  data_files:
  • split: train
    path: Verilog/*.parquet
• config_name: visual_basic
  data_files:
  • split: train
    path: Visual Basic/*.parquet
• config_name: vue
  data_files:
  • split: train
    path: Vue/*.parquet
• config_name: xml
  data_files:
  • split: train
    path: XML/*.parquet
• config_name: yaml
  data_files:
  • split: train
    path: YAML/*.parquet
• config_name: zig
  data_files:
  • split: train
    path: Zig/*.parquet

📜 Source Data & Attribution

This dataset is a processed derivative of nick007x/github-code-2025.

Origination

The original data was aggregated by nick007x from public GitHub repositories. We have retained the original content, file paths, and metadata while restructuring the format for easier consumption by language-specific models.

Processing Steps

To create this dataset, we performed the following processing on the source data:

1. Language Identification: We mapped file extensions (e.g., .py, .rs, .ts) to their respective programming languages using a comprehensive extension map.
2. Splitting: The dataset was sharded and split into separate sub-directories/categories by programming language to allow for targeted loading (e.g., loading only Python or Rust data).
3. Filtering: Binary files and ambiguous extensions were categorized as "Unknown" or removed to ensure text-based model compatibility.

Licensing Information

The data contained in this dataset belongs to the original authors of the code repositories on GitHub.

• Source Aggregation: The aggregation was provided by nick007x/github-code-2025.
• Individual Code Files: Each file typically retains the license of its original repository (MIT, Apache 2.0, BSD, etc.). Users of this dataset are responsible for adhering to the license terms of the individual code files contained within.

Citation

If you use this dataset, please cite the original source:

@misc{github-code-2025,
  author = {nick007x},
  title = {GitHub Code 2025 Dataset},
  year = {2025},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/nick007x/github-code-2025}}
}