license: cc-by-4.0
size_categories:

• n>1T
  task_categories:
• text-generation
  language:
• multilingual
  tags:
• common-crawl
• html-parsing
• markdown
• code
• math

🔧 🔧 Our New-Gen Html Parser MinerU-HTML Now Realease!

AICC: AI-ready Common Crawl Dataset

Paper | Project page

News

• [2025-12-24] 🔥 CC-MinerU-Code Updated! We have updated our specialized high-quality code dataset CC-MinerU-Code, containing 4.58M samples, also extracted from the full Common Crawl corpus.
  
  Download: CC-MinerU-Code
  
  Each record includes language, code_language, and Markdown-formatted content with fenced code blocks. Here is a sample:

  {
    "track_id": "30ffb0a6-312f-4151-a7b4-148b4e9dea9d",
    "url": "https://lightless.me/archives/61.html",
    "language": "zh",
    "code_language": "java",
    "content": "# Handler消息传递机制\n\nAndroid 平台不允许Activity启动的新线程访问该Activity中的界面组件，这样新启动的线程就无法改变界面组件的属性值。在这种情况下需要借助Handler消息传递机制来实现。\n\nHandler类的作用：\n\n- 在新启动的线程中发送消息\n- 在主线程中获取消息，处理消息\n\n为了让主线程能够在合适的时间获取的新线程发来的消息，只能通过回调的机制来实现，我们需要重写Handler类中处理消息的方法，当新启动的线程发送消息时，Handler类中处理消息的方法被自动回调。\n\nHandler类中包含的的常用方法主要有以下几个\n `void handleMessage(Message msg)` ：处理消息的方法，通常被重写\n `final boolean hasMessages(int what)` ：检查消息队列中，是否包含what属性指定的消息\n `final boolean hasMessages(int what, Object object)` ：检查消息队列中，是否包含what属性和object属性指定值的消息\n `Message obtainMessage()` ：该函数具有多个重载，用于获取消息\n `sendEmptyMessage(int what)` ：立即发送空消息\n `final boolean sendEmptyMessageDelayed(int what, long delayMills)` ：指定delayMills毫秒后发送空消息\n `final boolean sendMessage(Message msg)` ：立即发送消息\n `final boolean sendMessageDelayed(Message msg, long delayMills)` ：指定多少毫秒后发送消息\n\n```java\npackage me.lightless.handletest;\n\nimport android.app.Activity;\nimport android.os.Bundle;\nimport android.os.Message;\nimport android.os.Handler;\nimport android.view.Menu;\nimport android.view.MenuItem;\nimport android.widget.ImageView;\n\nimport java.util.Timer;\nimport java.util.TimerTask;\n//import java.util.logging.Handler;\nimport java.util.logging.LogRecord;\n\n\npublic class MyActivity extends Activity {\n\n    int[] imageIds = new int[] {\n            R.drawable.ajax,\n            R.drawable.classic,\n            R.drawable.ee,\n            R.drawable.ic_launcher,\n            R.drawable.java,\n            R.drawable.xml\n    };\n    int currentImageId = 0;\n\n    @Override\n    protected void onCreate(Bundle savedInstanceState) {\n        super.onCreate(savedInstanceState);\n        setContentView(R.layout.activity_my);\n\n        // Get ImageView\n        final ImageView show = (ImageView)findViewById(R.id.show);\n\n        final Handler myHandler = new Handler() {\n            @Override\n            public void handleMessage(Message msg) {\n                if (msg.what == 0x1233) {\n                    show.setImageResource(imageIds[currentImageId++]);\n                    if (currentImageId >= 5) {\n                        currentImageId = 0;\n                    }\n                }\n            }\n        };\n\n        // Set a timer to execute sth\n        new Timer().schedule(new TimerTask() {\n            @Override\n            public void run() {\n                Message msg = new Message();\n                msg.what = 0x1233;\n                myHandler.sendMessage(msg);\n            }\n        }, 0, 800)\n;\n    }\n\n\n    @Override\n    public boolean onCreateOptionsMenu(Menu menu) {\n        // Inflate the menu; this adds items to the action bar if it is present.\n        getMenuInflater().inflate(R.menu.my, menu);\n        return true;\n    }\n\n    @Override\n    public boolean onOptionsItemSelected(MenuItem item) {\n        // Handle action bar item clicks here. The action bar will\n        // automatically handle clicks on the Home/Up button, so long\n        // as you specify a parent activity in AndroidManifest.xml.\n        int id = item.getItemId();\n        if (id == R.id.action_settings) {\n            return true;\n        }\n        return super.onOptionsItemSelected(item);\n    }\n}\n```\n",
    "extract_method": "MinerU-HTML",
    "sub_path": "Code"
}

• [2025-12-12] 🔥 CC-MinerU-Formula Released! Beyond the general AICC corpus, we have launched the first part of our specialized high-quality data: the fine-grained web formula dataset CC-MinerU-Formula. This data is intelligently parsed and precisely extracted from full Common Crawl raw web structures using our self-developed MinerU-HTML semantic-aware HTML extraction engine. Compared to traditional heuristic extraction methods, MinerU-HTML comprehends HTML semantics and effectively preserves the original structural information of formulas, making this structured content highly suitable for Large Language Model scenarios such as mathematical understanding, reasoning, and fine-tuning.
  
  Download: CC-MinerU-Formula on Hugging Face
  
  We have collected 975,155 cross-disciplinary formula samples, covering mathematics, physics, chemistry, and engineering. Here is a sample.
  
  

AICC (AI-ready Common Crawl) is a large-scale, AI-ready web dataset derived from Common Crawl, containing semantically extracted Markdown-formatted main content from diverse web pages. The dataset is constructed using the MinerU-HTML, a web extraction pipeline developed by OpenDataLab.

• High-quality main content: High-fidelity main content extracted from diverse Common Crawl pages, including challenging types like forums, Q&A sites, and pages with tables or formulas.
• Precise structured elements: High-fidelity extraction of code blocks, mathematical formulas, and complex tables from real-world web pages, preserving syntax, formatting, and structural integrity.
• Proven downstream effectiveness: Pretraining a language model on AICC leads to higher accuracy across diverse benchmarks compared to training on datasets extracted with other methods.

🎉🎉🎉 Experience our online web extraction with your own HTML!!!

Dataset Creation

Raw Html Source This release includes the parsed results from two Common Crawl dumps:

• CC-MAIN-2025-08
• CC-MAIN-2025-13

MinerU-HTML Pipeline The detailed pipeline can be found in the AICC technical report:

Data Statistics

The AICC dataset contains only the successfully extracted AI-ready JSON records(each with a content field containing Markdown text).

For reference, the number of original pages in the corresponding Common Crawl dumps is also shown below.

Note that only the extracted JSON records are included in the released dataset.

Data Format

Data sample

  {
    "track_id": "5667aa9a-da8a-5c80-a678-d38609247cb5",
    "url": "https://cheezburger.com/14867205/dude-finds-giant-centipede-in-daughters-room-horrifies-people-with-the-footage",
    "language": "en",
    "content": "# Dude Finds Giant Centipede In Daughter's Room & Horrifies People With the Footage

Growing up in Brooklyn I was always terrified of house centipedes. And why wouldn't I be? The revoltingly speedy unibrow bugs were all over the place. They'd find their way onto my arm at slumber parties, inspiring blood-curdling screams more indicative of a murder than a creepy crawly. I'd find them in my shoes. My mother would tell me they're "good bugs" because they eat the "bad bugs," but to me they were the stuff of literal nightmares. Eventually my fear subsided, after having to live in a dank and dark East Village basement where they were a daily sighting. I'd almost forgotten about the critters - until Twitter user@VickGlaze horrified the internet with a video that features a far more sinister-looking centipede.
I almost had set the crib on fire yesterday….Y'all wouldn't believe what I found in my daughters room dawg. I just spent a 50ball at Lowe's on everything pest control. I found where it came in and sealed every window in the crib and some. pic.twitter.com/iAoN7uWYsY
— Miami Vice II (@VickGlaze) July 27, 2021
The alarmingly large centipede in Vick's video is known by a few names. Some call them Texas redheaded centipedes, others the Giant Desert Centipede. Whatever its called, it's freaking terrifying. And this scared father was not afraid to admit it. In a thread, he details all the measures he took to protect his home, and the terror the creature has struck in his heart. The responses (mostly in solidarity) are almost as entertaining as the video and accompanying story. As for me? I'm feeling pretty damn lucky that I grew up in New York and not in Austin, Texas, where these massive venomous critters roam.
",
    "extract_method": "MinerU-HTML",
    "sub_path": "CC-MAIN-2025-08"
  }

Citation

@misc{ma2025aiccparsehtmlfiner,
      title={AICC: Parse HTML Finer, Make Models Better -- A 7.3T AI-Ready Corpus Built by a Model-Based HTML Parser}, 
      author={Ren Ma and Jiantao Qiu and Chao Xu and Pei Chu and Kaiwen Liu and Pengli Ren and Yuan Qu and Jiahui Peng and Linfeng Hou and Mengjie Liu and Lindong Lu and Wenchang Ning and Jia Yu and Rui Min and Jin Shi and Haojiong Chen and Peng Zhang and Wenjian Zhang and Qian Jiang and Zengjie Hu and Guoqiang Yang and Zhenxiang Li and Fukai Shang and Runyuan Ma and Chenlin Su and Zhongying Tu and Wentao Zhang and Dahua Lin and Conghui He},
      year={2025},
      eprint={2511.16397},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2511.16397}, 
}

License and Ethics

This dataset is licensed under CC BY 4.0, requiring attribution when used. It is derived from Common Crawl web pages and may contain biased or sensitive content; users are responsible for ethical and lawful usage in research or applications.