AReaL: A Large-Scale Asynchronous Reinforcement Learning System

| Paper | Documentation | 中文文档 | Ask DeepWiki | 🤗 Models & Data | WeChat (微信) Group |

AReaL is a reinforcement learning (RL) infrastructure designed to bridge foundation model training with modern agent-based applications. It was originally developed by researchers and engineers from Tsinghua IIIS and the AReaL Team at Ant Group.

Built on a fully asynchronous RL training paradigm, AReaL is optimized for efficiency and scalability, making it particularly well-suited for training large-scale reasoning and agentic models.

AReaL’s mission is to make building AI agents accessible, efficient, and cost-effective for a broad community of developers and researchers.

Like milk tea - customizable, scalable, and enjoyable - we hope AReaL brings both flexibility and delight to your AI development experience. Cheers!

AReaL Highlights

• ⚡ Flexibility: Seamless customization for agentic RL and online RL training for black-box agent applications by simply replacing the base_url.
• 📈 Scalability: Stable fully asynchronous RL training with industry-leading speed.
• ✨ Cutting-Edge Performance: State-of-the-art math, coding, search, and customer service agents.

📰 News

[2026/04/23] 🚀 We’re excited to release our integration with Scaffoldings for agentic RL training - now live in our examples! Huge shoutout to @narutolhy and @WeiHaocheng for making this happen 🙌. The modular design of the Scaffoldings enables it to achieve a thorough decoupling of agent execution, reward calculation, and trajectory acquisition. This enables developers to reuse existing modules when implementing an agentic RL method, allowing them to focus on their own innovative modules.

[2026/04/18] We are thrilled to announce that AReaL's first Community Biweekly Meeting was successfully held! Thank you to everyone who joined us. Meeting materials are now available here. Our next meeting is scheduled for 2026/05/01 and will also be conducted in Chinese; English-language meetings will be scheduled in the future. We warmly welcome everyone to participate! See Community for more details.

[2026/03/02] We provide a complete example to train your own 🦞 OpenClaw agent by simply replacing the base_url and api_key with AReaL's RL service - no complicated dependencies, no code changes, works with any agentic runtime!

🚀 Getting Started

First, install the package:

git clone https://github.com/inclusionAI/AReaL
cd AReaL
pip install uv
# Install flash-attn pre-built wheel first to avoid compiling from source
# (pick the wheel matching your Python version; see https://github.com/mjun0812/flash-attention-prebuild-wheels/releases)
uv pip install "https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.7.16/flash_attn-2.8.3+cu128torch2.9-cp312-cp312-linux_x86_64.whl"
uv sync --extra cuda  # installs training packages + SGLang (default inference backend)
# For vLLM instead: cp pyproject.vllm.toml pyproject.toml && cp uv.vllm.lock uv.lock && uv sync --extra cuda

Our training scripts automatically download the required dataset (openai/gsm8k) and model (Qwen/Qwen2-1.5B-Instruct). To run on a single node:

python3 examples/math/gsm8k_rl.py --config examples/math/gsm8k_grpo.yaml scheduler.type=local

If you prefer to run experiments on a Ray cluster, update paths in the YAML file to point to your shared storage, and run:

python3 examples/math/gsm8k_rl.py --config examples/math/gsm8k_grpo.yaml \
  cluster.n_nodes=2 cluster.n_gpus_per_node=8 \
  cluster.fileroot=/path/to/nfs \
  scheduler.type=ray

For comprehensive setup instructions, see our quickstart guide.

📚 Examples

Math & Reasoning

Agentic RL

Vision-Language Models

Alignment & Infrastructure

🔧 Support Matrix

🧠 Algorithms

All RL algorithms support both asynchronous and synchronous versions by setting max_head_offpolicyness=0. See Asynchronous RL Guide.

Models

Check the AI Coding Assistant Guide and Archon Reference for how to integrate new models into AReaL.

Training Backends

Inference Backends

📖 Resources

Tutorial

• Installation
• Quickstart
• Agentic RL
• Evaluation
• Large MoE with Megatron
• Large MoE with PyTorch Archon

Code Walkthrough

• Running GRPO on GSM8K dataset

Best Practices

• Improving Algorithm Performance
• Agent Workflow Best Practices
• Debugging
• Handling OOM Issues
• Performance Profiling

Customization

• Customize Dataset
• Customize Agentic/RVLR Rollout Workflows

Algorithms

• Asynchronous RL Explained
• PPO, GRPO, and Related Algorithms
• M2PO

Reference

• CLI Configurations
• LoRA RL
• Checkpointing
• Metrics Tracking
• Allocation Mode
• Rollout Workflow
• Agent Workflow
• AI-Assisted Development

🤝 Contributing

We warmly welcome contributions from the community! Whether you're fixing bugs, adding features, improving documentation, or helping others, your contribution is valued. Please check our Contributing Guide for detailed information.

# Fork and clone the repository
git clone https://github.com/YOUR-USERNAME/AReaL
cd AReaL

# Install uv and sync dependencies
pip install uv
# Install flash-attn pre-built wheel to avoid compiling from source
uv pip install "https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.7.16/flash_attn-2.8.3+cu128torch2.9-cp312-cp312-linux_x86_64.whl"
# Use `--extra cuda` on Linux with CUDA (installs training packages + SGLang)
uv sync --extra cuda --group dev
# For vLLM instead:
# cp pyproject.vllm.toml pyproject.toml && cp uv.vllm.lock uv.lock && uv sync --extra cuda --group dev
# Or without CUDA support
# uv sync --group dev

# Set up pre-commit hooks (formatting, linting, commit message checks)
pre-commit install --install-hooks

# Make changes
git checkout -b feat/gpt-o5
git add .
# `git commit` will automatically check your files and commit messages
git commit -m "feat: implement gpt-o5 training loop"
git push

🗺️ Future Roadmap

• Full Roadmap
• 2026 Q1 Roadmap

AReaL is under active development with planned minor releases weekly and major releases monthly. We warmly welcome community engagement and contributions. We are also actively hiring interns and full-time employees with open positions in both the US and China.

🙏 Acknowledgments

We gratefully acknowledge that major contributors are from the AReaL Team at the Institute for Interdisciplinary Information Sciences (IIIS), Tsinghua University and Ant Group.

We have also received invaluable assistance from the following groups (listed alphabetically):

• The Data Intelligence Lab at Ant Research for their data support

• @HwVanICI for support on vLLM, LoRA, NPU integration, and more

• The Relaxed System Lab at HKUST for seamless collaboration on numerous system-related aspects

• The SGLang team for supporting custom weight update features and their contributions during AReaL-lite development

• The Super Computing Technology (SCT) team at Ant Group for their expertise in large-scale cluster operations and maintenance

• Special thanks to @Lyken17 for providing valuable suggestions throughout the API design process

We also deeply appreciate all pioneering work from the community, particularly the ReaLHF project from OpenPsi Inc. and other outstanding projects, including but not limited to DeepScaleR, Open-Reasoner-Zero, OpenRLHF, VeRL, SGLang, QwQ, Light-R1, and DAPO.

📜 License

This project is licensed under the Apache License 2.0.

📄 Citation

@inproceedings{mei2025real,
  author       = {Mei, Zhiyu and Fu, Wei and Li, Kaiwei and Wang, Guangju and Zhang, Huanchen and Wu, Yi},
  title        = {ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation},
  booktitle    = {Proceedings of the Eighth Conference on Machine Learning and Systems,
                  MLSys 2025, Santa Clara, CA, USA, May 12-15, 2025},
  publisher    = {mlsys.org},
  year         = {2025},
}

@misc{fu2025areal,
      title={AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning},
      author={Wei Fu and Jiaxuan Gao and Xujie Shen and Chen Zhu and Zhiyu Mei and Chuyi He and Shusheng Xu and Guo Wei and Jun Mei and Jiashu Wang and Tongkai Yang and Binhang Yuan and Yi Wu},
      year={2025},
      eprint={2505.24298},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2505.24298},
}

🚀 Quick Start

git clone https://github.com/inclusionAI/AReaL
cd AReaL
pip install uv
# Install flash-attn pre-built wheel first to avoid compiling from source
# (pick the wheel matching your Python version; see https://github.com/mjun0812/flash-attention-prebuild-wheels/releases)
uv pip install "https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.7.16/flash_attn-2.8.3+cu128torch2.9-cp312-cp312-linux_x86_64.whl"
uv sync --extra cuda  # installs training packages + SGLang (default inference backend)
# For vLLM instead: cp pyproject.vllm.toml pyproject.toml && cp uv.vllm.lock uv.lock && uv sync --extra cuda