license: cc-by-4.0
language:

• en
  tags:
• agents
• benchmarking
• finance
• legal
• management-consulting
• tool-use
• long-horizon
  pretty_name: apex-agents
  size_categories:
• n<1K

APEX–Agents

APEX–Agents is a benchmark from Mercor for evaluating whether AI agents can execute long-horizon, cross-application professional services tasks. Tasks were created by investment banking analysts, management consultants, and corporate lawyers, and require agents to navigate realistic work environments with files and tools (e.g., docs, spreadsheets, PDFs, email, chat, calendar).

• Tasks: 480 total (160 per job category)
• Worlds: 33 total (10 banking, 11 consulting, 12 law)
• Rubric criteria: binary, criterion-level grading; mean ~4 criteria per task
• Gold outputs: provided for every task
• World assets: included (files + metadata)
• License: CC-BY 4.0
• Intended use: APEX-Agents is intended exclusively for model evaluation. Any use of this dataset for training, fine-tuning, or parameter fitting is forbidden. Crawling or scraping the dataset is also forbidden.

Dataset overview

Each case corresponds to a task inside a world. A “world” is a realistic project scenario created by experts. Worlds contain files and tools required to complete tasks. Web search is disabled to keep evaluations reproducible.
Worlds contain applications such as: Calendar, Chat, Code Execution, Documents, File system, Mail, PDFs, Spreadsheets, Presentations. Some worlds include additional finance data applications.

A task includes:

• Prompt: single-turn instruction given to the agent
• Rubric: list of criteria (binary gradable statements) + grading target info
• Gold output(s): expert-created reference output (in the requested output format)
• Metadata: job/workflow tags, expected output type, estimated completion time, etc.
• World context: pointers/IDs for the world plus associated files/artifacts

Evaluation

APEX–Agents uses rubric-based grading:

• Each rubric contains multiple criteria (binary: Met / Not met).
• There are between 1 and 10 criteria, with a mean of 4.06.
• A judge model grades each criterion independently, using the prompt, the agent output, and relevant artifacts/changes.

Leaderboard baselines

Our paper reports results for eight agents against multiple metrics (Pass@1, Pass@8, mean score). The leaderboard uses Pass@1: the probability that a uniformly sampled task passes all criteria in a single run.

Where available, models have thinking / reasoning effort set to high.

Archipelago

Our service for executing and evalling agents is available open-source on Github.
✨View the code

How to load the dataset

from datasets import load_dataset

ds = load_dataset("mercor/apex-agents")  # replace if your org/name differs
print(ds)
print(ds["train"][0].keys())

Citation

@misc{vidgen2026apexagents,
  title        = {APEX--Agents},
  author       = {Vidgen, Bertie and Mann, Austin and Fennelly, Abby and Wright Stanly, John and Rothman, Lucas and Burstein, Marco and Benchek, Julien and Ostrofsky, David and Ravichandran, Anirudh and Sur, Debnil and Venugopal, Neel and Hsia, Alannah and Robinson, Isaac and Huang, Calix and Varones, Olivia and Khan, Daniyal and Haines, Michael and Richards, Zach and Mahapatra, Chirag and Foody, Brendan and Nitski, Osvald},
  year         = {2026},
  howpublished = {arXiv},
  url          = {https://arxiv.org/pdf/2601.14242}
}

Contact us

[email protected]

Legal disclaimer on the content of worlds

This material is provided for research, educational, and informational purposes only. It consists of hypothetical, simulated financial and legal and regulatory analyses and illustrative scenarios (including, without limitation, simulated leverage buyout structures, capital structures, financing terms, valuation ranges, projected returns, and potential mergers, acquisitions, divestitures, or other strategic transactions, legal memoranda, hypothetical legal advice to a company, hypothetical correspondences to regulatory agencies, etc.). No representation is made that any scenario described herein is likely to occur, is being contemplated by any person, or reflects an actual proposed or pending transaction or any legal, regulatory, or compliance risk.

This material does not constitute (and should not be construed as) financial, investment, legal, tax, accounting, or other professional advice, and is not intended to form the basis of any investment decision or any contract. The analyses and outputs in this material are based on assumptions, estimates, modeling methodologies, and hypothetical legal scenarios, that may prove incorrect. The financial and legal information is derived from publicly available information and third‑party sources that have not been independently verified. Any projections, forward‑looking statements, scenario outputs, similar financial information, and any legal documents, memoranda, and correspondence, are hypothetical and thus inherently uncertain and are provided solely to illustrate how results might change under different assumptions. No representation or warranty (express or implied) is made regarding this material, and it is provided on an “as‑is” and “as‑available” basis.

To the maximum extent permitted by applicable law, Mercor disclaims any liability for any direct or indirect losses or damages arising from or related to the use of (or reliance on) this material, including without limitation any loss of profits, loss of business, loss of goodwill, or consequential, incidental, special, punitive, or exemplary damages, even if advised of the possibility of such damages. Nothing in this disclaimer limits or excludes liability that cannot be limited or excluded under applicable law.

Robots Exclusion Statement (Human-Readable)

To all automated crawlers and bots:

User-Agent: *
Disallow: /

We ask that:
• You do not crawl, scrape, index, or download this dataset programmatically.
• You do not use this dataset for training models or any automated processing
without express permission from the dataset owner.