configs:

• config_name: default
  data_files:
  • split: train
    path: data/train-*
  • split: test
    path: data/test-*
  • split: valid
    path: data/valid-*
    dataset_info:
    features:
  • name: name
    dtype: large_string
  • name: description
    dtype: large_string
  • name: corner_cases
    list:
    • name: input
      struct:
      • name: stdin
        dtype: large_string
    • name: output
      struct:
      • name: stdout
        dtype: large_string
  • name: commands
    list: large_string
  • name: generator
    dtype: large_string
  • name: checker
    dtype: large_string
  • name: input_constraints_summary
    dtype: large_string
  • name: results
    list:
    • name: corner_cases
      list:
      • name: input
        struct:
        • name: stdin
          dtype: large_string
      • name: output
        struct:
        • name: stdout
          dtype: large_string
    • name: generate_commands
      list: large_string
    • name: commands_add
      list: large_string
    • name: commands_replace
      list: large_string
    • name: generate_case_inputs
      list: large_string
    • name: improved_generator
      dtype: large_string
    • name: search_replace_generator_blocks
      list: large_string
    • name: unmatched_blocks
      list: large_string
      splits:
  • name: train
    num_bytes: 692913981543
    num_examples: 11407
  • name: test
    num_bytes: 10751116880
    num_examples: 162
  • name: valid
    num_bytes: 10415379953
    num_examples: 114
    download_size: 324619170724
    dataset_size: 714080478376

CodeContests-O: Powering LLMs via Feedback-Driven Iterative Test Case Generation

Overview

CodeContests-O is a high-quality competitive programming dataset with iteratively refined test cases, designed to provide reliable verification signals for training and evaluating reasoning-centric Large Language Models (LLMs). Built upon the CodeContests dataset, CodeContests-O employs a novel Feedback-Driven Iterative Framework to systematically synthesize, validate, and refine test cases, achieving superior fidelity and discriminative power.

Key Features

• High-Quality Test Cases: Achieves $89.37%$ True Positive Rate (TPR) and $90.89%$ True Negative Rate (TNR), significantly outperforming CodeContests and CodeContests+ by margins of $4.32%$ and $9.37%$, respectively.
• Iterative Refinement: Test cases are progressively improved through a closed-loop process that leverages execution feedback from correct and incorrect solutions.
• Complete Generation Artifacts: Includes generators, checkers, commands, and full iteration history for transparency and reproducibility.
• Proven Effectiveness: Fine-tuning Qwen2.5-7B on CodeContests-O yields a $9.52%$ improvement on LiveCodeBench (Pass@1).

Dataset Statistics

Methodology

CodeContests-O is constructed using a Feedback-Driven Iterative Framework consisting of three primary stages:

1. Initial Test Case Generation

The framework adopts a Generator-Validator paradigm:

1. Problem Analysis: An LLM analyzes the problem description to identify input-output constraints and anticipate common algorithmic pitfalls (e.g., integer overflows, boundary conditions).
2. Generator Synthesis: A dedicated generator program $G^{(0)}$ is developed to ensure structural integrity and validity of test inputs.
3. Command Formulation: A series of execution commands $C^{(0)}$ are created, each defining unique arguments for diverse test case generation.

2. Execution and Feedback Collection

Generated test cases are executed against solution pools:

• Correct Solutions ($S^+$): Verify test case fidelity
• Incorrect Solutions ($S^-$): Measure discriminative power

The feedback report $R^{(i)} = {F^{(i)}, E^{(i)}}$ captures:

• $F^{(i)}$: False positives (incorrect solutions passing) and false negatives (correct solutions failing)
• $E^{(i)}$: Execution error logs and stack traces

3. Feedback-Guided Refinement

Based on the feedback report, the LLM performs root-cause analysis and:

• Updates the generator program $G^{(i)} \to G^{(i+1)}$ using a "search-and-replace" mechanism
• Redesigns execution commands $C^{(i)} \to C^{(i+1)}$ to explore new parameter spaces

The iterative loop terminates when TPR $\geq \alpha$ and TNR $\geq \beta$, or the maximum iteration limit $N_{\max}$ is reached.

Data Structure

Top-Level Fields

Corner Case Structure

Each corner case in corner_cases contains:

{
  "input": {
    "stdin": "5\n1 2 3 4 5\n"
  },
  "output": {
    "stdout": "15\n"
  }
}

Results (Iteration History)

The results field is a list where each element represents one iteration of the refinement process:

Note: Since results[0] corresponds to the Initial Test Case Generation phase, commands_add and commands_replace are empty for the first iteration.

Usage

Loading the Dataset

from datasets import load_dataset

Load the full dataset
dataset = load_dataset("caijanfeng/CodeContests-O")
Access different splits
train_data = dataset["train"]
test_data = dataset["test"]
valid_data = dataset["valid"]
Example: Access a single problem
problem = train_data[0]
print(f"Problem: {problem['name']}")
print(f"Number of test cases: {len(problem['corner_cases'])}")
print(f"Number of iterations: {len(problem['results'])}")

Extracting Test Cases

def get_test_cases(problem):
    """Extract all test cases from a problem."""
    test_cases = []
    for case in problem['corner_cases']:
        test_cases.append({
            'input': case['input']['stdin'],
            'output': case['output']['stdout']
        })
    return test_cases

Get test cases for a problem
cases = get_test_cases(train_data[0])
for i, case in enumerate(cases[:3]):
    print(f"Test Case {i+1}:")
    print(f"  Input: {case['input'][:50]}...")
    print(f"  Output: {case['output'][:50]}...")

Analyzing Iteration History

def analyze_iterations(problem):
    """Analyze the refinement process for a problem."""
    print(f"Problem: {problem['name']}")
    print(f"Total iterations: {len(problem['results'])}")
    

    

      

        text
        
          Copy
        
      
      
for i, result in enumerate(problem['results']):
    print(f"\nIteration {i}:")
    print(f"  - Corner cases generated: {len(result['corner_cases'])}")
    print(f"  - Commands generated: {len(result['generate_commands'])}")
    if i > 0:
        print(f"  - Commands added: {len(result['commands_add'])}")
        print(f"  - Commands replaced: {len(result['commands_replace'])}")
    print(f"  - Search-replace blocks: {len(result['search_replace_generator_blocks'])}")
    print(f"  - Unmatched blocks: {len(result['unmatched_blocks'])}")
    
  
analyze_iterations(train_data[0])

Performance Comparison

Test Case Quality (TPR/TNR)

Downstream RL Performance (LiveCodeBench Pass@1)

File Format Details

Generator Programs

Generators are implemented in C++ using testlib.h, a standard library for competitive programming problem creation:

#include "testlib.h"
#include <bits/stdc++.h>
using namespace std;

int main(int argc, char* argv[]) {
    registerGen(argc, argv, 1);

    

      

        text
        
          Copy
        
      
      
int n = opt<int>("n", rnd.next(1, 100000));
int maxVal = opt<int>("max", 1000000000);

println(n);
for (int i = 0; i < n; i++) {
    print(rnd.next(1, maxVal));
    if (i < n - 1) print(" ");
}
println();

return 0;
    
  
}

⚠️ Important: If you want to run the generators/checkers from our CodeContests-O dataset, please use the simplified testlib.h provided in our HuggingFace repository instead of the official version. The simplified version is optimized for compatibility with our generated code.

To support our generator/checker, we have extended ByteDance's SandboxFusion with additional features for competitive programming test case generation. Our enhanced version is available at: cai-jianfeng/SandboxFusion

Since CodeContests+ has already synthesized preliminary generators for each problem, we leverage these existing generators as a starting point rather than generating from scratch to reduce computational overhead.

Checker Programs

Checkers validate logical consistency rather than exact string matching, enabling proper evaluation of problems with multiple valid outputs.

Since CodeContests+ has already synthesized preliminary checkers for each problem, we leverage these existing checkers rather than generating from scratch to reduce computational overhead. In our preliminary experiments, we found that the LLM rarely made further modifications to these checkers, indicating their quality is already sufficiently high. Therefore, we chose not to iteratively refine the checker in sync with the generator, but instead consistently use the checker provided by CodeContests+ throughout the refinement process.

Command Format

Commands follow shell syntax with testlib-style arguments:

./gen --n 1000 --max 1000000000
./gen --n 1 --max 1
./gen --mode stress --iterations 100

Citation

We would appreciate it if you could cite our work when CodeContests-O is helpful for your research:

@misc{cai2026codecontestsopoweringllmsfeedbackdriven,
      title={CodeContests-O: Powering LLMs via Feedback-Driven Iterative Test Case Generation}, 
      author={Jianfeng Cai and Jinhua Zhu and Ruopei Sun and Kangwen Zhao and Dongyun Xue and Mingxiao Feng and Wengang Zhou and Houqiang Li},
      year={2026},
      eprint={2601.13682},
      archivePrefix={arXiv},
      primaryClass={cs.SE},
      url={https://arxiv.org/abs/2601.13682}, 
}

License

This dataset is released under the Apache License 2.0.

Acknowledgments

• Built upon CodeContests by DeepMind and CodeContests+ by ByteDance.
• Test case execution powered by SandboxFusion
• Generator programs use testlib.h by Mike Mirzayanov

Contact

For questions or issues, please open an issue on the repository or contact the authors.