Public Benchmarks

Evaluates AI agents' ability to solve real-world GitHub issues by producing code edits as patch files. Uses authentic software engineering problems from popular open-source repositories.

Attributions:

Carlos E. Jimenez, John Yang, Alexander Wettig

Human-validated subset of SWE-bench with 500 carefully verified samples, providing more reliable evaluation of AI models' software engineering capabilities.

Attributions:

Carlos E. Jimenez, John Yang, Alexander Wettig (original SWE-bench authors), OpenAI Preparedness Team

First multilingual code fix benchmark covering seven programming languages, designed to evaluate large models' self-debugging and code repair capabilities across diverse codebases.

Attributions:

Daoguang Zan, Zhirong Huang, Wei Liu

The original OpenAI benchmark for evaluating large language models trained on code, featuring carefully crafted evaluation sets that measure functional correctness.

Attributions:

Mark Chen, Jerry Tworek, Heewoo Jun

Evaluates LLMs on practical and challenging programming tasks with diverse function calls and complex instructions across 139 Python libraries.

Attributions:

Terry Yue Zhuo, Minh Chien Vu, Jenny Chim

Automated pipeline for generating large-scale software engineering training data, creating synthetic bug-fixing tasks from real codebases.

Attributions:

John Yang, Kyle Leret, Carlos E. Jimenez

Data science code generation benchmark with 1,000 problems spanning seven Python libraries including NumPy, Pandas, and Matplotlib

Attributions:

Yuhang Lai, Chengxi Li, Yiming Wang

AI-native automated bug fixing agent that achieves state-of-the-art performance on SWE-bench, demonstrating advanced software engineering capabilities.

Attributions:

Yuntong Liu, Peng Gao, Xinyu Wang