ASAC: A Benchmark for Algorithm Synthesis
Zhao Zhang, Yican Sun, Ruyi Ji, Siyuan Li, Xiaosong Peng, Zhechong Huang, Sizhe Li, Tianran Zhu, Yingfei Xiong · 2024
In this paper, we present the first benchmark for algorithm synthesis from formal specification: ASAC. ASAC consists of 136 tasks covering a wide range of algorithmic paradigms and various difficulty levels. Each task includes a formal specification and an efficiency requirement, and the program synthesizer is expected to produce a program that satisfies the formal specification and meets the efficiency requirement. Our evaluation of two state-of-the-art (SOTA) approaches in ASAC shows that ASAC exposes new challenges for future research on program synthesis. ASAC is available at https://auqwqua.github.io/ASACBenchmark, and the demo video is available at https://youtu.be/JXVleCdBh8U.