How Low Can You Go? The Data-Light SE Challenge
Kishan Kumar Ganguly, Tim Menzies · Proceedings of the ACM on software engineering. · 2026
Much of current Software Engineering (SE) research assumes that progress requires massive datasets and CPU-intensive optimizers. However, has this been rigorously tested? This work bears evidence that for over 100 SE optimization tasks, with only a few dozen labels, simple optimizers achieve ≈ 90% of the best reported results, matching complex optimizers such as SMAC, TPE, and DEHB while running orders of magnitude faster. We propose the data-light challenge: when will a handful of labels suffice for SE tasks? We contribute (1) a mathematical formalization of labeling costs, (2) a library of lightweight algorithms, and (3) empirical results across 100+ SE tasks showing when lightweight algorithms excel or fail.