Balancing Humans and Machines: A New Metric for CAPTCHA Recognition Under Adversarial Attacks
Qian Wang, Zainura Idrus, Shafaf Ibrahim · 2025
CAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart) protects user accounts and funds on websites, apps, and games by blocking bots. The security design of text-based CAPTCHA has faced a fundamental contradiction: while machine recognition capabilities are rapidly improving, the recognition experience of human users is often overlooked. We found that when using the Fast Gradient Sign Method (FGSM) method to add perturbations ($\varepsilon \leqslant 0.2$), human users can still maintain a recognition accuracy of $\mathbf{8 5. 7 4 \%}$. In comparison, the machine’s performance has dropped sharply to $47.65 \%$. Unlike prior metrics that evaluate security and usability separately, this study established the Security-Usability Trade-off Metric (SUTM), achieving a quantitative assessment of the balance between safety and usability. Experimental results indicated that the optimal equilibrium point was reached when $\varepsilon=0.15$: the pass rate of ordinary users remained at $92.69 \%$, while the success rate of automated attacks reduced to below $54.86 \%$ (SUTM = 0.7177). Unlike traditional separate evaluation metrics, SUTM establishes the quantitative benchmark for CAPTCHA evaluation, rather than relying solely on subjective judgments.