Diffing E2E Tests against User Traces for Continuous Improvement
Xavier Blanc, Thomas Degueule, Jean‐Rémy Falleri · HAL (Le Centre pour la Communication Scientifique Directe) · 2022
End-to-end testing (E2E) is a popular approach to uncover faults in web applications by mimicking the behavior of real users. Using dedicated frameworks (e.g., Playwright, Cypress), testers translate high-level validation scenarios into concrete browser interactions to evaluate the application's behavior. Testers are forced to make implementation choices when writing their tests: choosing one of several valid selectors to click a button, focusing an input field using a Click or Tab event, etc. In this paper, we show that the resulting tests may not always represent the behavioral of real users-some action sequences may even be infeasible for users. These discrepancies between tests and users hurt the very premise of E2E testing. To address this issue, we propose a novel method that enables testers to understand the true behavior of their E2E tests, and help them improve their tests by pinpointing when, where, and in which way they differ from the behavior of real users. Using a set of E2E tests provided by our industrial partner, we show that our tool is indeed able to identify discrepancies between the tests and the users and that, in several cases, the testers decided to update their tests to better match the behavior of their users.