Flakiness in automated web application end-to-end testing
Nelli Kiskonen · UTUPub (University of Turku) · 2020
Automated regression tests verify that changes in the software do not introduce new errors into previously tested components. Developers rely on the results of these tests to decide whether the code changes are ready to be submitted or the system is ready for deployment. One challenge with automated tests is that they can be flaky, which means that the same test can pass and fail on different test runs even though the tested code has not changed. The goal of this thesis is to study the flakiness issue in web application end-to-end testing. The research questions in this study are: What problems flaky tests cause? What are the main reasons for test flakiness? And how to fix flaky test cases? Flakiness issue was first examined by researching previous studies. In the second phase, end-to-end tests were implemented for a web application with Cypress testing tool. Different reasons that caused flakiness in the test cases, and their fixing strategies, are discussed in this thesis. At the end of the study, the fixing strategies were validated by running all tests 50 times in different modes. The results of this study were compared to the previous studies. 6 out of 14 issues that were described in the previous studies, also caused flakiness in this study. Additionally, three other reasons caused flakiness. The main conclusion of this work is that test flakiness is a big problem in end-to-end testing, it makes testing more complicated and increases costs. The root cause of flakiness is usually found from the test case, but flakiness can also be caused by the code of the application under test. In addition, flakiness can be related to the testing tool or be caused by external reasons, such as remote services.