Context-based A/B Test Validation
Michael Nolting, Jan Eike von Seggern · 2016
Data-driven and continuous development and deployment of modern web applications depend critically on registering changes as fast aspossible, paving the way for short innovation cycles. A/B testing is a popular tool for comparing the performance of different variants. Despite the widespread use of A/B tests, there is little research on how to assert the validity of such tests. Even small changes in the application's user base, hard- or software stack not related to the variants under test can transform on possibly hidden paths into significant disturbances of the overall evaluation criterion (OEC) of an A/B test and, hence, invalidate such a test. Therefore, the highly dynamic server and client run-time environments of modern web applications make it difficult to assert correctly the validity of an A/B test.