Continuous A/B Testing in Containers
Ádám Révész, Norbert Pataki · 2019
Software version ranking plays an important role in improved user experience and software quality. A/B testing is a technique to distinguish between the popularity and usability of two quite similar versions (A and B) of a product, marketing strategy, search ad, etc. It is a kind of two-sample hypothesis testing, used in the field of statistics. This controlled experiment can evaluate user engagement or satisfaction with a new service, feature, or product. A/B testing is typically used in evaluation of user-experience design in software technology. DevOps is an emerging software methodology in which the development and operations are not independent processes, they affect each other. DevOps emphasizes the usage of virtualization technologies (e.g. containers). Docker is widely-used technology for containerization. In this paper, we deal with a new approach for regular A/B testing via Docker containers. Our solution provides an API that can be available from many DevOps tools. This approach is DevOps-style A/B testing because after the evaluation the better version remains in production.