Continuous Experimentation Scenarios: A Case Study in e-Commerce
Rasmus Ros, Elizabeth Bjarnason · 2018
Controlled experiments on software variants enable e-commerce companies to increase sales by providing user-adapted functionality. Our goal is to understand how the context of experimentation influences tool support. We performed a case study at Apptus that develops algorithms for e-commerce. We investigated how the case company uses experiments through five semi-structured interviews. We identified four main scenarios of experimentation and found that there are stark differences in tool support for them. The scenarios illustrate that the aptness of tool support for experiments depend on four characteristics: (1) what the goal of the experiment is;validateoroptimize, (2) whether the experiment is performedinternallyin the organisation orexternally, (3) whether decisions are takenautomaticallyormanually, and finally (4) whether the experiment should be repeated or is asingleton. These insight can be used by practitioners with an interest in efficient experimentation and to form a basis for further research into a taxonomy of experiments for software.