On the Experimental Biases in User Behavior and QoE Assessment in the Lab
Werner Robitza, Peter Andras Kara, Maria G. Martini, Alexander Raake · 2016
User behavior is one of the key components of customer engagement and abandonment, which result from a good or bad Quality of Experience. However, methods to evoke and measure user behavior are still understudied. This paper presents an in-depth look at a study in which we measured user behavior during video streaming consumption in a controlled laboratory environment. We confronted subjects with typical streaming problems such as stalling and quality fluctuations. The subjects were not informed about the real purpose of the test; their behavior was tracked unobtrusively. The results suggest that the method can elicit responses to the inserted problems, such as seeking, pausing, or reloading the web page. However, a third of the subjects acted apprehensively, meaning that they changed their behavior due to being part of a test. In this contribution, we elaborate on the underlying reasons for those experimental biases. We discuss the suitability of different test designs for behavioral assessment and give guidelines on how to quantify and combat biasing factors introduced by the test procedure.