A Probabilistic, Mechanism-Indepedent Outlier Detection Method for Online Experimentation
Yan He, Miao Chen · 2017
Many web-facing companies rely on online experimentation for evaluating users' reaction to new product features as part of the product development cycle. The existence of outliers often complicates the analysis of experiment results. Identifying outliers in online experiments is challenging as the distribution of web data such as page views do not satisfy normality assumption and are highly right-skewed. In this paper, we first illustrate the impact of outliers on experiment results and demonstrate the importance of outlier detection. As a solution, a generic method is necessary to handle outlier filtering across various types of web data. We introduce a statistical algorithm for detecting outliers regardless of the mechanism of their generation. Evaluation based on actual and simulated web data demonstrates that the proposed algorithm outperforms the other existing methods.