Exceptional Contrast Set Mining: Moving Beyond the Deluge of the Obvious
Dang Nguyen, Wei Luo, Dinh Phung, Svetha Venkatesh · Lecture notes in computer science · 2016
Data scientists, with access to fast growing data and computing power, constantly look for algorithms with greater detection power to discover “novel” knowledge. But more often than not, their algorithms give them too many outputs that are either highly speculative or simply confirming what the domain experts already know. To escape this dilemma, we need algorithms that move beyond the obvious association analyses and leverage domain analytic objectives (aka. KPIs) to look for higher order connections. We propose a new technique Exceptional Contrast Set Mining that first gathers a succinct collection of affirmative contrast sets based on the principle of redundant information elimination. Then it discovers exceptional contrast sets that contradict the affirmative contrast sets. The algorithm has been successfully applied to several analytic consulting projects. In particular, during an analysis of a state-wide cancer registry, it discovered a surprising regional difference in breast cancer screening. These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.