Behavior Sequence Transformer Applied on SERP Evaluation and Model Interpretation
Xiaoyu Dong, Shen Shen, Jinkang Jia, Lianzhao Yu, Yifan Wang, Po Zhang · 2022
SERP (Search Engine Result Page) quality evaluation plays a vital role in industrial practice. With the rapid iterations of search engine, traditional page-level metrics like click ratio, dwell time are no longer suitable to evaluate user experience on various templates of results. To promote evaluation accuracy, we implement Transformer to capture the sequential patterns from behavior sequence data. In recent studies, approaches focusing on modeling behavior sequences have emerged. Some studies concentrate on feature engineering by extracting subsequence patterns, others focus on end-to-end deep learning models. While widely used, these two methods both have drawbacks, either a risk of distortion of true subsequence patterns or difficulty for interpretation. Here we implement Transformer to give considerations to both completeness of sequential patterns and model interpretation. To find the best way of modeling behavior sequence data with continuous features, we adopt two embedding methods to predict SERP quality evaluation, and the second one achieves good promotion. What’s more, we develop a novel interpretation method for transformer models and demonstrate its ability to make interpretations for subsequence patterns.