Transfer learning meets sales engagement email classification: Evaluation, analysis, and strategies
Yong Liu, Pavel Dmitriev, Yifei Huang, Andrew Brooks, Dong Li, Mengyue Liang, Zvi Boshernitzan, Jiwei Cao, Bobby Nguy · Concurrency and Computation Practice and Experience · 2020
Abstract Enterprise email classification in the sales engagement platform is a challenge due to its evolving asynchronous conversational context during the sales process and differences across industries and organizations. This is further exacerbated by the limited amount of labeled emails due to security and privacy constraints. The leaderboard success of using pretrained language models (LMs) such as BERT and various transfer learning techniques promises a paradigm shift to natural language processing, yet the recipe for applying high performance transfer learning (HPTL) in practical applications remains unclear. This article investigates applying HPTL to sales engagement email classification through a series of experiments and analysis. The experiment datasets include two different organizations' emails. The contribution of this paper is 4‐fold: (a) analysis and characterization of the email corpora from different organizations; (b) identification of the best combinations of pre‐trained LMs under different modeling architectures; (c) study of the impact and trade‐off of limited labeled data on the model accuracy and training time; and (d) characterization and study of the impact of different orgs' datasets on the model accuracy. Our results showed that a practical winning recipe that uses BERT‐finetuning with as few as 500 labeled training examples can consistently outperform significantly with reasonable training time among all models evaluated.