Impact of Deep Learning-Based Text Feature Extraction Methods on Binary Classification Quality of Customer Service Call Transcripts
Anna Kelm, Piotr Plebański, Robert Albert Kłopotek · 2024
This study examined the impact of various deep learning-based text feature extraction methods on the quality of binary classification of conversation transcripts. The classification aimed to determine whether a customer would recommend the company to family or friends, which is used to calculate the Net Promoter Score. Several text feature extraction techniques were investigated, including traditional methods like bag-of-words and TF-IDF, as well as deep learning approaches such as Word2Vec, Doc2Vec, FastText, GloVe, and transformer-based models like BERT and its variants. The methods were evaluated on a dataset of 30,000 transcribed customer service calls, with 20,000 used for training language models and 10,000 for classification. Performance was assessed using ROC-AUC and PR-AUC metrics with 3-fold cross-validation. Results showed that models trained specifically on the transcription data outperformed pre-trained models, with character-based n-gram models and Doc2Vec achieving the highest classification accuracy. The study highlights the importance of domain-specific training and considering the full conversation context for optimal performance in classifying customer service interactions.