An Improved Topic Extraction Method Based on Word Frequency Information Entropy for Multilingual Topic Attentional Division
Yuan Yue, Huaping Zhang · 2024
In the contemporary era of ubiquitous global information dissemination, a myriad of news articles are generated worldwide on a daily basis. The topics that capture the attention of different countries diverge due to variances in culture, values, and other influential factors. Analyzing these discrepancies in topic preferences across languages within specific timeframes holds paramount importance for comprehensively understanding and delineating the nuances of diverse national cultures. This paper proposes a novel statistical analysis methodology for extracting multi-language news topic keywords, leveraging the concept of word frequency information entropy. Our approach facilitates the identification of shared topics across different languages, as well as language-specific concerns, within extensive news datasets. Furthermore, we address a prevalent challenge encountered in existing topic modeling methodologies, namely output redundancy. Through the aggregation of synonymous terms, we effectively alleviate redundancy, thereby enhancing the quality of extracted topic keywords. Experimental evaluations are conducted on a meticulously collected multinational news dataset, wherein we assess the effectiveness of our approach in partitioning common and language-specific focus topics across multiple languages, while also quantifying the efficacy of redundancy elimination.