A New Method for Short Text Compression
Murat Aslanyürek, Altan Mesut · IEEE Access · 2023
In this study, a method calledWord-based Static Dictionary Compression (WSDC), which can compress short texts at a high ratio, and a model that uses iterative clustering to create static dictionaries used in this method are proposed. The number of static dictionaries to be created can vary by running the k-Means clustering algorithm iteratively according to some rules.Wikipedia article abstracts consisting of 5 different text size groups and 6 different languages were used as dataset. The developed WSDC method is compared with both general-purpose compression methods (Gzip, Bzip2, PPMd, Brotli and Zstd) and special methods used for compression of short texts (shoco, b64pack and smaz). According to the test results, it has been seen that WSDC gives better compression ratios than other compression methods, especially for very short texts.