Autonomous website categorization with pre-defined dictionary
Adsadawut Chanakitkarnchok, Kulit Na Nakorn, Kultida Rojviboolchai · 2016
In this technology emerging era, the number of websites is increasing dramatically. The content and category of information are overflowing the Internet World. Finding the right information from almost a billion of websites is considerably hard, but finding the accurate and quality one is even harder. Hence, the need of website categorization's demand is increasing tremendously. Unfortunately, the website categorization techniques in previous works are still immature and not good enough to satisfy the need. Additionally, a training dataset is a limitation of supervised learning algorithm and unsupervised learning algorithm also have a complex algorithm. Regularly, they can categorize into only 1 category but the content usually contains various types. Therefore, in this paper, we propose the simple yet powerful algorithm for website categorization which can give a multi-category results with confidence level, distributed systems supported, and does not even need to be trained because the algorithm uses word frequency in the content of each website to match with the categories in a pre-defined dictionary. The result shows that the accuracy of our proposed algorithm is over 95% when tested with Reuters dataset. The comparison of our algorithm and another Text Analysis API shows that our algorithm has more accuracy with less computation time. The accuracy can also be increased by improving the pre-defined dictionary and filtering noise words.