A Bangla Text Search Engine Using Pointwise Approach of Learn to Rank(LtR) Algorithm
Adiba Ibnat Hossain, Asaduzzaman Asaduzzaman · 2022
A search engine is the prime source of information in this modern time. Where a good number of monolingual and bilingual search engines are emerging, Bangla language falls behind in the area of information retrieval (IR) in the native language. Currently, Bangladesh has no active search engine as the first ever Bangla search engine Pipilika is not providing service for a long time. To fill the absence of such a platform we aim to initiate the development of a Bangla search engine through this work. We have developed a framework for the Bangla search engine that provides search results by performing syntax analysis of a given query. The information retrieval process of Pipilika was carried out using the score generated by a ranking function called BM25. But modern IR systems are acing with the aid of machine learning. Learn to Rank (LtR) is a class of techniques in machine learning to deal with the ranking problem in IR. The search engines employ any of the three approaches of LtR (pointwise, pairwise, listwise) to show the result on Search Engine Result Page (SERP) and we opted for the pointwise approach. In pointwise approach of LtR a numerical score (bid) is assigned to each keyword-document pair in the training set. Then our ranking problem can be treated as a regression problem. We achieved our pointwise approach by employing Random Forest Regressor which yields an accuracy of 70.14%. The dataset was prepared with the aid of feature extraction following the bid-based ranking approach of Google where we treat the BM25 score as one of the features. In the bid-based approach not only the BM25 score acts as one of the ranking factors but also other ranking factors such as TF-IDF score, keyword density, total words in a document contribute to generating the result. Our approach is capable of providing search results based on the ranking score which is generated by combining all the features.