Information retrieval from digital libraries in SQL
Carlos Garcia-Alvarado, CARLOS R. ORDÓÑEZ · 2008
Information retrieval techniques have been traditionally ex-ploited outside of relational database systems, due to storage overhead, the complexity of programming them inside the database system, and their slow performance in SQL im-plementations. This project supports the idea that search-ing and querying digital libraries with information retrieval models in relational database systems can be performed with optimized SQL queries and User-Defined Functions. In our research, we propose several techniques divided into two phases: storing and retrieving. The storing phase includes executing document pre-processing, stop-word removal and term extraction, and the retrieval phase is implemented with three fundamental IR models: the popular Vector Space Model, the Okapi Probabilistic Model, and the Dirichlet Prior Language Model. We conduct experiments using ar-ticle abstracts from the DBLP bibliography and the ACM Digital Library. We evaluate several query optimizations, compare the on-demand and the static weighting approaches, and we study the performance with conjunctive and disjunc-tive queries with the three ranking models. Our prototype proved to have linear scalability and a satisfactory perfor-mance with medium-sized document collections. Our im-plementation of the Vector Space Model is competitive with the two other models.