Development and Implementation of a Novel Hybrid Stemmer for Punjabi NLP: Integrating Rule-Based and Dictionary-Based Algorithms
Gurpej Singh, Rahul Bhandari, Prabhdeep Singh · 2024
This work introduces an advanced Punjabi language stemmer, marking a significant advancement in Natural Language Processing (NLP) capabilities for one of the most extensively spoken regional languages in India. Achieving an unparalleled accuracy of 92.5%, with only 3%(approx) overstemming and 4.5% understemming, our stemmer uniquely combines rule-based and dictionary-based methodologies to tackle the intricate morphological aspects of Punjabi. With more than 300 suffix rules, it adeptly handles the myriad grammatical scenarios intrinsic to the Punjabi language. The stemmer is supported by a comprehensive database containing over 50,000 alphabetically organized words, alongside meticulously curated linguistic resources for pronouns, adverbs, postpositions, vocabulary, place names, and other non-stemmed categories. This hybrid approach ensures meticulous processing and validation of stemmed outputs, meticulously preserving semantic integrity across a wide variety of grammatical scenarios including nouns, pronouns, adjectives, adverbs, verbs, proper names, as well as gender and orthographic variation distinctions. By effectively handling monolingual cases and maintaining the core semantic essence of words, the stemmer represents a monumental leap in linguistic tool development for regional languages. Our contribution, now accessible through Python package via pip install punjabi-stemmer, substantially advances Punjabi text processing technologies, establishing a new standard for linguistic tool development. This achievement not only illustrates the stemmer's adeptness at navigating Punjabi's morphological complexity but also establishes it as an indispensable asset for diverse NLP applications. This dissemination not only underscores our stemmer's pivotal role in enhancing the processing capabilities for morphologically complex languages but also invites widespread adoption and collaboration, promising to catalyze further innovation in the field globally.