Optimal N-gram Subset Extraction for Accelerating Evaluation Using Genetic Algorithm
Gaurav Goel, Harsh Bhardwaj, Ishika Hooda, Shailender Kumar · 2020 International Conference for Emerging Technology (INCET) · 2020
One of the standard methods of processing natural language in computational linguistics is to apply the ngram model. An n-gram model extracts unigrams, bigrams, trigrams, etc. from texts which are treated as features for various tasks such as feature selection in classification. One of the significant drawbacks of the n-gram model is that the number of n-grams generated can scale exponentially with the size of processing text. This results in complex processing, thus making the n-gram model infeasible on systems with limited computational resources. Here, we discuss a novel strategy to select an optimal subset of n-grams using a genetic algorithm. The subset so generated will be able to perform classification with similar accuracy to the original set and in significantly lower time. The method also leads to reduced memory & computational requirements, thus making it feasible for lowerend systems. The proposed algorithm named OSE produces significant performance gains and is seen as a pre-processing step, which opens up a whole new dimension for algorithms that may appear infeasible on a large corpus.