Segmentation based representation for tweet hashtag
Shaik Sharmila, P. Kola Sujatha · 2015
The effectiveness of segmenting the tweet hashtag for recognizing the Named entities is explored. Segmenting the hashtag, can offer huge help in recognizing named entities in the hashtags of tweets. The hashtag is segmented using sliding window of varying length, which provides the possible segmentations. Then the best segments are derived by Viterbiwordseg algorithm which is a dynamic programming algorithm. This algorithm derives the best segment using the unigram probabilities. The unigram language model is based on the Microsoft web N-Gram corpus, by which the probability of a segment is calculated. Named entities are recognized from the segmented hashtag using the method called EntityRegweb. This method recognizes the named entity by the global context, which is derived from the Web Page like Wikipedia. For recognizing the named entities from the segmented hashtag, Wikipedia is referred. Named entities such as person, location and organization is recognized by the method EntityRegweb. These are made in the Hadoop environment such that the larger data set is given as input.