A Method to Do Lexical Analysis while Word-position Tagging
YU Yue-meng · Computer Knowledge and Technology · 2012
A new kind of tags for the word-position tagging.word-position tagging and lexical analysis integration can be achieved using the new tags.the new kind of tags contains both the lexical information and the word-position information,before the training process,modify the trainning data using the new tags,then train the word segment model with the modified trainning data,after the trainning fin ished,use the model to do a word segment process on a sentence,then get the result from the model.the result will be both the word seg ment result and the lexical analysis result.This article also changed the viberti algorithm of HMM into N-viberti,which can find the top N results of a word segment graphy.Experiments show that through the model,we can get a good word segment result,and a lexical analysis result in the same time,though the accuracy is still relatively low.