Dynamically weighted hidden Markov model for spam deobfuscation
Seunghak Lee, Iryoung Jeong, Seungjin Choi · 2007
Spam deobfuscation is a processing to detect obfuscated words appeared in spam emails and to convert them back to the original words for correct recognition. Lexicon tree hidden Markov model (LT-HMM) was recently shown to be useful in spam deobfuscation. However, LT-HMM suffers from a huge number of states, which is not desirable for practical applications. In this paper we present a complexity-reduced HMM, referred to as dynamically weighted HMM (DW-HMM) where the states involving the same emission probability are grouped into super-states, while preserving state transition probabilities of the original HMM. DW-HMM dramatically reduces the number of states and its state transition probabilities are determined in the decoding phase. We illustrate how we convert a LT-HMM to its associated DW-HMM. We confirm the useful behavior of DW-HMM in the task of spam deobfuscation, showing that it significantly reduces the number of states while maintaining the high accuracy. 1