BPE beyond Word Boundary: How NOT to use Multi Word Expressions in Neural Machine Translation
Dipesh Kumar, Avijit Thawani · 2022
BPE tokenization merges characters into longer tokens by finding frequently occurring contiguous patterns within the word boundary.An intuitive relaxation would be to extend a BPE vocabulary with multi-word expressions (MWEs): bigrams (in_a), trigrams (out_of _the), and skip-grams (he•his).In the context of Neural Machine Translation (NMT), we replace the least frequent subword/whole-word tokens with the most frequent MWEs.We find that these modifications to BPE end up hurting the model, resulting in a net drop of BLEU and chrF scores across two language pairs.We observe that naively extending BPE beyond word boundaries results in incoherent tokens which are themselves better represented as individual words.Moreover, we find that Pointwise Mutual Information (PMI) instead of frequency finds better MWEs (e.g., N ew_Y ork, Statue_of _Liberty, neither•nor) which consistently improves translation performance.