morphemepiece: Morpheme Tokenization
Jonathan Bratt, Jon Harmon · 2021
Tokenize text into morphemes. The morphemepiece algorithm uses a lookup table to determine the morpheme breakdown of words, and falls back on a modified wordpiece tokenization algorithm for words not found in the lookup table.