tok: Fast Text Tokenization
Daniel Falbel · 2023
Interfaces with the 'Hugging Face' tokenizers library to provide implementations of today's most used tokenizers such as the 'Byte-Pair Encoding' algorithm . It's extremely fast for both training new vocabularies and tokenizing texts.