rtiktoken: A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models
David Zimmermann-Kollenda, Roger Zurawicki, Authors of the dependent Rust crates · 2024
A thin wrapper around the tiktoken-rs crate, allowing to encode text into Byte-Pair-Encoding (BPE) tokens and decode tokens back to text. This is useful to understand how Large Language Models (LLMs) perceive text.