Tokenizers are stranger than they look

Byte-pair encoding gets described as "merge the most frequent pair, repeat." That is accurate and also hides nearly everything that makes a real tokenizer difficult.

Whitespace is load-bearing

Most implementations attach the leading space to the following token, so "hello" and " hello" are different entries in the vocabulary. Get this wrong and every sentence after the first word tokenizes differently than the model expects.

Unicode makes it worse. Working at the byte level instead of the character level means you never fail on an unknown codepoint, but it also means a single emoji can cost four tokens before any merging happens.

The merge table is the model

Once trained, the vocabulary is frozen and the ordering of merges matters as much as their contents. Applying them out of order produces a different tokenization of the same string, which is a genuinely annoying bug to find.