BPE: How Models See Text

When you type a sentence into an LLM, the model never sees your letters. It sees a sequence of integers. Before any attention head fires or any logit is computed, a tokenizer chops your text into pieces called tokens and looks up each one’s ID in a fixed vocabulary. The word “tokenization” might become ["token", "ization"] → [3323, 2065]. Everything the model knows about language, it knows in terms of these tokens. It has no direct access to the characters underneath.

The algorithm that builds this vocabulary is Byte-Pair Encoding (BPE). It’s a small, almost mundane piece of machinery — a greedy loop that repeatedly merges the most common adjacent pair of symbols — but it sits underneath every modern LLM, and its quirks leak upward into behavior you’ve probably noticed: models that can’t count the letters in “strawberry,” that stumble on arithmetic, that cost twice as much to run in Japanese as in English. This post is about how BPE works and why those quirks are a direct consequence of it.

The Vocabulary Dilemma

Before BPE, there are two obvious ways to turn text into tokens, and both are bad.

Character-level. Give every character its own token. The vocabulary is tiny (a few hundred entries) and can represent anything. But sequences become enormous — a 1,000-word document is ~5,000 tokens — and the model has to learn everything about word structure from scratch, one character at a time. Attention is quadratic in sequence length, so long sequences are expensive.

Word-level. Give every word its own token. Sequences are short, but the vocabulary explodes. English has hundreds of thousands of words, plus every misspelling, name, and neologism. Worse, you inevitably hit words at inference time that weren’t in your training vocabulary — the dreaded out-of-vocabulary problem — and all you can do is emit an <UNK> token and lose the information.

BPE threads the needle. It’s a subword tokenizer: common words get a single token, rare words get split into a few meaningful pieces, and truly novel strings fall back to smaller and smaller fragments until, in the worst case, individual bytes. Frequent things are cheap, rare things are still representable, and nothing is ever unrepresentable. A typical vocabulary lands around 50,000–200,000 tokens, and English text averages roughly four characters per token.

The Algorithm

BPE was originally a data compression algorithm from 1994, repurposed for tokenization by Sennrich et al. in 2015. Training a tokenizer means learning a set of merge rules from a corpus. The procedure is:

1. Start with a base vocabulary of individual symbols (characters, or bytes — more on that below).
2. Split every word in the corpus into those symbols.
3. Count every adjacent pair of symbols across the corpus.
4. Merge the single most frequent pair into a new symbol, and record the merge.
5. Repeat from step 3 until you reach your target vocabulary size.

Consider a toy corpus of five words with these frequencies: hug (10), pug (5), pun (12), bun (4), hugs (5). We begin with each word as a sequence of characters:

h u g     ×10
p u g     ×5
p u n     ×12
b u n     ×4
h u g s   ×5

Now count adjacent pairs. (u, g) appears in hug, pug, and hugs: 10 + 5 + 5 = 20 times. (u, n) appears in pun and bun: 12 + 4 = 16. (p, u) shows up 17 times, (h, u) 15 times. The winner is (u, g) at 20, so we merge it into a new token ug and record the rule u + g → ug:

h ug      ×10
p ug      ×5
p u n     ×12
b u n     ×4
h ug s    ×5

Recount. Now (u, n) leads at 16, so we merge u + n → un:

h ug      ×10
p ug      ×5
p un      ×12
b un      ×4
h ug s    ×5

Recount again. (h, ug) now appears 15 times (in hug and hugs), beating everything else, so we merge h + ug → hug. After three merges our learned rules are, in order:

u + g  → ug
u + n  → un
h + ug → hug

The order matters enormously, because it’s not just training output — it’s the program the tokenizer runs at inference time.

Encoding With a Trained Tokenizer

Once the merge rules exist, encoding new text is deterministic. Split the input into base symbols, then apply merge rules in priority order — earliest-learned first — repeatedly, until no rule applies. The learned rules are essentially a ranked list, and encoding is: “find the highest-priority merge that’s currently applicable, apply it everywhere, repeat.”

To tokenize the new word bug with the rules above: it starts as b u g. The highest-priority applicable rule is u + g → ug, giving b ug. No further rule applies (there’s no rule starting with b), so bug tokenizes as ["b", "ug"]. The word hug collapses all the way to a single token ["hug"], while bug — which never appeared in training — still gets a sensible two-piece split. That’s the whole trick: frequency in the training corpus buys you shorter token sequences.

Why Bytes, Not Characters

The classic version of BPE operates on Unicode characters. That works until your input contains a character you’ve never seen — an obscure CJK ideograph, a rare emoji, a symbol from a script that wasn’t in your corpus. You’re back to the out-of-vocabulary problem at the character level.

GPT-2 solved this with byte-level BPE. Instead of starting from characters, it starts from the 256 possible byte values. Any string in any language, any emoji, any binary garbage, is ultimately a sequence of bytes, so a base vocabulary of 256 byte-tokens can represent literally anything — there is no such thing as an out-of-vocabulary input. BPE merges then build up from bytes rather than characters. A common English word becomes one token; a rare Unicode character that takes three bytes in UTF-8 becomes, in the worst case, three byte-tokens, but it’s always representable.

Two implementation details ride along with this. First, before BPE runs, GPT-2 applies a pre-tokenization regex that splits text on whitespace and punctuation boundaries. This prevents merges from ever spanning across words — you never get a single token for "the cat" — which keeps the vocabulary focused on within-word structure. Second, leading spaces are folded into the following token: the tokenizer represents the space before a word as part of that word’s token (rendered as Ġ in GPT-2’s debugging output). This is why "hello" and " hello" are different tokens, and why token counts can shift depending on spacing.

The previous post on special tokens covered the other end of the vocabulary: hand-crafted tokens like <|im_start|> that are injected above the BPE range and never produced by merges. BPE builds the natural-language bulk of the vocabulary; special tokens are the manually-added structural grammar bolted on top.

The Consequences Leak Upward

Here’s where tokenization stops being an implementation detail and starts explaining model behavior.

Models can’t see letters

Ask a model how many R’s are in “strawberry” and it has historically struggled. The reason is structural: the model never sees s-t-r-a-w-b-e-r-r-y. It sees something like ["str", "aw", "berry"] — three opaque integers. The individual letters are fused inside the tokens, invisible to the model unless it has separately memorized the spelling of each token from training text. Counting characters, reversing strings, detecting rhymes, doing pig latin — every task that requires character-level access is fighting against the tokenizer, which threw that information away. It’s not that the model is dumb; it’s that you asked it to count something it can’t see.

Arithmetic is at the mercy of digit splits

How a tokenizer chops numbers has an outsized effect on arithmetic. Early tokenizers split numbers inconsistently — 1234 might be one token, while 1235 splits as ["123", "5"], purely as an accident of which digit sequences were frequent in the training corpus. That inconsistency makes it hard for a model to learn place-value algorithms, because “the same” number wears a different token costume depending on its digits. Newer tokenizers impose regularity: many now force numbers to split into individual digits or fixed three-digit groups, sometimes right-to-left so that place value lines up. It’s a small change to the tokenizer that measurably improves arithmetic, precisely because it gives the model a consistent representation to compute over.

Whitespace and code

GPT-2 was mediocre at code, and part of the reason was tokenization. Each run of indentation spaces became a pile of separate space-tokens, inflating sequence length and burying structure. Later tokenizers added dedicated tokens for common whitespace runs (two spaces, four spaces, a tab), which both shortens code sequences and gives the model cleaner signal about nesting. Tokenizer design and coding ability turn out to be linked.

Multilingual inequality

Because BPE merges are learned from a training corpus that is overwhelmingly English, English gets the most efficient representation — common English words are single tokens. Text in languages with non-Latin scripts fragments much harder, often down toward the byte level, because those character sequences were rare during merge training. The practical result is a fairness and cost problem: the same meaning expressed in, say, Burmese or Telugu can consume several times as many tokens as its English translation. Since you pay per token and the context window is measured in tokens, non-English users get less effective context and higher bills for identical content. The tokenizer quietly encodes the demographics of its training data.

Glitch tokens

The strangest consequence is glitch tokens. The most famous is SolidGoldMagikarp. It exists as a single token in GPT-2/GPT-3’s vocabulary — which means the string was frequent enough during tokenizer training to earn its own merge. It turned out to be a Reddit username, common in a counting subreddit that was scraped for tokenizer training but then largely filtered out of the language model’s training data. So the token existed in the vocabulary, but its embedding was almost never updated during model training. Asking the model to repeat SolidGoldMagikarp produced bizarre results — evasion, insults, hallucinated other words — because the model was being asked to reason about a token it had essentially never seen used. Glitch tokens are a direct fingerprint of the split between the tokenizer’s training data and the model’s.

Takeaway

BPE is a greedy compression algorithm doing an unglamorous job: find the most common adjacent pair, merge it, repeat, until you have a vocabulary that balances short sequences against a manageable number of tokens. Byte-level BPE makes that vocabulary universal by building up from raw bytes, so nothing is ever unrepresentable. It’s an elegant solution to the vocabulary dilemma, and it’s under every model you use.

But the tokenizer is also a lens with distortions, and the model only ever sees the world through it. The letters fused inside a token are invisible; the digits chopped one way learn differently than the digits chopped another; the languages underrepresented at merge-training time pay a permanent tax; and a username that slipped through the corpus filter becomes a haunted token. When a model does something inexplicable with text, the tokenizer is often the first place to look — because for the model, the tokens are the text.