Every large language model begins with the same quiet step: turning text into numbers. The algorithm that does it is called Byte Pair Encoding, and it is small enough to fit on one screen. This tutorial builds it line by line — you press step, the line lights up, and you watch exactly what it does to the data.
Each section has a stepper: real code on the left, a running commentary and the live variable state on the right. Press Step to advance one line at a time, or Play to watch it run. Nothing is pre-recorded — the algorithm genuinely executes as you step, and the highlighted line is the line producing what you see.
A transformer is a stack of matrix multiplications. Matrices consume numbers, not letters.
So before a single layer of an LLM runs, some piece of code must convert
"The cat sat" into something like [976, 9059, 7731].
That piece of code is the tokenizer, and it is a completely separate program from the
model — with its own training data, its own training algorithm, and its own saved file.
The tokenizer exposes exactly two operations to the rest of the system:
encode(text) → ids going in, and decode(ids) → text coming out.
Everything in between is the model's business. Everything about how text becomes those ids
is the tokenizer's business — and it turns out that a surprising number of an LLM's
strangest failures are the tokenizer's fault, not the model's. We will come back to that
in Part 11, once you can read the code that causes it.
The simplest possible tokenizer. Collect every distinct character in your corpus, sort them, and number them. For a corpus of English text you end up with roughly 100 symbols — a tiny, tidy vocabulary.
The catch is sequence length. Every single character costs one slot in the model's context window, and attention cost grows with the square of the sequence length. A 2,000-word essay becomes ~11,000 tokens. You have spent your entire context on a few pages.
Sequences get beautifully short — one token per word. But now the vocabulary explodes to hundreds of thousands of entries, and the model needs an embedding row for every one of them.
Worse, it is brittle. Any word not seen during training — a typo, a new product name,
a rare surname — has no id at all. And run, running,
runs become three unrelated ids that share nothing.
Small vocabulary → long sequences → expensive attention and less text per context window.
Large vocabulary → short sequences → an enormous embedding table, and rare tokens that
almost never get trained.
BPE is the compromise. It starts from the smallest honest vocabulary that can
represent anything at all — the 256 byte values — and then buys shorter sequences
by spending vocabulary slots on whichever character sequences actually turn out to be
common in real text. You choose the budget. The algorithm spends it optimally-ish, greedily,
one merge at a time.
Drag the vocabulary budget. Watch what it costs and what it buys. (Numbers are modelled on realistic English text; the shape of the curve is what matters, not the third decimal.)
Before BPE can merge anything, we need a starting alphabet that can represent any text — every language, every emoji, every symbol — without ever failing. Python gives us two candidates, and only one of them is practical.
A Python string is a sequence of Unicode code points. ord("A") is
65; ord("🐱") is 128049. Universal, but the
alphabet has ~150,000 entries before we have learned anything — most of them
vanishingly rare — and the standard gains new characters every year, so the vocabulary
would be a moving target.
UTF-8 encodes any code point as 1–4 bytes, and a byte has only 256 possible values. The alphabet is small, fixed forever, and nothing is unrepresentable. ASCII characters cost one byte each; other scripts cost two to four.
This is why the base vocabulary of a BPE tokenizer is always exactly 256, and why
token ids for learned merges start at 256.
Type anything — try your name, then some emoji, then a non-Latin script — and watch the two representations diverge.
If you write in English, one character costs one byte. If you write in Japanese, Hindi or Arabic, one character often costs three. Before a tokenizer learns anything at all, non-English text is already three times longer in the units BPE operates on — and because tokenizers are trained mostly on English, they will also learn far fewer merges that help those languages. Both effects compound. This is the mechanical reason LLMs are worse, slower and more expensive in most of the world's languages.
Find the pair of adjacent symbols that occurs most often, replace every occurrence of that pair with a brand-new symbol, and repeat. That is the entire algorithm. It was published in 1994 as a data-compression trick; it now underpins essentially every production language model.
aaabdaaabacThe classic textbook example. Our string is 11 characters. We will run three merges and watch it shrink to 5 tokens. Press step to advance one merge at a time.
Merge 2 combines token 256 — which did not exist a moment ago — with
a. New tokens are built out of previously-built tokens, so a single token
id can unpack into a whole tree of bytes.
At each step we take the currently most frequent pair. We never reconsider. This is not guaranteed to produce the best possible vocabulary — it is simply fast, deterministic, and good enough.
Training produces one thing: an ordered list of merge rules. That list is the tokenizer. Everything else — the vocabulary, the encoder, the decoder — is derived from it.
In merge 1 the pair (a,a) is counted 4 times, and the run
aaa becomes [256, a] — not [256, 256]. Merging walks
left to right and consumes two symbols at a time, so overlapping occurrences cannot both
be taken. The leftover a is exactly what merge 2 then picks up. Watch for it
in the animation above.
get_statsTo merge the most common pair, we first have to count pairs. This function takes a list of token ids and returns a dictionary mapping every adjacent pair to how many times it occurs. It is five lines long, and one of those lines is the cleverest line in the whole algorithm.
Step through it below. The commentary on the right changes with the highlighted line, and the strip underneath shows which two tokens are being looked at right now.
zip(ids, ids[1:])
ids is the list. ids[1:] is the same list shifted left by one.
Zipping them pairs each element with its right-hand neighbour — every adjacent pair, in
order, with no index arithmetic. A list of n items yields n−1 pairs.
counts.get(pair, 0) + 1
"Look up the current count for this pair; if we have never seen it, treat it as 0; add
one; store it back." A defaultdict or Counter would do this
automatically, but writing it out keeps the mechanics visible.
In aaa, this function reports (a,a): 2, because positions
(0,1) and (1,2) are both adjacent pairs. But when we merge, we can only take one
of them. Counting and merging use different rules, and that mismatch is deliberate — it
keeps both functions trivially simple. It is also why frequency counts drop faster than
you might predict as training proceeds.
mergeNow the other half. Given the list of ids, a pair to replace, and the new id to replace it with, walk the list and rewrite it. This is the function that actually makes the sequence shorter, and it contains the single most common off-by-one bug in the algorithm.
i < len(ids) - 1?
The condition reads ids[i+1]. If i is the last index, that
would run off the end of the list. The guard stops us one position early. Forget it and
you get an IndexError on every input whose final token happens to match
the first half of the pair.
i += 2 vs i += 1?
On a match we consume both symbols, so we skip forward two. Otherwise we copy
one symbol and advance one. This is precisely what makes merging non-overlapping and
left-to-right greedy — and what leaves the stray a behind in
aaa.
merge([7,7,7,7], (7,7), 99) return?i=0 the pair matches, so we append 99 and jump to
i=2. At i=2 the pair matches again, so we append 99 and
jump to i=4, which ends the loop. Four symbols became two. With an
odd run — [7,7,7] — you would get [99, 7], with the last
one stranded.
trainWe have a counter and a rewriter. Training is just the two of them in a loop: count, pick the winner, mint a new id, rewrite, record. Run that N times and you have a tokenizer.
vocab_size = 256 + num_merges
You choose vocab_size; it determines how many times the loop runs. GPT-2 used
50,257 = 256 bytes + 50,000 merges + 1 special token. That is the whole budget, laid bare.
merges maps (a, b) → new_id and is used by encode: given
text, which rules apply? vocab maps id → bytes and is used by
decode: given an id, what bytes does it stand for? Both are built in the same loop,
and vocab carries no independent information — it can always be rebuilt from
merges, which is why a saved tokenizer file only needs to store the merge
list.
decodeIds back into text. We do decode before encode because it is the easier direction, and because it exposes an important subtlety about bytes that encode will rely on.
errors="replace" is not optional
A token can hold a fragment of a multi-byte character. If a model emits a token
sequence whose bytes do not form valid UTF-8 — easy to do, since nothing forces the model
to respect character boundaries — a strict decode raises
UnicodeDecodeError and your application crashes. With
errors="replace" you get the � replacement character
instead. This is the mechanism behind those stray � symbols you
occasionally see in streamed LLM output: the text arrived mid-character.
encodeText into ids — and the one place where the algorithm does something genuinely subtle. Training picked the pair with the highest count. Encoding must pick the pair with the lowest merge index. Getting this backwards is the classic BPE bug.
The reason: merges were learned in a specific order, and later merges are built out of tokens that earlier merges created. Merge #400 might combine two ids that only exist because merges #12 and #77 ran first. So to encode new text we must replay the merges in the order they were learned, earliest first, until no learned rule applies any more.
self.merges.get(p, inf)
Score each candidate pair by when it was learned. Pairs we never learned score
infinity, so min will never pick one unless nothing else is available —
which is exactly the case the next line catches.
if pair not in self.merges: break
When every pair scores infinity, min still returns something — arbitrarily,
the first one. Without this membership check we would merge a pair we never learned and
corrupt the output. It is the loop's real termination condition.
(a,b)→256 first, then (256,c)→257.
You encode "abc". Why can't we just scan for (256,c) straight away?
[a, b, c] — there is no
256 anywhere in it. Only after the first merge rewrites the list to
[256, c] does the pair (256, c) physically exist to be matched.
Merge order is a dependency chain, and encode walks it from the beginning.
Everything above, running at full speed on real text. Edit the corpus, choose a vocabulary
budget, and train. The merges stream in as they are learned — watch how the early ones are
all boring high-frequency fragments (e , th,
t) and how real words only start appearing after a few dozen.
= 256 bytes + 20 merges
Train to verify decode(encode(t)) == t
Compression climbs steeply at first, then flattens — the first few hundred merges do most of the work, and each additional merge buys less than the one before. That curve is the real reason vocabulary sizes settled around 50k–100k rather than a million: past a point you are paying for embedding rows that barely shorten anything. Try the Python corpus and watch indentation get eaten. Try mixed languages and watch how few merges the non-English text earns.
The algorithm as written has a flaw. Nothing stops it merging dog with
. to make a dog. token, and separately dog! and
dog? and dog,. The model then has to learn four unrelated ids that
all mean the same animal, and we have wasted vocabulary encoding punctuation four times.
The fix used by GPT-2 onward: before training, split the text with a regular
expression into chunks — words, numbers, punctuation runs, whitespace — and only ever
merge within a chunk. Pairs that straddle a chunk boundary are never counted, so
they can never be merged. Two small changes to train, and the vocabulary gets
dramatically cleaner.
Each coloured box is a chunk. Merges can happen inside a box, never between boxes. Compare how the two generations of pattern handle the same text.
| Improvement | Effect |
|---|---|
| Case-insensitive contractions | GPT-2 split HOW'S into three pieces but how's into two. GPT-4 handles both identically. |
| Digits capped at 3, no leading space | \p{N}{1,3} forces consistent digit grouping. GPT-2 could swallow a ten-digit number whole, giving the model no consistent structure for arithmetic. |
| Better whitespace and newlines | GPT-2 turned each run of indentation into its own tokens, which is why it was disproportionately bad at Python. GPT-4 groups them sensibly. |
These patterns use \p{L} and \p{N} — Unicode property escapes
for "any letter" and "any number" in any script. Python's built-in re module
does not support them; the third-party regex package does. If you implement
this yourself, pip install regex and import regex as re.
The patterns also use possessive quantifiers (?+, ++)
and inline case-insensitive groups ((?i:…)), which JavaScript does not
support — so the live demo above uses close equivalents. The chunk boundaries match the
real patterns on ordinary text, but treat it as an illustration, not a reference
implementation.
You can now read the code that causes most of the famous failure modes. None of these are failures of the transformer. All of them are consequences of the five functions you just stepped through.
| The symptom | The tokenizer explanation |
|---|---|
| Can't spell, can't count letters | Ask how many rs are in "strawberry" and the model may not see letters at all — the word arrives as two or three opaque ids. The character identities were consumed by merge during training and are not recoverable from the id alone. |
| Can't reverse a string | Same cause. Reversal is a character operation; the model only has chunks. |
| Bad at arithmetic | Digit grouping is arbitrary. Whether 677 is one token or two depends on frequency counts from the tokenizer's training corpus, not on arithmetic structure. Column-aligned addition is exactly the thing this destroys. GPT-4's \p{N}{1,3} is a partial mitigation. |
| Worse in non-English languages | Two compounding effects, both visible in Part 2 and Part 9: UTF-8 already makes the text 2–3× longer in bytes, and a mostly-English training corpus means few merges are learned that would shorten it again. More tokens per idea means more cost, less context, and weaker performance. |
| GPT-2 was oddly bad at Python | Its regex turned runs of indentation into separate tokens, so every indented line burned context on whitespace. A pure pre-tokenization problem, fixed in GPT-4. |
| Trailing whitespace breaks things | The regex attaches a space to the word that follows it, so " hello" is one natural token. End your prompt with a space and you have created a token combination the model rarely saw in training — it is now off-distribution. |
Typing <|endoftext|> can break an app |
Special tokens bypass BPE entirely and are matched as literal strings before the algorithm runs. Untrusted user text that contains one can inject a control signal into the stream. |
The SolidGoldMagikarp phenomenon |
The clearest proof that the tokenizer is a separate program. A username frequent enough in the tokenizer's corpus earned its own token — but that data was filtered out of the model's corpus, so its embedding row was never meaningfully trained and stayed near random initialization. Feeding it in produced bizarre, undefined behaviour. Train the two on different data and this is what you get. |
| YAML is cheaper than JSON | JSON's dense punctuation tokenizes less efficiently. Same information, more tokens, higher bill. |
The tokenizer is trained separately from the model, on possibly different data, using a greedy frequency heuristic that knows nothing about meaning, spelling, or arithmetic. Every item in that table follows from those facts. When an LLM does something inexplicable with text, "check the tokenizer" is a genuinely good first instinct.
You have seen every line. Open a notebook and reproduce get_stats,
merge, train, encode, decode from
memory. Train on any text file you have. Assert
decode(encode(t)) == t.
Upgrade train to work on a list of chunks instead of one flat list. Two
changes: sum counts across chunks, and merge within each chunk. Confirm no token ever
crosses a category boundary.
Load GPT-4's published merges with tiktoken and check your
encode produces identical ids. When it does, you have genuinely
reimplemented a production tokenizer.
Credit. This tutorial follows the approach Andrej Karpathy takes in his lecture "Let's build the GPT Tokenizer" and the accompanying minbpe repository (MIT licensed) — build the smallest honest version of the algorithm, then explain every line of it. The code here is written in that spirit and stays close to his structure so that the repository reads as familiar when you get there. The BPE algorithm itself is due to Philip Gage (1994), adapted to language-model tokenization by Sennrich, Haddow & Birch (2016).
Further reading. karpathy/minbpe · The lecture (2h13m) · tiktokenizer (paste text, see real GPT tokens) · openai/tiktoken
Prepared for Alayna Leck · INQUIRE Lab · Summer 2026 · Questions to the lab coordinator.