Why the Tokenizer Prefers English

A tokenizer is a compression scheme, and compression schemes reward whatever was seen most often.

That sentence is the entire mechanism. Byte-pair encoding takes a corpus, counts adjacent pairs, merges the most frequent ones, and repeats until a vocabulary budget is exhausted. Tokens that mark frequent strings become short. Tokens that mark rare strings stay long, or split into bytes. Nothing in the procedure contains a theory of language. The preference for English, where it appears, is arithmetic.

The simplest tokenizer: split on spaces#

The naive approach treats a word as a unit and whitespace as the delimiter.

def naive_tokenize(text: str) -> list[str]:
    return text.split()
 
def naive_count(text: str) -> int:
    return len(naive_tokenize(text))
 
for sentence in [
    "The quick brown fox jumps",
    "नमस्ते संसार",
    "ሰላም ዓለም",
]:
    print(sentence, naive_count(sentence))

The space-split works for toy demonstrations and fails immediately in production. Three bottlenecks appear at once.

• Punctuation attaches to words: "fox," and "fox" become different tokens.
• Unknown words at inference time have no vocabulary entry and fall out entirely.
• The vocabulary grows without bound: every new name, typo, and hashtag needs a new entry.

The bottleneck is coverage. A closed vocabulary of whole words cannot represent open-ended text. The fix begins by dropping down a level, to characters and then to bytes.

Bytes are the floor#

Every text string, in every language and script, is stored as bytes before it reaches a model. A byte-level tokenizer uses that fact.

def byte_tokenize(text: str) -> list[int]:
    return list(text.encode("utf-8"))
 
for sentence in ["hello", "नमस्ते", "ሰላም"]:
    encoded = byte_tokenize(sentence)
    print(sentence, encoded, len(encoded))

Byte tokenization has perfect coverage: any UTF-8 string can be encoded, in any language, with zero unknown tokens. The bottleneck is length. An English word of five characters costs five tokens. A Devanagari or Ge'ez word costs more per character because each glyph occupies multiple bytes, and each byte becomes a separate token. The window fills with fragments.

The next step is compression without losing coverage: keep the byte floor, but learn multi-byte units for frequent strings.

BPE from first principles#

Byte-pair encoding is a compression loop with two inputs: a corpus of bytes and a vocabulary budget. The loop runs as follows.

from collections import Counter, defaultdict
 
def get_pairs(splits: list[list[str]]) -> Counter:
    pairs = Counter()
    for word in splits:
        for i in range(len(word) - 1):
            pairs[(word[i], word[i + 1])] += 1
    return pairs
 
def train_bpe(corpus: list[str], num_merges: int) -> dict[tuple[str, str], str]:
    splits = [list(word) + ["</w>"] for word in corpus]
    merges: dict[tuple[str, str], str] = {}
    for _ in range(num_merges):
        pairs = get_pairs(splits)
        if not pairs:
            break
        most_common = pairs.most_common(1)[0][0]
        new_token = "".join(most_common)
        merges[most_common] = new_token
        for i, word in enumerate(splits):
            j = 0
            while j < len(word) - 1:
                if (word[j], word[j + 1]) == most_common:
                    word[j:j + 2] = [new_token]
                else:
                    j += 1
    return merges
 
merges = train_bpe(["the ", "the ", "the ", "thing ", "thing "], num_merges=5)
print(merges)

Three properties follow from the loop structure.

• The first merges are always the most frequent pairs in the corpus.
• Each merge creates a longer token, so later merges build on earlier ones.
• Training stops at a fixed vocabulary size, so low-frequency strings never earn a merge.

The loop contains no knowledge of English. Change the corpus and the merge order changes. In an English-heavy corpus, " th", " the", "ing", and "tion" merge early and become single tokens. Pairs from other scripts merge late, or never.

Bottleneck 1: a finite vocabulary spends its budget on frequency#

Vocabulary size is the hard constraint. A tokenizer with 50k entries cannot give every language its common substrings. The allocation rule is frequency in the training corpus, not linguistic regularity.

def tokenize_with_merges(text: str, merges: set[str]) -> list[str]:
    tokens = list(text)
    changed = True
    while changed:
        changed = False
        for token in sorted(merges, key=len, reverse=True):
            if token in "".join(tokens):
                text = "".join(tokens).replace(token, f"\0{token}\0")
                tokens = [t for t in text.split("\0") if t]
                changed = True
    return tokens
 
english_merges = {" the", "ing", "tion", " th"}
print(tokenize_with_merges(" the thing", english_merges))
print(tokenize_with_merges("नमस्ते संसार", english_merges))

An English word frequently occupies one token. A Nepali or Amharic or Burmese word that rarely appeared in training remains as bytes or small fragments. Morphological regularity does not help, because the loop counts surface frequency, not structure.

The measurable consequence is called fertility: the average number of tokens required per word, per character, or per fixed passage, in a given language. Low fertility means long learned tokens. High fertility means fragmentation.

Measuring fertility#

Fertility turns the allocation into a number that can be compared across languages.

def fertility(passages: list[str], tokenizer) -> float:
    total_tokens = sum(len(tokenizer.encode(p)) for p in passages)
    total_words = sum(len(p.split()) for p in passages)
    return total_tokens / max(total_words, 1)
 
passages_en = ["The history of the city is well documented."]
passages_ne = ["शहरको इतिहास राम्ररी दस्तावेज गरिएको छ।"]
 
# A tokenizer trained on English-heavy data produces, illustratively:
print("en fertility ≈ 1.0 tokens/word")
print("ne fertility ≈ 2.5–4.0 tokens/word")

Fertility is an average over a passage, not a property of a single word. The important comparison holds the passage fixed and varies the language. A sentence that costs 20 tokens in English may cost 60 to 80 tokens in a language the tokenizer never compressed well. Both sequences enter the model as integer IDs. The model sees only the length difference.

Bottleneck 2: fragmentation costs more than length#

Length is the first cost. A high-fertility passage puts more tokens between the same start and end of an idea, so copying, reasoning over distance, and staying inside a context limit all become harder. The second cost is fragmentation itself.

# The same idea, at two fertility levels
ideas = {
    "compact": ["नमस्ते"],            # one token carries the word
    "fragmented": ["न", "म", "स्ते"], # several tokens must combine
}
 
for name, tokens in ideas.items():
    print(name, "tokens:", len(tokens), "whole-token identity:", name == "compact")

A word broken into five tokens has no single embedding. The representation must be assembled by the model from fragments, and assembly for rare fragments is less reliable than retrieval of a whole token. Rare fragments also appear in few training contexts, so the statistics learned for them are thinner.

Neither cost is a verdict on the model. Training distribution compounds tokenizer distribution: a corpus dominated by English also trains English embeddings more thoroughly. The two preferences stack, which is why fertility gaps persist even after multilingual training begins.

Fix (a): byte-level vocabulary with a larger floor#

One fix enlarges the base vocabulary so that common byte sequences in more scripts exist from the start.

from transformers import GPT2TokenizerFast
 
tok_large = GPT2TokenizerFast.from_pretrained("gpt2")
assert tok_large.vocab_size >= 50_000
 
for text in ["The history of the city", "नमस्ते संसार", "ሰላም ዓለም"]:
    ids = tok_large.encode(text)
    print(text, "->", len(ids), "tokens", ids)

A byte-level vocabulary of 50k entries still allocates most entries by frequency, but the floor is higher than a 256-byte fallback. Frequent multi-byte sequences in lower-resource scripts can exist as tokens if the training corpus contained enough of them. The fix helps only if the corpus already covered the script. Vocabulary size alone does not correct an imbalanced corpus.

Fix (b): reweight the corpus before merging#

The direct fix changes what the loop counts. Instead of sampling the corpus proportionally to availability, the training sample upweights low-resource languages.

import random
 
def sample_multilingual(pools: dict[str, list[str]], total: int) -> list[str]:
    # Equal share per language instead of share per available text
    per_lang = max(1, total // len(pools))
    sample: list[str] = []
    for lang, texts in pools.items():
        sample.extend(random.choices(texts, k=per_lang))
    return sample
 
pools = {
    "en": ["The history of the city."] * 1000,
    "ne": ["शहरको इतिहास।"] * 100,
    "am": ["ሰላም።"] * 50,
}
balanced = sample_multilingual(pools, total=3000)
print(len(balanced))

Equal sampling shifts merges toward lower-resource scripts and lowers their fertility, at a small cost to English fertility. The trade exists because the vocabulary budget is fixed. A script-aware tokenizer, per-language vocabularies, or a larger byte-level vocabulary move the trade but do not remove the budget.

What to measure#

Fertility is the primary metric, measured per language, per script, and per domain:

def fertility_suite(passages_by_lang: dict[str, list[str]], tokenizer) -> dict[str, float]:
    results = {}
    for lang, passages in passages_by_lang.items():
        tokens = sum(len(tokenizer.encode(p)) for p in passages)
        chars = sum(len(p) for p in passages)
        results[lang] = tokens / max(chars, 1)  # tokens per character
    return results
 
benchmark = {
    "en": ["The committee met on Tuesday to discuss the report."],
    "ne": ["समिति मङ्गलबार प्रतिवेदन छलफल गर्न भेला भयो।"],
    "am": ["ኮሚቴው ማክሰናት ለሪፖርት ለመወያየት ተሰብስቧል።"],
}
print("Run the suite before and after any tokenizer change.")

Four checks cover most failures. Fertility per language on identical passages. Unknown-token rate, which must stay at zero for byte-level systems. Sequence-length inflation, the ratio of token count to character count. Downstream task accuracy per language, because fertility is a proxy and accuracy is the outcome.

A checklist for a tokenizer change:

• Report fertility for every supported language, not a global average.
• Verify byte-level fallback: any UTF-8 string encodes without errors.
• Compare token counts on identical passages across languages.
• Check downstream accuracy per language, not overall accuracy.
• Record corpus proportions; the merge order follows from them.

Fertility measures the distance between a language and the corpus the tokenizer was fitted on. Low fertility for English is not evidence that English is more tokenizable. It is evidence that the merges were fitted to English text. Change the corpus and the gap moves. Change the procedure and the gap moves again. The preference is contingent on what was counted, in what proportion, and for how long.

The vocabulary is a map of the training corpus drawn in token lengths. English sits near the center of that map because the corpus was centered there.