The Token Is Not the Word

A language model vocabulary is a compression scheme learned from text. The units are tokens. Tokens often look like words, so the confusion is natural. They are not words, and the difference matters for languages where the words are not made of the same letters.

The structure follows the gold-standard shape: a naive split on spaces, the byte layer underneath, a byte-pair encoder built from first principles, the fertility arithmetic, then the split between English morphology mismatch and Urdu and Nepali coverage mismatch.

The naive tokenizer: split on spaces#

The simplest tokenizer splits on whitespace. Each word becomes one token. That equality fails on punctuation, contractions, and unknown forms.

def tokenize_naive(text: str) -> list[str]:
    return text.split()
 
sentences = [
    "unhappiness spreads quietly",
    "children playing, trombonist practicing",
    "محبت پرانی کہانی ہے۔",
    "नमस्ते, माया सबैभन्दा पुरानो कथा हो।",
]
 
for s in sentences:
    print(len(s.split()), s.split())

A space split has no mechanism for a word outside the vocabulary. A new name, a hyphenated compound, or a spelling variant becomes an unknown token with no internal structure. The model receives a constant vector where a word should be. The fix must represent any byte sequence with a finite vocabulary.

The byte layer: what text is before tokenization#

Digital text is bytes. Unicode assigns code points. UTF-8 encodes code points in one to four bytes. A byte-level tokenizer starts from 256 byte symbols so any script is representable.

def byte_len(text: str) -> int:
    return len(text.encode("utf-8"))
 
for s in sentences:
    print(byte_len(s), len(s), repr(s))

The asymmetry appears immediately. English letters cost one byte each. Urdu letters in the Arabic block cost two bytes. Devanagari letters cost three bytes. A byte tokenizer starts with unequal raw sizes for equal content. Merges must recover the difference. When the training corpus has little Urdu or Devanagari text, recovery stays incomplete.

def chars_bytes(text: str):
    return len(text), len(text.encode("utf-8"))
 
print("love", chars_bytes("love"))
print("محبت", chars_bytes("محبت"))
print("नमस्ते", chars_bytes("नमस्ते"))

BPE from first principles: frequency builds the vocabulary#

Byte-pair encoding starts with individual bytes and merges the most frequent adjacent pairs until a target size is reached. The training loop fits on one page.

from collections import Counter
 
def train_bpe(corpus: list[list[int]], merges: int):
    vocab = list(range(256))
    merge_list = []
    for _ in range(merges):
        counts = Counter()
        for seq in corpus:
            counts.update(zip(seq, seq[1:]))
        if not counts:
            break
        pair, _count = counts.most_common(1)[0]
        new_id = len(vocab)
        vocab.append(pair)
        merge_list.append((pair, new_id))
        corpus = [merge_into(seq, pair, new_id) for seq in corpus]
    return vocab, merge_list, corpus
 
def merge_into(seq, pair, new_id):
    out = []
    i = 0
    while i < len(seq):
        if i + 1 < len(seq) and seq[i] == pair[0] and seq[i + 1] == pair[1]:
            out.append(new_id)
            i += 2
        else:
            out.append(seq[i])
            i += 1
    return out
 
def encode_bytes(data: list[int], merge_list) -> list[int]:
    for pair, new_id in merge_list:
        data = merge_into(data, pair, new_id)
    return data

Frequency decides the merge order. Overlap decides which occurrence is consumed first, scanning left to right. Corpus composition decides which language receives long symbols. An English-heavy corpus produces English pieces such as ing, tion, and whole words such as the. A rare true morpheme may never merge. A frequent non-morpheme merges early and survives.

The bottleneck is speed. Each round scans the full corpus, re-counts pairs, and rewrites every sequence. With a large corpus and thousands of merges, the cost grows as merges × corpus length. A production trainer updates counters incrementally around replaced positions instead of rescanning.

def pair_counts(corpus):
    counts = Counter()
    for seq in corpus:
        for j in range(len(seq) - 1):
            counts[(seq[j], seq[j + 1])] += 1
    return counts

What splitting does#

The first consequence is that token boundaries rarely coincide with morpheme boundaries. unhappiness may split as un + happi + ness. children may split as child + ren. A rare word such as trombonist may become five fragments with no fragment tied to the instrument.

long_words = [
    "unhappiness",
    "children",
    "trombonist",
    "nationalization",
    "नमस्ते",
    "محبت",
]
 
for w in long_words:
    print(w, len(w.encode("utf-8")), "bytes", [ord(c) for c in w])

The model copes. Transformers compose tokens into representations, and composition works in practice. Composition still spends capacity on what the user treats as a single unit. The tokenizer prices each word by corpus frequency, not by meaning.

A consistent split helps. A tokenizer that splits happi the same way in happiness, happy, and happily lets the model share statistical strength. An inconsistent split scatters the same morpheme across unrelated token ids. Counting recurring pieces across a word family measures consistency directly.

def recurring_pieces(words, encode) -> Counter:
    pieces = Counter()
    for w in words:
        ids = encode(list(w.encode("utf-8")))
        pieces.update(ids)
    return pieces

Two different problems#

English: morphology mismatch. The script is ASCII-friendly and the corpus is large. The merges align with English morphology often enough that the damage stays mild. The model performs extra composition work that a perfect vocabulary would avoid. The inefficiency is real but bounded.

Urdu and Nepali: coverage mismatch. The script uses a different alphabet. Characters encode in two or three bytes. The corpus contains far less text. The tokenizer falls back to small pieces, sometimes single bytes. The pieces are too small to correspond to morphemes. Splits follow byte lines, not linguistic lines.

The distinction decides the fix. More merges help English marginally. More merges do not fix Urdu or Nepali, because the binding constraint is training data. Merges without data produce a larger vocabulary of rare strings. Data without merges produces long raw byte sequences. Both together change fertility.

The unit budget#

Tokenization budgets sequence length. A fixed context window holds a fixed number of tokens. For English, that is a generous number of words. For Urdu in Arabic script, the same window holds fewer words because each word costs more. Chess notation, code, and tables obey the same arithmetic.

def window_words(context: int, avg_tokens_per_word: float) -> float:
    return context / avg_tokens_per_word
 
for name, tpw in [("english", 1.3), ("urdu_script", 3.0),
                  ("nepali", 4.0), ("roman_urdu", 1.4)]:
    print(name, round(window_words(4096, tpw)))

Each downstream quantity inherits the ratio. Context coverage, latency, training cost, and API cost all scale with token count. A model evaluated at a fixed window therefore receives less content in high-fertility languages. Sentence truncation arrives earlier. Documents are cut sooner. The bias is arithmetic.

What to measure#

Fertility is the first measurement: tokens per word on a parallel corpus, reported by language.

The second measurement is token-level consistency. For each word in a family, record the piece ids. Shared morphemes should share ids. Divergence across the family signals fragmentation.

def family_split(words, encode_ids):
    for w in words:
        print(w, encode_ids(list(w.encode("utf-8"))))

The third measurement is byte fallback rate: the fraction of tokens that decode to a single byte. A high fallback rate marks a coverage gap. A low fallback rate with high fertility marks a different problem: the vocabulary exists but the merges are still short.

Why it persists#

Tokenizers are old, stable, and load-bearing. A pre-training run conditions on a fixed vocabulary. Changing the vocabulary means retraining from the start. The tokenizer chosen at the beginning constrains later decisions long after the original cost-benefit analysis is forgotten. That path dependence turns a corpus statistic into a permanent budget.

A checklist#

• Report fertility by language on parallel text before comparing multilingual scores.
• Track byte fallback rate per language as a direct coverage diagnostic.
• Check split consistency across word families that share a morpheme.
• Budget context, latency, and training tokens in token counts, not word counts.
• Adapt merges and training data together when fixing a coverage gap.
• Treat vocabulary changes as full retrains, not patches.

The representation is a choice. BPE chooses corpus frequency as the criterion. The choice is defensible for compression. The cost per language must be measured before the costs are treated as inevitable.