What a Tokenizer Does to Urdu

A tokenizer is a pricing scheme.

It decides what counts as a unit of text, and the unit determines how much context, compute, and memory a sentence costs. Two sentences that say the same thing can have very different prices. The difference is not in the meaning. It is in the script, the vocabulary, and the corpus the tokenizer was trained on.

The structure that follows starts from the simplest tokenizer, derives the byte layer underneath it, trains a byte-pair encoding from scratch, and then measures the cost of Urdu, Nepali, and Roman Urdu under identical code.

The naive tokenizer: split on spaces#

The simplest tokenizer splits a string on whitespace. Each word becomes one token. The implementation is three lines and appears in introductory examples.

def tokenize_naive(text: str) -> list[str]:
    return text.split()

Applied to text, the function produces a flat list:

samples = {
    "english": "love is the oldest story",
    "urdu_script": "محبت پرانی کہانی ہے۔",
    "urdu_roman": "muhabbat purani kahani hai",
    "nepali": "नमस्ते, माया सबैभन्दा पुरानो कथा हो।",
}
 
for name, text in samples.items():
    tokens = tokenize_naive(text)
    print(name, len(tokens), tokens)

The naive tokenizer has one virtue: the token equals the word. Multilingual text, punctuation, and contractions break the virtue immediately. The comma in نमस्ते, attaches to the word. The zero-width joiner, the danda, and the Urdu full stop each sit inside a token or outside it depending on spacing. A model cannot learn from a unit that changes meaning when a space moves.

The naive version also fails on unknown words. A dictionary tokenizer needs the whole vocabulary in advance. A new name, a code-switched word, or a spelling variant produces an out-of-vocabulary token that carries no structure. The model receives a constant vector where a word should be. The next step replaces the dictionary with a compression scheme that can represent anything.

The byte layer: what text is before tokenization#

Digital text is bytes. Unicode assigns each character a code point. UTF-8 encodes each code point in one to four bytes. A byte-level tokenizer starts from bytes rather than characters, so the alphabet has 256 possible symbols and any script can be represented.

def show_bytes(text: str) -> list[int]:
    return list(text.encode("utf-8"))
 
for name, text in samples.items():
    data = show_bytes(text)
    print(name)
    print("  chars:", len(text), "bytes:", len(data))
    print("  hex:", bytes(data).hex())

The asymmetry appears at this layer. An English letter in the ASCII range occupies one byte. An Urdu letter in the Arabic block occupies two bytes. A Devanagari letter occupies three bytes. The word محبت (muhabbat, love) has four letters and eight bytes. The word नमस्ते (namaste) has six Devanagari code points and eighteen bytes. English love has four letters and four bytes.

A byte-level tokenizer therefore starts with unequal raw sizes for equal linguistic content. The vocabulary must recover the difference through merges. When the training corpus contains little Urdu or Devanagari text, the recovery is incomplete. The byte layer is not a bug. It is a universal starting point with script-dependent costs.

def bytes_per_char(text: str) -> float:
    data = text.encode("utf-8")
    return len(data) / max(len(text), 1)
 
print({name: round(bytes_per_char(text), 2)
       for name, text in samples.items()})

BPE from first principles: the merge loop#

Byte-pair encoding builds symbols by merging frequent adjacent pairs. The training loop is short enough to write in full.

from collections import Counter
 
def train_bpe_naive(corpus: list[list[int]], merges: int):
    # corpus: a list of byte sequences.
    # Start: every unit is a single byte.
    vocab = list(range(256))
    for _ in range(merges):
        counts = Counter()
        for seq in corpus:
            for j in range(len(seq) - 1):
                counts[(seq[j], seq[j + 1])] += 1
        if not counts:
            break
        pair, _ = counts.most_common(1)[0]
        new_id = len(vocab)
        vocab.append(pair)
        # Rewrite every sequence, replacing the chosen pair.
        updated = []
        for seq in corpus:
            out = []
            j = 0
            while j < len(seq):
                if j + 1 < len(seq) and (seq[j], seq[j + 1]) == pair:
                    out.append(new_id)
                    j += 2
                else:
                    out.append(seq[j])
                    j += 1
            updated.append(out)
        corpus = updated
    return vocab, corpus

The loop makes three decisions explicit. Frequency decides which pair merges first. Overlap decides which occurrence is consumed first, scanning left to right. Corpus composition decides which language receives long symbols. An English-heavy corpus produces English long symbols first. Urdu and Devanagari pairs remain short because their counts are lower.

The bottleneck is speed. Each iteration scans the full corpus, counts every adjacent pair, and rewrites every sequence. With a large corpus and thousands of merges, the cost grows as merges × corpus length. A production trainer replaces the scan with incremental counters updated only around replaced positions.

def apply_pair(seq: list[int], pair, new_id: int) -> list[int]:
    out = []
    j = 0
    while j < len(seq):
        if j + 1 < len(seq) and seq[j] == pair[0] and seq[j + 1] == pair[1]:
            out.append(new_id)
            j += 2
        else:
            out.append(seq[j])
            j += 1
    return out
 
def encode_with_merges(data: list[int], merge_list) -> list[int]:
    for pair, new_id in merge_list:
        data = apply_pair(data, pair, new_id)
    return data

Encoding runs the same rewrite in merge order. The order carries the priority: earlier merges apply first and protect their output from later splits. Training and encoding share one primitive, which keeps the implementation testable.

Fertility: the price of a script#

Fertility is tokens per word. The formula is elementary:

A parallel sentence in two languages should carry the same content. The fertility ratio between the two encodings measures the tokenizer price difference. Measurement needs a sample of parallel text, a fixed word count, and one tokenizer applied to both sides.

def fertility(text: str, encode) -> float:
    words = text.split()
    tokens = encode(text.encode("utf-8"))
    return len(tokens) / max(len(words), 1)
 
parallel = [
    ("love is the oldest story", "محبت پرانی کہانی ہے۔"),
    ("hello, maya is the oldest story", "नमस्ते, माया सबैभन्दा पुरानो कथा हो।"),
]
 
def make_encoder(merge_list, table):
    def encode(data: list[int]) -> list[int]:
        return encode_with_merges(data, merge_list)
    return encode

The ratio predicts downstream cost without any model. A fixed context window holds fewer high-fertility words. Attention spans more positions for the same sentence. Training consumes more tokens per sentence, so a fixed token budget covers fewer sentences. Inference latency grows with sequence length. A fertility ratio of two therefore becomes a context ratio, a latency ratio, and a data-efficiency ratio before modeling begins.

The diagnostic value is that fertility separates tokenizer effects from model effects. A model that performs worse on Urdu at equal word counts may simply receive longer, more fragmented input. Normalizing scores by token count removes the tokenizer contribution and exposes the remainder.

Nastaliq, code points, and normalisation#

Urdu is written in Nastaliq, a calligraphic script with deep slants and stacking. The rendered word is a shaped object. The tokenizer never sees the shape. The tokenizer sees Unicode code points. Shaping happens later, in the font and the text renderer. A merge can split inside a joined cluster because the merge rule knows byte frequency, not letter joining.

Normalisation adds a second layer of cost. Several characters exist in Urdu and Arabic variants that look similar:

variants = {
    "Urdu ye": "ی",
    "Arabic ye": "ي",
    "Urdu kaf": "ک",
    "Arabic kaf": "ك",
    "Urdu he": "ہ",
    "Arabic he": "ه",
    "Bari ye": "ے",
}
 
for name, char in variants.items():
    code = f"U+{ord(char):04X}"
    data = list(char.encode("utf-8"))
    print(name, code, data)

Scraped text contains both variants. Keyboard layouts differ. Diacritics are often omitted and, when present, are encoded as separate combining marks. A tokenizer trained on one normalisation treats the other as rare. The rare form falls back to small pieces or bytes. The model learns that a common word is uncommon when the spelling is merely different.

A correct pipeline normalises before encoding. The rule is a fixed mapping applied to every input, with display text preserved separately when display matters.

CANON = {
    "ي": "ی",
    "ك": "ک",
    "ه": "ہ",
}
 
def normalise(text: str) -> str:
    return "".join(CANON.get(ch, ch) for ch in text)

Normalisation does not add vocabulary. It reduces the number of distinct forms that the vocabulary must cover. The gain appears in fertility, where a canonical form shares merges across previously split spellings.

Roman Urdu is cheaper#

A large share of everyday Urdu on the internet uses Latin script: mujhe, kya, nahi, yaar. The register is common and functional. For a byte-level tokenizer, Roman Urdu looks like English. The letters are ASCII, the bigrams are familiar, and fertility is low. The Arabic-script form costs more tokens for the same words.

pairs = [
    ("mujhe pyaar hai", "مجھے پیار ہے۔"),
    ("kya tum theek ho", "کیا تم ٹھیک ہو۔"),
]
 
for roman, script in pairs:
    print(repr(roman), "->", len(roman.encode("utf-8")), "bytes")
    print(repr(script), "->", len(script.attach() if False else script.encode("utf-8")), "bytes")

The script form occupies roughly twice the bytes. The merge vocabulary is also richer for Latin text. The token gap is therefore larger than the byte gap. A model frequently performs better at the Romanised form because the script form is more expensive and less represented. A user who writes Urdu in script pays more and receives a weaker answer. The mechanism is budget, not intent.

The same effect appears in code-switched text. A sentence with Urdu content and English words mixes two fertility regimes. A benchmark of formal newspaper prose misses the failure modes of chat text, messages, and captions.

What to measure#

Fertility on a parallel corpus, reported by language, is the first measurement. Token count per script, token count per word, and context usage at a fixed window complete the picture.

def audit(pairs, encode):
    rows = []
    for a, b in pairs:
        fa = fertility(a, encode)
        fb = fertility(b, encode)
        rows.append((fa, fb, fb / fa if fa else float("inf")))
    return rows

Consistency is the second measurement. A tokenizer that splits a morpheme the same way across words allows composition. A tokenizer that splits the same morpheme differently blocks composition. Counting split patterns across a word family exposes the difference.

A checklist#

• Report bytes per character and tokens per word for each target script on parallel text.
• Train or adapt merges on the target script so frequent pieces exist for Urdu and Devanagari.
• Normalise variant characters before encoding and keep display text separate.
• Evaluate on code-switched and Roman Urdu text, not only formal prose.
• Report context usage by language alongside any multilingual accuracy score.
• Treat a tokenizer change as a training change. A new vocabulary means a new run.

The tokenizer is not a neutral reader. It is a budget in code form. The budget decides which languages speak in full symbols and which are spelled out byte by byte. Measuring the budget comes before debating the model.