Transliteration Is a Lossy Codec

Transliteration is an encoder from one script to another, and whether it is lossless depends entirely on whether anyone designed the decoder.

Scholarly schemes such as IAST and ISO 15919 are designed to be reversed. Chat romanisation, the kya haal hai and k cha khabar of messages and comments, is designed to be typed on a Latin keyboard, and is reversed only by a reader who already knows the language. This post builds a Devanagari-to-Latin encoder in Python, measures round-trip loss under three schemes, adds the context rules that make romanisation sound right and decode wrong, and then looks at Roman Urdu and at what spelling variation does to a tokenizer. The byte-level and Unicode details are in Devanagari encoding; the token-cost side of Urdu script is in what a tokenizer does to Urdu. This post is about the mapping between scripts.

The smallest encoder: one letter, one string#

Devanagari is an abugida. A consonant letter carries an inherent vowel, the schwa a, unless a vowel sign (matra) replaces it or a virama (्) cancels it. So the unit of encoding is a consonant plus a vowel slot:

import unicodedata
 
CONS = dict(zip("कखगघङचछजझञटठडढणतथदधनपफबभमयरलवशषसह",
    "k kh g gh ṅ c ch j jh ñ ṭ ṭh ḍ ḍh ṇ t th d dh n p ph b bh m y r l v ś ṣ s h".split()))
CONS.update({"ड़": "ṛ", "ढ़": "ṛh"})            # flaps: consonant + nukta
VOWELS = dict(zip("अआइईउऊऋएऐओऔ", "a ā i ī u ū ṛ e ai o au".split()))
MATRAS = dict(zip("ािीुूृेैोौ", "ā i ī u ū ṛ e ai o au".split()))
MARKS = {"ं": "ṃ", "ँ": "m̐", "ः": "ḥ"}                   # anusvara, candrabindu, visarga
VIRAMA, NUKTA = "्", "़"
 
def units(word):
    """[consonant, vowel] pairs. vowel None = inherent schwa, '' = virama (no vowel)."""
    word = unicodedata.normalize("NFD", word)               # split precomposed ड़ into ड + ़
    out, i = [], 0
    while i < len(word):
        ch = word[i]
        if word[i + 1:i + 2] == NUKTA:
            ch, i = ch + NUKTA, i + 1
        if ch in CONS:     out.append([CONS[ch], None])
        elif ch in MATRAS: out[-1][1] = MATRAS[ch]
        elif ch == VIRAMA: out[-1][1] = ""
        elif ch in VOWELS: out.append(["", VOWELS[ch]])
        elif ch in MARKS:  out.append(["", MARKS[ch]])
        i += 1
    return out
 
def render(u):
    return "".join(c + ("a" if v is None else v) for c, v in u)
 
def naive(word):
    return render(units(word))

The map is IAST-flavoured: a macron for vowel length, a dot below for retroflex consonants, h for aspiration. naive("कमल") gives kamala, naive("नमस्ते") gives namaste, naive("आँख") gives ām̐kha. Every schwa the script implies is written out.

The decoder, and the first losses#

A decoder reads Latin greedily, longest token first, and inserts a virama whenever two consonants meet:

INV_C = {v: k for k, v in CONS.items()}
INV_V = {v: k for k, v in VOWELS.items()}
INV_M = {v: k for k, v in MATRAS.items()}
INV_K = {v: k for k, v in MARKS.items()}
TOKENS = sorted(set(INV_C) | set(INV_V) | set(INV_K), key=len, reverse=True)
 
def decode(latin):
    """Greedy longest match. A consonant with no following vowel gets a virama."""
    out, i, bare = [], 0, False
    while i < len(latin):
        t = next(t for t in TOKENS if latin.startswith(t, i)); i += len(t)
        if t in INV_C:                                      # consonants win ties (ṛ!)
            if bare: out.append(VIRAMA)
            out.append(INV_C[t]); bare = True
        elif t in INV_V:
            if not bare: out.append(INV_V[t])
            elif t != "a": out.append(INV_M[t])             # inherent a writes nothing
            bare = False
        else:
            out.append(INV_K[t])
    if bare: out.append(VIRAMA)
    return unicodedata.normalize("NFC", "".join(out))

On a list of 41 common Hindi words, decode(naive(w)) == w for all 41. The scheme looks lossless. It is not; the test set simply missed the collisions. Three inputs break it:

ISO 15919 exists largely to close these gaps. It writes vocalic ऋ as r̥ (a ring below) so that ṛ is free for the flap, distinguishes long ē/ō from short e/o for languages that need it, writes the anusvara as ṁ, and reserves a separator (a colon) for the k:h and a:i cases. Changing the two ṛ entries for ऋ to r̥ in the code above fixes the first row immediately: kr̥pā decodes back to कृपा, and r̥ṣi to ऋषि. A reversible romanisation is an engineering artefact: every collision has to be found and assigned a distinct spelling.

Context rule 1: schwa deletion#

kamala is a faithful encoding of कमल and a wrong pronunciation of it. Hindi speakers say kamal, samajhnā, dhaṛkan. The inherent vowel is written but often not spoken. The standard approximation has two parts: drop a word-final schwa, and drop a medial schwa in the context VC_CV, scanning right to left so that two neighbouring schwas are not both removed.

def has_vowel(u):
    return u[1] is None or u[1] not in ("", "ṃ", "m̐", "ḥ")
 
def delete_schwas(u):
    syl = [k for k, x in enumerate(u) if has_vowel(x)]
    if len(syl) > 1 and syl[-1] == len(u) - 1 and u[-1][0] and u[-1][1] is None:
        u[-1][1] = ""                                       # kamala -> kamal
    for j in range(len(u) - 2, 0, -1):
        c, v = u[j]
        if c and v is None and has_vowel(u[j - 1]) and u[j + 1][0] and has_vowel(u[j + 1]):
            u[j][1] = ""                                    # samajhanā -> samajhnā
    return u
 
def schwa_delete(word):
    return render(delete_schwas(units(word)))

The output now matches speech, and the round trip collapses: only 8 of the 41 words survive decode(schwa_delete(w)) == w. The decoder sees sarkār and writes सर्कार्, with a virama the original never had. The information that distinguishes "schwa written but not spoken" from "no vowel at all" lives in the script, not in the sound, so a pronunciation-faithful encoder must discard it.

The rule is also approximate and language-specific. Even for Hindi it over-deletes after a conjunct: सत्य is spoken satya, but the code above produces saty. For Nepali it is worse. Nepali orthography marks many vowel-less endings explicitly with a final virama (हलन्त): गर्छन् and भन्छन् carry it in the spelling, so garchan and bhanchan come straight from the letters. A final consonant without हलन्त is then often pronounced with its vowel: गर्छ is garcha, and the Hindi-tuned rule turns it into garch. The deletion rule cannot be separated from the language it serves, and the same Devanagari string needs a different encoder depending on whether it is Hindi or Nepali.

Context rule 2: nasalisation and aspiration#

Two more distinctions sit on top of the consonant inventory.

• Nasalisation. The anusvara ं marks a nasal consonant whose place matches the next consonant, or a nasalised vowel. The candrabindu ँ marks a nasalised vowel. In practice both spellings occur for the same word: हिंदी (hiṃdī) and हिन्दी (hindī) are the same word written two ways. A scholarly scheme keeps the spellings apart. Chat romanisation writes both as hindi, and आँख as ankh, where the n no longer says whether it is a consonant or a vowel colour.
• Aspiration. Devanagari distinguishes four-way series: क ख ग घ (k kh g gh), त थ द ध, and the retroflex ट ठ ड ढ. Aspiration survives romanisation reasonably well because h is cheap to type. Retroflexion does not: त and ट both become t.

Chat-style romanisation: the encoder nobody inverts#

Chat romanisation applies schwa deletion, then drops every diacritic:

CHAT_C = {"ṭ": "t", "ṭh": "th", "ḍ": "d", "ḍh": "dh", "ṇ": "n", "ś": "sh", "ṣ": "sh",
          "c": "ch", "ch": "chh", "ṛ": "r", "ṛh": "rh", "ñ": "n", "ṅ": "n"}
CHAT_V = {"ā": "a", "ī": "i", "ū": "u", "ṛ": "ri", "ṃ": "n", "m̐": "n", "ḥ": "h"}
 
def chat(word):
    u = delete_schwas(units(word))
    return "".join(CHAT_C.get(c, c) + ("a" if v is None else CHAT_V.get(v, v)) for c, v in u)

The same 41 words produce only 32 distinct chat spellings. The collisions:

Vowel length is the biggest casualty, because the macron is the first thing a Latin keyboard drops. Real chat text compensates by doubling (kaam, daal, meel), but inconsistently, which brings the next problem.

Roman Urdu: many letters, one Latin letter#

Urdu's script carries Arabic and Persian consonants that merged in pronunciation. Romanisation merges them on the page too:

khana is کھانا (khānā, food, to eat) and also خانہ (khāna, house, compartment, as in dawā-khāna). The two words differ in the first letter, the vowel length of the last syllable and the final letter, and Roman Urdu writes them identically. sher is both شیر (lion) and شعر (couplet); Hindi शेर also commonly serves for both, so here the Devanagari spelling loses the distinction too. Context resolves these for a human reader instantly. For a model, the context has to do all of the work that the spelling stopped doing.

What spelling variation does to a tokenizer#

Chat romanisation has no standard, so one word arrives in many spellings: nahin, nahi, nhi, nai. A subword tokenizer learns merges from frequency, and variation splits each word's frequency across its spellings. A small byte-pair-encoding experiment measures this: 14 common Roman Hindi words with three to five chat variants each, a 20,000-word training corpus, 40 merges, evaluated on 5,000 held-out words drawn the same way.

VARIANTS = {"nahin": ["nahi", "nhi", "nai", "nahee"], "hai": ["he", "h", "hain"],
            "acchha": ["accha", "acha", "achha", "achchha"], "bahut": ["bohot", "bhut", "bahot"],
            "kyunki": ["kyuki", "kyonki", "kyoki"], "theek": ["thik", "theek", "thick"]}  # ...14 in all
 
def corpus(n_words, p_variant, rng):
    keys = list(VARIANTS)
    return [rng.choice(VARIANTS[w]) if rng.random() < p_variant else w
            for w in (rng.choice(keys) for _ in range(n_words))]

The vocabulary budget is the same in every row. With consistent spelling, 40 merges are enough to make most words whole tokens. With variation, the same budget is spent on fragments shared across spellings, and most words stay in pieces. At full scale the effect is diluted by a much larger vocabulary, but the mechanism is the same: the model sees nahin and nhi as different sequences and has to learn from data that they mean the same thing, while also learning from context that khana sometimes means two different things. Romanisation makes some different words look the same and some identical words look different, and both errors land on the model.

What to measure#

1. Round-trip exactness of any transliterator on a word list that includes vocalic ऋ, nukta letters, hiatus vowels, anusvara versus explicit nasal clusters, and Nepali words with final हलन्त. A pass on common words proves little.
2. Collision rate of the Latin side: distinct Latin forms divided by distinct source words. It is 32/41 for the chat scheme above and 41/41 for the scholarly one.
3. Variant fan-out in real chat data: how many spellings per lemma, and how the frequency splits across them. This predicts tokenizer fragmentation better than byte counts.
4. Tokens per word on Roman, native-script and normalised-Roman versions of the same text, using the tokenizer you will actually ship.
5. Disambiguation accuracy on minimal pairs (khana as food versus house, sher as lion versus couplet) after back-transliteration to script. This is where lossy encoding shows up as a model error rather than a spelling curiosity, and where code-switched text makes it harder still.