Devanagari Encoding

नमस्ते looks like a single word and is, to a human reader, one. To the machine, it is a sequence of code points that a font engine assembles into a shape. The assembly is not the encoding. The encoding is a list; the shape is a computation.

That distinction runs through all of Devanagari, and it is why text in the script is harder to measure, search, and tokenize than text in Latin script. This post starts from a three-line program that gets it wrong, derives the three layers underneath — code points, grapheme clusters, glyphs — and rebuilds length, comparison, search, and tokenization on the right layer. All code is Python 3 with the standard library plus the regex package.

The simplest model: a string is a list of letters#

The naive model treats a string as an array whose elements are letters. That model works for ASCII. A Latin letter is one code point, one byte in UTF-8, and one visible glyph. Indexing behaves as expected.

The simplest program written against that model is already broken:

s = "नमस्ते"
print(len(s))      # 6   -- code points, not letters
print(s[0:3])      # नमस -- the slice drops the virama and leaves a bare स
print(s[:-1])      # नमस्त -- the vowel sign is gone; the word changed

len counts code points. A slice by index can separate a consonant from the virama or vowel sign that modifies it, and the result is not a shorter version of the word; it is a different word. A system that passes English smoke tests fails on the first Devanagari string it meets.

Deriving the storage: what Unicode actually holds#

Print every code point with its name and its UTF-8 size:

import unicodedata
 
for ch in "नमस्ते":
    print(f"U+{ord(ch):04X}  {unicodedata.name(ch):28}  {len(ch.encode())} bytes")

Every consonant letter carries an inherent vowel. The virama removes it, so स् is a bare s that joins the following त to form the conjunct स्त. The vowel sign े replaces the inherent vowel of त. The reader sees na-ma-ste: three orthographic syllables built from six code points.

Three units must be kept apart.

Code points. The abstract characters. Six here. Storage, normalization, and regular expressions operate on code points.

Bytes. Every code point in the Devanagari block (U+0900–U+097F) occupies three bytes in UTF-8. Six code points become eighteen bytes; the Latin transliteration namaste is seven. Byte length is a transport and tokenizer metric, not a text metric.

Grapheme clusters. The user-perceived units. Unicode's segmentation rules (UAX #29) define them. Since Unicode 15.1 the rules keep a virama-joined conjunct together, so नमस्ते has three clusters: न, म, स्ते. Libraries built on older Unicode data split it into four: न, म, स्, ते.

Deriving the renderer: shaping is a computation#

The visible form is produced by a shaping engine — HarfBuzz, DirectWrite, Core Text — that takes the code point sequence and produces positioned glyphs. The engine applies the script's rules: conjunct formation, half forms, mark positioning, and reordering.

Reordering is the rule that breaks visual intuitions. The vowel sign i (ि, U+093F) is stored after its consonant but drawn before it. In किताब (kitāb, book) the logical order is क, ि, त, ा, ब; the eye sees ि first. Unicode always stores Devanagari in pronunciation order. The font moves the glyph.

The architecture is deliberate. Unicode stores the content, the font stores the appearance, the engine performs the mapping. A search engine, a text extractor, or a language model operates on content. A human reads appearance. Neither layer is the word, and bugs live in the gap between them.

Step 1: measure clusters, not code points#

Bottleneck: user-facing limits, truncation, and cursor movement treat code points as letters.

Fix: segment with the Unicode grapheme rules. In the regex package, \X matches one extended grapheme cluster:

import regex  # pip install regex; supports \X with current Unicode data
 
def clusters(text: str) -> list[str]:
    return regex.findall(r"\X", text)
 
def truncate(text: str, limit: int) -> str:
    # Cut at a cluster boundary so no mark is separated from its base.
    parts = clusters(text)
    return text if len(parts) <= limit else "".join(parts[:limit])
 
print(clusters("नमस्ते"))        # ['न', 'म', 'स्ते']
print(truncate("नमस्ते", 2))     # 'नम'

A limit of 280 "characters" becomes a limit of 280 clusters. Truncation never leaves an orphaned vowel sign or a dangling virama. The same rule fixes cursor movement and backspace: one keypress, one cluster.

Pin the library version in tests. A dependency upgrade that moves from pre-15.1 to post-15.1 Unicode data changes the cluster count of every conjunct, and therefore every stored length.

Step 2: normalize before comparing#

Bottleneck: two encodings of what looks like the same text compare unequal.

Devanagari has a concrete case. The letter क़ (qa, used in Urdu and Persian loanwords) exists both as a single code point, U+0958, and as क plus the nukta sign, U+0915 U+093C. They render identically.

import unicodedata
 
precomposed = "क़"          # क़ as one code point
decomposed  = "क़"    # क + nukta
 
print(precomposed == decomposed)                       # False
nfc = lambda t: unicodedata.normalize("NFC", t)
print(nfc(precomposed) == nfc(decomposed))             # True
print([f"U+{ord(c):04X}" for c in nfc(precomposed)])   # ['U+0915', 'U+093C']

Note the direction: NFC decomposes U+0958. The precomposed nukta letters are composition exclusions, so the normalized form is always base plus nukta. Code that assumes NFC means "fewest code points" is wrong for this script.

The practical rule: normalize at ingestion, store normalized, compare normalized. A cache, a deduplication pass, or a username check on raw strings is a bug waiting for the second encoding.

Step 3: strip the invisible joiners from search keys#

Bottleneck: normalization does not touch format characters. ZERO WIDTH JOINER (U+200D) and ZERO WIDTH NON-JOINER (U+200C) change how a conjunct is drawn — a half form instead of a full ligature, or an explicit virama — without changing which word it is.

a = "क्ष"            # क + virama + ष : the ligature क्ष
b = "क्‍ष"      # same letters with ZWJ: drawn as a half form
print(a == b)        # False, and NFC leaves both unchanged

Fix: build search and deduplication keys that are invariant to these presentation choices, while keeping the original text for display.

import unicodedata
 
PRESENTATION_ONLY = dict.fromkeys(map(ord, "‌‍"))
 
def search_key(text: str) -> str:
    text = unicodedata.normalize("NFC", text)
    text = text.translate(PRESENTATION_ONLY)   # drop ZWJ / ZWNJ
    return text.casefold()                     # harmless for Devanagari, needed for mixed text
 
def contains(doc: str, query: str) -> bool:
    return search_key(query) in search_key(doc)

Two more decisions belong in the key, depending on the application. Whether nukta is significant (ज and ज़ are distinct sounds, but users often omit the dot), and whether candrabindu and anusvara are interchangeable in informal spelling. Each is a policy choice. Write it down in the key function rather than discovering it in a bug report.

Sorting has the same shape. Code point order is not dictionary order for Nepali. Use a collation library with locale data — ICU via PyICU, for instance — rather than sorted().

Step 4: account for tokenization cost#

Bottleneck: a byte-level tokenizer sees UTF-8 bytes. Every Devanagari code point is three bytes, so before any merges are learned a Devanagari word starts at roughly three times the byte count of its Latin transliteration.

for w in ["नमस्ते", "namaste", "किताब", "kitab"]:
    print(f"{w:10} code points={len(w):2}  bytes={len(w.encode()):2}")
# नमस्ते     code points= 6  bytes=18
# namaste    code points= 7  bytes= 7
# किताब      code points= 5  bytes=15
# kitab      code points= 5  bytes= 5

Whether those bytes merge into a few tokens depends on how much Devanagari the tokenizer saw during training. Measure it against clusters, not code points:

import regex
 
def fertility(words: list[str], tokenize) -> dict[str, float]:
    tokens = sum(len(tokenize(w)) for w in words)
    n_clusters = sum(len(regex.findall(r"\X", w)) for w in words)
    return {
        "tokens_per_word": tokens / len(words),
        "tokens_per_cluster": tokens / max(n_clusters, 1),
    }

A model trained on fragmented input reassembles the word internally, and it pays for that in context length and compute. The same sentence costs more tokens in Devanagari than in Roman Nepali for most widely used tokenizers. That is one of the reasons Roman Nepali is so common online: it is not only a transliteration, it is a cheaper encoding for the same language.

Step 5: test on the script#

Bottleneck: unit tests written in ASCII cannot fail on any of the above.

Fix: keep a small fixture of hard cases and run every text function against it.

import unicodedata
from clusters import clusters, truncate
from search_key import search_key
 
CASES = ["नमस्ते", "किताब", "क्ष", "क्‍ष", "क़", "क़", "श्री"]
 
def test_truncate_never_orphans_marks():
    for s in CASES:
        for n in range(len(clusters(s)) + 1):
            out = truncate(s, n)
            assert not out or not unicodedata.category(out[0]).startswith("M")
            assert s.startswith(out)
 
def test_equivalent_forms_share_a_key():
    assert search_key("क़") == search_key("क़")
    assert search_key("क्ष") == search_key("क्‍ष")

What to measure#

1. Code point length and cluster length, logged side by side. Divergence shows where limits are wrong.
2. Byte length for storage and transport budgets. Three bytes per code point is normal.
3. Normalization drift. The fraction of stored strings that change under NFC. Anything above zero means ingestion is not normalizing.
4. Tokens per cluster for each tokenizer in use, on Devanagari and on Roman Nepali.
5. Search recall on visual queries. Queries typed by people, not copied from the database.
6. Unicode version of every segmentation library, pinned and recorded.

A checklist#

• Count clusters when the limit is "characters". Never len() on user-facing text.
• Split, truncate, and move the cursor on cluster boundaries.
• Normalize to NFC at ingestion, and remember that NFC decomposes the nukta letters.
• Build search keys that drop ZWJ and ZWNJ; keep the original for display.
• Decide nukta and nasalization policy in one function, not in scattered comparisons.
• Sort with locale-aware collation, not code point order.
• Budget tokens per cluster, not per word, when planning context length.
• Keep a fixture of hard strings and test every text function against it.