Urdu Tokenization: Pak-LLM's Nastaliq BPE vs a Western BPE Vocabulary
A Western-centric BPE vocabulary fragments Urdu words into many small pieces because Perso-Arabic script and Nastaliq ligatures are rare in its training corpus. A Perso-Arabic vocabulary covers common Urdu words and letter joins as single tokens, which reduces token count per page and therefore cost, while preserving diacritics and ligature fidelity.
- Written for:
- Teams budgeting Urdu workloads, and engineers evaluating tokenizer quality for regional scripts.
- Last reviewed:
- Author:
- Saad Bin Farrukh
Why does Urdu use more tokens on a Western tokenizer?
Byte-pair encoding allocates vocabulary slots according to frequency in the training corpus. Because Urdu is a small share of most English-centric corpora, common Urdu words and joined letter forms are split into multiple subword pieces, so the same sentence consumes more tokens.
- Rare scripts get fewer dedicated vocabulary entries.
- Nastaliq joins and diacritics are frequently split apart.
- More tokens per sentence means higher cost and shorter effective context.
What changes with a Perso-Arabic vocabulary?
A vocabulary built with substantial Perso-Arabic coverage represents frequent Urdu words, suffixes, and joined forms as single tokens. The result is fewer tokens for the same text, which lowers cost per document and leaves more of the context window for actual content.
- Fewer tokens per Urdu sentence at equal meaning.
- Diacritics and ligatures survive tokenization intact.
- More effective context available for long Urdu documents.
- Lower PKR cost per page at a fixed per-token rate.
How should I measure the difference for my own corpus?
Tokenize a representative sample of your own documents with both vocabularies and compare token counts, not published averages. Corpus mix matters: legal Urdu, Roman Urdu chat, and religious text behave very differently.
- Sample at least a few thousand words per document type.
- Compare token counts, not character counts.
- Check that diacritics and ligatures survive round-trip.
- Re-measure after any vocabulary change.
Does a better tokenizer make the model smarter?
No. Tokenization affects cost, context efficiency, and script fidelity; it does not add reasoning capability. A model can tokenize Urdu efficiently and still be weaker than a frontier model on a complex reasoning task.
- Tokenizer efficiency is an economics and fidelity property.
- Reasoning quality comes from training, data, and architecture.
- Evaluate both separately before choosing a platform.
Frequently asked questions
How much cheaper is Urdu on a Perso-Arabic tokenizer?
It depends entirely on your corpus. Pak-LLM reports compression gains on its own Urdu evaluation sets, but you should tokenize a sample of your own documents with both vocabularies to get a number you can defend internally.
Does the tokenizer handle Roman Urdu?
Roman Urdu is Latin-script and tokenizes differently from Nastaliq Urdu. Pak-LLM handles both, but the compression advantage is concentrated in Perso-Arabic script rather than Roman transliteration.
Will switching tokenizers break my prompts?
Token boundaries change, so token-count-based limits and truncation behaviour will shift. Re-measure your prompt sizes and context budgets after any tokenizer change.
Is the tokenizer open source?
The tokenizer and model weights are proprietary and licensed for use through the platform or an on-premise agreement. They are not published as open weights.