A random thought hit me the other day: if the information is identical, which human language requires the fewest LLM tokens? You’d think this is a linguistics question. It turns out to be an economics question wearing a linguistics costume.
The intuition
Chinese should win, right? 猫 is one character; “cat” is three. Classical information-density arguments say logographic scripts pack more meaning per glyph. If tokens tracked meaning, dense languages would be cheap.
The reality
Tokens don’t track meaning. They track what the tokenizer saw during training — and tokenizers grew up reading mostly English.
Tokenizers don’t operate on characters; they operate on sub-word units, so character-level efficiency doesn’t automatically translate to token efficiency. A Chinese character can get fragmented into multiple tokens — sometimes one token per part of a character — because the byte-level vocabulary never merged it into a unit. Meanwhile “the” gets its own cozy token because English dominates the training corpus. arXivMedium
The numbers bear it out. A study using 2 million professionally translated sentences found that every CJK language uses more tokens than English — no exceptions. On GPT’s cl100k_base tokenizer, Chinese pays roughly 15% more tokens for equivalent content, and low-resource languages fare far worse — Khmer, Lao, and Burmese show notably high length ratios on most open-source LLMs. arxivGitHub
So English wins?
On most Western tokenizers, yes — but for the least satisfying reason possible: home-field advantage. The proof: GLM’s Chinese-native tokenizer inverts the pattern, producing fewer tokens for Chinese than English (ratio 0.923). Same languages, opposite winner. The answer to “which language is most efficient” is: whichever one your tokenizer grew up speaking. GitHub
(For fun: one recent paper asked whether Sanskrit — famously compact — beats everyone. With bias-controlled tokenizers trained at equal vocabulary sizes, Sanskrit does show superior density. The ancient grammarians would be pleased with their compression ratio.) Sicheng Ouyang
Why a LaTeX person cares
I write in Korean, English, and LaTeX daily, often through texspark’s AI panel — where the attach-document toggle ships my source to a model with a 60 KB cap and a token bill. My source files are trilingual: Korean prose, English terms, LaTeX markup. Every \begin{equation} is a flat tax in any language.
The practical takeaway is almost embarrassing: prompting in English is usually cheapest, not because English is better, but because the meter was calibrated in it. Somewhere in that fact is a whole essay about soft power. This is not that essay. This was just a random thought that turned out to have a real answer.
Leave a Reply