The least efficient tokenizer produces ~1.8 times as many tokens as the most efficient one at the median.

Tokens per character vary by script and tokenizer. At the median tokenizer, a character of translation is 0.36 tokens in the Latin-script languages and 0.42 in the Cyrillic ones, ~0.5 in Chinese, Japanese and Korean, and 0.5 to 1.15 in the scripts of India, Sri Lanka and Southeast Asia. Against the English source on the same tokenizer, a character of translation is ~1.4 times as many tokens in the Latin-script languages, ~2 times in Chinese, Japanese and Korean, and ~5 times in Amharic.

Tokenizers differ least in the Latin script and most in the scripts with the most tokens per character. Within a language, the least efficient tokenizer produces ~1.7 times as many tokens as the most efficient one in the Latin-script languages, ~2 times in the Cyrillic ones, and ~5.5 times in Malayalam, Lao and Odia.

Chart data
Tokens per character of translated text, by script
ScriptLowest, tokens per character of the translationLower quartile, tokens per character of the translationMedian, tokens per character of the translationUpper quartile, tokens per character of the translationHighest, tokens per character of the translationAgainst EnglishNote
Latin0.290.310.360.40.511.4358 languages
Greek0.290.340.380.610.861.65Greek
Cyrillic0.320.360.420.510.591.6510 languages
Arabic0.320.330.460.620.851.865 languages
Thai0.30.370.480.760.842.01Thai
Hangul0.320.450.490.750.832.08Korean
Tamil0.270.340.50.761.162.16Tamil
Hebrew0.360.430.520.630.882.03Hebrew, Yiddish
Armenian0.30.470.530.831.152.11Armenian
Han0.370.480.530.690.772.1Chinese Simplified, Chinese Traditional
Devanagari0.290.320.540.720.891.95Hindi, Marathi, Nepali
Japanese0.310.440.550.670.812.17Japanese
Bengali0.310.360.620.831.082.36Assamese, Bengali
Georgian0.320.40.630.821.522.4Georgian
Gujarati0.350.370.730.921.632.92Gujarati
Telugu0.350.370.730.791.622.76Telugu
Gurmukhi0.410.480.771.581.863.39Punjabi
Sinhala0.410.490.811.041.423.12Sinhala
Khmer0.370.520.830.961.522.86Khmer
Malayalam0.320.340.831.121.772.79Malayalam
Myanmar0.370.470.870.911.672.58Burmese
Kannada0.360.360.921.412.053.61Kannada
Lao0.350.981.061.331.924.51Lao
Odia0.390.811.151.392.113.85Odia (Oriya)
Ethiopic0.490.961.241.532.35.35Amharic

Relative tokenizer efficiency is similar across languages. The OpenAI models produce ~0.86 times the tokens of the median tokenizer of the language, and more than the median in at most 6 of the 100 languages. The Google models produce ~0.92 times. Mistral Large 3 and Medium 3.1 produce ~1.2 times, and Claude Sonnet 5, Fable 5 and Opus 4.8 ~1.4 times, more than the median in 98 to 99 of the 100 languages.

The tokenizer of Anthropic models changed with the model generation. Claude Haiku 4.5 and Sonnet 4.6 produce ~1.13 times the tokens of the median tokenizer, Claude Sonnet 5, Fable 5 and Opus 4.8 ~1.40 times, also in English (0.26 against 0.35 tokens per character).

Chart data
Tokens per character against the median tokenizer of the same language, by model
ModelLowest, tokens per character against the median tokenizer of the same languageLower quartile, tokens per character against the median tokenizer of the same languageMedianUpper quartile, tokens per character against the median tokenizer of the same languageHighest, tokens per character against the median tokenizer of the same languageLanguages with more tokens than the medianLanguagesNote
Cohere Tiny Aya0.370.770.820.881.13269more tokens in 2 of 69
OpenAI GPT-4.10.390.760.860.911.486100more tokens in 6 of 100
OpenAI GPT-5.50.390.750.860.911.254100more tokens in 4 of 100
OpenAI GPT-5.6 Luna0.390.750.860.91.244100more tokens in 4 of 100
OpenAI GPT-5.6 Sol0.390.760.860.91.275100more tokens in 5 of 100
OpenAI GPT-5.6 Terra0.390.760.860.91.255100more tokens in 5 of 100
MiniMax M2.50.570.860.910.962.1620100more tokens in 20 of 100
Google Gemini 3.1 Pro Preview0.390.820.920.9514100more tokens in 4 of 100
Google Gemini 3.5 Flash0.390.820.920.9510100more tokens in 0 of 100
Google Gemma 40.390.810.920.9410100more tokens in 0 of 100
xAI Grok 4.20 (non-reasoning)0.650.890.950.991.2612100more tokens in 12 of 100
xAI Grok 4.20 (reasoning)0.640.890.950.991.238100more tokens in 8 of 100
Cohere Command A Translate0.770.961.021.131.9561100more tokens in 61 of 100
Widn AI Tower Anthill0.730.871.021.051.41120more tokens in 11 of 20
DeepSeek V3.20.6711.031.051.3672100more tokens in 72 of 100
z.ai GLM-50.9311.031.072.2573100more tokens in 73 of 100
MoonshotAI Kimi K2.50.5911.041.081.6766100more tokens in 66 of 100
Widn AI Tower Sugarloaf0.881.031.041.081.411922more tokens in 19 of 22
Widn AI Tower Vesuvius0.911.031.041.081.471720more tokens in 17 of 20
Alibaba Cloud Qwen3-235B-A22B0.881.041.061.211.947279more tokens in 72 of 79
Anthropic Claude Haiku 4.50.781.071.121.242.0493100more tokens in 93 of 100
Anthropic Claude Sonnet 4.60.781.081.141.252.0895100more tokens in 95 of 100
AI21 Jamba Large 1.70.331.081.171.432.4284100more tokens in 84 of 100
Mistral Large 31.111.171.221.462.4100100more tokens in 100 of 100
Mistral Medium 3.11.131.191.231.52.4100100more tokens in 100 of 100
Anthropic Claude Sonnet 50.791.311.391.512.0898100more tokens in 98 of 100
Anthropic Claude Fable 50.81.31.41.52.0899100more tokens in 99 of 100
Anthropic Claude Opus 4.80.791.311.411.52.0999100more tokens in 99 of 100

Tokens per character of translated text, by language

LanguageScriptMedian modelMost efficient tokenizerLeast efficient tokenizerLeast against most efficientAgainst EnglishModels
ModelTokens per characterModelTokens per character
AmharicEthiopic1.24Google Gemini 3.5 Flash0.49AI21 Jamba Large 1.72.34.75.424
Odia (Oriya)Odia1.15AI21 Jamba Large 1.70.39Mistral Large 32.115.43.924
LaoLao1.06AI21 Jamba Large 1.70.35Mistral Large 31.925.54.525
KannadaKannada0.92Google Gemini 3.5 Flash0.36AI21 Jamba Large 1.72.055.83.624
BurmeseMyanmar0.87Google Gemma 40.37MiniMax M2.51.674.62.625
KhmerKhmer0.83Cohere Tiny Aya0.37Cohere Command A Translate1.524.12.925
MalayalamMalayalam0.83Google Gemini 3.1 Pro Preview0.32Mistral Large 31.775.52.824
SinhalaSinhala0.81Google Gemini 3.5 Flash0.41Cohere Command A Translate1.423.43.124
PunjabiGurmukhi0.77Cohere Tiny Aya0.41Mistral Medium 3.11.864.53.425
GujaratiGujarati0.73OpenAI GPT-4.10.35Mistral Large 31.634.72.925
TeluguTelugu0.73Google Gemini 3.5 Flash0.35AI21 Jamba Large 1.71.624.62.825
AssameseBengali0.66AI21 Jamba Large 1.70.35Cohere Command A Translate1.0732.524
UyghurArabic0.64OpenAI GPT-4.10.44Mistral Large 30.92.12.423
GeorgianGeorgian0.63OpenAI GPT-5.50.32AI21 Jamba Large 1.71.524.82.424
YiddishHebrew0.62OpenAI GPT-5.6 Luna0.36z.ai GLM-50.932.62.323
BengaliBengali0.59Google Gemini 3.5 Flash0.27AI21 Jamba Large 1.71.14.12.225
Chinese TraditionalHan0.57Widn AI Tower Anthill0.41Mistral Medium 3.10.781.92.228
HindiDevanagari0.56Google Gemma 40.28Mistral Large 30.893.21.928
JapaneseJapanese0.55MiniMax M2.50.31Mistral Medium 3.10.812.62.228
NepaliDevanagari0.54Cohere Tiny Aya0.3Mistral Medium 3.10.893225
ArmenianArmenian0.53OpenAI GPT-5.6 Luna0.3MiniMax M2.51.153.82.124
MarathiDevanagari0.53Google Gemini 3.5 Flash0.29Mistral Large 30.893.1225
YorubaLatin0.52OpenAI GPT-5.50.43Anthropic Claude Fable 50.81.82.224
MongolianCyrillic0.51OpenAI GPT-5.50.35AI21 Jamba Large 1.70.862.41.923
TamilTamil0.5Google Gemini 3.5 Flash0.27AI21 Jamba Large 1.71.164.22.225
KazakhCyrillic0.49OpenAI GPT-5.6 Luna0.31AI21 Jamba Large 1.70.842.71.825
KoreanHangul0.49MiniMax M2.50.32Mistral Medium 3.10.832.62.128
Chinese SimplifiedHan0.49MiniMax M2.50.32Anthropic Claude Fable 50.752.4228
PashtoArabic0.48AI21 Jamba Large 1.70.32Mistral Large 30.92.81.923
ThaiThai0.48Cohere Tiny Aya0.3Mistral Medium 3.10.842.8225
TatarCyrillic0.46OpenAI GPT-5.50.34Mistral Large 30.571.71.924
UrduArabic0.46Google Gemma 40.32Mistral Medium 3.10.852.61.925
KyrgyzCyrillic0.45OpenAI GPT-4.10.32AI21 Jamba Large 1.70.832.61.823
HebrewHebrew0.43Cohere Tiny Aya0.36Mistral Medium 3.10.832.31.825
IgboLatin0.43OpenAI GPT-5.6 Terra0.32Anthropic Claude Fable 50.631.91.824
TajikCyrillic0.43OpenAI GPT-5.6 Luna0.33AI21 Jamba Large 1.70.722.21.724
AzerbaijaniLatin0.42OpenAI GPT-5.50.31Anthropic Claude Opus 4.80.561.81.724
IcelandicLatin0.42OpenAI GPT-5.50.33Anthropic Claude Sonnet 50.511.61.627
BelarusianCyrillic0.41AI21 Jamba Large 1.70.32Anthropic Claude Sonnet 50.531.61.624
Kurdish (Kurmanji)Latin0.4MiniMax M2.50.35Anthropic Claude Sonnet 50.551.61.623
LatvianLatin0.4Cohere Tiny Aya0.31Anthropic Claude Sonnet 50.531.71.525
WolofLatin0.4xAI Grok 4.20 (reasoning)0.32Anthropic Claude Fable 50.561.81.624
IrishLatin0.39OpenAI GPT-5.6 Sol0.32Anthropic Claude Fable 50.521.61.525
TurkmenLatin0.39OpenAI GPT-5.6 Terra0.31Anthropic Claude Sonnet 50.561.81.623
WelshLatin0.38OpenAI GPT-5.6 Luna0.33Anthropic Claude Sonnet 50.551.71.625
GreekGreek0.38Cohere Tiny Aya0.29Mistral Medium 3.10.862.91.625
FarsiArabic0.38Cohere Tiny Aya0.31Mistral Medium 3.10.832.71.625
Scottish GaelicLatin0.38MiniMax M2.50.32Anthropic Claude Fable 50.531.71.523
LithuanianLatin0.38Cohere Tiny Aya0.31Anthropic Claude Fable 50.521.71.525
MalteseLatin0.38AI21 Jamba Large 1.70.31Anthropic Claude Sonnet 50.541.81.625
SomaliLatin0.38OpenAI GPT-5.50.31Anthropic Claude Sonnet 50.551.81.523
AlbanianLatin0.38OpenAI GPT-4.10.33Anthropic Claude Opus 4.80.511.51.524
Arabic (MSA)Arabic0.37MiniMax M2.50.26Mistral Medium 3.10.7831.625
HungarianLatin0.37Cohere Tiny Aya0.28Anthropic Claude Sonnet 50.481.71.428
MacedonianCyrillic0.37OpenAI GPT-5.6 Sol0.33AI21 Jamba Large 1.70.561.71.524
KinyarwandaLatin0.37OpenAI GPT-5.6 Luna0.31Anthropic Claude Sonnet 50.551.71.523
ShonaLatin0.37OpenAI GPT-4.10.31Anthropic Claude Sonnet 50.531.71.524
UkrainianCyrillic0.37xAI Grok 4.20 (reasoning)0.3AI21 Jamba Large 1.70.6121.428
UzbekLatin0.37OpenAI GPT-5.6 Sol0.32Anthropic Claude Sonnet 4.60.6321.524
XhosaLatin0.37Cohere Tiny Aya0.31Anthropic Claude Opus 4.80.551.81.624
ZuluLatin0.37Cohere Tiny Aya0.31Anthropic Claude Opus 4.80.551.81.524
CzechLatin0.36Cohere Tiny Aya0.28Anthropic Claude Opus 4.80.471.71.528
EstonianLatin0.36Cohere Tiny Aya0.3Anthropic Claude Opus 4.80.51.71.425
FinnishLatin0.36Cohere Tiny Aya0.29Anthropic Claude Opus 4.80.521.81.528
HausaLatin0.36OpenAI GPT-4.10.31Anthropic Claude Opus 4.80.531.71.524
CroatianLatin0.36Cohere Tiny Aya0.28Anthropic Claude Fable 50.471.71.325
Haitian CreoleLatin0.36MiniMax M2.50.3Anthropic Claude Opus 4.80.531.71.524
SerbianLatin0.36OpenAI GPT-4.10.3Anthropic Claude Opus 4.80.481.61.425
SesothoLatin0.36AI21 Jamba Large 1.70.3Anthropic Claude Fable 50.531.81.523
SwahiliLatin0.36Cohere Tiny Aya0.29Anthropic Claude Opus 4.80.531.81.525
BosnianLatin0.35Cohere Tiny Aya0.28Anthropic Claude Opus 4.80.471.71.325
MalagasyLatin0.35MiniMax M2.50.3Anthropic Claude Fable 50.511.71.524
SlovakLatin0.35Cohere Tiny Aya0.3Anthropic Claude Opus 4.80.491.71.425
SlovenianLatin0.35Cohere Tiny Aya0.29Anthropic Claude Opus 4.80.481.71.425
BulgarianCyrillic0.34MiniMax M2.50.31AI21 Jamba Large 1.70.551.81.425
BasqueLatin0.34Cohere Tiny Aya0.3Anthropic Claude Fable 50.51.61.425
FrisianLatin0.33MiniMax M2.50.3Anthropic Claude Fable 50.491.61.423
SundaneseLatin0.33OpenAI GPT-4.10.28Anthropic Claude Fable 50.511.81.424
AfrikaansLatin0.32OpenAI GPT-5.6 Sol0.29Anthropic Claude Sonnet 50.471.61.324
CebuanoLatin0.32OpenAI GPT-4.10.28Anthropic Claude Opus 4.80.51.71.324
DanishLatin0.32Cohere Tiny Aya0.26Anthropic Claude Opus 4.80.451.71.328
JavaneseLatin0.32Cohere Tiny Aya0.21Anthropic Claude Sonnet 50.52.41.425
NorwegianLatin0.32Cohere Tiny Aya0.26Anthropic Claude Opus 4.80.461.71.328
PolishLatin0.32xAI Grok 4.20 (reasoning)0.27Anthropic Claude Sonnet 50.481.81.428
RomanianLatin0.32xAI Grok 4.20 (reasoning)0.27Anthropic Claude Fable 50.441.61.228
SwedishLatin0.32Cohere Tiny Aya0.27Anthropic Claude Opus 4.80.441.61.228
TagalogLatin0.32Cohere Tiny Aya0.26Anthropic Claude Sonnet 50.491.81.325
TurkishLatin0.31xAI Grok 4.20 (reasoning)0.27Anthropic Claude Fable 50.521.91.425
CatalanLatin0.3MiniMax M2.50.27Anthropic Claude Opus 4.80.421.51.225
DutchLatin0.29OpenAI GPT-5.6 Luna0.24Anthropic Claude Fable 50.451.91.228
VietnameseLatin0.29Cohere Tiny Aya0.27Mistral Medium 3.10.592.21.325
GermanLatin0.28OpenAI GPT-5.6 Luna0.24Anthropic Claude Opus 4.80.4821.228
MalayLatin0.28Cohere Tiny Aya0.22Anthropic Claude Sonnet 50.482.21.225
RussianCyrillic0.28OpenAI GPT-5.6 Luna0.26AI21 Jamba Large 1.70.542.11.226
GalicianLatin0.27MiniMax M2.50.24Anthropic Claude Sonnet 50.421.71.125
IndonesianLatin0.27MiniMax M2.50.21Anthropic Claude Opus 4.80.482.31.225
ItalianLatin0.27xAI Grok 4.20 (non-reasoning)0.24Anthropic Claude Fable 50.431.81.226
Portuguese (European)Latin0.27MiniMax M2.50.23Anthropic Claude Opus 4.80.421.81.128
Spanish (European)Latin0.26MiniMax M2.50.21Anthropic Claude Opus 4.80.421.91.128
FrenchLatin0.26MiniMax M2.50.22Anthropic Claude Sonnet 50.41.81.128

Output tokens per character of the translation, counted with the tokenizer of each model (or its family tokenizer), translation without context. Against English: tokens per character in the language divided by tokens per character of the English source on the same tokenizer, median over the models.

Automate translation to meet your requirements

See Intento Language Hub in action.

Book a demo
Leave feedback