The State of Translation Automation 2026Models

The best open-weight LLM translation has more errors than the best other LLM translation in all 50 languages with results for both.

← back

Google Gemma 4 has the best open-weight solution in 37 of the 50 languages, OpenAI gpt-oss-120b in 11 and Meta Llama 4 Maverick in 2.

The gap is widest in Welsh. There, translations from the best open-weight LLM have ~15 times the errors of translations from the best other LLM.

Errors in the best open-weight and other LLM solutions, by language

LanguageOpen-weight LLM with the fewest errorsIts weighted errorsOther LLM with the fewest errorsIts weighted errorsError ratio, open-weight to other LLM
IndonesianGoogle Gemma 417.8Google Gemini 3.1 Pro Preview12.51.42
SwahiliGoogle Gemma 454.7Anthropic Claude Fable 536.31.51
TagalogGoogle Gemma 435.9OpenAI GPT-5.6 Terra21.41.68
CebuanoGoogle Gemma 461.3Google Gemini 3.1 Pro Preview34.91.76
KoreanGoogle Gemma 440.5OpenAI GPT-5.6 Terra21.41.89
RussianGoogle Gemma 424.7Anthropic Claude Fable 512.91.91
SerbianGoogle Gemma 437.2Anthropic Claude Fable 517.82.09
AfrikaansGoogle Gemma 463.9OpenAI GPT-5.528.72.23
HindiGoogle Gemma 426.4Anthropic Claude Fable 511.52.3
TurkishGoogle Gemma 428.3Anthropic Claude Fable 512.22.32
DutchOpenAI gpt-oss-120b33.9OpenAI GPT-5.514.52.34
GujaratiGoogle Gemma 445.2Anthropic Claude Fable 518.82.4
Chinese SimplifiedOpenAI gpt-oss-120b54.4OpenAI GPT-5.6 Terra22.72.4
FrenchOpenAI gpt-oss-120b54.1Anthropic Claude Fable 522.12.45
FarsiGoogle Gemma 432Google Gemini 3.1 Pro Preview12.92.48
UrduGoogle Gemma 445.8Anthropic Claude Fable 518.52.48
KannadaGoogle Gemma 459.3OpenAI GPT-5.6 Terra23.72.5
ThaiGoogle Gemma 449.8Anthropic Claude Opus 4.819.82.52
NorwegianGoogle Gemma 425OpenAI GPT-5.59.92.53
ItalianGoogle Gemma 428.7OpenAI GPT-5.511.22.56
VietnameseGoogle Gemma 435.3Anthropic Claude Opus 4.813.82.56
BosnianGoogle Gemma 446.1Google Gemini 3.1 Pro Preview17.52.63
Arabic (MSA)Google Gemma 443.2OpenAI GPT-5.516.22.67
SundaneseOpenAI gpt-oss-120b97.9OpenAI GPT-5.535.32.77
AlbanianGoogle Gemma 451.7Anthropic Claude Fable 518.12.86
UkrainianGoogle Gemma 428.3Anthropic Claude Fable 59.92.86
KhmerGoogle Gemma 453.4OpenAI GPT-5.6 Sol18.52.89
MalayalamGoogle Gemma 451.7OpenAI GPT-5.6 Sol16.83.08
Portuguese (European)Meta Llama 4 Maverick46.1Anthropic Claude Fable 514.83.11
AssameseGoogle Gemma 495.6OpenAI GPT-5.530.33.16
PolishGoogle Gemma 443.5OpenAI GPT-5.6 Terra13.53.22
MarathiGoogle Gemma 454.4OpenAI GPT-5.6 Sol16.23.36
AzerbaijaniGoogle Gemma 456OpenAI GPT-5.516.53.39
Odia (Oriya)Google Gemma 479.1OpenAI GPT-5.6 Luna23.13.42
BengaliGoogle Gemma 459OpenAI GPT-5.517.13.45
GermanMeta Llama 4 Maverick36.9Google Gemini 3.1 Pro Preview10.53.51
SinhalaGoogle Gemma 460Anthropic Claude Fable 517.13.51
MacedonianGoogle Gemma 442.8OpenAI GPT-5.6 Terra11.93.6
CatalanOpenAI gpt-oss-120b46.8Google Gemini 3.1 Pro Preview11.93.93
MalayOpenAI gpt-oss-120b43.5OpenAI GPT-5.510.93.99
PunjabiGoogle Gemma 469.9OpenAI GPT-5.6 Sol16.84.16
AmharicGoogle Gemma 4104.5OpenAI GPT-5.6 Sol23.14.52
HebrewOpenAI gpt-oss-120b75.8Anthropic Claude Fable 516.54.59
TeluguGoogle Gemma 467.6Anthropic Claude Fable 514.54.66
CroatianGoogle Gemma 447.8OpenAI GPT-5.59.64.98
BulgarianOpenAI gpt-oss-120b63.3Google Gemini 3.1 Pro Preview11.25.65
TamilGoogle Gemma 465.6OpenAI GPT-5.6 Sol10.56.25
TatarOpenAI gpt-oss-120b206Google Gemini 3.1 Pro Preview306.87
PashtoOpenAI gpt-oss-120b182.6OpenAI GPT-5.6 Sol26.46.92
WelshOpenAI gpt-oss-120b189.5Google Gemini 3.1 Pro Preview12.914.69

One-step solutions with an Intento LQA result. Open-weight LLMs: Google Gemma 4, OpenAI gpt-oss-120b, Meta Llama 4 Maverick, Alibaba Qwen3-235B-A22B and Swiss AI Apertus-8B (SpeakLeash Bielik-11B has no solution on our shortlists). Weighted errors: minor 1, major 5, critical 10.

Automate translation to meet your requirements

See Intento Language Hub in action.

Book a demo
Leave feedback