Critical errors occur more often in MT engine translations than in LLM translations.
Among LLMs, 71% to 100% of the solutions of each model have no critical error. Translations from xAI Grok 4.20 (reasoning) and MoonshotAI Kimi K2.5 have no critical error in any language where we shortlisted them, and translations from Anthropic Claude Opus 4.8, Claude Fable 5 and OpenAI GPT-5.6 Sol have none in 97% of their languages or more. Translations from DeepSeek V3.2 have none in 71% of the languages where we shortlisted it.
Among MT engines, translations from DeepL Next Gen have no critical error in 83% of the languages, against 59% for Microsoft Translator. Translations from Google Translation LLM carry ~2.8 critical errors per 1,000 words.
Critical and major translation errors, by model
| Model | Type | Languages | Solutions without a critical error | Major errors per 1,000 words | Critical errors per 1,000 words |
|---|---|---|---|---|---|
| xAI Grok 4.20 (reasoning) | LLM | 15 | 100% | 3.1 | 0 |
| MoonshotAI Kimi K2.5 | LLM | 11 | 100% | 3.8 | 0 |
| Anthropic Claude Opus 4.8 | LLM | 45 | 98% | 1.55 | 0.01 |
| OpenAI GPT-5.6 Sol | LLM | 32 | 97% | 0.64 | 0.01 |
| Anthropic Claude Fable 5 | LLM | 60 | 97% | 0.7 | 0.01 |
| z.ai GLM-5 | LLM | 33 | 97% | 3.24 | 0.01 |
| Google Gemini 3.1 Pro Preview | LLM | 75 | 96% | 0.91 | 0.02 |
| Anthropic Claude Sonnet 5 | LLM | 35 | 94% | 2.09 | 0.02 |
| OpenAI GPT-5.5 | LLM | 59 | 93% | 0.77 | 0.04 |
| OpenAI GPT-4.1 | LLM | 43 | 91% | 2.55 | 0.05 |
| OpenAI GPT-5.6 Terra | LLM | 20 | 90% | 0.73 | 0.03 |
| Anthropic Claude Sonnet 4.6 | LLM | 31 | 87% | 1.87 | 0.07 |
| Mistral Large 3 | LLM | 28 | 86% | 3.59 | 0.06 |
| OpenAI gpt-oss-120b | Open-weight LLM | 14 | 86% | 7.09 | 0.21 |
| OpenAI GPT-5.6 Luna | LLM | 26 | 85% | 1.28 | 0.06 |
| Google Gemini 3.5 Flash | LLM | 63 | 83% | 1.4 | 0.16 |
| DeepL Next Gen | LLM-based MT engine | 69 | 83% | 3.29 | 0.07 |
| Cohere Command A Translate | LLM | 22 | 82% | 3.58 | 1.39 |
| xAI Grok 4.20 (non-reasoning) | LLM | 8 | 75% | 6.01 | 2.1 |
| Translated Lara 2 | LLM-based MT engine | 44 | 73% | 3.98 | 0.11 |
| DeepSeek V3.2 | LLM | 14 | 71% | 3.23 | 0.14 |
| Widn AI Tower Sugarloaf | LLM-based MT engine | 10 | 70% | 5.9 | 0.13 |
| Google Gemma 4 | Open-weight LLM | 37 | 68% | 2.81 | 0.2 |
| Microsoft Translator | MT engine | 32 | 59% | 9.36 | 0.22 |
| Google Translation LLM | LLM-based MT engine | 32 | 0% | 3.75 | 2.77 |
| Amazon Translate | MT engine | 7 | 0% | 7.44 | 1.69 |
| ModernMT Enterprise Edition | MT engine | 5 | 0% | 7.71 | 0.46 |
Models with one-step solutions in five or more of the 82 languages. Errors per 1,000 words, the mean over the languages of the model.