Some translation errors and unmet requirements recur across languages for the same model.
Missed tags in LLM translations are few. In an LLM translation that misses tag handling, a median of 4 of the 508 tags is missing or extra, against 18 in an MT engine translation off the shelf and 25 adjusted. In translations from Anthropic Claude Opus 4.8, Claude Sonnet 4.6 and Claude Fable 5, nearly all missed tags are in the JSON document.
Error types differ by model. Translations from OpenAI GPT-4.1 and Mistral Large 3 have about twice as many omissions as the other LLMs shortlisted for the same languages, and translations from Google Gemini 3.5 Flash and Google Gemma 4 about twice as many additions. Translations from Microsoft Translator have about twice as many mistranslations and grammar errors. Translations from OpenAI gpt-oss-120b carry non-breaking hyphens in place of hyphens, and translations from Google Translation LLM carry HTML codes in place of quotation marks in the JSON document.
Format settings per MT engine. We set the format parameter for tagged content where the API has one. The LLMs have no format parameter, so in the prompt we ask them to keep every HTML tag of the source in place.
| Model | Format setting |
|---|---|
| DeepL Next Gen | tag_handling: html |
| Google Cloud Advanced Translation API | mimeType: text/html |
| Google Translation LLM | mimeType: text/html |
| Microsoft Translator | textType: html |
| ModernMT Enterprise Edition | format: text/html |
| Yandex | format: HTML |
| Elia | content_type: html |
| Translated Lara 2 | content_type: text/html |
| Amazon, ALL-Lab, Widn AI Tower Sugarloaf | not set |
Share of one-step solutions meeting each requirement, by model
| Model | Type | Languages | General translation quality | Tone of voice | Terminology (languages) | Tag handling | Formatting | Valid JSON |
|---|---|---|---|---|---|---|---|---|
| Google Gemini 3.1 Pro Preview | LLM, adjusted | 76 | 29% | 92% | 100% (15) | 96% | 87% | 100% |
| Google Gemini 3.5 Flash | LLM, adjusted | 63 | 8% | 94% | 75% (8) | 56% | 90% | 98% |
| OpenAI GPT-5.5 | LLM, adjusted | 62 | 39% | 89% | 83% (6) | 89% | 76% | 100% |
| Anthropic Claude Fable 5 | LLM, adjusted | 60 | 55% | 95% | 58% (12) | 68% | 95% | 98% |
| DeepL Next Gen | MT engine, off the shelf | 56 | 0% | 68% | 0% (1) | 0% | 0% | 46% |
| Anthropic Claude Opus 4.8 | LLM, adjusted | 46 | 26% | 89% | 62% (8) | 24% | 91% | 100% |
| OpenAI GPT-4.1 | LLM, adjusted | 45 | 2% | 93% | 60% (10) | 71% | 98% | 98% |
| Translated Lara 2 | LLM, adjusted | 44 | 0% | 91% | 33% (6) | 70% | 0% | 0% |
| Google Gemma 4 | LLM, adjusted | 37 | 3% | 86% | 50% (6) | 30% | 78% | 100% |
| Anthropic Claude Sonnet 5 | LLM, adjusted | 35 | 3% | 97% | 20% (10) | 31% | 86% | 100% |
| GLM-5 | LLM, adjusted | 33 | 3% | 97% | 75% (4) | 88% | 91% | 97% |
| OpenAI GPT-5.6 Sol | LLM, adjusted | 33 | 61% | 94% | 71% (7) | 61% | 97% | 100% |
| Microsoft Translator | MT engine, off the shelf | 32 | 0% | 50% | 0% (8) | 0% | 0% | 0% |
| Anthropic Claude Sonnet 4.6 | LLM, adjusted | 31 | 0% | 90% | 67% (9) | 19% | 68% | 68% |
| Google Translation LLM | MT engine, adjusted | 28 | 0% | 68% | 0% | 0% | 0% | |
| Mistral Large 3 | LLM, adjusted | 28 | 0% | 96% | 29% (7) | 36% | 43% | 96% |
| OpenAI GPT-5.6 Luna | LLM, adjusted | 26 | 23% | 88% | 75% (4) | 46% | 85% | 100% |
| Cohere Command A (command-a-translate-08-2025) | LLM, adjusted | 22 | 5% | 64% | 0% (13) | 27% | 91% | 95% |
| OpenAI GPT-5.6 Terra | LLM, adjusted | 20 | 40% | 90% | 25% (4) | 80% | 85% | 100% |
| Grok 4.20 (reasoning) | LLM, adjusted | 16 | 6% | 81% | 100% (3) | 38% | 88% | 100% |
| DeepL Next Gen | MT engine, adjusted | 15 | 13% | 60% | 29% (14) | 0% | 13% | 93% |
| DeepSeek V3.2 | LLM, adjusted | 14 | 0% | 86% | 0% (1) | 57% | 93% | 93% |
| MoonshotAI Kimi K2.5 | LLM, adjusted | 12 | 0% | 67% | 100% (1) | 42% | 25% | 92% |
| OpenAI gpt-oss-120b | LLM, adjusted | 12 | 0% | 83% | 0% (3) | 42% | 83% | 100% |
| Widn AI Tower Sugarloaf | LLM, adjusted | 10 | 0% | 50% | 57% (7) | 0% | 0% | 0% |
| Grok 4.20 (non-reasoning) | LLM, adjusted | 9 | 0% | 89% | 11% | 67% | 89% | |
| Amazon | MT engine, off the shelf | 7 | 0% | 43% | 0% (7) | 0% | 0% | 0% |
| Google Cloud Advanced Translation API | MT engine, off the shelf | 4 | 0% | 0% | 0% | 0% | 0% | |
| Google Translation LLM | MT engine, off the shelf | 4 | 0% | 25% | 0% (3) | 0% | 0% | 0% |
| Mistral Medium 3.1 | LLM, adjusted | 4 | 0% | 50% | 50% (2) | 0% | 0% | 100% |
| ModernMT Enterprise Edition | MT engine, off the shelf | 4 | 0% | 50% | 0% | 0% | 75% | |
| AI21 Jamba Large 1.7 | LLM, adjusted | 2 | 0% | 50% | 100% (1) | 0% | 100% | 100% |
| Alibaba Cloud Qwen3-235B-A22B | LLM, adjusted | 2 | 0% | 0% | 0% (2) | 0% | 100% | 100% |
| Anthropic Claude Haiku 4.5 | LLM, adjusted | 2 | 0% | 100% | 0% (1) | 0% | 50% | 50% |
| Meta Llama 4 Maverick | LLM, adjusted | 2 | 0% | 50% | 0% (2) | 0% | 100% | 100% |
| Yandex | MT engine, off the shelf | 2 | 0% | 100% | 0% (2) | 0% | 0% | 0% |
| ALL-Lab | MT engine, off the shelf | 1 | 0% | 0% | 0% (1) | 100% | 100% | 100% |
| Apertus-8B-2509 | LLM, adjusted | 1 | 0% | 100% | 0% (1) | 0% | 100% | 100% |
| Elia Elhuyar Machine Translation System | MT engine, off the shelf | 1 | 0% | 0% | 0% (1) | 0% | 0% | 0% |
| M2.5 | LLM, adjusted | 1 | 0% | 100% | 0% | 0% | 0% | |
| ModernMT Enterprise Edition | MT engine, adjusted | 1 | 0% | 100% | 0% (1) | 0% | 0% | 0% |
Share of the one-step solutions of the model with a conclusive LQA result that meet the requirement, one solution per language. Terminology only in the languages with a glossary. Valid JSON is one of the formatting requirements. Adjusted MT engines: a glossary and a formality setting through the API. Adjusted LLMs: tone of voice, terminology and other context in the request.
General translation quality errors against the other LLMs shortlisted for the same languages, by model and error type
| Model | Type | Languages | Errors against the other LLMs | More frequent error types | Less frequent error types |
|---|---|---|---|---|---|
| OpenAI GPT-5.6 Sol | LLM, adjusted | 32 | ×0.5 | grammar (×0.4) | |
| Anthropic Claude Fable 5 | LLM, adjusted | 60 | ×0.6 | omission (×0.5) | |
| OpenAI GPT-5.5 | LLM, adjusted | 59 | ×0.6 | omission (×0.4), addition (×0.4) | |
| OpenAI GPT-5.6 Terra | LLM, adjusted | 20 | ×0.6 | ||
| OpenAI GPT-5.6 Luna | LLM, adjusted | 26 | ×0.7 | ||
| Google Gemini 3.1 Pro Preview | LLM, adjusted | 75 | ×0.7 | omission (×0.5) | |
| Anthropic Claude Opus 4.8 | LLM, adjusted | 44 | ×0.8 | omission (×0.4), addition (×0.5) | |
| DeepL Next Gen | MT engine, adjusted | 15 | ×0.9 | ||
| Google Gemini 3.5 Flash | LLM, adjusted | 62 | ×1.0 | addition (×2.1) | |
| Anthropic Claude Sonnet 5 | LLM, adjusted | 35 | ×1.1 | ||
| Anthropic Claude Sonnet 4.6 | LLM, adjusted | 31 | ×1.1 | ||
| DeepSeek V3.2 | LLM, adjusted | 14 | ×1.1 | addition (×0.4) | |
| GLM-5 | LLM, adjusted | 33 | ×1.2 | ||
| Translated Lara 2 | LLM, adjusted | 44 | ×1.2 | stray markup (×9.1), character encoding (×6.5) | |
| Cohere Command A (command-a-translate-08-2025) | LLM, adjusted | 22 | ×1.2 | ||
| Grok 4.20 (reasoning) | LLM, adjusted | 15 | ×1.2 | ||
| Google Gemma 4 | LLM, adjusted | 37 | ×1.3 | addition (×1.8) | |
| OpenAI GPT-4.1 | LLM, adjusted | 43 | ×1.3 | omission (×2.3), under-translation (×2.0) | |
| Mistral Large 3 | LLM, adjusted | 28 | ×1.4 | omission (×2.2) | |
| MoonshotAI Kimi K2.5 | LLM, adjusted | 11 | ×1.4 | grammar (×1.9) | |
| Google Translation LLM | MT engine, adjusted | 28 | ×1.4 | character encoding (×9.6), addition (×3.3), corrupted output (×2.5), omission (×2.5) | |
| Widn AI Tower Sugarloaf | LLM, adjusted | 10 | ×1.4 | grammar (×1.9), stray markup (×1.9) | |
| Grok 4.20 (non-reasoning) | LLM, adjusted | 8 | ×1.5 | untranslated text (×2.0), mistranslation (×1.8) | |
| Microsoft Translator | MT engine, off the shelf | 32 | ×1.8 | mistranslation (×2.2), grammar (×2.1), punctuation (×1.8) | addition (×0.3), over-translation (×0.4) |
| OpenAI gpt-oss-120b | LLM, adjusted | 12 | ×2.1 | character encoding (×4.1), omission (×3.2), grammar (×2.2), mistranslation (×2.2), under-translation (×1.8) |
×N compares the general translation quality errors found by Intento LQA in the translations from the model with the average of the other adjusted LLMs shortlisted for the same language, averaged over the languages of the model: ×2.1 means 2.1 times as many errors, ×1.0 as many, ×0.4 means 60% fewer. More frequent error types are those at ×1.8 or more, less frequent at ×0.5 or less.