Glossary terminology in LLM and MT engine translations
LLM translations meet glossary requirements more often than MT engine translations, and additional steps can correct terminology errors.
How do LLMs and MT engines compare on glossary terminology?
In the 16 languages with a glossary, translations from LLMs have terminology errors in 43% of the cases, from LLM-based MT engines in 68%, from open-weight LLMs in 80% and from other MT engines in nearly every case. At the median, translations from LLMs have 0 terminology errors per 1,000 words, against ~0.5 for open-weight LLMs and LLM-based MT engines and 12.6 for other MT engines. Translations from Google Gemini 3.1 Pro Preview use every term in all 15 glossary languages where we shortlisted them.
Translations into eight of the 16 glossary languages have a median of 0 terminology errors per 1,000 words
These are Arabic, Belarusian, Dutch, German, Italian, Japanese, Korean and European Spanish. Translations into Chinese Simplified have the highest median, 1.1 terminology errors per 1,000 words, and into Ukrainian 0.8.
Translations from Microsoft Translator have a median of 13.9 terminology errors per 1,000 words, and translations from most LLMs 0
Translations from Google Gemini 3.1 Pro Preview, Anthropic Claude Fable 5, OpenAI GPT-4.1, Anthropic Claude Sonnet 4.6, Anthropic Claude Opus 4.8 and Google Gemini 3.5 Flash have a median of 0. Translations from DeepL Next Gen have 0.5, from Anthropic Claude Sonnet 5 0.8 and from Cohere Command A Translate 1.0.
Every one-step solution with terminology errors meets the requirement as a two-step or three-step solution
On the same solutions, the one-step solutions with terminology errors have no terminology errors as two-step solutions (translation + post-edit) and as three-step solutions (translation + LQA + repair). The repair in three-step solutions can also change a term that was right, most often by paraphrasing it. We measured these fixes on LLM solutions, since we leave out the MT engine solutions as too far off.
Disclaimers
- Terminology applies in the 16 languages with a glossary. We score terminology with an LLM check that accepts declined forms, since in many languages a glossary term changes form inside a sentence.
- We assess only the shortlisted one-step solutions (10 to 13 per language by COMET-Kiwi or MetricX, with at least three real-time solutions among them), so a model has no result in a language where it is not on the shortlist.
- The results come from four business texts of ~3,000 words. Other content can show other results.
- We build two-step and three-step solutions only in the languages with fewer than two solutions that meet every Standard requirement, so their results describe those languages.
Terminology of one-step solutions, by model
| Model | Type | Languages where we shortlisted it | Languages with no terminology errors | Terminology errors per 1,000 words, median |
|---|---|---|---|---|
| Google Gemini 3.1 Pro Preview | LLM | 15 | 15 | 0 |
| Anthropic Claude Fable 5 | LLM | 12 | 7 | 0 |
| OpenAI GPT-4.1 | LLM | 10 | 6 | 0 |
| Anthropic Claude Sonnet 4.6 | LLM | 9 | 6 | 0 |
| Anthropic Claude Opus 4.8 | LLM | 8 | 5 | 0 |
| Google Gemini 3.5 Flash | LLM | 8 | 6 | 0 |
| DeepL Next Gen | LLM-based MT engine | 15 | 4 | 0.5 |
| Anthropic Claude Sonnet 5 | LLM | 10 | 2 | 0.8 |
| Cohere Command A Translate | LLM | 13 | 0 | 1 |
| Microsoft Translator | MT engine | 8 | 0 | 13.9 |
| OpenAI GPT-5.6 Sol | LLM | 7 | 5 | 0 |
| Widn AI Tower Sugarloaf | LLM-based MT engine | 7 | 4 | 0 |
| OpenAI GPT-5.5 | LLM | 6 | 5 | 0 |
| OpenAI GPT-5.6 Luna | LLM | 4 | 3 | 0 |
| z.ai GLM-5 | LLM | 4 | 3 | 0 |
| xAI Grok 4.20 (reasoning) | LLM | 3 | 3 | 0 |
| AI21 Jamba Large 1.7 | LLM | 1 | 1 | 0 |
| MoonshotAI Kimi K2.5 | LLM | 1 | 1 | 0 |
| Google Gemma 4 | Open-weight LLM | 6 | 3 | 0.1 |
| Mistral Medium 3.1 | LLM | 2 | 1 | 0.3 |
| Meta Llama 4 Maverick | Open-weight LLM | 2 | 0 | 0.4 |
| Translated Lara 2 | LLM-based MT engine | 6 | 2 | 0.5 |
| DeepSeek V3.2 | LLM | 1 | 0 | 0.5 |
| ModernMT Enterprise Edition | MT engine | 1 | 0 | 0.5 |
| Alibaba Qwen3-235B-A22B | Open-weight LLM | 2 | 0 | 0.6 |
| Mistral Large 3 | LLM | 7 | 2 | 0.8 |
| Anthropic Claude Haiku 4.5 | LLM | 1 | 0 | 0.9 |
| OpenAI GPT-5.6 Terra | LLM | 4 | 1 | 1 |
| OpenAI gpt-oss-120b | Open-weight LLM | 4 | 0 | 2.4 |
| Swiss AI Apertus-8B-2509 | Open-weight LLM | 1 | 0 | 8.1 |
| ALL-Lab | MT engine | 1 | 0 | 9.6 |
| Google Translation LLM | LLM-based MT engine | 3 | 0 | 10.1 |
| Elia Machine Translation System | MT engine | 1 | 0 | 10.2 |
| Yandex Cloud Translate v2 | MT engine | 2 | 0 | 15.8 |
| Amazon Translate | MT engine | 7 | 0 | 16 |
One-step solutions with an Intento LQA result in 82 languages, the models on fewer than eight shortlists after the others. Terminology applies in the 16 languages with a glossary, 121 glossary terms in the four texts.
Terminology of one-step solutions, by language
| Language | Terminology errors per 1,000 words, median | Models with no terminology errors |
|---|---|---|
| Arabic (MSA) | 0 |
|
| Belarusian | 0 |
|
| Dutch | 0 |
|
| German | 0 |
|
| Italian | 0 |
|
| Japanese | 0 |
|
| Korean | 0 |
|
| Spanish (European) | 0 |
|
| French | 0.2 |
|
| Romanian | 0.2 |
|
| Russian | 0.2 |
|
| Portuguese (European) | 0.4 |
|
| Estonian | 0.5 |
|
| Serbian | 0.5 |
|
| Ukrainian | 0.8 |
|
| Chinese Simplified | 1.1 |
|
One-step solutions with an Intento LQA result, in the languages where the requirement applies.