Added HTML entities in LLM and MT engine translations
Translations from Translated Lara 2 and Google Translation LLM contain added HTML entities in every language assessed.
How do LLMs and MT engines compare on HTML entities?
Translations from Translated Lara 2 and Google Translation LLM carry HTML entities that are not in the source in every language where we shortlisted them, and translations from DeepL Next Gen in 11 of 69. Translations from LLM-based MT engines carry them in 56% of the cases, from other MT engines in 18%, from LLMs in 2% and from open-weight LLMs never, and translations from each LLM in three languages or fewer.
Translations from at least one shortlisted model carry added HTML entities in 68 of the 82 languages
In Frisian, these are translations from Google Gemini 3.1 Pro Preview, Anthropic Claude Opus 4.8 and MoonshotAI Kimi K2.5, and in Belarusian translations from Translated Lara 2, Google Translation LLM and Yandex Cloud Translate v2.
Translations from DeepL Next Gen carry added HTML entities in 11 of 69 languages
Translations from ModernMT Enterprise Edition carry them in all 5 languages where we shortlisted them, from MoonshotAI Kimi K2.5 in 2 of 11 and from Google Gemini 3.1 Pro Preview in 3 of 75. Translations from Google Translation LLM have " in place of quotation marks in the JSON document.
The repair in two-step and three-step solutions removes added HTML entities
Every two-step solution (translation + post-edit) and three-step solution (translation + LQA + repair) built on a one-step solution with added entities is free of them. In two-step solutions, we also repair tags and formatting by code. We measured these fixes on LLM solutions, since we leave out the MT engine solutions as too far off.
Disclaimers
- An added HTML entity is an HTML character reference, such as
"for a quotation mark, that is in the translation but not in the source. - We assess only the shortlisted one-step solutions (10 to 13 per language by COMET-Kiwi or MetricX, with at least three real-time solutions among them), so a model has no result in a language where it is not on the shortlist.
- The results come from four business texts of ~3,000 words. Other content can show other results.
- We build two-step and three-step solutions only in the languages with fewer than two solutions that meet every Standard requirement, so their results describe those languages.
HTML entities of one-step solutions, by model
| Model | Type | Languages where we shortlisted it | Languages with added HTML entities |
|---|---|---|---|
| Translated Lara 2 | LLM-based MT engine | 44 | 44 |
| Google Translation LLM | LLM-based MT engine | 32 | 32 |
| DeepL Next Gen | LLM-based MT engine | 69 | 11 |
| ModernMT Enterprise Edition | MT engine | 5 | 5 |
| Google Gemini 3.1 Pro Preview | LLM | 75 | 3 |
| Anthropic Claude Opus 4.8 | LLM | 45 | 3 |
| Google Gemini 3.5 Flash | LLM | 63 | 2 |
| MoonshotAI Kimi K2.5 | LLM | 11 | 2 |
| Google Cloud Advanced Translation API | MT engine | 2 | 2 |
| Yandex Cloud Translate v2 | MT engine | 2 | 2 |
| OpenAI GPT-5.5 | LLM | 59 | 1 |
| Anthropic Claude Sonnet 5 | LLM | 35 | 1 |
| OpenAI GPT-5.6 Luna | LLM | 26 | 1 |
| OpenAI GPT-5.6 Terra | LLM | 20 | 1 |
| xAI Grok 4.20 (reasoning) | LLM | 15 | 1 |
| Anthropic Claude Fable 5 | LLM | 60 | 0 |
| OpenAI GPT-4.1 | LLM | 43 | 0 |
| Google Gemma 4 | Open-weight LLM | 37 | 0 |
| z.ai GLM-5 | LLM | 33 | 0 |
| Microsoft Translator | MT engine | 32 | 0 |
| OpenAI GPT-5.6 Sol | LLM | 32 | 0 |
| Anthropic Claude Sonnet 4.6 | LLM | 31 | 0 |
| Mistral Large 3 | LLM | 28 | 0 |
| Cohere Command A Translate | LLM | 22 | 0 |
| DeepSeek V3.2 | LLM | 14 | 0 |
| OpenAI gpt-oss-120b | Open-weight LLM | 14 | 0 |
| Widn AI Tower Sugarloaf | LLM-based MT engine | 10 | 0 |
| xAI Grok 4.20 (non-reasoning) | LLM | 8 | 0 |
| Amazon Translate | MT engine | 7 | 0 |
| Mistral Medium 3.1 | LLM | 4 | 0 |
| Alibaba Qwen3-235B-A22B | Open-weight LLM | 2 | 0 |
| Anthropic Claude Haiku 4.5 | LLM | 2 | 0 |
| Meta Llama 4 Maverick | Open-weight LLM | 2 | 0 |
| AI21 Jamba Large 1.7 | LLM | 1 | 0 |
| ALL-Lab | MT engine | 1 | 0 |
| Elia Machine Translation System | MT engine | 1 | 0 |
| MiniMax M2.5 | LLM | 1 | 0 |
| Swiss AI Apertus-8B-2509 | Open-weight LLM | 1 | 0 |
One-step solutions with an Intento LQA result in 82 languages, the models on fewer than eight shortlists after the others. Met: the requirement holds in all four texts.
HTML entities of one-step solutions, by language
| Language | Models with added HTML entities |
|---|---|
| Belarusian |
|
| Dutch |
|
| Frisian |
|
| Polish |
|
| Serbian |
|
| Tagalog |
|
| Albanian |
|
| Amharic |
|
| Bosnian |
|
| Czech |
|
| Estonian |
|
| French |
|
| Galician |
|
| Georgian |
|
| Greek |
|
| Hungarian |
|
| Irish |
|
| Kannada |
|
| Khmer |
|
| Lao |
|
| Latvian |
|
| Lithuanian |
|
| Marathi |
|
| Mongolian |
|
| Punjabi |
|
| Romanian |
|
| Russian |
|
| Sinhala |
|
| Slovak |
|
| Slovenian |
|
| Swedish |
|
| Tajik |
|
| Tamil |
|
| Thai |
|
| Turkish |
|
| Ukrainian |
|
| Vietnamese |
|
| Afrikaans |
|
| Arabic (MSA) |
|
| Armenian |
|
| Azerbaijani |
|
| Basque |
|
| Bulgarian |
|
| Burmese |
|
| Catalan |
|
| Cebuano |
|
| Croatian |
|
| Danish |
|
| Farsi |
|
| Finnish |
|
| German |
|
| Hebrew |
|
| Hindi |
|
| Icelandic |
|
| Indonesian |
|
| Italian |
|
| Korean |
|
| Kyrgyz |
|
| Macedonian |
|
| Malayalam |
|
| Nepali |
|
| Norwegian |
|
| Odia (Oriya) |
|
| Portuguese (European) |
|
| Swahili |
|
| Tatar |
|
| Telugu |
|
| Urdu |
|
| Assamese | |
| Bengali | |
| Chinese Simplified | |
| Chinese Traditional | |
| Gujarati | |
| Japanese | |
| Kazakh | |
| Malay | |
| Pashto | |
| Spanish (European) | |
| Sundanese | |
| Uzbek | |
| Welsh | |
| Yiddish |
One-step solutions with an Intento LQA result, in the languages where the requirement applies.