HTML tag handling in LLM and MT engine translations

How do LLMs and MT engines compare on tag handling?

Translations from LLMs lose tags in 41% of the cases, from open-weight LLMs in 71%, from LLM-based MT engines in 80% and from other MT engines in 98%. A tag error is a tag of the source lost or changed, checked with its attributes, and at the median, translations from LLMs have 0 tag errors per 1,000 words, against 1.3 for open-weight LLMs, 5.9 for MT engines and 8.2 for LLM-based MT engines. Translations from Google Gemini 3.1 Pro Preview keep every tag in 72 of the 75 languages where we shortlisted them, and translations from OpenAI GPT-5.5 in 52 of 59.

Translations into 33 of the 82 languages have a median of 0 tag errors per 1,000 words

Translations into European Spanish, Japanese, Chinese Simplified and Marathi have the highest medians, 2.0 to 2.3.

Translations from Google Gemini 3.1 Pro Preview keep every tag in 72 of 75 languages, and translations from Translated Lara 2 in 31 of 44

Of the 28 MT engines we tested, 11 reach a shortlist with an Intento LQA result. Translations from DeepL Next Gen, Microsoft Translator, Google Translation LLM and Widn AI Tower Sugarloaf have tag errors in every language where we shortlisted them, with medians of 5.9 to 59.8 tag errors per 1,000 words, as do translations from Amazon Translate, ModernMT Enterprise Edition, Google Cloud Advanced Translation API, Yandex Cloud Translate v2 and Elia Machine Translation System, which are on fewer shortlists. Among LLMs, translations from Anthropic Claude Opus 4.8 keep every tag in 11 of 45 languages and from Anthropic Claude Sonnet 4.6 in 6 of 31.

One-step solutions below the bar have a median of 1.3 tag errors per 1,000 words, 0.7 as two-step solutions and 0 as three-step solutions

On the same solutions, 59% keep every tag as two-step solutions (translation + post-edit) and 56% as three-step solutions (translation + LQA + repair). Where tags or formatting were the only problem, we repair the tags by code, without an LLM, and 90% of these two-step solutions keep every tag. We measured these fixes on LLM solutions, since we leave out the MT engine solutions as too far off.

Disclaimers

• We check tags strictly, every tag instance of the source with its attributes. We send every text as HTML with the format or tag-handling parameter of the API set wherever one exists, and we tell the LLMs in the prompt to keep every tag in place.
• We assess only the shortlisted one-step solutions (10 to 13 per language by COMET-Kiwi or MetricX, with at least three real-time solutions among them), so a model has no result in a language where it is not on the shortlist.
• The results come from four business texts of ~3,000 words. Other content can show other results.
• We build two-step and three-step solutions only in the languages with fewer than two solutions that meet every Standard requirement, so their results describe those languages.