Unicode character errors in LLM and MT engine translations

How do LLMs and MT engines compare on Unicode character integrity?

Translations from LLMs have Unicode errors in 3% of the cases, from MT engines in 4%, from LLM-based MT engines in 6% and from open-weight LLMs in 16%. Translations from 9 of the 25 models on eight or more shortlists keep Unicode characters intact in every language where we shortlisted them, LLMs and MT engines among them. Most Unicode errors are look-alike letters from another script, and the others are corrupted characters.

Translations from every shortlisted model keep Unicode characters intact in 60 of the 82 languages

In Kyrgyz and Tatar, translations from five shortlisted models each carry a look-alike letter from another script, OpenAI GPT-5.5 in both, and in Macedonian translations from Google Gemma 4, Anthropic Claude Sonnet 5 and DeepL Next Gen. In Slovak, translations from Google Gemini 3.5 Flash, z.ai GLM-5, DeepL Next Gen and Google Translation LLM carry corrupted characters.

Translations from Google Gemma 4 keep Unicode characters intact in 30 of 37 languages and from xAI Grok 4.20 (non-reasoning) in 6 of 8

Translations from Google Translation LLM keep them intact in 27 of 32 languages and from OpenAI gpt-oss-120b in 12 of 14. Translations from Anthropic Claude Fable 5, Google Gemini 3.1 Pro Preview, OpenAI GPT-4.1, OpenAI GPT-5.6 Sol and OpenAI GPT-5.6 Luna keep them intact in every language where we shortlisted them.

Three-step solutions keep Unicode characters intact

As three-step solutions (translation + LQA + repair), every one-step solution with a Unicode error keeps characters intact. We measured these fixes on LLM solutions, since we leave out the MT engine solutions as too far off.

Disclaimers

• Unicode character integrity covers replacement characters, control characters, mojibake (text decoded with the wrong encoding), stray byte-order marks and look-alike letters from another script. We check look-alike letters in languages written in Latin, Cyrillic or Greek script.
• We assess only the shortlisted one-step solutions (10 to 13 per language by COMET-Kiwi or MetricX, with at least three real-time solutions among them), so a model has no result in a language where it is not on the shortlist.
• The results come from four business texts of ~3,000 words. Other content can show other results.
• We build two-step and three-step solutions only in the languages with fewer than two solutions that meet every Standard requirement, so their results describe those languages.