Formal tone of voice in LLM and MT engine translations

How do LLMs and MT engines compare on tone of voice?

Translations from LLMs have register errors in 9% of the cases, from open-weight LLMs in 21%, from LLM-based MT engines in 28% and from other MT engines in 52%. A register error is a sentence in the second person or the imperative that is not in the formal register. At the mean, translations from LLMs have 0.1 register errors per 1,000 words, against 0.6 for open-weight LLMs, 1.1 for LLM-based MT engines and 3.5 for other MT engines.

Translations into 51 of the 82 languages have no register errors

Translations into Cebuano, Finnish and Sundanese have the most, a mean of 7.8, 5.3 and 3.9 register errors per 1,000 words. In 80 of the 82 languages, two or more one-step solutions keep the formal register in every sentence where it applies.

Translations from Microsoft Translator have a mean of 3.8 register errors per 1,000 words, the most of any model

Translations from Widn AI Tower Sugarloaf have 1.9, from DeepL Next Gen 1.4 and from Google Translation LLM 1.3, while translations from most LLMs have 0.2 or fewer.

One-step solutions below the bar have a median of 0.7 register errors per 1,000 words, 0.3 as two-step solutions and 0 as three-step solutions

A one-step solution is below the bar with a single register error, since tone of voice requires the formal register in every sentence where it applies. On the same solutions, 68% have no register errors as two-step solutions (translation + post-edit) and 73% as three-step solutions (translation + LQA + repair). We measured these fixes on LLM solutions, since we leave out the MT engine solutions as too far off.

Disclaimers

• Tone of voice applies to the 47 of the 177 sentences in the second person or the imperative. Every instruction opens with the same sentence about a formal register, and a line for the language follows it, naming the forms to use.
• We report the mean by model and by language, since the median is 0 for nearly every model.
• We assess only the shortlisted one-step solutions (10 to 13 per language by COMET-Kiwi or MetricX, with at least three real-time solutions among them), so a model has no result in a language where it is not on the shortlist.
• The results come from four business texts of ~3,000 words. Other content can show other results.
• We build two-step and three-step solutions only in the languages with fewer than two solutions that meet every Standard requirement, so their results describe those languages.