General translation quality of LLMs and MT engines

How do LLMs and MT engines compare on general translation quality?

A solution meets general translation quality with no critical error, at most three major errors and at most 20 weighted errors per 1,000 words in the four texts (minor 1, major 5, critical 10). Translations from LLMs meet it in 21% of the cases, from open-weight LLMs in 2%, from LLM-based MT engines in 1% and from other MT engines never. At the median, translations from LLMs have 30 weighted errors per 1,000 words, against 47 for LLM-based MT engines, 52 for open-weight LLMs and 74 for other MT engines.

Two or more one-step solutions meet general translation quality in 41 of the 82 languages

Translations into Indonesian and European Spanish have the lowest median, 19 weighted errors per 1,000 words, and translations into Odia and Tatar the highest, 73 and 78. In 13 other languages a single one-step solution meets general translation quality. In the remaining 28, every one-step solution misses it, and these are Afrikaans, Amharic, Armenian, Assamese, Basque, Burmese, Cebuano, Chinese Simplified, Chinese Traditional, French, Frisian, Icelandic, Irish, Japanese, Kannada, Korean, Kyrgyz, Malayalam, Mongolian, Odia, Pashto, Sundanese, Swahili, Tagalog, Tajik, Tatar, Uzbek and Yiddish.

Translations from six LLMs have a median of 25 weighted errors per 1,000 words or fewer

These are OpenAI GPT-5.6 Sol, Anthropic Claude Fable 5, OpenAI GPT-5.6 Terra, OpenAI GPT-5.5, Anthropic Claude Opus 4.8 and Google Gemini 3.1 Pro Preview. Among MT engines, translations from Translated Lara 2 have the lowest median, 38.2 weighted errors per 1,000 words, and only translations from DeepL Next Gen meet general translation quality, in German and Romanian.

One-step solutions below the bar have a median of 36 weighted errors per 1,000 words, 28 as two-step solutions and 16 as three-step solutions

On the same solutions, 24% meet general translation quality as two-step solutions (translation + post-edit) and 60% as three-step solutions (translation + LQA + repair), and the median of the three-step solutions is below the bar of 20 weighted errors per 1,000 words. We measured these fixes on LLM solutions, since we leave out the MT engine solutions as too far off.

Disclaimers

• We assess only the shortlisted one-step solutions (10 to 13 per language by COMET-Kiwi or MetricX, with at least three real-time solutions among them), so a model has no result in a language where it is not on the shortlist.
• The results come from four business texts of ~3,000 words. Other content can show other results.
• We build two-step and three-step solutions only in the languages with fewer than two solutions that meet every Standard requirement, so their results describe those languages.