The State of Translation Automation 2026 · Languages
What it takes to automate translation from English into Zulu against 10 requirements in three levels: the solutions that meet them, what they cost and how fast they are, from the 2026 benchmark of 57 AI models and MT engines.
A solution is a model plus its harness: prompts, glossaries and other context, pre- and post-processing. It meets a quality level when it meets every requirement of the level.
| Requirement | Description | Met when |
|---|---|---|
| Basic requirements for any company | ||
| General translation quality | Translation errors of every type, such as mistranslations, omissions and grammar errors, classified by type and severity following MQM, the industry standard. | No critical error, at most 3 major errors in the four test texts (3,034 words) and at most 20 errors per 1,000 words, weighted by severity (minor 1, major 5, critical 10). |
| Standard requirements adds the rules companies set for their content | ||
| Terminology | The approved translation of every glossary term, in any of its correct forms. | 100% of the glossary terms. We test it in the 16 languages with a glossary, so it does not apply in Zulu. |
| Tone of voice | The formal register in every sentence in the second person or the imperative. | 100% of those sentences. |
| Tags | Every HTML tag of the source, with its attributes, in the translation. | 100% of the tags. |
| Unicode character integrity | Every character intact, with no corrupted characters and no letter from another script that only looks like the right one. | 100% of the characters. |
| Locale conventions | The punctuation marks of the target language in place of the English ones. | No pass-or-fail bar. We report it as a share, because whether to localize punctuation is a matter of house style. |
| HTML entities | No HTML character code, such as the code for a quotation mark, that the source does not carry. | No added HTML entity. |
| Valid JSON | The translated JSON document still parses. | The document parses. |
| Spacing at tag boundaries | The spaces next to HTML tags stay as in the source. | 100% of the tag boundaries. |
| Advanced requirements adds a requirement for some companies and use cases | ||
| Gender neutrality | A gender-neutral translation wherever the English source leaves the gender of a person open, as in “your manager” or “the expert”. | 100% of the sentences that refer to a person. |
| Solution | Sentences on which the three judges agree (%) | Status |
|---|---|---|
| OpenAI GPT-5.6 Luna | 89.3% | Does not pass |
| OpenAI GPT-5.5 | 87.6% | Does not pass |
| Anthropic Claude Fable 5 | 85.3% | Does not pass |
| Anthropic Claude Opus 4.8 | 85.3% | Does not pass |
| xAI Grok 4.20 (reasoning) | 84.7% | Does not pass |
| Google Gemini 3.1 Pro Preview | 84.2% | Does not pass |
| DeepL Next Gen | 81.9% | Does not pass |
| OpenAI gpt-oss-120b | 81.4% | Does not pass |
| Translated Lara 2 | 79.1% | Does not pass |
| OpenAI GPT-4.1 | 78% | Does not pass |
The three LQA judges agree most often on the translations of the solution on OpenAI GPT-5.6 Luna, on 89.3% of the sentences, just under the 90% they must reach. They agree least often on the translations of the solution on OpenAI GPT-4.1, on 78.0% of the sentences. The similarity pair agrees at 0.4852, with an interval of 0.4719 to 0.4981 above its threshold of 0.4613. So the automatic scores can rank models within Zulu.
Pick a quality level, or your own requirements, bars and constraints, to see the shortlisted solutions that meet them in Zulu, with a column for each requirement you select. Under Optimize for, choose which of them comes first: the one with the lowest cost or the shortest time. Click a column heading to sort by it.
The shortlisted solutions on 10 models, with their results on tone of voice, tags and formatting, which do not depend on the agreement of the three LQA judges, and on general translation quality only where the judges agree. We cannot assess translation quality in Zulu: why. The models that did not make the shortlist have automatic scores only: the shortlist. Inconclusive: the three LQA judges agree on fewer than 90% of the sentences of the solution, so it has no general translation quality result (09-02). Gender neutrality: the Advanced requirements count the two best solutions of the language, with the gender-neutrality agent where they need it. The results of the other solutions are shown for information.
Of the three shortlisted solutions that meet tone of voice, tag handling and formatting in Zulu, the one on OpenAI GPT-5.6 Luna costs $0.46 per million characters and takes 18 seconds per 1,000 words. The ones on Anthropic Claude Opus 4.8 and Anthropic Claude Fable 5 cost $16.86 and $33.28 per million characters and take 122 and 100 seconds per 1,000 words. Of the three, a cost cap of at most $16.85 per million characters or a time limit under 100 seconds per 1,000 words leaves only the one on OpenAI GPT-5.6 Luna.
Three more shortlisted solutions in Zulu meet tone of voice and formatting and miss tag handling by a small margin: translations from xAI Grok 4.20 (reasoning) have 99.6% of the tags right, from OpenAI gpt-oss-120b 99.2% and from OpenAI GPT-4.1 98.4%.
We cannot assess general translation quality in Zulu: how confident we are.
We cannot assess general translation quality in Zulu: how confident we are.
We cannot assess general translation quality in Zulu: how confident we are.
| Model | MetricX scores of the shortlisted solutions in Zulu (lower is better) | Class |
|---|---|---|
| OpenAI gpt-oss-120b | 1.9 | Open-weight LLMs |
| Anthropic Claude Opus 4.8 | 1.89 | LLMs |
| Translated Lara 2 | 1.82 | LLM-based MT engines |
| OpenAI GPT-5.5 | 1.81 | LLMs |
| OpenAI GPT-4.1 | 1.78 | LLMs |
| xAI Grok 4.20 (reasoning) | 1.75 | LLMs |
| Google Gemini 3.1 Pro Preview | 1.75 | LLMs |
| DeepL Next Gen | 1.67 | LLM-based MT engines |
| Anthropic Claude Fable 5 | 1.65 | LLMs |
| OpenAI GPT-5.6 Luna | 1.65 | LLMs |
No translation into Zulu: AI21 Jamba Large 1.7 (garbage or near-empty), ALL-Lab (far too short), Cohere Command A Translate (garbage or near-empty).
The automatic scores rank models within Zulu and are not a requirement result. Of the three LLM-based MT engines, DeepL Next Gen is 3rd on MetricX, Translated Lara 2 8th and Google Translation LLM 16th. Of the three open-weight LLMs, OpenAI gpt-oss-120b is 10th, Google Gemma 4 18th and Swiss AI Apertus-8B-2509 26th. After Microsoft Translator in 12th place, the other four MT engines are 17th to 24th. LLMs hold the other seven places in the top 9.
| Rank | Model | MetricX (lower is better) | 83% confidence interval, from | to | Status |
|---|---|---|---|---|---|
| 1 | Anthropic Claude Fable 5 | 1.65 | 1.45 | 1.84 | Shortlisted |
| 2 | OpenAI GPT-5.6 Luna | 1.65 | 1.43 | 1.87 | Shortlisted |
| 3 | DeepL Next Gen | 1.67 | 1.46 | 1.88 | Shortlisted |
| 4 | xAI Grok 4.20 (reasoning) | 1.75 | 1.54 | 1.96 | Shortlisted |
| 5 | Google Gemini 3.1 Pro Preview | 1.75 | 1.54 | 1.96 | Shortlisted |
| 6 | OpenAI GPT-4.1 | 1.78 | 1.56 | 2.01 | Shortlisted |
| 7 | OpenAI GPT-5.5 | 1.81 | 1.55 | 2.08 | Shortlisted |
| 8 | Translated Lara 2 | 1.82 | 1.59 | 2.05 | Shortlisted |
| 9 | Anthropic Claude Opus 4.8 | 1.89 | 1.65 | 2.13 | Shortlisted |
| 10 | OpenAI gpt-oss-120b | 1.90 | 1.67 | 2.13 | Shortlisted |
| 11 | Google Gemini 3.5 Flash | 1.92 | 1.68 | 2.15 | Not shortlisted |
| 12 | Microsoft Translator | 1.95 | 1.72 | 2.19 | Not shortlisted |
| 13 | Anthropic Claude Sonnet 4.6 | 1.96 | 1.75 | 2.18 | Not shortlisted |
| 14 | Anthropic Claude Sonnet 5 | 2.04 | 1.82 | 2.26 | Not shortlisted |
| 15 | DeepSeek V3.2 | 2.11 | 1.86 | 2.36 | Not shortlisted |
| 16 | Google Translation LLM | 2.16 | 1.76 | 2.56 | Not shortlisted |
| 17 | ModernMT Enterprise Edition | 2.17 | 1.90 | 2.43 | Not shortlisted |
| 18 | Google Gemma 4 | 2.27 | 2.02 | 2.51 | Not shortlisted |
| 19 | z.ai GLM-5 | 2.42 | 2.16 | 2.69 | Not shortlisted |
| 20 | Google Cloud Advanced Translation API | 2.56 | 2.06 | 3.07 | Not shortlisted |
| 21 | MoonshotAI Kimi K2.5 | 2.84 | 2.59 | 3.09 | Not shortlisted |
| 22 | Mistral Large 3 | 2.89 | 2.60 | 3.18 | Not shortlisted |
| 23 | Alibaba Machine Translation API | 3.15 | 2.81 | 3.48 | Not shortlisted |
| 24 | Yandex Cloud Translate v2 | 3.37 | 3.00 | 3.74 | Not shortlisted |
| 25 | Anthropic Claude Haiku 4.5 | 3.46 | 3.15 | 3.77 | Not shortlisted |
| 26 | Swiss AI Apertus-8B-2509 | 4.17 | 3.72 | 4.62 | Not shortlisted |
| 27 | MiniMax M2.5 | 4.24 | 3.83 | 4.65 | Not shortlisted |
| 28 | Mistral Medium 3.1 | 4.25 | 3.91 | 4.60 | Not shortlisted |
| 29 | Cohere Tiny Aya | 8.11 | 7.06 | 9.15 | Not shortlisted |
| Model | Solution complexity | MetricX rank | MetricX | Shortlist |
|---|---|---|---|---|
| Anthropic Claude Fable 5 | Adjusted LLM | 1 | 1.65 | Top 10 by MetricX |
| OpenAI GPT-5.6 Luna | Adjusted LLM | 2 | 1.65 | Top 10 by MetricX |
| DeepL Next Gen | Off-the-shelf NMT | 3 | 1.67 | Top 10 by MetricX |
| xAI Grok 4.20 (reasoning) | Adjusted LLM | 4 | 1.75 | Top 10 by MetricX |
| Google Gemini 3.1 Pro Preview | Adjusted LLM | 5 | 1.75 | Top 10 by MetricX |
| OpenAI GPT-4.1 | Adjusted LLM | 6 | 1.78 | Top 10 by MetricX |
| OpenAI GPT-5.5 | Adjusted LLM | 7 | 1.81 | Top 10 by MetricX |
| Translated Lara 2 | Adjusted NMT | 8 | 1.82 | Top 10 by MetricX |
| Anthropic Claude Opus 4.8 | Adjusted LLM | 9 | 1.89 | Top 10 by MetricX |
| OpenAI gpt-oss-120b | Adjusted LLM | 10 | 1.90 | Top 10 by MetricX |
| Google Gemini 3.5 Flash | Adjusted LLM | 11 | 1.92 | Not shortlisted |
| Microsoft Translator | Off-the-shelf NMT | 12 | 1.95 | Not shortlisted |
| Anthropic Claude Sonnet 4.6 | Adjusted LLM | 13 | 1.96 | Not shortlisted |
| Anthropic Claude Sonnet 5 | Adjusted LLM | 14 | 2.04 | Not shortlisted |
| DeepSeek V3.2 | Adjusted LLM | 15 | 2.11 | Not shortlisted |
| Google Translation LLM | Adjusted NMT | 16 | 2.16 | Not shortlisted |
| ModernMT Enterprise Edition | Off-the-shelf NMT | 17 | 2.17 | Not shortlisted |
| Google Gemma 4 | Adjusted LLM | 18 | 2.27 | Not shortlisted |
| z.ai GLM-5 | Adjusted LLM | 19 | 2.42 | Not shortlisted |
| Google Cloud Advanced Translation API | Off-the-shelf NMT | 20 | 2.56 | Not shortlisted |
| MoonshotAI Kimi K2.5 | Adjusted LLM | 21 | 2.84 | Not shortlisted |
| Mistral Large 3 | Adjusted LLM | 22 | 2.89 | Not shortlisted |
| Alibaba Machine Translation API | Off-the-shelf NMT | 23 | 3.15 | Not shortlisted |
| Yandex Cloud Translate v2 | Off-the-shelf NMT | 24 | 3.37 | Not shortlisted |
| Anthropic Claude Haiku 4.5 | Adjusted LLM | 25 | 3.46 | Not shortlisted |
| Swiss AI Apertus-8B-2509 | Adjusted LLM | 26 | 4.17 | Not shortlisted |
| MiniMax M2.5 | Adjusted LLM | 27 | 4.24 | Not shortlisted |
| Mistral Medium 3.1 | Adjusted LLM | 28 | 4.25 | Not shortlisted |
| Cohere Tiny Aya | Adjusted LLM | 29 | 8.11 | Not shortlisted |
| OpenAI GPT-5.6 Sol | Adjusted LLM | — | 1.75 | Same model family as OpenAI GPT-5.6 Luna |
| OpenAI GPT-5.6 Terra | Adjusted LLM | — | 1.73 | Same model family as OpenAI GPT-5.6 Luna |
| xAI Grok 4.20 (non-reasoning) | Adjusted LLM | — | 2.12 | Same model family as xAI Grok 4.20 (reasoning) |
Every model we got a translation from, as the shortlist compared it: off the shelf, without context in the request, or adjusted, with it. Ranks among 29 models, with each model family counted once, and the top 10 in bold. Lower is better on MetricX. The scores rank models within this language and are not a requirement result.
Nearest the cut is Google Gemini 3.5 Flash, 11th at 1.92 on MetricX, against 1.90 for OpenAI gpt-oss-120b, 10th. OpenAI GPT-5.6 Terra, of the same model family as OpenAI GPT-5.6 Luna, is at 1.73 on MetricX, ahead of 4th place at 1.75. The ranks do not follow the requirement results: the solution on Anthropic Claude Opus 4.8, 9th, meets tone of voice, tag handling and formatting, and the one on DeepL Next Gen, 3rd, meets tone of voice only.
| Model | US dollars per million characters | Class |
|---|---|---|
| Google Gemma 4 | $0.15 | Open-weight LLMs |
| OpenAI GPT-5.6 Luna | $0.45 | LLMs |
| MiniMax M2.5 | $0.64 | LLMs |
| Mistral Large 3 | $0.84 | LLMs |
| DeepSeek V3.2 | $1.04 | LLMs |
| Mistral Medium 3.1 | $1.05 | LLMs |
| xAI Grok 4.20 (non-reasoning) | $1.26 | LLMs |
| xAI Grok 4.20 (reasoning) | $1.28 | LLMs |
| MoonshotAI Kimi K2.5 | $1.36 | LLMs |
| z.ai GLM-5 | $2.13 | LLMs |
| Anthropic Claude Haiku 4.5 | $2.4 | LLMs |
| OpenAI GPT-4.1 | $3.09 | LLMs |
| Google Gemini 3.5 Flash | $3.93 | LLMs |
| Yandex Cloud Translate v2 | $3.94 | MT engines |
| OpenAI GPT-5.6 Terra | $4.54 | LLMs |
| Google Gemini 3.1 Pro Preview | $5.12 | LLMs |
| Anthropic Claude Sonnet 5 | $6.54 | LLMs |
| Alibaba Machine Translation API | $7 | MT engines |
| Anthropic Claude Sonnet 4.6 | $7.44 | LLMs |
| OpenAI GPT-5.6 Sol | $7.81 | LLMs |
| Microsoft Translator | $10 | MT engines |
| OpenAI GPT-5.5 | $11.39 | LLMs |
| Anthropic Claude Opus 4.8 | $16.53 | LLMs |
| Google Cloud Advanced Translation API | $20 | MT engines |
| Google Translation LLM | $21.79 | LLM-based MT engines |
| Translated Lara 2 | $24.99 | LLM-based MT engines |
| DeepL Next Gen | $27.5 | LLM-based MT engines |
| Anthropic Claude Fable 5 | $32.74 | LLMs |
| Model | Tokens per character | Class |
|---|---|---|
| Anthropic Claude Opus 4.8 | 0.546 | LLMs |
| Anthropic Claude Sonnet 5 | 0.543 | LLMs |
| Anthropic Claude Fable 5 | 0.539 | LLMs |
| Mistral Large 3 | 0.438 | LLMs |
| Mistral Medium 3.1 | 0.437 | LLMs |
| Anthropic Claude Sonnet 4.6 | 0.414 | LLMs |
| Anthropic Claude Haiku 4.5 | 0.41 | LLMs |
| DeepSeek V3.2 | 0.39 | LLMs |
| MoonshotAI Kimi K2.5 | 0.387 | LLMs |
| z.ai GLM-5 | 0.386 | LLMs |
| xAI Grok 4.20 (reasoning) | 0.368 | LLMs |
| xAI Grok 4.20 (non-reasoning) | 0.367 | LLMs |
| Google Gemini 3.5 Flash | 0.361 | LLMs |
| Google Gemma 4 | 0.359 | Open-weight LLMs |
| Google Gemini 3.1 Pro Preview | 0.358 | LLMs |
| OpenAI GPT-5.5 | 0.314 | LLMs |
| OpenAI GPT-5.6 Terra | 0.314 | LLMs |
| OpenAI GPT-4.1 | 0.312 | LLMs |
| OpenAI GPT-5.6 Luna | 0.312 | LLMs |
| OpenAI GPT-5.6 Sol | 0.312 | LLMs |
| MiniMax M2.5 | 0.311 | LLMs |
| Cohere Tiny Aya | 0.307 | LLMs |
Across the 22 models we counted tokens for, a translation into Zulu has a median 0.37 tokens per character, 1.5 times as many as English text. The tokenizer of Anthropic Claude Fable 5 produces 0.539 tokens per character, nearly as many as that of Anthropic Claude Opus 4.8. Yet off the shelf, translations from Anthropic Claude Opus 4.8 cost $16.53 per million characters and from Anthropic Claude Fable 5 at least $32.74.
| Model | Seconds per 1,000 words | Class |
|---|---|---|
| Microsoft Translator | 0.2 | MT engines |
| Google Cloud Advanced Translation API | 0.2 | MT engines |
| ModernMT Enterprise Edition | 0.4 | MT engines |
| DeepL Next Gen | 0.7 | LLM-based MT engines |
| Google Translation LLM | 0.7 | LLM-based MT engines |
| Translated Lara 2 | 1.6 | LLM-based MT engines |
| OpenAI gpt-oss-120b | 2.3 | Open-weight LLMs |
| Yandex Cloud Translate v2 | 2.9 | MT engines |
| OpenAI GPT-5.6 Luna | 18 | LLMs |
| Google Gemini 3.5 Flash | 22 | LLMs |
| Alibaba Machine Translation API | 25 | MT engines |
| OpenAI GPT-4.1 | 26 | LLMs |
| Cohere Tiny Aya | 27 | LLMs |
| xAI Grok 4.20 (non-reasoning) | 41 | LLMs |
| OpenAI GPT-5.6 Terra | 52 | LLMs |
| Anthropic Claude Haiku 4.5 | 53 | LLMs |
| Swiss AI Apertus-8B-2509 | 64 | Open-weight LLMs |
| DeepSeek V3.2 | 75 | LLMs |
| MiniMax M2.5 | 77 | LLMs |
| Mistral Large 3 | 86 | LLMs |
| Anthropic Claude Sonnet 5 | 92 | LLMs |
| OpenAI GPT-5.5 | 96 | LLMs |
| Anthropic Claude Sonnet 4.6 | 97 | LLMs |
| xAI Grok 4.20 (reasoning) | 99 | LLMs |
| Anthropic Claude Fable 5 | 100 | LLMs |
| Anthropic Claude Opus 4.8 | 122 | LLMs |
| Google Gemini 3.1 Pro Preview | 129 | LLMs |
| OpenAI GPT-5.6 Sol | 136 | LLMs |
| Google Gemma 4 | 180 | Open-weight LLMs |
| Mistral Medium 3.1 | 267 | LLMs |
| MoonshotAI Kimi K2.5 | 338 | LLMs |
| z.ai GLM-5 | 370 | LLMs |
Real-time models are those the shortlist counts as real-time, under 2 seconds per text across the languages we timed. The chart shows the time of each model into Zulu.
All three LLM-based MT engines are real-time in Zulu, and so are three of the five MT engines we timed. With the other two MT engines, a translation takes 2.9 and 25 seconds per 1,000 words. Of the LLMs, only the open-weight OpenAI gpt-oss-120b is real-time. Translations from OpenAI GPT-5.6 Luna, the next fastest LLM, take 18 seconds per 1,000 words. The four real-time models off the shortlist are 12th to 20th on MetricX. These automatic scores rank models within Zulu and are not a requirement result.
Of the 10 shortlisted solutions in Zulu, 9 meet tone of voice, and translations from the other one, on OpenAI GPT-5.5, have 97.9% of the sentences in the formal register. For information, translations from 9 of the 10 have 100% of the sentences about people gender-neutral, and those from OpenAI gpt-oss-120b 99.1%.
Leave feedback