The State of Translation Automation 2026 · Languages
What it takes to automate translation from English into Xhosa against 10 requirements in three levels: the solutions that meet them, what they cost and how fast they are, from the 2026 benchmark of 57 AI models and MT engines.
A solution is a model plus its harness: prompts, glossaries and other context, pre- and post-processing. It meets a quality level when it meets every requirement of the level.
| Requirement | Description | Met when |
|---|---|---|
| Basic requirements for any company | ||
| General translation quality | Translation errors of every type, such as mistranslations, omissions and grammar errors, classified by type and severity following MQM, the industry standard. | No critical error, at most 3 major errors in the four test texts (3,034 words) and at most 20 errors per 1,000 words, weighted by severity (minor 1, major 5, critical 10). |
| Standard requirements adds the rules companies set for their content | ||
| Terminology | The approved translation of every glossary term, in any of its correct forms. | 100% of the glossary terms. We test it in the 16 languages with a glossary, so it does not apply in Xhosa. |
| Tone of voice | The formal register in every sentence in the second person or the imperative. | 100% of those sentences. |
| Tags | Every HTML tag of the source, with its attributes, in the translation. | 100% of the tags. |
| Unicode character integrity | Every character intact, with no corrupted characters and no letter from another script that only looks like the right one. | 100% of the characters. |
| Locale conventions | The punctuation marks of the target language in place of the English ones. | No pass-or-fail bar. We report it as a share, because whether to localize punctuation is a matter of house style. |
| HTML entities | No HTML character code, such as the code for a quotation mark, that the source does not carry. | No added HTML entity. |
| Valid JSON | The translated JSON document still parses. | The document parses. |
| Spacing at tag boundaries | The spaces next to HTML tags stay as in the source. | 100% of the tag boundaries. |
| Advanced requirements adds a requirement for some companies and use cases | ||
| Gender neutrality | A gender-neutral translation wherever the English source leaves the gender of a person open, as in “your manager” or “the expert”. | 100% of the sentences that refer to a person. |
| Solution | Sentences on which the three judges agree (%) | Status |
|---|---|---|
| OpenAI GPT-5.6 Luna | 89.8% | Does not pass |
| Google Gemini 3.1 Pro Preview | 89.3% | Does not pass |
| OpenAI GPT-4.1 | 88.7% | Does not pass |
| OpenAI gpt-oss-120b | 88.1% | Does not pass |
| Anthropic Claude Sonnet 4.6 | 87% | Does not pass |
| OpenAI GPT-5.5 | 85.9% | Does not pass |
| DeepSeek V3.2 | 85.9% | Does not pass |
| Anthropic Claude Opus 4.8 | 85.3% | Does not pass |
| Google Gemini 3.5 Flash | 84.7% | Does not pass |
| DeepL Next Gen | 84.7% | Does not pass |
| z.ai GLM-5 | 84.2% | Does not pass |
| xAI Grok 4.20 (non-reasoning) | 83.6% | Does not pass |
| Anthropic Claude Sonnet 5 | 83.1% | Does not pass |
The pair that does not agree in Xhosa, COMET-Kiwi and MetricX, is the pair the shortlist rests on. The LQA judges agree most on the translations from OpenAI GPT-5.6 Luna, on 89.8% of the sentences, just under the bar. They agree least on those from Anthropic Claude Sonnet 5, on 83.1% of the sentences.
Pick a quality level, or your own requirements, bars and constraints, to see the shortlisted solutions that meet them in Xhosa, with a column for each requirement you select. Under Optimize for, choose which of them comes first: the one with the lowest cost or the shortest time. Click a column heading to sort by it.
The shortlisted solutions on 13 models, with their results on tone of voice, tags and formatting, which do not depend on the agreement of the three LQA judges, and on general translation quality only where the judges agree. We cannot assess translation quality in Xhosa: why. The models that did not make the shortlist have automatic scores only: the shortlist. Inconclusive: the three LQA judges agree on fewer than 90% of the sentences of the solution, so it has no general translation quality result (09-02). Gender neutrality: the Advanced requirements count the two best solutions of the language, with the gender-neutrality agent where they need it. The results of the other solutions are shown for information.
Six shortlisted solutions meet tone of voice, tag handling and formatting in Xhosa. Among them, cost and time rise together, up to $16.80 per million characters and 122 seconds per 1,000 words on Anthropic Claude Opus 4.8. With these three selected and general translation quality left out, Optimize for selects the one on OpenAI GPT-5.6 Luna for cost and latency. Two of the six, on Google Gemini 3.5 Flash and Anthropic Claude Sonnet 5, are in the top 8 of both COMET-Kiwi and MetricX. These automatic scores are not a requirement result.
Below 17 seconds per 1,000 words, the shortlist has only its two real-time (<2 sec) solutions, the one on DeepL Next Gen at 0.9 and the one on OpenAI gpt-oss-120b at 2.4 seconds per 1,000 words. Of the two, the one on DeepL Next Gen meets tone of voice alone, and the one on OpenAI gpt-oss-120b tone of voice and formatting. A cost cap of $1.08 per million characters keeps two solutions that meet tone of voice, tag handling and formatting, on OpenAI GPT-5.6 Luna and DeepSeek V3.2.
Four more shortlisted solutions meet tone of voice and formatting, with 98.0% to 99.6% of the tags kept, short of the 100% bar. They are on xAI Grok 4.20 (non-reasoning), z.ai GLM-5, OpenAI GPT-4.1 and OpenAI gpt-oss-120b.
We cannot assess general translation quality in Xhosa: how confident we are.
We cannot assess general translation quality in Xhosa: how confident we are.
We cannot assess general translation quality in Xhosa: how confident we are.
| Model | COMET-Kiwi scores of the shortlisted solutions in Xhosa (higher is better) | Class |
|---|---|---|
| OpenAI gpt-oss-120b | 0.62 | Open-weight LLMs |
| Anthropic Claude Sonnet 4.6 | 0.618 | LLMs |
| DeepL Next Gen | 0.618 | LLM-based MT engines |
| Google Gemini 3.5 Flash | 0.614 | LLMs |
| z.ai GLM-5 | 0.61 | LLMs |
| DeepSeek V3.2 | 0.61 | LLMs |
| Anthropic Claude Sonnet 5 | 0.61 | LLMs |
| xAI Grok 4.20 (non-reasoning) | 0.61 | LLMs |
| OpenAI GPT-5.5 | 0.608 | LLMs |
| OpenAI GPT-4.1 | 0.608 | LLMs |
| Google Gemini 3.1 Pro Preview | 0.608 | LLMs |
| Anthropic Claude Opus 4.8 | 0.602 | LLMs |
| OpenAI GPT-5.6 Luna | 0.601 | LLMs |
No translation into Xhosa: AI21 Jamba Large 1.7 (garbage or near-empty), ALL-Lab (far too short).
The automatic scores rank models within Xhosa and are not a requirement result. The top 8 on COMET-Kiwi are six LLMs, the open-weight OpenAI gpt-oss-120b and one LLM-based MT engine, DeepL Next Gen, which is also 2nd on MetricX. The other LLM-based MT engine, Translated Lara 2, is 18th on COMET-Kiwi and 10th on MetricX. The six MT engines we got a translation from are 17th to 27th of 30 on COMET-Kiwi, and the highest of them, Microsoft Translator, is 11th on MetricX.
| Model | COMET-Kiwi | Rank on COMET-Kiwi | MetricX | Rank on MetricX | Shortlisted | Shortlist place |
|---|---|---|---|---|---|---|
| Anthropic Claude Fable 5 | 0.604 | 12 | 2.01 | 9 | No | Not shortlisted |
| Google Gemma 4 | 0.603 | 13 | 2.83 | 19 | No | Not shortlisted |
| Mistral Large 3 | 0.602 | 15 | 3.18 | 20 | No | Not shortlisted |
| Microsoft Translator | 0.601 | 17 | 2.03 | 11 | No | Not shortlisted |
| Translated Lara 2 | 0.600 | 18 | 2.02 | 10 | No | Not shortlisted |
| MoonshotAI Kimi K2.5 | 0.600 | 19 | 3.39 | 22 | No | Not shortlisted |
| MiniMax M2.5 | 0.597 | 20 | 3.94 | 25 | No | Not shortlisted |
| ModernMT Enterprise Edition | 0.587 | 21 | 2.42 | 15 | No | Not shortlisted |
| Mistral Medium 3.1 | 0.584 | 22 | 4.20 | 26 | No | Not shortlisted |
| Google Cloud Advanced Translation API | 0.583 | 23 | 2.77 | 18 | No | Not shortlisted |
| Anthropic Claude Haiku 4.5 | 0.582 | 24 | 4.57 | 27 | No | Not shortlisted |
| Yandex Cloud Translate v2 | 0.574 | 25 | 3.31 | 21 | No | Not shortlisted |
| NiuTrans | 0.565 | 26 | 3.44 | 23 | No | Not shortlisted |
| Alibaba Machine Translation API | 0.555 | 27 | 3.62 | 24 | No | Not shortlisted |
| Swiss AI Apertus-8B-2509 | 0.551 | 28 | 5.10 | 28 | No | Not shortlisted |
| Cohere Tiny Aya | 0.511 | 29 | 5.81 | 29 | No | Not shortlisted |
| Cohere Command A Translate | 0.487 | 30 | 9.61 | 30 | No | Not shortlisted |
| OpenAI gpt-oss-120b | 0.620 | 1 | 2.57 | 17 | Yes | Top 8 by COMET-Kiwi |
| Anthropic Claude Sonnet 4.6 | 0.618 | 2 | 2.12 | 13 | Yes | Top 8 by COMET-Kiwi |
| DeepL Next Gen | 0.618 | 3 | 1.58 | 2 | Yes | Top 8 by both metrics |
| Google Gemini 3.5 Flash | 0.614 | 4 | 1.67 | 3 | Yes | Top 8 by both metrics |
| z.ai GLM-5 | 0.610 | 5 | 2.55 | 16 | Yes | Top 8 by COMET-Kiwi |
| DeepSeek V3.2 | 0.610 | 6 | 2.20 | 14 | Yes | Top 8 by COMET-Kiwi |
| Anthropic Claude Sonnet 5 | 0.610 | 7 | 1.94 | 8 | Yes | Top 8 by both metrics |
| xAI Grok 4.20 (non-reasoning) | 0.610 | 8 | 2.05 | 12 | Yes | Top 8 by COMET-Kiwi |
| OpenAI GPT-5.5 | 0.608 | 9 | 1.77 | 5 | Yes | Top 8 by MetricX |
| OpenAI GPT-4.1 | 0.608 | 10 | 1.86 | 6 | Yes | Top 8 by MetricX |
| Google Gemini 3.1 Pro Preview | 0.608 | 11 | 1.58 | 1 | Yes | Top 8 by MetricX |
| Anthropic Claude Opus 4.8 | 0.602 | 14 | 1.93 | 7 | Yes | Top 8 by MetricX |
| OpenAI GPT-5.6 Luna | 0.601 | 16 | 1.69 | 4 | Yes | Top 8 by MetricX |
| Model | Solution complexity | COMET-Kiwi rank | COMET-Kiwi | MetricX rank | MetricX | Shortlist |
|---|---|---|---|---|---|---|
| OpenAI gpt-oss-120b | Adjusted LLM | 1 | 0.620 | 17 | 2.57 | Top 8 by COMET-Kiwi |
| Anthropic Claude Sonnet 4.6 | Adjusted LLM | 2 | 0.618 | 13 | 2.12 | Top 8 by COMET-Kiwi |
| DeepL Next Gen | Off-the-shelf NMT | 3 | 0.618 | 2 | 1.58 | Top 8 by both metrics |
| Google Gemini 3.5 Flash | Adjusted LLM | 4 | 0.614 | 3 | 1.67 | Top 8 by both metrics |
| z.ai GLM-5 | Adjusted LLM | 5 | 0.610 | 16 | 2.55 | Top 8 by COMET-Kiwi |
| DeepSeek V3.2 | Adjusted LLM | 6 | 0.610 | 14 | 2.20 | Top 8 by COMET-Kiwi |
| Anthropic Claude Sonnet 5 | Adjusted LLM | 7 | 0.610 | 8 | 1.94 | Top 8 by both metrics |
| xAI Grok 4.20 (non-reasoning) | Adjusted LLM | 8 | 0.610 | 12 | 2.05 | Top 8 by COMET-Kiwi |
| OpenAI GPT-5.5 | Adjusted LLM | 9 | 0.608 | 5 | 1.77 | Top 8 by MetricX |
| OpenAI GPT-4.1 | Adjusted LLM | 10 | 0.608 | 6 | 1.86 | Top 8 by MetricX |
| Google Gemini 3.1 Pro Preview | Adjusted LLM | 11 | 0.608 | 1 | 1.58 | Top 8 by MetricX |
| Anthropic Claude Fable 5 | Adjusted LLM | 12 | 0.604 | 9 | 2.01 | Not shortlisted |
| Google Gemma 4 | Adjusted LLM | 13 | 0.603 | 19 | 2.83 | Not shortlisted |
| Anthropic Claude Opus 4.8 | Adjusted LLM | 14 | 0.602 | 7 | 1.93 | Top 8 by MetricX |
| Mistral Large 3 | Adjusted LLM | 15 | 0.602 | 20 | 3.18 | Not shortlisted |
| OpenAI GPT-5.6 Luna | Adjusted LLM | 16 | 0.601 | 4 | 1.69 | Top 8 by MetricX |
| Microsoft Translator | Off-the-shelf NMT | 17 | 0.601 | 11 | 2.03 | Not shortlisted |
| Translated Lara 2 | Adjusted NMT | 18 | 0.600 | 10 | 2.02 | Not shortlisted |
| MoonshotAI Kimi K2.5 | Adjusted LLM | 19 | 0.600 | 22 | 3.39 | Not shortlisted |
| MiniMax M2.5 | Adjusted LLM | 20 | 0.597 | 25 | 3.94 | Not shortlisted |
| ModernMT Enterprise Edition | Off-the-shelf NMT | 21 | 0.587 | 15 | 2.42 | Not shortlisted |
| Mistral Medium 3.1 | Adjusted LLM | 22 | 0.584 | 26 | 4.20 | Not shortlisted |
| Google Cloud Advanced Translation API | Off-the-shelf NMT | 23 | 0.583 | 18 | 2.77 | Not shortlisted |
| Anthropic Claude Haiku 4.5 | Off-the-shelf LLM | 24 | 0.582 | 27 | 4.57 | Not shortlisted |
| Yandex Cloud Translate v2 | Off-the-shelf NMT | 25 | 0.574 | 21 | 3.31 | Not shortlisted |
| NiuTrans | Off-the-shelf NMT | 26 | 0.565 | 23 | 3.44 | Not shortlisted |
| Alibaba Machine Translation API | Off-the-shelf NMT | 27 | 0.555 | 24 | 3.62 | Not shortlisted |
| Swiss AI Apertus-8B-2509 | Off-the-shelf LLM | 28 | 0.551 | 28 | 5.10 | Not shortlisted |
| Cohere Tiny Aya | Off-the-shelf LLM | 29 | 0.511 | 29 | 5.81 | Not shortlisted |
| Cohere Command A Translate | Off-the-shelf LLM | 30 | 0.487 | 30 | 9.61 | Not shortlisted |
| OpenAI GPT-5.6 Sol | Adjusted LLM | — | 0.599 | — | 1.79 | Same model family as OpenAI GPT-5.6 Luna |
| OpenAI GPT-5.6 Terra | Adjusted LLM | — | 0.598 | — | 1.77 | Same model family as OpenAI GPT-5.6 Luna |
| xAI Grok 4.20 (reasoning) | Adjusted LLM | — | 0.608 | — | 2.00 | Same model family as xAI Grok 4.20 (non-reasoning) |
Every model we got a translation from, as the shortlist compared it: off the shelf, without context in the request, or adjusted, with it. A model that did not make the shortlist is shown adjusted, unless off the shelf it scores higher by 0.03 or more on COMET-Kiwi. Ranks among 30 models, with each model family counted once, and the top 8 of each metric in bold. Higher is better on COMET-Kiwi, lower on MetricX. The scores rank models within this language and are not a requirement result.
The two metrics place some models far apart. MetricX ranks OpenAI gpt-oss-120b 17th, and z.ai GLM-5, 5th on COMET-Kiwi, 16th. COMET-Kiwi ranks Google Gemini 3.1 Pro Preview 11th, and OpenAI GPT-5.6 Luna, 4th on MetricX, 16th. Nearest the cut is Anthropic Claude Fable 5, 12th on COMET-Kiwi and 9th on MetricX at 2.01, against 1.94 for Anthropic Claude Sonnet 5 in 8th place.
| Model | US dollars per million characters | Class |
|---|---|---|
| Google Gemma 4 | $0.16 | Open-weight LLMs |
| OpenAI GPT-5.6 Luna | $0.46 | LLMs |
| MiniMax M2.5 | $0.63 | LLMs |
| Mistral Large 3 | $0.85 | LLMs |
| Mistral Medium 3.1 | $1.04 | LLMs |
| DeepSeek V3.2 | $1.05 | LLMs |
| xAI Grok 4.20 (non-reasoning) | $1.27 | LLMs |
| xAI Grok 4.20 (reasoning) | $1.28 | LLMs |
| MoonshotAI Kimi K2.5 | $1.36 | LLMs |
| z.ai GLM-5 | $2.13 | LLMs |
| Anthropic Claude Haiku 4.5 | $2.39 | LLMs |
| OpenAI GPT-4.1 | $3.12 | LLMs |
| Yandex Cloud Translate v2 | $3.94 | MT engines |
| Google Gemini 3.5 Flash | $4 | LLMs |
| OpenAI GPT-5.6 Terra | $4.7 | LLMs |
| Google Gemini 3.1 Pro Preview | $5.33 | LLMs |
| Anthropic Claude Sonnet 5 | $6.58 | LLMs |
| Alibaba Machine Translation API | $7 | MT engines |
| OpenAI GPT-5.6 Sol | $7.94 | LLMs |
| Microsoft Translator | $10 | MT engines |
| Anthropic Claude Sonnet 4.6 | $10.26 | LLMs |
| OpenAI GPT-5.5 | $11.47 | LLMs |
| Anthropic Claude Opus 4.8 | $16.45 | LLMs |
| Google Cloud Advanced Translation API | $20 | MT engines |
| Translated Lara 2 | $24.99 | LLM-based MT engines |
| DeepL Next Gen | $27.5 | LLM-based MT engines |
| Anthropic Claude Fable 5 | $32.92 | LLMs |
| Model | Tokens per character | Class |
|---|---|---|
| Anthropic Claude Opus 4.8 | 0.551 | LLMs |
| Anthropic Claude Fable 5 | 0.548 | LLMs |
| Anthropic Claude Sonnet 5 | 0.548 | LLMs |
| Anthropic Claude Sonnet 4.6 | 0.458 | LLMs |
| Mistral Medium 3.1 | 0.442 | LLMs |
| Mistral Large 3 | 0.441 | LLMs |
| Anthropic Claude Haiku 4.5 | 0.413 | LLMs |
| DeepSeek V3.2 | 0.393 | LLMs |
| z.ai GLM-5 | 0.391 | LLMs |
| MoonshotAI Kimi K2.5 | 0.389 | LLMs |
| Google Gemini 3.1 Pro Preview | 0.373 | LLMs |
| Cohere Command A Translate | 0.371 | LLMs |
| xAI Grok 4.20 (reasoning) | 0.371 | LLMs |
| Google Gemini 3.5 Flash | 0.37 | LLMs |
| xAI Grok 4.20 (non-reasoning) | 0.369 | LLMs |
| Google Gemma 4 | 0.368 | Open-weight LLMs |
| MiniMax M2.5 | 0.334 | LLMs |
| OpenAI GPT-5.6 Sol | 0.323 | LLMs |
| OpenAI GPT-5.6 Terra | 0.323 | LLMs |
| OpenAI GPT-5.6 Luna | 0.321 | LLMs |
| OpenAI GPT-5.5 | 0.32 | LLMs |
| OpenAI GPT-4.1 | 0.317 | LLMs |
| Cohere Tiny Aya | 0.313 | LLMs |
Over the tokenizers of 23 models, a translation into Xhosa has a median 0.37 tokens per character, 1.6 times as many as English text. Tokens do not set the cost alone. Translations from xAI Grok 4.20 (non-reasoning) and Google Gemini 3.1 Pro Preview have 0.369 and 0.373 tokens per character, and cost $1.27 and $5.33 per million characters off the shelf. Three models have no price and are not on the chart: Swiss AI Apertus-8B-2509, Cohere Tiny Aya and OpenAI gpt-oss-120b.
| Model | Seconds per 1,000 words | Class |
|---|---|---|
| Google Cloud Advanced Translation API | 0.1 | MT engines |
| Microsoft Translator | 0.2 | MT engines |
| ModernMT Enterprise Edition | 0.7 | MT engines |
| DeepL Next Gen | 0.9 | LLM-based MT engines |
| Translated Lara 2 | 1.8 | LLM-based MT engines |
| OpenAI gpt-oss-120b | 2.4 | Open-weight LLMs |
| Yandex Cloud Translate v2 | 2.8 | MT engines |
| Cohere Tiny Aya | 17 | LLMs |
| OpenAI GPT-5.6 Luna | 18 | LLMs |
| Alibaba Machine Translation API | 22 | MT engines |
| xAI Grok 4.20 (non-reasoning) | 23 | LLMs |
| OpenAI GPT-4.1 | 39 | LLMs |
| Swiss AI Apertus-8B-2509 | 49 | Open-weight LLMs |
| Anthropic Claude Haiku 4.5 | 50 | LLMs |
| Mistral Medium 3.1 | 51 | LLMs |
| OpenAI GPT-5.6 Terra | 57 | LLMs |
| xAI Grok 4.20 (reasoning) | 57 | LLMs |
| DeepSeek V3.2 | 58 | LLMs |
| NiuTrans | 60 | MT engines |
| Google Gemini 3.5 Flash | 65 | LLMs |
| MiniMax M2.5 | 68 | LLMs |
| Cohere Command A Translate | 78 | LLMs |
| Google Gemini 3.1 Pro Preview | 86 | LLMs |
| Mistral Large 3 | 93 | LLMs |
| Anthropic Claude Sonnet 5 | 95 | LLMs |
| Anthropic Claude Sonnet 4.6 | 101 | LLMs |
| Anthropic Claude Fable 5 | 104 | LLMs |
| OpenAI GPT-5.5 | 108 | LLMs |
| OpenAI GPT-5.6 Sol | 119 | LLMs |
| Anthropic Claude Opus 4.8 | 122 | LLMs |
| MoonshotAI Kimi K2.5 | 188 | LLMs |
| Google Gemma 4 | 207 | Open-weight LLMs |
| z.ai GLM-5 | 278 | LLMs |
Real-time models are those the shortlist counts as real-time, under 2 seconds per text across the languages we timed. The chart shows the time of each model into Xhosa.
The six real-time models in Xhosa are three of the six MT engines we timed, the two LLM-based MT engines DeepL Next Gen and Translated Lara 2, and the open-weight OpenAI gpt-oss-120b. Translations from the other three MT engines are slower, at 2.8 to 60 seconds per 1,000 words. After OpenAI gpt-oss-120b, the fastest LLM is Cohere Tiny Aya, at 17 seconds per 1,000 words. Translations from the three open-weight LLMs are at 2.4 to 207 seconds per 1,000 words, from OpenAI gpt-oss-120b to Google Gemma 4, the second slowest model.
In Xhosa, gender neutrality comes down to avoiding gendered terms and not adding a gender the source does not specify. For information, translations from 12 of the 13 shortlisted solutions have 100% of the sentences about people gender-neutral, and those from DeepSeek V3.2 99.1%.
Leave feedback