The State of Translation Automation 2026 · Languages
What it takes to automate translation from English into Shona against 10 requirements in three levels: the solutions that meet them, what they cost and how fast they are, from the 2026 benchmark of 57 AI models and MT engines.
A solution is a model plus its harness: prompts, glossaries and other context, pre- and post-processing. It meets a quality level when it meets every requirement of the level.
| Requirement | Description | Met when |
|---|---|---|
| Basic requirements for any company | ||
| General translation quality | Translation errors of every type, such as mistranslations, omissions and grammar errors, classified by type and severity following MQM, the industry standard. | No critical error, at most 3 major errors in the four test texts (3,034 words) and at most 20 errors per 1,000 words, weighted by severity (minor 1, major 5, critical 10). |
| Standard requirements adds the rules companies set for their content | ||
| Terminology | The approved translation of every glossary term, in any of its correct forms. | 100% of the glossary terms. We test it in the 16 languages with a glossary, so it does not apply in Shona. |
| Tone of voice | The formal register in every sentence in the second person or the imperative. | 100% of those sentences. |
| Tags | Every HTML tag of the source, with its attributes, in the translation. | 100% of the tags. |
| Unicode character integrity | Every character intact, with no corrupted characters and no letter from another script that only looks like the right one. | 100% of the characters. |
| Locale conventions | The punctuation marks of the target language in place of the English ones. | No pass-or-fail bar. We report it as a share, because whether to localize punctuation is a matter of house style. |
| HTML entities | No HTML character code, such as the code for a quotation mark, that the source does not carry. | No added HTML entity. |
| Valid JSON | The translated JSON document still parses. | The document parses. |
| Spacing at tag boundaries | The spaces next to HTML tags stay as in the source. | 100% of the tag boundaries. |
| Advanced requirements adds a requirement for some companies and use cases | ||
| Gender neutrality | A gender-neutral translation wherever the English source leaves the gender of a person open, as in “your manager” or “the expert”. | 100% of the sentences that refer to a person. |
| Solution | Sentences on which the three judges agree (%) | Status |
|---|---|---|
| OpenAI GPT-5.5 | 93.8% | Passes (90% or more) |
| Anthropic Claude Opus 4.8 | 87.6% | Does not pass |
| Google Cloud Advanced Translation API | 85.9% | Does not pass |
| Google Gemini 3.1 Pro Preview | 85.3% | Does not pass |
| Translated Lara 2 | 85.3% | Does not pass |
| OpenAI GPT-5.6 Luna | 84.7% | Does not pass |
| xAI Grok 4.20 (non-reasoning) | 84.2% | Does not pass |
| OpenAI gpt-oss-120b | 83.6% | Does not pass |
| Google Gemini 3.5 Flash | 83.1% | Does not pass |
| OpenAI GPT-4.1 | 82.5% | Does not pass |
| Anthropic Claude Fable 5 | 81.4% | Does not pass |
Only the solution on OpenAI GPT-5.5 passes the agreement check, at 93.8% of the sentences. Of the rest, the one on Anthropic Claude Opus 4.8 comes nearest to the bar, at 87.6% of the sentences. The similarity pair agrees at 0.55 against a threshold of 0.46. That is enough for Shona to pass the check of the automatic metrics.
Pick a quality level, or your own requirements, bars and constraints, to see the shortlisted solutions that meet them in Shona, with a column for each requirement you select. Under Optimize for, choose which of them comes first: the one with the lowest cost or the shortest time. Click a column heading to sort by it.
The shortlisted solutions on 11 models, with their results on tone of voice, tags and formatting, which do not depend on the agreement of the three LQA judges, and on general translation quality only where the judges agree. We cannot assess translation quality in Shona: why. The models that did not make the shortlist have automatic scores only: the shortlist. Inconclusive: the three LQA judges agree on fewer than 90% of the sentences of the solution, so it has no general translation quality result (09-02). Gender neutrality: the Advanced requirements count the two best solutions of the language, with the gender-neutrality agent where they need it. The results of the other solutions are shown for information.
The shortlisted solution on Google Gemini 3.5 Flash meets tone of voice, tag handling and the formatting requirements at $4.02 per million characters and 25 seconds per 1,000 words. Three shortlisted solutions cost less, on OpenAI GPT-5.6 Luna, xAI Grok 4.20 (non-reasoning) and OpenAI GPT-4.1, and four are faster: the one on OpenAI GPT-5.6 Luna and the three real-time (<2 sec) solutions. None of these six meet all three requirements. A cost cap of $5.26 per million characters also keeps the one on Google Gemini 3.1 Pro Preview.
We cannot assess general translation quality in Shona: how confident we are.
We cannot assess general translation quality in Shona: how confident we are.
We cannot assess general translation quality in Shona: how confident we are.
| Model | MetricX scores of the shortlisted solutions in Shona (lower is better) | Class |
|---|---|---|
| Google Cloud Advanced Translation API | 3.68 | MT engines |
| Translated Lara 2 | 3 | LLM-based MT engines |
| xAI Grok 4.20 (non-reasoning) | 2.78 | LLMs |
| Anthropic Claude Opus 4.8 | 2.72 | LLMs |
| Anthropic Claude Fable 5 | 2.58 | LLMs |
| OpenAI gpt-oss-120b | 2.58 | Open-weight LLMs |
| OpenAI GPT-5.5 | 2.42 | LLMs |
| OpenAI GPT-4.1 | 2.42 | LLMs |
| Google Gemini 3.5 Flash | 2.41 | LLMs |
| Google Gemini 3.1 Pro Preview | 2.4 | LLMs |
| OpenAI GPT-5.6 Luna | 2.39 | LLMs |
No translation into Shona: AI21 Jamba Large 1.7 (garbage or near-empty), ALL-Lab (far too short, wrong language), Swiss AI Apertus-8B-2509 (garbage or near-empty), Cohere Command A Translate (garbage or near-empty).
The MetricX scores rank models within Shona and are not a requirement result. The models in the top 9 places are all LLMs, one of them the open-weight OpenAI gpt-oss-120b in 7th place. Translated Lara 2, an LLM-based MT engine, is 10th. Google Gemma 4, also an open-weight LLM, is 11th, just outside the top 10. The four MT engines are 16th, 18th, 19th and 21st of 25: Google Cloud Advanced Translation API, Microsoft Translator, NiuTrans and ModernMT Enterprise Edition.
| Rank | Model | MetricX (lower is better) | 83% confidence interval, from | to | Status |
|---|---|---|---|---|---|
| 1 | OpenAI GPT-5.6 Luna | 2.39 | 2.14 | 2.64 | Shortlisted |
| 2 | Google Gemini 3.1 Pro Preview | 2.40 | 2.14 | 2.66 | Shortlisted |
| 3 | Google Gemini 3.5 Flash | 2.41 | 2.16 | 2.65 | Shortlisted |
| 4 | OpenAI GPT-5.5 | 2.42 | 2.13 | 2.70 | Shortlisted |
| 5 | OpenAI GPT-4.1 | 2.42 | 2.17 | 2.66 | Shortlisted |
| 6 | Anthropic Claude Fable 5 | 2.58 | 2.34 | 2.82 | Shortlisted |
| 7 | OpenAI gpt-oss-120b | 2.58 | 2.34 | 2.83 | Shortlisted |
| 8 | Anthropic Claude Opus 4.8 | 2.72 | 2.42 | 3.02 | Shortlisted |
| 9 | xAI Grok 4.20 (non-reasoning) | 2.78 | 2.51 | 3.05 | Shortlisted |
| 10 | Translated Lara 2 | 3.00 | 2.70 | 3.31 | Shortlisted |
| 11 | Google Gemma 4 | 3.08 | 2.80 | 3.36 | Not shortlisted |
| 12 | Anthropic Claude Sonnet 4.6 | 3.13 | 2.87 | 3.39 | Not shortlisted |
| 13 | Anthropic Claude Sonnet 5 | 3.25 | 2.94 | 3.55 | Not shortlisted |
| 14 | DeepSeek V3.2 | 3.33 | 3.03 | 3.63 | Not shortlisted |
| 15 | z.ai GLM-5 | 3.47 | 3.16 | 3.77 | Not shortlisted |
| 16 | Google Cloud Advanced Translation API | 3.68 | 3.16 | 4.21 | Shortlisted |
| 17 | MoonshotAI Kimi K2.5 | 4.17 | 3.87 | 4.47 | Not shortlisted |
| 18 | Microsoft Translator | 4.25 | 3.85 | 4.65 | Not shortlisted |
| 19 | NiuTrans | 4.65 | 4.25 | 5.05 | Not shortlisted |
| 20 | Mistral Large 3 | 4.68 | 4.36 | 5.01 | Not shortlisted |
| 21 | ModernMT Enterprise Edition | 5.47 | 5.00 | 5.94 | Not shortlisted |
| 22 | Anthropic Claude Haiku 4.5 | 6.36 | 5.95 | 6.78 | Not shortlisted |
| 23 | Cohere Tiny Aya | 6.49 | 5.66 | 7.33 | Not shortlisted |
| 24 | Mistral Medium 3.1 | 7.91 | 7.43 | 8.38 | Not shortlisted |
| 25 | MiniMax M2.5 | 9.73 | 8.92 | 10.54 | Not shortlisted |
| Model | Solution complexity | MetricX rank | MetricX | Shortlist |
|---|---|---|---|---|
| OpenAI GPT-5.6 Luna | Adjusted LLM | 1 | 2.39 | Top 10 by MetricX |
| Google Gemini 3.1 Pro Preview | Adjusted LLM | 2 | 2.40 | Top 10 by MetricX |
| Google Gemini 3.5 Flash | Adjusted LLM | 3 | 2.41 | Top 10 by MetricX |
| OpenAI GPT-5.5 | Adjusted LLM | 4 | 2.42 | Top 10 by MetricX |
| OpenAI GPT-4.1 | Adjusted LLM | 5 | 2.42 | Top 10 by MetricX |
| Anthropic Claude Fable 5 | Adjusted LLM | 6 | 2.58 | Top 10 by MetricX |
| OpenAI gpt-oss-120b | Adjusted LLM | 7 | 2.58 | Top 10 by MetricX |
| Anthropic Claude Opus 4.8 | Adjusted LLM | 8 | 2.72 | Top 10 by MetricX |
| xAI Grok 4.20 (non-reasoning) | Off-the-shelf LLM | 9 | 2.78 | Top 10 by MetricX |
| Translated Lara 2 | Adjusted NMT | 10 | 3.00 | Top 10 by MetricX |
| Google Gemma 4 | Adjusted LLM | 11 | 3.08 | Not shortlisted |
| Anthropic Claude Sonnet 4.6 | Adjusted LLM | 12 | 3.13 | Not shortlisted |
| Anthropic Claude Sonnet 5 | Adjusted LLM | 13 | 3.25 | Not shortlisted |
| DeepSeek V3.2 | Adjusted LLM | 14 | 3.33 | Not shortlisted |
| z.ai GLM-5 | Adjusted LLM | 15 | 3.47 | Not shortlisted |
| Google Cloud Advanced Translation API | Off-the-shelf NMT | 16 | 3.68 | Real-time place |
| MoonshotAI Kimi K2.5 | Adjusted LLM | 17 | 4.17 | Not shortlisted |
| Microsoft Translator | Off-the-shelf NMT | 18 | 4.25 | Not shortlisted |
| NiuTrans | Off-the-shelf NMT | 19 | 4.65 | Not shortlisted |
| Mistral Large 3 | Adjusted LLM | 20 | 4.68 | Not shortlisted |
| ModernMT Enterprise Edition | Off-the-shelf NMT | 21 | 5.47 | Not shortlisted |
| Anthropic Claude Haiku 4.5 | Adjusted LLM | 22 | 6.36 | Not shortlisted |
| Cohere Tiny Aya | Off-the-shelf LLM | 23 | 6.49 | Not shortlisted |
| Mistral Medium 3.1 | Adjusted LLM | 24 | 7.91 | Not shortlisted |
| MiniMax M2.5 | Adjusted LLM | 25 | 9.73 | Not shortlisted |
| OpenAI GPT-5.6 Sol | Adjusted LLM | — | 2.46 | Same model family as OpenAI GPT-5.6 Luna |
| OpenAI GPT-5.6 Terra | Adjusted LLM | — | 2.63 | Same model family as OpenAI GPT-5.6 Luna |
| xAI Grok 4.20 (reasoning) | Adjusted LLM | — | 2.79 | Same model family as xAI Grok 4.20 (non-reasoning) |
Every model we got a translation from, as the shortlist compared it: off the shelf, without context in the request, or adjusted, with it. Ranks among 25 models, with each model family counted once, and the top 10 in bold. Lower is better on MetricX. The scores rank models within this language and are not a requirement result.
On MetricX the top five scores lie between 2.39 and 2.42, so the first five places sit close together. Nearest the cut is Google Gemma 4, 11th at 3.08, against 3.00 for Translated Lara 2 in 10th place. The three model-family duplicates score from 2.46 to 2.79, better than Translated Lara 2, as lower is better on MetricX. We added Google Cloud Advanced Translation API, 16th at 3.68, as a real-time solution, and five models ranked above it stay in the tail.
| Model | US dollars per million characters | Class |
|---|---|---|
| Google Gemma 4 | $0.15 | Open-weight LLMs |
| OpenAI GPT-5.6 Luna | $0.45 | LLMs |
| MiniMax M2.5 | $0.6 | LLMs |
| Mistral Medium 3.1 | $0.76 | LLMs |
| Mistral Large 3 | $0.82 | LLMs |
| DeepSeek V3.2 | $0.98 | LLMs |
| xAI Grok 4.20 (non-reasoning) | $1.19 | LLMs |
| xAI Grok 4.20 (reasoning) | $1.2 | LLMs |
| MoonshotAI Kimi K2.5 | $1.33 | LLMs |
| z.ai GLM-5 | $2.01 | LLMs |
| Anthropic Claude Haiku 4.5 | $2.31 | LLMs |
| OpenAI GPT-4.1 | $2.97 | LLMs |
| Google Gemini 3.5 Flash | $3.85 | LLMs |
| OpenAI GPT-5.6 Terra | $4.55 | LLMs |
| Google Gemini 3.1 Pro Preview | $4.97 | LLMs |
| Anthropic Claude Sonnet 5 | $6.19 | LLMs |
| Anthropic Claude Sonnet 4.6 | $7.05 | LLMs |
| OpenAI GPT-5.6 Sol | $7.77 | LLMs |
| Microsoft Translator | $10 | MT engines |
| OpenAI GPT-5.5 | $11.07 | LLMs |
| Anthropic Claude Opus 4.8 | $15.57 | LLMs |
| Google Cloud Advanced Translation API | $20 | MT engines |
| Translated Lara 2 | $24.99 | LLM-based MT engines |
| Anthropic Claude Fable 5 | $31.21 | LLMs |
| Model | Tokens per character | Class |
|---|---|---|
| Anthropic Claude Sonnet 5 | 0.532 | LLMs |
| Anthropic Claude Fable 5 | 0.53 | LLMs |
| Anthropic Claude Opus 4.8 | 0.53 | LLMs |
| Mistral Large 3 | 0.429 | LLMs |
| Mistral Medium 3.1 | 0.426 | LLMs |
| Anthropic Claude Sonnet 4.6 | 0.408 | LLMs |
| Anthropic Claude Haiku 4.5 | 0.392 | LLMs |
| MoonshotAI Kimi K2.5 | 0.383 | LLMs |
| DeepSeek V3.2 | 0.38 | LLMs |
| z.ai GLM-5 | 0.379 | LLMs |
| Google Gemini 3.1 Pro Preview | 0.356 | LLMs |
| Google Gemini 3.5 Flash | 0.356 | LLMs |
| xAI Grok 4.20 (non-reasoning) | 0.353 | LLMs |
| xAI Grok 4.20 (reasoning) | 0.352 | LLMs |
| Google Gemma 4 | 0.344 | Open-weight LLMs |
| Cohere Tiny Aya | 0.333 | LLMs |
| MiniMax M2.5 | 0.319 | LLMs |
| OpenAI GPT-5.6 Terra | 0.317 | LLMs |
| OpenAI GPT-5.6 Luna | 0.315 | LLMs |
| OpenAI GPT-5.6 Sol | 0.315 | LLMs |
| OpenAI GPT-5.5 | 0.313 | LLMs |
| OpenAI GPT-4.1 | 0.31 | LLMs |
Across the 22 models we counted tokens for, a translation into Shona has a median 0.36 tokens per character, 1.5 times as many as English text. Tokens per character do not set the cost alone. The tokenizers of Anthropic Claude Opus 4.8 and Anthropic Claude Fable 5 both produce 0.53 tokens per character. Yet off the shelf, translations from the two cost $15.57 and at least $31.21 per million characters.
| Model | Seconds per 1,000 words | Class |
|---|---|---|
| Google Cloud Advanced Translation API | 0.1 | MT engines |
| Microsoft Translator | 0.2 | MT engines |
| ModernMT Enterprise Edition | 0.3 | MT engines |
| Translated Lara 2 | 1.8 | LLM-based MT engines |
| OpenAI gpt-oss-120b | 2.3 | Open-weight LLMs |
| OpenAI GPT-5.6 Luna | 18 | LLMs |
| Cohere Tiny Aya | 18 | LLMs |
| Google Gemini 3.5 Flash | 25 | LLMs |
| xAI Grok 4.20 (non-reasoning) | 29 | LLMs |
| OpenAI GPT-4.1 | 38 | LLMs |
| Anthropic Claude Haiku 4.5 | 45 | LLMs |
| OpenAI GPT-5.6 Terra | 51 | LLMs |
| xAI Grok 4.20 (reasoning) | 54 | LLMs |
| NiuTrans | 60 | MT engines |
| Mistral Medium 3.1 | 60 | LLMs |
| DeepSeek V3.2 | 78 | LLMs |
| Anthropic Claude Sonnet 5 | 86 | LLMs |
| Mistral Large 3 | 88 | LLMs |
| Anthropic Claude Sonnet 4.6 | 92 | LLMs |
| Anthropic Claude Fable 5 | 93 | LLMs |
| Google Gemini 3.1 Pro Preview | 94 | LLMs |
| OpenAI GPT-5.5 | 112 | LLMs |
| Anthropic Claude Opus 4.8 | 114 | LLMs |
| OpenAI GPT-5.6 Sol | 127 | LLMs |
| MiniMax M2.5 | 196 | LLMs |
| Google Gemma 4 | 222 | Open-weight LLMs |
| z.ai GLM-5 | 263 | LLMs |
| MoonshotAI Kimi K2.5 | 331 | LLMs |
Real-time models are those the shortlist counts as real-time, under 2 seconds per text across the languages we timed. The chart shows the time of each model into Shona.
Three of the five real-time models are MT engines. The other two are Translated Lara 2, an LLM-based MT engine, and the open-weight LLM OpenAI gpt-oss-120b. Of the three real-time solutions on the shortlist, the one on Translated Lara 2 meets tag handling, and the one on OpenAI gpt-oss-120b meets the formatting requirements. Translations from NiuTrans, an MT engine, take 60 seconds per 1,000 words. After the real-time models, the fastest translations are from OpenAI GPT-5.6 Luna and Cohere Tiny Aya, at 18 seconds per 1,000 words each.
For tone of voice in Shona we check the formal register, with a bar of 100%. Of the 11 shortlisted solutions, 4 meet it. Translations from OpenAI GPT-5.5, Anthropic Claude Opus 4.8 and xAI Grok 4.20 (non-reasoning) have 97.9% of the sentences in the formal register. Shona has no grammatical gender for persons, so the requirement is to avoid gendered terms. In translations from all 11, 100% of the sentences about people are gender-neutral. Shona has no Advanced result, so this is for information.
Leave feedback