The State of Translation Automation 2026 · Languages
What it takes to automate translation from English into Yoruba against 10 requirements in three levels: the solutions that meet them, what they cost and how fast they are, from the 2026 benchmark of 57 AI models and MT engines.
A solution is a model plus its harness: prompts, glossaries and other context, pre- and post-processing. It meets a quality level when it meets every requirement of the level.
| Requirement | Description | Met when |
|---|---|---|
| Basic requirements for any company | ||
| General translation quality | Translation errors of every type, such as mistranslations, omissions and grammar errors, classified by type and severity following MQM, the industry standard. | No critical error, at most 3 major errors in the four test texts (3,034 words) and at most 20 errors per 1,000 words, weighted by severity (minor 1, major 5, critical 10). |
| Standard requirements adds the rules companies set for their content | ||
| Terminology | The approved translation of every glossary term, in any of its correct forms. | 100% of the glossary terms. We test it in the 16 languages with a glossary, so it does not apply in Yoruba. |
| Tone of voice | The formal register (ẹ (respectful plural)) in every sentence in the second person or the imperative. | 100% of those sentences. |
| Tags | Every HTML tag of the source, with its attributes, in the translation. | 100% of the tags. |
| Unicode character integrity | Every character intact, with no corrupted characters and no letter from another script that only looks like the right one. | 100% of the characters. |
| Locale conventions | The punctuation marks of the target language in place of the English ones. | No pass-or-fail bar. We report it as a share, because whether to localize punctuation is a matter of house style. |
| HTML entities | No HTML character code, such as the code for a quotation mark, that the source does not carry. | No added HTML entity. |
| Valid JSON | The translated JSON document still parses. | The document parses. |
| Spacing at tag boundaries | The spaces next to HTML tags stay as in the source. | 100% of the tag boundaries. |
| Advanced requirements adds a requirement for some companies and use cases | ||
| Gender neutrality | A gender-neutral translation wherever the English source leaves the gender of a person open, as in “your manager” or “the expert”. | 100% of the sentences that refer to a person. |
| Solution | Sentences on which the three judges agree (%) | Status |
|---|---|---|
| MoonshotAI Kimi K2.5 | 91.5% | Passes (90% or more) |
| OpenAI GPT-5.5 | 89.3% | Does not pass |
| OpenAI GPT-5.6 Terra | 89.3% | Does not pass |
| OpenAI gpt-oss-120b | 89.3% | Does not pass |
| DeepSeek V3.2 | 88.1% | Does not pass |
| Microsoft Translator | 88.1% | Does not pass |
| Anthropic Claude Fable 5 | 87% | Does not pass |
| Anthropic Claude Sonnet 4.6 | 86.4% | Does not pass |
| Google Gemini 3.1 Pro Preview | 84.2% | Does not pass |
| xAI Grok 4.20 (reasoning) | 83.6% | Does not pass |
| Translated Lara 2 | 83.1% | Does not pass |
| Anthropic Claude Opus 4.8 | 80.2% | Does not pass |
The similarity pair agrees at 0.519, with its whole interval above the threshold, so Yoruba counts as measurable by the automatic metrics. The three LQA judges agree on 91.5% of the sentences for the solution on MoonshotAI Kimi K2.5, the only one that passes the agreement check. The solutions on OpenAI GPT-5.5, OpenAI GPT-5.6 Terra and OpenAI gpt-oss-120b come nearest to passing it, with the judges agreeing on 89.3% of the sentences.
Pick a quality level, or your own requirements, bars and constraints, to see the shortlisted solutions that meet them in Yoruba, with a column for each requirement you select. Under Optimize for, choose which of them comes first: the one with the lowest cost or the shortest time. Click a column heading to sort by it.
The shortlisted solutions on 12 models, with their results on tone of voice, tags and formatting, which do not depend on the agreement of the three LQA judges, and on general translation quality only where the judges agree. We cannot assess translation quality in Yoruba: why. The models that did not make the shortlist have automatic scores only: the shortlist. Inconclusive: the three LQA judges agree on fewer than 90% of the sentences of the solution, so it has no general translation quality result (09-02). Gender neutrality: the Advanced requirements count the two best solutions of the language, with the gender-neutrality agent where they need it. The results of the other solutions are shown for information.
Of the five shortlisted solutions in Yoruba that meet tag handling and formatting, the one on Anthropic Claude Fable 5 also meets tone of voice and costs the most, $45.47 per million characters. The one on xAI Grok 4.20 (reasoning) is the fastest, at 63 seconds per 1,000 words, with 74.5% of the sentences in the formal register, at $1.66 per million characters.
None of the shortlisted solutions under $45.47 per million characters meet tone of voice, tag handling and formatting together. Three cost $1.79 per million characters or less, on DeepSeek V3.2, xAI Grok 4.20 (reasoning) and MoonshotAI Kimi K2.5, and the first two meet tag handling and formatting. In the explorer, five shortlisted solutions take 65 seconds per 1,000 words or less. Two of them, on xAI Grok 4.20 (reasoning) and OpenAI GPT-5.6 Terra, meet tag handling and formatting. All 12 shortlisted solutions are one-step solutions.
We cannot assess general translation quality in Yoruba: how confident we are.
We cannot assess general translation quality in Yoruba: how confident we are.
We cannot assess general translation quality in Yoruba: how confident we are.
| Model | MetricX scores of the shortlisted solutions in Yoruba (lower is better) | Class |
|---|---|---|
| OpenAI gpt-oss-120b | 4.85 | Open-weight LLMs |
| Translated Lara 2 | 4.76 | LLM-based MT engines |
| MoonshotAI Kimi K2.5 | 4.57 | LLMs |
| Anthropic Claude Sonnet 4.6 | 4.53 | LLMs |
| DeepSeek V3.2 | 4.46 | LLMs |
| Anthropic Claude Fable 5 | 4.38 | LLMs |
| Google Gemini 3.1 Pro Preview | 4.37 | LLMs |
| OpenAI GPT-5.5 | 4.36 | LLMs |
| xAI Grok 4.20 (reasoning) | 4.32 | LLMs |
| Anthropic Claude Opus 4.8 | 4.22 | LLMs |
| OpenAI GPT-5.6 Terra | 4.17 | LLMs |
| Microsoft Translator | 3.55 | MT engines |
No translation into Yoruba: AI21 Jamba Large 1.7 (garbage or near-empty, wrong language), ALL-Lab (far too short).
| Rank | Model | MetricX (lower is better) | 83% confidence interval, from | to | Status |
|---|---|---|---|---|---|
| 1 | Microsoft Translator | 3.55 | 3.17 | 3.93 | Shortlisted |
| 2 | OpenAI GPT-5.6 Terra | 4.17 | 3.89 | 4.45 | Shortlisted |
| 3 | Anthropic Claude Opus 4.8 | 4.22 | 3.96 | 4.49 | Shortlisted |
| 4 | xAI Grok 4.20 (reasoning) | 4.32 | 4.05 | 4.59 | Shortlisted |
| 5 | OpenAI GPT-5.5 | 4.36 | 4.03 | 4.68 | Shortlisted |
| 6 | Google Gemini 3.1 Pro Preview | 4.37 | 4.05 | 4.70 | Shortlisted |
| 7 | Anthropic Claude Fable 5 | 4.38 | 4.09 | 4.67 | Shortlisted |
| 8 | DeepSeek V3.2 | 4.46 | 4.20 | 4.73 | Shortlisted |
| 9 | Anthropic Claude Sonnet 4.6 | 4.53 | 4.21 | 4.85 | Shortlisted |
| 10 | MoonshotAI Kimi K2.5 | 4.57 | 4.30 | 4.83 | Shortlisted |
| 11 | z.ai GLM-5 | 4.59 | 4.25 | 4.94 | Not shortlisted |
| 12 | Google Gemma 4 | 4.73 | 4.40 | 5.06 | Not shortlisted |
| 13 | Translated Lara 2 | 4.76 | 4.40 | 5.11 | Shortlisted |
| 14 | Anthropic Claude Sonnet 5 | 4.78 | 4.43 | 5.14 | Not shortlisted |
| 15 | OpenAI gpt-oss-120b | 4.85 | 4.53 | 5.17 | Shortlisted |
| 16 | OpenAI GPT-4.1 | 4.94 | 4.61 | 5.28 | Not shortlisted |
| 17 | Anthropic Claude Haiku 4.5 | 4.95 | 4.67 | 5.24 | Not shortlisted |
| 18 | Mistral Large 3 | 5.07 | 4.79 | 5.35 | Not shortlisted |
| 19 | MiniMax M2.5 | 5.09 | 4.71 | 5.48 | Not shortlisted |
| 20 | ModernMT Enterprise Edition | 5.22 | 4.80 | 5.65 | Not shortlisted |
| 21 | Google Cloud Advanced Translation API | 5.55 | 5.01 | 6.10 | Not shortlisted |
| 22 | Swiss AI Apertus-8B-2509 | 6.01 | 5.50 | 6.52 | Not shortlisted |
| 23 | Cohere Command A Translate | 7.80 | 7.12 | 8.48 | Not shortlisted |
| 24 | Google Gemini 3.5 Flash | 8.38 | 7.60 | 9.16 | Not shortlisted |
| 25 | Cohere Tiny Aya | 8.48 | 7.72 | 9.23 | Not shortlisted |
| 26 | Alibaba Machine Translation API | 10.02 | 9.48 | 10.57 | Not shortlisted |
| 27 | Mistral Medium 3.1 | 10.06 | 9.33 | 10.80 | Not shortlisted |
| Model | Solution complexity | MetricX rank | MetricX | Shortlist |
|---|---|---|---|---|
| Microsoft Translator | Off-the-shelf NMT | 1 | 3.55 | Top 10 by MetricX |
| OpenAI GPT-5.6 Terra | Adjusted LLM | 2 | 4.17 | Top 10 by MetricX |
| Anthropic Claude Opus 4.8 | Adjusted LLM | 3 | 4.22 | Top 10 by MetricX |
| xAI Grok 4.20 (reasoning) | Adjusted LLM | 4 | 4.32 | Top 10 by MetricX |
| OpenAI GPT-5.5 | Adjusted LLM | 5 | 4.36 | Top 10 by MetricX |
| Google Gemini 3.1 Pro Preview | Adjusted LLM | 6 | 4.37 | Top 10 by MetricX |
| Anthropic Claude Fable 5 | Adjusted LLM | 7 | 4.38 | Top 10 by MetricX |
| DeepSeek V3.2 | Adjusted LLM | 8 | 4.46 | Top 10 by MetricX |
| Anthropic Claude Sonnet 4.6 | Adjusted LLM | 9 | 4.53 | Top 10 by MetricX |
| MoonshotAI Kimi K2.5 | Adjusted LLM | 10 | 4.57 | Top 10 by MetricX |
| z.ai GLM-5 | Adjusted LLM | 11 | 4.59 | Not shortlisted |
| Google Gemma 4 | Adjusted LLM | 12 | 4.73 | Not shortlisted |
| Translated Lara 2 | Adjusted NMT | 13 | 4.76 | Real-time place |
| Anthropic Claude Sonnet 5 | Adjusted LLM | 14 | 4.78 | Not shortlisted |
| OpenAI gpt-oss-120b | Adjusted LLM | 15 | 4.85 | Real-time place |
| OpenAI GPT-4.1 | Adjusted LLM | 16 | 4.94 | Not shortlisted |
| Anthropic Claude Haiku 4.5 | Adjusted LLM | 17 | 4.95 | Not shortlisted |
| Mistral Large 3 | Adjusted LLM | 18 | 5.07 | Not shortlisted |
| MiniMax M2.5 | Adjusted LLM | 19 | 5.09 | Not shortlisted |
| ModernMT Enterprise Edition | Off-the-shelf NMT | 20 | 5.22 | Not shortlisted |
| Google Cloud Advanced Translation API | Off-the-shelf NMT | 21 | 5.55 | Not shortlisted |
| Swiss AI Apertus-8B-2509 | Off-the-shelf LLM | 22 | 6.01 | Not shortlisted |
| Cohere Command A Translate | Adjusted LLM | 23 | 7.80 | Not shortlisted |
| Google Gemini 3.5 Flash | Adjusted LLM | 24 | 8.38 | Not shortlisted |
| Cohere Tiny Aya | Off-the-shelf LLM | 25 | 8.48 | Not shortlisted |
| Alibaba Machine Translation API | Off-the-shelf NMT | 26 | 10.02 | Not shortlisted |
| Mistral Medium 3.1 | Adjusted LLM | 27 | 10.06 | Not shortlisted |
| OpenAI GPT-5.6 Luna | Adjusted LLM | — | 4.71 | Same model family as OpenAI GPT-5.6 Terra |
| OpenAI GPT-5.6 Sol | Adjusted LLM | — | 4.46 | Same model family as OpenAI GPT-5.6 Terra |
| xAI Grok 4.20 (non-reasoning) | Adjusted LLM | — | 4.67 | Same model family as xAI Grok 4.20 (reasoning) |
Every model we got a translation from, as the shortlist compared it: off the shelf, without context in the request, or adjusted, with it. Ranks among 27 models, with each model family counted once, and the top 10 in bold. Lower is better on MetricX. The scores rank models within this language and are not a requirement result.
These MetricX ranks are not a requirement result. On MetricX, where lower is better, nearest the cut is z.ai GLM-5, 11th at 4.59, then Google Gemma 4, 12th at 4.73. The real-time places went to Translated Lara 2, 13th on MetricX at 4.76, and OpenAI gpt-oss-120b, 15th at 4.85. Of the models counted once with their family, OpenAI GPT-5.6 Sol is at 4.46 on MetricX, xAI Grok 4.20 (non-reasoning) at 4.67 and OpenAI GPT-5.6 Luna at 4.71.
| Model | US dollars per million characters | Class |
|---|---|---|
| Google Gemma 4 | $0.19 | Open-weight LLMs |
| OpenAI GPT-5.6 Luna | $0.59 | LLMs |
| MiniMax M2.5 | $0.7 | LLMs |
| Mistral Large 3 | $1.19 | LLMs |
| xAI Grok 4.20 (non-reasoning) | $1.39 | LLMs |
| DeepSeek V3.2 | $1.45 | LLMs |
| xAI Grok 4.20 (reasoning) | $1.45 | LLMs |
| Mistral Medium 3.1 | $1.52 | LLMs |
| MoonshotAI Kimi K2.5 | $1.75 | LLMs |
| z.ai GLM-5 | $2.43 | LLMs |
| OpenAI GPT-4.1 | $3.68 | LLMs |
| Anthropic Claude Haiku 4.5 | $3.84 | LLMs |
| Google Gemini 3.5 Flash | $5.19 | LLMs |
| OpenAI GPT-5.6 Terra | $5.9 | LLMs |
| Google Gemini 3.1 Pro Preview | $6.86 | LLMs |
| Alibaba Machine Translation API | $7 | MT engines |
| Anthropic Claude Sonnet 5 | $8.59 | LLMs |
| Microsoft Translator | $10 | MT engines |
| OpenAI GPT-5.6 Sol | $10.18 | LLMs |
| Anthropic Claude Sonnet 4.6 | $11.5 | LLMs |
| OpenAI GPT-5.5 | $14.31 | LLMs |
| Google Cloud Advanced Translation API | $20 | MT engines |
| Anthropic Claude Opus 4.8 | $22.07 | LLMs |
| Translated Lara 2 | $24.99 | LLM-based MT engines |
| Anthropic Claude Fable 5 | $43.81 | LLMs |
| Model | Tokens per character | Class |
|---|---|---|
| Anthropic Claude Fable 5 | 0.796 | LLMs |
| Anthropic Claude Opus 4.8 | 0.793 | LLMs |
| Anthropic Claude Sonnet 5 | 0.786 | LLMs |
| Anthropic Claude Haiku 4.5 | 0.738 | LLMs |
| Anthropic Claude Sonnet 4.6 | 0.729 | LLMs |
| Mistral Medium 3.1 | 0.715 | LLMs |
| Mistral Large 3 | 0.711 | LLMs |
| Cohere Command A Translate | 0.668 | LLMs |
| DeepSeek V3.2 | 0.602 | LLMs |
| Cohere Tiny Aya | 0.544 | LLMs |
| MoonshotAI Kimi K2.5 | 0.541 | LLMs |
| Google Gemini 3.1 Pro Preview | 0.524 | LLMs |
| Google Gemini 3.5 Flash | 0.518 | LLMs |
| z.ai GLM-5 | 0.503 | LLMs |
| Google Gemma 4 | 0.502 | Open-weight LLMs |
| xAI Grok 4.20 (reasoning) | 0.486 | LLMs |
| xAI Grok 4.20 (non-reasoning) | 0.457 | LLMs |
| OpenAI GPT-5.6 Sol | 0.454 | LLMs |
| OpenAI GPT-5.6 Luna | 0.451 | LLMs |
| OpenAI GPT-5.6 Terra | 0.45 | LLMs |
| MiniMax M2.5 | 0.448 | LLMs |
| OpenAI GPT-4.1 | 0.438 | LLMs |
| OpenAI GPT-5.5 | 0.435 | LLMs |
Across the 23 models we counted tokens for, a translation into Yoruba has a median 0.52 tokens per character, 2.2 times as many as English text. The tokens alone do not set the cost. Translations from Mistral Large 3 have 0.711 tokens per character and cost $1.19 per million characters off the shelf, and those from Anthropic Claude Sonnet 4.6 have 0.729 tokens per character and cost $11.50 off the shelf. The shortlisted solution on Anthropic Claude Opus 4.8 costs $22.62 per million characters, against $22.07 for the model off the shelf.
| Model | Seconds per 1,000 words | Class |
|---|---|---|
| Google Cloud Advanced Translation API | 0.1 | MT engines |
| Microsoft Translator | 0.2 | MT engines |
| ModernMT Enterprise Edition | 0.3 | MT engines |
| OpenAI gpt-oss-120b | 3 | Open-weight LLMs |
| Translated Lara 2 | 3.2 | LLM-based MT engines |
| OpenAI GPT-5.6 Luna | 26 | LLMs |
| xAI Grok 4.20 (non-reasoning) | 27 | LLMs |
| Alibaba Machine Translation API | 30 | MT engines |
| Cohere Tiny Aya | 37 | LLMs |
| Swiss AI Apertus-8B-2509 | 49 | Open-weight LLMs |
| Anthropic Claude Haiku 4.5 | 62 | LLMs |
| xAI Grok 4.20 (reasoning) | 63 | LLMs |
| OpenAI GPT-5.6 Terra | 64 | LLMs |
| OpenAI GPT-4.1 | 76 | LLMs |
| Mistral Medium 3.1 | 76 | LLMs |
| MiniMax M2.5 | 82 | LLMs |
| DeepSeek V3.2 | 93 | LLMs |
| Cohere Command A Translate | 95 | LLMs |
| Anthropic Claude Sonnet 5 | 106 | LLMs |
| Google Gemini 3.1 Pro Preview | 107 | LLMs |
| Anthropic Claude Fable 5 | 120 | LLMs |
| Anthropic Claude Sonnet 4.6 | 132 | LLMs |
| Anthropic Claude Opus 4.8 | 136 | LLMs |
| Mistral Large 3 | 145 | LLMs |
| OpenAI GPT-5.5 | 146 | LLMs |
| Google Gemini 3.5 Flash | 150 | LLMs |
| OpenAI GPT-5.6 Sol | 152 | LLMs |
| Google Gemma 4 | 234 | Open-weight LLMs |
| MoonshotAI Kimi K2.5 | 237 | LLMs |
| z.ai GLM-5 | 350 | LLMs |
Real-time models are those the shortlist counts as real-time, under 2 seconds per text across the languages we timed. The chart shows the time of each model into Yoruba.
Three of the five real-time models are MT engines, at 0.1 to 0.3 seconds per 1,000 words, and the other MT engine we timed, Alibaba Machine Translation API, takes 30. The other two are Translated Lara 2, an LLM-based MT engine, and the open-weight OpenAI gpt-oss-120b. They have the two real-time places on the shortlist, and Microsoft Translator a MetricX place. Only the solution on OpenAI gpt-oss-120b meets formatting, and translations from the three have 21.3%, 76.6% and 10.6% of the sentences in the formal register.
Only the solutions on Anthropic Claude Fable 5 and OpenAI GPT-5.5 meet tone of voice, with all of the sentences in the formal register, the bar of 100%. Translations from Google Gemini 3.1 Pro Preview and Anthropic Claude Opus 4.8 have 97.9% of the sentences in the formal register. For information, translations from all 12 shortlisted solutions have 100% of the sentences about people gender-neutral.
Leave feedback