The State of Translation Automation 2026 · Languages
What it takes to automate translation from English into Igbo against 10 requirements in three levels: the solutions that meet them, what they cost and how fast they are, from the 2026 benchmark of 57 AI models and MT engines.
A solution is a model plus its harness: prompts, glossaries and other context, pre- and post-processing. It meets a quality level when it meets every requirement of the level.
| Requirement | Description | Met when |
|---|---|---|
| Basic requirements for any company | ||
| General translation quality | Translation errors of every type, such as mistranslations, omissions and grammar errors, classified by type and severity following MQM, the industry standard. | No critical error, at most 3 major errors in the four test texts (3,034 words) and at most 20 errors per 1,000 words, weighted by severity (minor 1, major 5, critical 10). |
| Standard requirements adds the rules companies set for their content | ||
| Terminology | The approved translation of every glossary term, in any of its correct forms. | 100% of the glossary terms. We test it in the 16 languages with a glossary, so it does not apply in Igbo. |
| Tone of voice | The formal register in every sentence in the second person or the imperative. | 100% of those sentences. |
| Tags | Every HTML tag of the source, with its attributes, in the translation. | 100% of the tags. |
| Unicode character integrity | Every character intact, with no corrupted characters and no letter from another script that only looks like the right one. | 100% of the characters. |
| Locale conventions | The punctuation marks of the target language in place of the English ones. | No pass-or-fail bar. We report it as a share, because whether to localize punctuation is a matter of house style. |
| HTML entities | No HTML character code, such as the code for a quotation mark, that the source does not carry. | No added HTML entity. |
| Valid JSON | The translated JSON document still parses. | The document parses. |
| Spacing at tag boundaries | The spaces next to HTML tags stay as in the source. | 100% of the tag boundaries. |
| Advanced requirements adds a requirement for some companies and use cases | ||
| Gender neutrality | A gender-neutral translation wherever the English source leaves the gender of a person open, as in “your manager” or “the expert”. | 100% of the sentences that refer to a person. |
| Solution | Sentences on which the three judges agree (%) | Status |
|---|---|---|
| xAI Grok 4.20 (reasoning) | 88.1% | Does not pass |
| OpenAI gpt-oss-120b | 88.1% | Does not pass |
| OpenAI GPT-5.5 | 85.9% | Does not pass |
| OpenAI GPT-5.6 Luna | 84.2% | Does not pass |
| DeepL Next Gen | 82.5% | Does not pass |
| DeepSeek V3.2 | 82.5% | Does not pass |
| Anthropic Claude Fable 5 | 79.1% | Does not pass |
| Google Gemini 3.5 Flash | 78% | Does not pass |
| OpenAI GPT-4.1 | 76.8% | Does not pass |
| Google Gemini 3.1 Pro Preview | 75.1% | Does not pass |
| Google Translation LLM | 61% | Does not pass |
The highest agreement of the three LQA judges is 88.1% of the sentences, for the solutions on xAI Grok 4.20 (reasoning) and OpenAI gpt-oss-120b. The lowest is 61% of the sentences, for the solution on Google Translation LLM. Every quality level includes general translation quality, so all three levels stay unassessed in Igbo. The agreement of the similarity pair is 0.509, above the minimum of 0.461 and not borderline.
Pick a quality level, or your own requirements, bars and constraints, to see the shortlisted solutions that meet them in Igbo, with a column for each requirement you select. Under Optimize for, choose which of them comes first: the one with the lowest cost or the shortest time. Click a column heading to sort by it.
The shortlisted solutions on 11 models, with their results on tone of voice, tags and formatting, which do not depend on the agreement of the three LQA judges, and on general translation quality only where the judges agree. We cannot assess translation quality in Igbo: why. The models that did not make the shortlist have automatic scores only: the shortlist. Inconclusive: the three LQA judges agree on fewer than 90% of the sentences of the solution, so it has no general translation quality result (09-02). Gender neutrality: the Advanced requirements count the two best solutions of the language, with the gender-neutrality agent where they need it. The results of the other solutions are shown for information.
Of the eight shortlisted solutions in Igbo that meet tone of voice, tag handling and formatting, the fastest and the one MetricX ranks first are different. The one on OpenAI gpt-oss-120b takes 2.3 seconds per 1,000 words, and MetricX ranks it 12th. The one on OpenAI GPT-5.6 Luna takes 20 seconds per 1,000 words, costs $0.45 per million characters and is 1st on MetricX. The other six cost $1.12 to $35.30 per million characters and take 24 to 117 seconds per 1,000 words. MetricX ranks are not a requirement result.
Of the ten shortlisted solutions in Igbo with a cost, a cost cap of $1.39 per million characters leaves three, on OpenAI GPT-5.6 Luna, DeepSeek V3.2 and xAI Grok 4.20 (reasoning). All three meet tone of voice, tag handling and formatting. A time limit of 20 seconds per 1,000 words leaves four: the three real-time (<2 sec) solutions and the one on OpenAI GPT-5.6 Luna. Of these four, the solutions on OpenAI GPT-5.6 Luna and OpenAI gpt-oss-120b meet tag handling and formatting. All 11 shortlisted solutions are one-step solutions.
Of tone of voice, tag handling and formatting, tag handling is the one the fewest shortlisted solutions in Igbo meet: 8 out of 11, against 9 for formatting and all 11 for tone of voice. The solutions on DeepL Next Gen and Google Translation LLM meet neither tag handling nor formatting, with 95.3% and 95.1% of the tags kept. The solution on OpenAI GPT-4.1 meets formatting but not tag handling, with 99.2% of the tags kept against a bar of 100%.
We cannot assess general translation quality in Igbo: how confident we are.
We cannot assess general translation quality in Igbo: how confident we are.
We cannot assess general translation quality in Igbo: how confident we are.
| Model | MetricX scores of the shortlisted solutions in Igbo (lower is better) | Class |
|---|---|---|
| OpenAI gpt-oss-120b | 2.69 | Open-weight LLMs |
| DeepL Next Gen | 2.59 | LLM-based MT engines |
| xAI Grok 4.20 (reasoning) | 2.55 | LLMs |
| DeepSeek V3.2 | 2.53 | LLMs |
| OpenAI GPT-4.1 | 2.49 | LLMs |
| Google Translation LLM | 2.49 | LLM-based MT engines |
| OpenAI GPT-5.5 | 2.45 | LLMs |
| Google Gemini 3.1 Pro Preview | 2.37 | LLMs |
| Anthropic Claude Fable 5 | 2.28 | LLMs |
| OpenAI GPT-5.6 Luna | 2.21 | LLMs |
| Google Gemini 3.5 Flash | 2.21 | LLMs |
No translation into Igbo: AI21 Jamba Large 1.7 (garbage or near-empty), ALL-Lab (far too short, wrong language), Swiss AI Apertus-8B-2509 (garbage or near-empty).
MetricX ranks models within Igbo and is not a requirement result. On MetricX, where lower is better, the six LLMs at the top score 2.21 to 2.49. Of the LLM-based MT engines, Google Translation LLM is 7th on MetricX at 2.49, DeepL Next Gen 10th at 2.59, and Translated Lara 2 is 13th at 2.69. The open-weight OpenAI gpt-oss-120b is 12th on MetricX at 2.69 and Google Gemma 4 20th at 3.94. The four ranked MT engines score 3.22 to 6.81 on MetricX, from Microsoft Translator in 17th place to Alibaba Machine Translation API in 25th.
| Rank | Model | MetricX (lower is better) | 83% confidence interval, from | to | Status |
|---|---|---|---|---|---|
| 1 | OpenAI GPT-5.6 Luna | 2.21 | 1.98 | 2.43 | Shortlisted |
| 2 | Google Gemini 3.5 Flash | 2.21 | 1.99 | 2.43 | Shortlisted |
| 3 | Anthropic Claude Fable 5 | 2.28 | 2.06 | 2.51 | Shortlisted |
| 4 | Google Gemini 3.1 Pro Preview | 2.37 | 2.15 | 2.59 | Shortlisted |
| 5 | OpenAI GPT-5.5 | 2.45 | 2.16 | 2.74 | Shortlisted |
| 6 | OpenAI GPT-4.1 | 2.49 | 2.26 | 2.71 | Shortlisted |
| 7 | Google Translation LLM | 2.49 | 2.12 | 2.87 | Shortlisted |
| 8 | DeepSeek V3.2 | 2.53 | 2.29 | 2.78 | Shortlisted |
| 9 | xAI Grok 4.20 (reasoning) | 2.55 | 2.31 | 2.79 | Shortlisted |
| 10 | DeepL Next Gen | 2.59 | 2.32 | 2.85 | Shortlisted |
| 11 | z.ai GLM-5 | 2.61 | 2.35 | 2.86 | Not shortlisted |
| 12 | OpenAI gpt-oss-120b | 2.69 | 2.46 | 2.91 | Shortlisted |
| 13 | Translated Lara 2 | 2.69 | 2.39 | 2.99 | Not shortlisted |
| 14 | Anthropic Claude Opus 4.8 | 2.85 | 2.59 | 3.11 | Not shortlisted |
| 15 | Anthropic Claude Sonnet 5 | 3.00 | 2.74 | 3.26 | Not shortlisted |
| 16 | Anthropic Claude Sonnet 4.6 | 3.03 | 2.79 | 3.26 | Not shortlisted |
| 17 | Microsoft Translator | 3.22 | 2.85 | 3.59 | Not shortlisted |
| 18 | Google Cloud Advanced Translation API | 3.41 | 2.93 | 3.89 | Not shortlisted |
| 19 | MoonshotAI Kimi K2.5 | 3.72 | 3.42 | 4.03 | Not shortlisted |
| 20 | Google Gemma 4 | 3.94 | 3.63 | 4.25 | Not shortlisted |
| 21 | Anthropic Claude Haiku 4.5 | 4.65 | 4.33 | 4.97 | Not shortlisted |
| 22 | MiniMax M2.5 | 4.87 | 4.50 | 5.25 | Not shortlisted |
| 23 | Mistral Large 3 | 5.05 | 4.70 | 5.39 | Not shortlisted |
| 24 | ModernMT Enterprise Edition | 5.89 | 5.38 | 6.40 | Not shortlisted |
| 25 | Alibaba Machine Translation API | 6.81 | 6.34 | 7.28 | Not shortlisted |
| 26 | Mistral Medium 3.1 | 8.89 | 8.47 | 9.32 | Not shortlisted |
| 27 | Cohere Tiny Aya | 11.42 | 10.31 | 12.52 | Not shortlisted |
| 28 | Cohere Command A Translate | 13.14 | 12.36 | 13.91 | Not shortlisted |
| Model | Solution complexity | MetricX rank | MetricX | Shortlist |
|---|---|---|---|---|
| OpenAI GPT-5.6 Luna | Adjusted LLM | 1 | 2.21 | Top 10 by MetricX |
| Google Gemini 3.5 Flash | Adjusted LLM | 2 | 2.21 | Top 10 by MetricX |
| Anthropic Claude Fable 5 | Adjusted LLM | 3 | 2.28 | Top 10 by MetricX |
| Google Gemini 3.1 Pro Preview | Adjusted LLM | 4 | 2.37 | Top 10 by MetricX |
| OpenAI GPT-5.5 | Adjusted LLM | 5 | 2.45 | Top 10 by MetricX |
| OpenAI GPT-4.1 | Adjusted LLM | 6 | 2.49 | Top 10 by MetricX |
| Google Translation LLM | Adjusted NMT | 7 | 2.49 | Top 10 by MetricX |
| DeepSeek V3.2 | Adjusted LLM | 8 | 2.53 | Top 10 by MetricX |
| xAI Grok 4.20 (reasoning) | Adjusted LLM | 9 | 2.55 | Top 10 by MetricX |
| DeepL Next Gen | Off-the-shelf NMT | 10 | 2.59 | Top 10 by MetricX |
| z.ai GLM-5 | Adjusted LLM | 11 | 2.61 | Not shortlisted |
| OpenAI gpt-oss-120b | Off-the-shelf LLM | 12 | 2.69 | Real-time place |
| Translated Lara 2 | Adjusted NMT | 13 | 2.69 | Not shortlisted |
| Anthropic Claude Opus 4.8 | Adjusted LLM | 14 | 2.85 | Not shortlisted |
| Anthropic Claude Sonnet 5 | Adjusted LLM | 15 | 3.00 | Not shortlisted |
| Anthropic Claude Sonnet 4.6 | Adjusted LLM | 16 | 3.03 | Not shortlisted |
| Microsoft Translator | Off-the-shelf NMT | 17 | 3.22 | Not shortlisted |
| Google Cloud Advanced Translation API | Off-the-shelf NMT | 18 | 3.41 | Not shortlisted |
| MoonshotAI Kimi K2.5 | Adjusted LLM | 19 | 3.72 | Not shortlisted |
| Google Gemma 4 | Adjusted LLM | 20 | 3.94 | Not shortlisted |
| Anthropic Claude Haiku 4.5 | Adjusted LLM | 21 | 4.65 | Not shortlisted |
| MiniMax M2.5 | Adjusted LLM | 22 | 4.87 | Not shortlisted |
| Mistral Large 3 | Adjusted LLM | 23 | 5.05 | Not shortlisted |
| ModernMT Enterprise Edition | Off-the-shelf NMT | 24 | 5.89 | Not shortlisted |
| Alibaba Machine Translation API | Off-the-shelf NMT | 25 | 6.81 | Not shortlisted |
| Mistral Medium 3.1 | Adjusted LLM | 26 | 8.89 | Not shortlisted |
| Cohere Tiny Aya | Adjusted LLM | 27 | 11.42 | Not shortlisted |
| Cohere Command A Translate | Adjusted LLM | 28 | 13.14 | Not shortlisted |
| OpenAI GPT-5.6 Sol | Adjusted LLM | — | 2.41 | Same model family as OpenAI GPT-5.6 Luna |
| OpenAI GPT-5.6 Terra | Adjusted LLM | — | 2.54 | Same model family as OpenAI GPT-5.6 Luna |
| xAI Grok 4.20 (non-reasoning) | Adjusted LLM | — | 2.88 | Same model family as xAI Grok 4.20 (reasoning) |
Every model we got a translation from, as the shortlist compared it: off the shelf, without context in the request, or adjusted, with it. Ranks among 28 models, with each model family counted once, and the top 10 in bold. Lower is better on MetricX. The scores rank models within this language and are not a requirement result.
| Model | US dollars per million characters | Class |
|---|---|---|
| Google Gemma 4 | $0.16 | Open-weight LLMs |
| OpenAI GPT-5.6 Luna | $0.44 | LLMs |
| MiniMax M2.5 | $0.71 | LLMs |
| Mistral Large 3 | $0.88 | LLMs |
| Mistral Medium 3.1 | $1.06 | LLMs |
| DeepSeek V3.2 | $1.1 | LLMs |
| xAI Grok 4.20 (non-reasoning) | $1.33 | LLMs |
| xAI Grok 4.20 (reasoning) | $1.35 | LLMs |
| MoonshotAI Kimi K2.5 | $1.47 | LLMs |
| z.ai GLM-5 | $2.26 | LLMs |
| Anthropic Claude Haiku 4.5 | $2.8 | LLMs |
| OpenAI GPT-4.1 | $2.94 | LLMs |
| Google Gemini 3.5 Flash | $4.18 | LLMs |
| OpenAI GPT-5.6 Terra | $4.32 | LLMs |
| Google Gemini 3.1 Pro Preview | $5.49 | LLMs |
| Youdao Translation Service API | $6.71 | MT engines |
| Anthropic Claude Sonnet 5 | $6.81 | LLMs |
| Alibaba Machine Translation API | $7 | MT engines |
| OpenAI GPT-5.6 Sol | $7.61 | LLMs |
| Anthropic Claude Sonnet 4.6 | $8.49 | LLMs |
| Microsoft Translator | $10 | MT engines |
| OpenAI GPT-5.5 | $11.03 | LLMs |
| Anthropic Claude Opus 4.8 | $17.18 | LLMs |
| Google Cloud Advanced Translation API | $20 | MT engines |
| Google Translation LLM | $21.29 | LLM-based MT engines |
| Translated Lara 2 | $24.99 | LLM-based MT engines |
| DeepL Next Gen | $27.5 | LLM-based MT engines |
| Anthropic Claude Fable 5 | $34.73 | LLMs |
| Model | Tokens per character | Class |
|---|---|---|
| Anthropic Claude Fable 5 | 0.626 | LLMs |
| Anthropic Claude Opus 4.8 | 0.626 | LLMs |
| Anthropic Claude Sonnet 5 | 0.624 | LLMs |
| Anthropic Claude Haiku 4.5 | 0.544 | LLMs |
| Anthropic Claude Sonnet 4.6 | 0.54 | LLMs |
| Mistral Large 3 | 0.522 | LLMs |
| Mistral Medium 3.1 | 0.502 | LLMs |
| DeepSeek V3.2 | 0.456 | LLMs |
| MoonshotAI Kimi K2.5 | 0.455 | LLMs |
| z.ai GLM-5 | 0.451 | LLMs |
| xAI Grok 4.20 (reasoning) | 0.439 | LLMs |
| xAI Grok 4.20 (non-reasoning) | 0.429 | LLMs |
| Google Gemini 3.1 Pro Preview | 0.423 | LLMs |
| Google Gemini 3.5 Flash | 0.422 | LLMs |
| Google Gemma 4 | 0.42 | Open-weight LLMs |
| Cohere Command A Translate | 0.418 | LLMs |
| MiniMax M2.5 | 0.372 | LLMs |
| Cohere Tiny Aya | 0.334 | LLMs |
| OpenAI GPT-5.5 | 0.327 | LLMs |
| OpenAI GPT-4.1 | 0.326 | LLMs |
| OpenAI GPT-5.6 Sol | 0.326 | LLMs |
| OpenAI GPT-5.6 Luna | 0.324 | LLMs |
| OpenAI GPT-5.6 Terra | 0.324 | LLMs |
Over the 23 models we counted tokens for, a translation into Igbo has a median 0.43 tokens per character, 1.8 times as many per character as the English source. Tokens alone do not set the cost. The tokenizers of Google Gemma 4, Google Gemini 3.5 Flash and Google Gemini 3.1 Pro Preview produce 0.420 to 0.423 tokens per character. Off the shelf, translations from them cost $0.16, $4.18 and $5.49 per million characters. On the shortlist, one-step solutions on LLMs cost a median $3.68 per million characters, and those on LLM-based MT engines $24.39.
| Model | Seconds per 1,000 words | Class |
|---|---|---|
| Google Cloud Advanced Translation API | 0.1 | MT engines |
| Microsoft Translator | 0.3 | MT engines |
| ModernMT Enterprise Edition | 0.4 | MT engines |
| Google Translation LLM | 0.9 | LLM-based MT engines |
| DeepL Next Gen | 0.9 | LLM-based MT engines |
| OpenAI gpt-oss-120b | 2.3 | Open-weight LLMs |
| Translated Lara 2 | 2.8 | LLM-based MT engines |
| OpenAI GPT-5.6 Luna | 20 | LLMs |
| Google Gemini 3.5 Flash | 24 | LLMs |
| Cohere Tiny Aya | 26 | LLMs |
| xAI Grok 4.20 (non-reasoning) | 27 | LLMs |
| Alibaba Machine Translation API | 29 | MT engines |
| OpenAI GPT-4.1 | 32 | LLMs |
| Anthropic Claude Haiku 4.5 | 51 | LLMs |
| OpenAI GPT-5.6 Terra | 51 | LLMs |
| Mistral Medium 3.1 | 58 | LLMs |
| MiniMax M2.5 | 62 | LLMs |
| Cohere Command A Translate | 76 | LLMs |
| DeepSeek V3.2 | 80 | LLMs |
| xAI Grok 4.20 (reasoning) | 80 | LLMs |
| Anthropic Claude Sonnet 5 | 92 | LLMs |
| Google Gemini 3.1 Pro Preview | 94 | LLMs |
| Mistral Large 3 | 95 | LLMs |
| Anthropic Claude Sonnet 4.6 | 98 | LLMs |
| Anthropic Claude Fable 5 | 99 | LLMs |
| Anthropic Claude Opus 4.8 | 114 | LLMs |
| OpenAI GPT-5.5 | 117 | LLMs |
| OpenAI GPT-5.6 Sol | 125 | LLMs |
| Google Gemma 4 | 204 | Open-weight LLMs |
| z.ai GLM-5 | 248 | LLMs |
| MoonshotAI Kimi K2.5 | 253 | LLMs |
Real-time models are those the shortlist counts as real-time, under 2 seconds per text across the languages we timed. The chart shows the time of each model into Igbo.
Of the 7 real-time models in Igbo, 3 are MT engines, 3 are LLM-based MT engines and 1, OpenAI gpt-oss-120b, is an open-weight LLM. Translations from the other MT engine, Alibaba Machine Translation API, take 29 seconds per 1,000 words. Of the other LLMs, translations from OpenAI GPT-5.6 Luna are the fastest, at 20 seconds per 1,000 words, and those from Google Gemini 3.5 Flash next, at 24. Translations from Google Gemma 4, also an open-weight LLM, take 204 seconds per 1,000 words.
For tone of voice in Igbo we check the formal register, with a bar of 100% of the sentences. Translations from all 11 shortlisted solutions have 100% of the sentences in the formal register. The gender-neutrality requirement for Igbo is that a translation does not add a gender the source does not specify, never a default gender. Igbo has no grammatical gender for persons, so translations are neutral by default and the requirement comes down to avoiding gendered terms.
Leave feedback