The State of Translation Automation 2026 · Languages
What it takes to automate translation from English into Hausa against 10 requirements in three levels: the solutions that meet them, what they cost and how fast they are, from the 2026 benchmark of 57 AI models and MT engines.
A solution is a model plus its harness: prompts, glossaries and other context, pre- and post-processing. It meets a quality level when it meets every requirement of the level.
| Requirement | Description | Met when |
|---|---|---|
| Basic requirements for any company | ||
| General translation quality | Translation errors of every type, such as mistranslations, omissions and grammar errors, classified by type and severity following MQM, the industry standard. | No critical error, at most 3 major errors in the four test texts (3,034 words) and at most 20 errors per 1,000 words, weighted by severity (minor 1, major 5, critical 10). |
| Standard requirements adds the rules companies set for their content | ||
| Terminology | The approved translation of every glossary term, in any of its correct forms. | 100% of the glossary terms. We test it in the 16 languages with a glossary, so it does not apply in Hausa. |
| Tone of voice | The formal register in every sentence in the second person or the imperative. | 100% of those sentences. |
| Tags | Every HTML tag of the source, with its attributes, in the translation. | 100% of the tags. |
| Unicode character integrity | Every character intact, with no corrupted characters and no letter from another script that only looks like the right one. | 100% of the characters. |
| Locale conventions | The punctuation marks of the target language in place of the English ones. | No pass-or-fail bar. We report it as a share, because whether to localize punctuation is a matter of house style. |
| HTML entities | No HTML character code, such as the code for a quotation mark, that the source does not carry. | No added HTML entity. |
| Valid JSON | The translated JSON document still parses. | The document parses. |
| Spacing at tag boundaries | The spaces next to HTML tags stay as in the source. | 100% of the tag boundaries. |
| Advanced requirements adds a requirement for some companies and use cases | ||
| Gender neutrality | A gender-neutral translation wherever the English source leaves the gender of a person open, as in “your manager” or “the expert”. | 100% of the sentences that refer to a person. |
| Solution | Sentences on which the three judges agree (%) | Status |
|---|---|---|
| OpenAI GPT-5.6 Sol | 93.2% | Passes (90% or more) |
| DeepL Next Gen | 93.2% | Passes (90% or more) |
| OpenAI GPT-5.5 | 91.5% | Passes (90% or more) |
| Translated Lara 2 | 88.1% | Does not pass |
| Anthropic Claude Fable 5 | 87% | Does not pass |
| Google Gemini 3.5 Flash | 87% | Does not pass |
| z.ai GLM-5 | 86.4% | Does not pass |
| Google Gemini 3.1 Pro Preview | 85.9% | Does not pass |
| Anthropic Claude Opus 4.8 | 85.3% | Does not pass |
| Google Gemma 4 | 84.2% | Does not pass |
| Google Translation LLM | 77.4% | Does not pass |
In Hausa the agreement of the similarity pair, 0.506, and of the quality-estimation pair, 0.567, stays above the minimum of each, 0.461 and 0.534, across its whole band, 0.491 to 0.520 and 0.554 to 0.581. So the automatic scores can rank the models within Hausa. The solutions on OpenAI GPT-5.6 Sol, DeepL Next Gen and OpenAI GPT-5.5 pass the check, at 91.5% to 93.2% of the sentences. Of the rest, the three judges agree most often on the translations from Translated Lara 2, on 88.1% of the sentences, short of 90%.
Pick a quality level, or your own requirements, bars and constraints, to see the shortlisted solutions that meet them in Hausa, with a column for each requirement you select. Under Optimize for, choose which of them comes first: the one with the lowest cost or the shortest time. Click a column heading to sort by it.
The shortlisted solutions on 11 models, with their results on tone of voice, tags and formatting, which do not depend on the agreement of the three LQA judges, and on general translation quality only where the judges agree. We cannot assess translation quality in Hausa: why. The models that did not make the shortlist have automatic scores only: the shortlist. Inconclusive: the three LQA judges agree on fewer than 90% of the sentences of the solution, so it has no general translation quality result (09-02). Gender neutrality: the Advanced requirements count the two best solutions of the language, with the gender-neutrality agent where they need it. The results of the other solutions are shown for information.
A cost cap under $2.08 per million characters leaves one shortlisted solution in Hausa, on Google Gemma 4 at $0.15. A time limit under 73 seconds per 1,000 words leaves four: the solution on Google Gemini 3.5 Flash, at 65 seconds, and the three real-time (<2 sec) solutions. None of these five meet tone of voice, tag handling and formatting together. A cap of $5.02 per million characters keeps two solutions that do, on z.ai GLM-5 and Google Gemini 3.1 Pro Preview.
Of tone of voice, tag handling and formatting, formatting is the one the fewest shortlisted solutions meet: 7 out of 11, against 8 for tone of voice and 9 for tag handling. The solutions on OpenAI GPT-5.5 and Translated Lara 2 meet tone of voice and tag handling but miss formatting. The solution on Translated Lara 2 is one of the three real-time solutions.
We cannot assess general translation quality in Hausa: how confident we are.
We cannot assess general translation quality in Hausa: how confident we are.
We cannot assess general translation quality in Hausa: how confident we are.
| Model | COMET-Kiwi scores of the shortlisted solutions in Hausa (higher is better) | Class |
|---|---|---|
| OpenAI GPT-5.6 Sol | 0.678 | LLMs |
| OpenAI GPT-5.5 | 0.672 | LLMs |
| Google Gemini 3.1 Pro Preview | 0.672 | LLMs |
| DeepL Next Gen | 0.671 | LLM-based MT engines |
| Anthropic Claude Opus 4.8 | 0.67 | LLMs |
| Anthropic Claude Fable 5 | 0.669 | LLMs |
| Google Gemma 4 | 0.669 | Open-weight LLMs |
| Google Gemini 3.5 Flash | 0.668 | LLMs |
| Google Translation LLM | 0.668 | LLM-based MT engines |
| Translated Lara 2 | 0.666 | LLM-based MT engines |
| z.ai GLM-5 | 0.66 | LLMs |
No translation into Hausa: AI21 Jamba Large 1.7 (garbage or near-empty, wrong language), Swiss AI Apertus-8B-2509 (garbage or near-empty).
Of the three LLM-based MT engines, DeepL Next Gen is 4th on COMET-Kiwi and 2nd on MetricX, Google Translation LLM 11th and 15th, and Translated Lara 2 12th and 7th. The open-weight Google Gemma 4 is 7th and 6th, and the open-weight OpenAI gpt-oss-120b 17th and 13th. The MT engine ranked highest, ALL-Lab, is 18th and 17th.
| Model | COMET-Kiwi | Rank on COMET-Kiwi | MetricX | Rank on MetricX | Shortlisted | Shortlist place |
|---|---|---|---|---|---|---|
| Anthropic Claude Sonnet 5 | 0.668 | 9 | 2.50 | 11 | No | Not shortlisted |
| xAI Grok 4.20 (reasoning) | 0.668 | 10 | 2.50 | 12 | No | Not shortlisted |
| OpenAI GPT-4.1 | 0.662 | 13 | 2.58 | 16 | No | Not shortlisted |
| Anthropic Claude Sonnet 4.6 | 0.659 | 15 | 2.53 | 14 | No | Not shortlisted |
| DeepSeek V3.2 | 0.658 | 16 | 2.78 | 19 | No | Not shortlisted |
| OpenAI gpt-oss-120b | 0.657 | 17 | 2.51 | 13 | No | Not shortlisted |
| ALL-Lab | 0.650 | 18 | 2.65 | 17 | No | Not shortlisted |
| Microsoft Translator | 0.646 | 19 | 2.71 | 18 | No | Not shortlisted |
| MoonshotAI Kimi K2.5 | 0.643 | 20 | 2.94 | 20 | No | Not shortlisted |
| Google Cloud Advanced Translation API | 0.638 | 21 | 3.30 | 23 | No | Not shortlisted |
| Mistral Large 3 | 0.637 | 22 | 3.29 | 22 | No | Not shortlisted |
| MiniMax M2.5 | 0.619 | 23 | 3.39 | 25 | No | Not shortlisted |
| Amazon Translate | 0.619 | 24 | 3.30 | 24 | No | Not shortlisted |
| SYSTRAN PNMT API | 0.618 | 25 | 3.16 | 21 | No | Not shortlisted |
| Cohere Tiny Aya | 0.598 | 26 | 4.21 | 26 | No | Not shortlisted |
| Anthropic Claude Haiku 4.5 | 0.589 | 27 | 4.39 | 27 | No | Not shortlisted |
| ModernMT Enterprise Edition | 0.551 | 28 | 4.99 | 29 | No | Not shortlisted |
| Mistral Medium 3.1 | 0.542 | 29 | 4.52 | 28 | No | Not shortlisted |
| Alibaba Machine Translation API | 0.498 | 30 | 6.85 | 30 | No | Not shortlisted |
| Cohere Command A Translate | 0.459 | 31 | 7.73 | 31 | No | Not shortlisted |
| OpenAI GPT-5.6 Sol | 0.678 | 1 | 2.36 | 4 | Yes | Top 8 by both metrics |
| OpenAI GPT-5.5 | 0.672 | 2 | 2.49 | 10 | Yes | Top 8 by COMET-Kiwi |
| Google Gemini 3.1 Pro Preview | 0.672 | 3 | 2.22 | 1 | Yes | Top 8 by both metrics |
| DeepL Next Gen | 0.671 | 4 | 2.29 | 2 | Yes | Top 8 by both metrics |
| Anthropic Claude Opus 4.8 | 0.670 | 5 | 2.39 | 5 | Yes | Top 8 by both metrics |
| Anthropic Claude Fable 5 | 0.669 | 6 | 2.49 | 9 | Yes | Top 8 by COMET-Kiwi |
| Google Gemma 4 | 0.669 | 7 | 2.47 | 6 | Yes | Top 8 by both metrics |
| Google Gemini 3.5 Flash | 0.668 | 8 | 2.36 | 3 | Yes | Top 8 by both metrics |
| Google Translation LLM | 0.668 | 11 | 2.56 | 15 | Yes | Real-time place |
| Translated Lara 2 | 0.666 | 12 | 2.47 | 7 | Yes | Top 8 by MetricX |
| z.ai GLM-5 | 0.660 | 14 | 2.49 | 8 | Yes | Top 8 by MetricX |
| Model | Solution complexity | COMET-Kiwi rank | COMET-Kiwi | MetricX rank | MetricX | Shortlist |
|---|---|---|---|---|---|---|
| OpenAI GPT-5.6 Sol | Adjusted LLM | 1 | 0.678 | 4 | 2.36 | Top 8 by both metrics |
| OpenAI GPT-5.5 | Adjusted LLM | 2 | 0.672 | 10 | 2.49 | Top 8 by COMET-Kiwi |
| Google Gemini 3.1 Pro Preview | Adjusted LLM | 3 | 0.672 | 1 | 2.22 | Top 8 by both metrics |
| DeepL Next Gen | Off-the-shelf NMT | 4 | 0.671 | 2 | 2.29 | Top 8 by both metrics |
| Anthropic Claude Opus 4.8 | Adjusted LLM | 5 | 0.670 | 5 | 2.39 | Top 8 by both metrics |
| Anthropic Claude Fable 5 | Adjusted LLM | 6 | 0.669 | 9 | 2.49 | Top 8 by COMET-Kiwi |
| Google Gemma 4 | Adjusted LLM | 7 | 0.669 | 6 | 2.47 | Top 8 by both metrics |
| Google Gemini 3.5 Flash | Adjusted LLM | 8 | 0.668 | 3 | 2.36 | Top 8 by both metrics |
| Anthropic Claude Sonnet 5 | Adjusted LLM | 9 | 0.668 | 11 | 2.50 | Not shortlisted |
| xAI Grok 4.20 (reasoning) | Adjusted LLM | 10 | 0.668 | 12 | 2.50 | Not shortlisted |
| Google Translation LLM | Adjusted NMT | 11 | 0.668 | 15 | 2.56 | Real-time place |
| Translated Lara 2 | Adjusted NMT | 12 | 0.666 | 7 | 2.47 | Top 8 by MetricX |
| OpenAI GPT-4.1 | Adjusted LLM | 13 | 0.662 | 16 | 2.58 | Not shortlisted |
| z.ai GLM-5 | Adjusted LLM | 14 | 0.660 | 8 | 2.49 | Top 8 by MetricX |
| Anthropic Claude Sonnet 4.6 | Adjusted LLM | 15 | 0.659 | 14 | 2.53 | Not shortlisted |
| DeepSeek V3.2 | Adjusted LLM | 16 | 0.658 | 19 | 2.78 | Not shortlisted |
| OpenAI gpt-oss-120b | Adjusted LLM | 17 | 0.657 | 13 | 2.51 | Not shortlisted |
| ALL-Lab | Off-the-shelf NMT | 18 | 0.650 | 17 | 2.65 | Not shortlisted |
| Microsoft Translator | Off-the-shelf NMT | 19 | 0.646 | 18 | 2.71 | Not shortlisted |
| MoonshotAI Kimi K2.5 | Adjusted LLM | 20 | 0.643 | 20 | 2.94 | Not shortlisted |
| Google Cloud Advanced Translation API | Off-the-shelf NMT | 21 | 0.638 | 23 | 3.30 | Not shortlisted |
| Mistral Large 3 | Adjusted LLM | 22 | 0.637 | 22 | 3.29 | Not shortlisted |
| MiniMax M2.5 | Adjusted LLM | 23 | 0.619 | 25 | 3.39 | Not shortlisted |
| Amazon Translate | Off-the-shelf NMT | 24 | 0.619 | 24 | 3.30 | Not shortlisted |
| SYSTRAN PNMT API | Off-the-shelf NMT | 25 | 0.618 | 21 | 3.16 | Not shortlisted |
| Cohere Tiny Aya | Off-the-shelf LLM | 26 | 0.598 | 26 | 4.21 | Not shortlisted |
| Anthropic Claude Haiku 4.5 | Off-the-shelf LLM | 27 | 0.589 | 27 | 4.39 | Not shortlisted |
| ModernMT Enterprise Edition | Off-the-shelf NMT | 28 | 0.551 | 29 | 4.99 | Not shortlisted |
| Mistral Medium 3.1 | Adjusted LLM | 29 | 0.542 | 28 | 4.52 | Not shortlisted |
| Alibaba Machine Translation API | Off-the-shelf NMT | 30 | 0.498 | 30 | 6.85 | Not shortlisted |
| Cohere Command A Translate | Adjusted LLM | 31 | 0.459 | 31 | 7.73 | Not shortlisted |
| OpenAI GPT-5.6 Luna | Adjusted LLM | — | 0.672 | — | 2.28 | Same model family as OpenAI GPT-5.6 Sol |
| OpenAI GPT-5.6 Terra | Adjusted LLM | — | 0.671 | — | 2.33 | Same model family as OpenAI GPT-5.6 Sol |
| xAI Grok 4.20 (non-reasoning) | Adjusted LLM | — | 0.657 | — | 2.64 | Same model family as xAI Grok 4.20 (reasoning) |
Every model we got a translation from, as the shortlist compared it: off the shelf, without context in the request, or adjusted, with it. A model that did not make the shortlist is shown adjusted, unless off the shelf it scores higher by 0.03 or more on COMET-Kiwi. Ranks among 31 models, with each model family counted once, and the top 8 of each metric in bold. Higher is better on COMET-Kiwi, lower on MetricX. The scores rank models within this language and are not a requirement result.
The places of OpenAI GPT-5.5 and Anthropic Claude Fable 5 come from COMET-Kiwi alone, and those of Translated Lara 2 and z.ai GLM-5 from MetricX alone. The solution on z.ai GLM-5, 14th on COMET-Kiwi, meets tone of voice, tag handling and formatting, and the solution on OpenAI GPT-5.5, 2nd, misses formatting. Nearest the cut are Anthropic Claude Sonnet 5 and xAI Grok 4.20 (reasoning), 9th and 10th on COMET-Kiwi and 11th and 12th on MetricX. On COMET-Kiwi both round to 0.668, as does Google Gemini 3.5 Flash in 8th place.
| Model | US dollars per million characters | Class |
|---|---|---|
| Google Gemma 4 | $0.14 | Open-weight LLMs |
| OpenAI GPT-5.6 Luna | $0.44 | LLMs |
| MiniMax M2.5 | $0.63 | LLMs |
| Mistral Large 3 | $0.78 | LLMs |
| Mistral Medium 3.1 | $0.97 | LLMs |
| DeepSeek V3.2 | $0.98 | LLMs |
| xAI Grok 4.20 (reasoning) | $1.22 | LLMs |
| xAI Grok 4.20 (non-reasoning) | $1.23 | LLMs |
| MoonshotAI Kimi K2.5 | $1.3 | LLMs |
| z.ai GLM-5 | $2.04 | LLMs |
| OpenAI GPT-4.1 | $2.93 | LLMs |
| Google Gemini 3.5 Flash | $3.65 | LLMs |
| Anthropic Claude Haiku 4.5 | $3.67 | LLMs |
| OpenAI GPT-5.6 Terra | $4.43 | LLMs |
| Google Gemini 3.1 Pro Preview | $4.87 | LLMs |
| Anthropic Claude Sonnet 5 | $6.12 | LLMs |
| Alibaba Machine Translation API | $7 | MT engines |
| Anthropic Claude Sonnet 4.6 | $7.22 | LLMs |
| OpenAI GPT-5.6 Sol | $7.64 | LLMs |
| Microsoft Translator | $10 | MT engines |
| OpenAI GPT-5.5 | $11.06 | LLMs |
| Amazon Translate | $15 | MT engines |
| SYSTRAN PNMT API | $15.49 | MT engines |
| Anthropic Claude Opus 4.8 | $15.54 | LLMs |
| Google Cloud Advanced Translation API | $20 | MT engines |
| Google Translation LLM | $22.24 | LLM-based MT engines |
| Translated Lara 2 | $24.99 | LLM-based MT engines |
| DeepL Next Gen | $27.5 | LLM-based MT engines |
| Anthropic Claude Fable 5 | $30.9 | LLMs |
| ALL-Lab | $100 | MT engines |
| Model | Tokens per character | Class |
|---|---|---|
| Anthropic Claude Opus 4.8 | 0.531 | LLMs |
| Anthropic Claude Fable 5 | 0.53 | LLMs |
| Anthropic Claude Sonnet 5 | 0.522 | LLMs |
| Mistral Large 3 | 0.423 | LLMs |
| Mistral Medium 3.1 | 0.422 | LLMs |
| Anthropic Claude Sonnet 4.6 | 0.42 | LLMs |
| Cohere Command A Translate | 0.411 | LLMs |
| Anthropic Claude Haiku 4.5 | 0.391 | LLMs |
| DeepSeek V3.2 | 0.383 | LLMs |
| z.ai GLM-5 | 0.375 | LLMs |
| MoonshotAI Kimi K2.5 | 0.374 | LLMs |
| xAI Grok 4.20 (non-reasoning) | 0.365 | LLMs |
| xAI Grok 4.20 (reasoning) | 0.365 | LLMs |
| Cohere Tiny Aya | 0.361 | LLMs |
| Google Gemini 3.1 Pro Preview | 0.352 | LLMs |
| Google Gemini 3.5 Flash | 0.348 | LLMs |
| Google Gemma 4 | 0.345 | Open-weight LLMs |
| MiniMax M2.5 | 0.327 | LLMs |
| OpenAI GPT-5.6 Sol | 0.318 | LLMs |
| OpenAI GPT-5.5 | 0.316 | LLMs |
| OpenAI GPT-5.6 Terra | 0.316 | LLMs |
| OpenAI GPT-5.6 Luna | 0.314 | LLMs |
| OpenAI GPT-4.1 | 0.313 | LLMs |
Over the 23 models we counted tokens for, a translation into Hausa has a median 0.36 tokens per character, 1.5 times that of English text. Translations from Anthropic Claude Opus 4.8 and Anthropic Claude Fable 5 have 0.53 tokens per character but cost $15.54 and at least $30.90 per million characters off the shelf. So the tokens of the translation alone do not set the cost. Off the shelf, translations from the LLM-based MT engines cost $22.24 to $27.50 per million characters, and from the MT engines other than ALL-Lab $7 to $20.
| Model | Seconds per 1,000 words | Class |
|---|---|---|
| Google Cloud Advanced Translation API | 0.1 | MT engines |
| Microsoft Translator | 0.2 | MT engines |
| ModernMT Enterprise Edition | 0.4 | MT engines |
| Google Translation LLM | 0.6 | LLM-based MT engines |
| DeepL Next Gen | 0.8 | LLM-based MT engines |
| Amazon Translate | 1.1 | MT engines |
| SYSTRAN PNMT API | 1.2 | MT engines |
| Translated Lara 2 | 1.7 | LLM-based MT engines |
| OpenAI gpt-oss-120b | 2.3 | Open-weight LLMs |
| OpenAI GPT-5.6 Luna | 17 | LLMs |
| Cohere Tiny Aya | 22 | LLMs |
| Alibaba Machine Translation API | 30 | MT engines |
| ALL-Lab | 31 | MT engines |
| OpenAI GPT-4.1 | 36 | LLMs |
| OpenAI GPT-5.6 Terra | 46 | LLMs |
| DeepSeek V3.2 | 47 | LLMs |
| Anthropic Claude Haiku 4.5 | 59 | LLMs |
| MiniMax M2.5 | 60 | LLMs |
| Mistral Medium 3.1 | 60 | LLMs |
| Google Gemini 3.5 Flash | 65 | LLMs |
| xAI Grok 4.20 (reasoning) | 67 | LLMs |
| Google Gemini 3.1 Pro Preview | 73 | LLMs |
| Cohere Command A Translate | 83 | LLMs |
| Mistral Large 3 | 86 | LLMs |
| Anthropic Claude Sonnet 5 | 88 | LLMs |
| Anthropic Claude Sonnet 4.6 | 92 | LLMs |
| OpenAI GPT-5.5 | 93 | LLMs |
| Anthropic Claude Fable 5 | 97 | LLMs |
| OpenAI GPT-5.6 Sol | 101 | LLMs |
| Anthropic Claude Opus 4.8 | 115 | LLMs |
| Google Gemma 4 | 180 | Open-weight LLMs |
| xAI Grok 4.20 (non-reasoning) | 181 | LLMs |
| MoonshotAI Kimi K2.5 | 212 | LLMs |
| z.ai GLM-5 | 215 | LLMs |
Real-time models are those the shortlist counts as real-time, under 2 seconds per text across the languages we timed. The chart shows the time of each model into Hausa.
All three LLM-based MT engines are real-time in Hausa, at 0.6 to 1.7 seconds per 1,000 words, or 0.48 to 1.28 seconds per text. Two of the seven MT engines we timed are not, at 30 and 31 seconds per 1,000 words. Of the LLMs, only the open-weight OpenAI gpt-oss-120b is real-time. The next fastest LLM, OpenAI GPT-5.6 Luna, takes 17 seconds per 1,000 words. After z.ai GLM-5, the slowest model is MoonshotAI Kimi K2.5, at 212 seconds per 1,000 words.
Of the 11 shortlisted solutions, 8 meet tone of voice in Hausa. The solutions on DeepL Next Gen, Google Gemini 3.5 Flash and Google Gemma 4 miss it at 97.87%, against a bar of 100%.
Leave feedback