The State of Translation Automation 2026 · Languages
What it takes to automate translation from English into Somali against 10 requirements in three levels: the solutions that meet them, what they cost and how fast they are, from the 2026 benchmark of 57 AI models and MT engines.
A solution is a model plus its harness: prompts, glossaries and other context, pre- and post-processing. It meets a quality level when it meets every requirement of the level.
| Requirement | Description | Met when |
|---|---|---|
| Basic requirements for any company | ||
| General translation quality | Translation errors of every type, such as mistranslations, omissions and grammar errors, classified by type and severity following MQM, the industry standard. | No critical error, at most 3 major errors in the four test texts (3,034 words) and at most 20 errors per 1,000 words, weighted by severity (minor 1, major 5, critical 10). |
| Standard requirements adds the rules companies set for their content | ||
| Terminology | The approved translation of every glossary term, in any of its correct forms. | 100% of the glossary terms. We test it in the 16 languages with a glossary, so it does not apply in Somali. |
| Tone of voice | The formal register in every sentence in the second person or the imperative. | 100% of those sentences. |
| Tags | Every HTML tag of the source, with its attributes, in the translation. | 100% of the tags. |
| Unicode character integrity | Every character intact, with no corrupted characters and no letter from another script that only looks like the right one. | 100% of the characters. |
| Locale conventions | The punctuation marks of the target language in place of the English ones. | No pass-or-fail bar. We report it as a share, because whether to localize punctuation is a matter of house style. |
| HTML entities | No HTML character code, such as the code for a quotation mark, that the source does not carry. | No added HTML entity. |
| Valid JSON | The translated JSON document still parses. | The document parses. |
| Spacing at tag boundaries | The spaces next to HTML tags stay as in the source. | 100% of the tag boundaries. |
| Advanced requirements adds a requirement for some companies and use cases | ||
| Gender neutrality | A gender-neutral translation wherever the English source leaves the gender of a person open, as in “your manager” or “the expert”. | 100% of the sentences that refer to a person. |
| Solution | Sentences on which the three judges agree (%) | Status |
|---|---|---|
| OpenAI GPT-4.1 | 90.4% | Passes (90% or more) |
| OpenAI GPT-5.5 | 88.1% | Does not pass |
| OpenAI GPT-5.6 Terra | 87.6% | Does not pass |
| Google Translation LLM | 87% | Does not pass |
| Google Gemini 3.1 Pro Preview | 86.4% | Does not pass |
| Translated Lara 2 | 85.9% | Does not pass |
| Google Gemma 4 | 85.9% | Does not pass |
| ALL-Lab | 85.3% | Does not pass |
| Google Gemini 3.5 Flash | 84.7% | Does not pass |
| Anthropic Claude Fable 5 | 84.2% | Does not pass |
| Microsoft Translator | 83.1% | Does not pass |
Neither pair of automatic metrics is borderline in Somali, so the automatic scores can order the models within Somali. That order is not a requirement result. The one solution that passes the agreement check is on OpenAI GPT-4.1, at 90.4% of the sentences. Every quality level includes general translation quality, so the page cannot say which solutions meet a level.
Pick a quality level, or your own requirements, bars and constraints, to see the shortlisted solutions that meet them in Somali, with a column for each requirement you select. Under Optimize for, choose which of them comes first: the one with the lowest cost or the shortest time. Click a column heading to sort by it.
The shortlisted solutions on 11 models, with their results on tone of voice, tags and formatting, which do not depend on the agreement of the three LQA judges, and on general translation quality only where the judges agree. We cannot assess translation quality in Somali: why. The models that did not make the shortlist have automatic scores only: the shortlist. Inconclusive: the three LQA judges agree on fewer than 90% of the sentences of the solution, so it has no general translation quality result (09-02). Gender neutrality: the Advanced requirements count the two best solutions of the language, with the gender-neutrality agent where they need it. The results of the other solutions are shown for information.
Two shortlisted solutions meet tone of voice, tag handling and formatting. The one on Google Gemini 3.1 Pro Preview costs $5.52 per million characters and takes 106 seconds per 1,000 words, against $35.05 and 111 seconds on Anthropic Claude Fable 5. Those on OpenAI GPT-4.1 and Google Gemini 3.5 Flash cost $3.27 and $4.17 per million characters and take 29 and 22 seconds per 1,000 words, but miss tag handling, at 99.6% of the tags right. Only the one on OpenAI GPT-4.1 has a general translation quality result, and it is a miss.
None of the shortlisted solutions under $5.51 per million characters or under 106 seconds per 1,000 words meet tone of voice, tag handling and formatting together. Six solutions meet tag handling and formatting. Of these, the one on the open-weight Google Gemma 4 costs the least, $0.16 per million characters, and takes the longest, 174 seconds per 1,000 words. The one on ALL-Lab is the fastest of the six, at 32 seconds per 1,000 words, and the dearest, at $100 per million characters.
Fewer shortlisted solutions meet tone of voice in Somali than tag handling or formatting: 6 out of 11, against 7 and 8. The solutions on OpenAI GPT-5.5, OpenAI GPT-5.6 Terra, Google Gemma 4 and ALL-Lab meet tag handling and formatting. Translations from them have 97.9% of the sentences in the formal register. Two of them, on Google Gemma 4 and OpenAI GPT-5.6 Terra, cost $0.16 and $4.89 per million characters, less than the two that meet tone of voice, tag handling and formatting.
We cannot assess general translation quality in Somali: how confident we are.
We cannot assess general translation quality in Somali: how confident we are.
We cannot assess general translation quality in Somali: how confident we are.
| Model | COMET-Kiwi scores of the shortlisted solutions in Somali (higher is better) | Class |
|---|---|---|
| OpenAI GPT-5.6 Terra | 0.68 | LLMs |
| Google Gemini 3.1 Pro Preview | 0.678 | LLMs |
| Google Gemma 4 | 0.676 | Open-weight LLMs |
| Anthropic Claude Fable 5 | 0.673 | LLMs |
| Google Translation LLM | 0.673 | LLM-based MT engines |
| OpenAI GPT-5.5 | 0.672 | LLMs |
| Google Gemini 3.5 Flash | 0.671 | LLMs |
| OpenAI GPT-4.1 | 0.671 | LLMs |
| Translated Lara 2 | 0.664 | LLM-based MT engines |
| Microsoft Translator | 0.66 | MT engines |
| ALL-Lab ALL-Lab | 0.66 | MT engines |
No translation into Somali: AI21 Jamba Large 1.7 (garbage or near-empty, wrong language).
The automatic scores rank models within Somali and are not a requirement result. The two LLM-based MT engines, Google Translation LLM and Translated Lara 2, are 5th and 9th on COMET-Kiwi and 9th and 13th on MetricX. Of the three open-weight LLMs, Google Gemma 4 is 3rd on COMET-Kiwi, OpenAI gpt-oss-120b 18th and Swiss AI Apertus-8B-2509 23rd. The MT engine ranked highest on COMET-Kiwi is Microsoft Translator, 10th, and on MetricX ALL-Lab, 7th.
| Model | COMET-Kiwi | Rank on COMET-Kiwi | MetricX | Rank on MetricX | Shortlisted | Shortlist place |
|---|---|---|---|---|---|---|
| xAI Grok 4.20 (reasoning) | 0.660 | 11 | 3.27 | 16 | No | Not shortlisted |
| Anthropic Claude Sonnet 5 | 0.657 | 13 | 3.02 | 10 | No | Not shortlisted |
| Anthropic Claude Opus 4.8 | 0.657 | 14 | 3.24 | 15 | No | Not shortlisted |
| DeepSeek V3.2 | 0.654 | 15 | 3.23 | 14 | No | Not shortlisted |
| z.ai GLM-5 | 0.652 | 16 | 3.43 | 18 | No | Not shortlisted |
| Anthropic Claude Sonnet 4.6 | 0.649 | 17 | 3.31 | 17 | No | Not shortlisted |
| OpenAI gpt-oss-120b | 0.643 | 18 | 3.06 | 11 | No | Not shortlisted |
| Google Cloud Advanced Translation API | 0.634 | 19 | 3.78 | 21 | No | Not shortlisted |
| MoonshotAI Kimi K2.5 | 0.634 | 20 | 3.54 | 19 | No | Not shortlisted |
| Mistral Large 3 | 0.628 | 21 | 3.57 | 20 | No | Not shortlisted |
| ModernMT Enterprise Edition | 0.624 | 22 | 3.90 | 22 | No | Not shortlisted |
| Swiss AI Apertus-8B-2509 | 0.585 | 23 | 5.32 | 23 | No | Not shortlisted |
| MiniMax M2.5 | 0.581 | 24 | 5.65 | 26 | No | Not shortlisted |
| Anthropic Claude Haiku 4.5 | 0.563 | 25 | 6.12 | 27 | No | Not shortlisted |
| NiuTrans | 0.545 | 26 | 5.47 | 25 | No | Not shortlisted |
| Mistral Medium 3.1 | 0.543 | 27 | 5.36 | 24 | No | Not shortlisted |
| Amazon Translate | 0.481 | 28 | 8.13 | 28 | No | Not shortlisted |
| Cohere Command A Translate | 0.412 | 29 | 9.28 | 29 | No | Not shortlisted |
| OpenAI GPT-5.6 Terra | 0.680 | 1 | 2.57 | 1 | Yes | Top 8 by both metrics |
| Google Gemini 3.1 Pro Preview | 0.678 | 2 | 2.83 | 5 | Yes | Top 8 by both metrics |
| Google Gemma 4 | 0.676 | 3 | 2.88 | 6 | Yes | Top 8 by both metrics |
| Anthropic Claude Fable 5 | 0.673 | 4 | 2.73 | 3 | Yes | Top 8 by both metrics |
| Google Translation LLM | 0.673 | 5 | 2.99 | 9 | Yes | Top 8 by COMET-Kiwi |
| OpenAI GPT-5.5 | 0.672 | 6 | 2.98 | 8 | Yes | Top 8 by both metrics |
| Google Gemini 3.5 Flash | 0.671 | 7 | 2.71 | 2 | Yes | Top 8 by both metrics |
| OpenAI GPT-4.1 | 0.671 | 8 | 2.82 | 4 | Yes | Top 8 by both metrics |
| Translated Lara 2 | 0.664 | 9 | 3.14 | 13 | Yes | Real-time place |
| Microsoft Translator | 0.660 | 10 | 3.09 | 12 | Yes | Real-time place |
| ALL-Lab | 0.660 | 12 | 2.96 | 7 | Yes | Top 8 by MetricX |
| Model | Solution complexity | COMET-Kiwi rank | COMET-Kiwi | MetricX rank | MetricX | Shortlist |
|---|---|---|---|---|---|---|
| OpenAI GPT-5.6 Terra | Adjusted LLM | 1 | 0.680 | 1 | 2.57 | Top 8 by both metrics |
| Google Gemini 3.1 Pro Preview | Adjusted LLM | 2 | 0.678 | 5 | 2.83 | Top 8 by both metrics |
| Google Gemma 4 | Adjusted LLM | 3 | 0.676 | 6 | 2.88 | Top 8 by both metrics |
| Anthropic Claude Fable 5 | Adjusted LLM | 4 | 0.673 | 3 | 2.73 | Top 8 by both metrics |
| Google Translation LLM | Adjusted NMT | 5 | 0.673 | 9 | 2.99 | Top 8 by COMET-Kiwi |
| OpenAI GPT-5.5 | Adjusted LLM | 6 | 0.672 | 8 | 2.98 | Top 8 by both metrics |
| Google Gemini 3.5 Flash | Adjusted LLM | 7 | 0.671 | 2 | 2.71 | Top 8 by both metrics |
| OpenAI GPT-4.1 | Adjusted LLM | 8 | 0.671 | 4 | 2.82 | Top 8 by both metrics |
| Translated Lara 2 | Adjusted NMT | 9 | 0.664 | 13 | 3.14 | Real-time place |
| Microsoft Translator | Off-the-shelf NMT | 10 | 0.660 | 12 | 3.09 | Real-time place |
| xAI Grok 4.20 (reasoning) | Adjusted LLM | 11 | 0.660 | 16 | 3.27 | Not shortlisted |
| ALL-Lab | Off-the-shelf NMT | 12 | 0.660 | 7 | 2.96 | Top 8 by MetricX |
| Anthropic Claude Sonnet 5 | Adjusted LLM | 13 | 0.657 | 10 | 3.02 | Not shortlisted |
| Anthropic Claude Opus 4.8 | Adjusted LLM | 14 | 0.657 | 15 | 3.24 | Not shortlisted |
| DeepSeek V3.2 | Adjusted LLM | 15 | 0.654 | 14 | 3.23 | Not shortlisted |
| z.ai GLM-5 | Adjusted LLM | 16 | 0.652 | 18 | 3.43 | Not shortlisted |
| Anthropic Claude Sonnet 4.6 | Adjusted LLM | 17 | 0.649 | 17 | 3.31 | Not shortlisted |
| OpenAI gpt-oss-120b | Adjusted LLM | 18 | 0.643 | 11 | 3.06 | Not shortlisted |
| Google Cloud Advanced Translation API | Off-the-shelf NMT | 19 | 0.634 | 21 | 3.78 | Not shortlisted |
| MoonshotAI Kimi K2.5 | Adjusted LLM | 20 | 0.634 | 19 | 3.54 | Not shortlisted |
| Mistral Large 3 | Adjusted LLM | 21 | 0.628 | 20 | 3.57 | Not shortlisted |
| ModernMT Enterprise Edition | Off-the-shelf NMT | 22 | 0.624 | 22 | 3.90 | Not shortlisted |
| Swiss AI Apertus-8B-2509 | Off-the-shelf LLM | 23 | 0.585 | 23 | 5.32 | Not shortlisted |
| MiniMax M2.5 | Off-the-shelf LLM | 24 | 0.581 | 26 | 5.65 | Not shortlisted |
| Anthropic Claude Haiku 4.5 | Off-the-shelf LLM | 25 | 0.563 | 27 | 6.12 | Not shortlisted |
| NiuTrans | Off-the-shelf NMT | 26 | 0.545 | 25 | 5.47 | Not shortlisted |
| Mistral Medium 3.1 | Adjusted LLM | 27 | 0.543 | 24 | 5.36 | Not shortlisted |
| Amazon Translate | Off-the-shelf NMT | 28 | 0.481 | 28 | 8.13 | Not shortlisted |
| Cohere Command A Translate | Adjusted LLM | 29 | 0.412 | 29 | 9.28 | Not shortlisted |
| OpenAI GPT-5.6 Luna | Adjusted LLM | — | 0.675 | — | 2.88 | Same model family as OpenAI GPT-5.6 Terra |
| OpenAI GPT-5.6 Sol | Adjusted LLM | — | 0.680 | — | 2.72 | Same model family as OpenAI GPT-5.6 Terra |
| xAI Grok 4.20 (non-reasoning) | Adjusted LLM | — | 0.627 | — | 4.68 | Same model family as xAI Grok 4.20 (reasoning) |
Every model we got a translation from, as the shortlist compared it: off the shelf, without context in the request, or adjusted, with it. A model that did not make the shortlist is shown adjusted, unless off the shelf it scores higher by 0.03 or more on COMET-Kiwi. Ranks among 29 models, with each model family counted once, and the top 8 of each metric in bold. Higher is better on COMET-Kiwi, lower on MetricX. The scores rank models within this language and are not a requirement result.
The place of Google Translation LLM comes from COMET-Kiwi alone, where it is 5th, and the place of ALL-Lab from MetricX alone, where it is 7th. The two real-time places go to Translated Lara 2 and Microsoft Translator, 9th and 10th on COMET-Kiwi. Nearest the cut are xAI Grok 4.20 (reasoning), 11th on COMET-Kiwi at 0.660, and Anthropic Claude Sonnet 5, 10th on MetricX at 3.02.
| Model | US dollars per million characters | Class |
|---|---|---|
| Google Gemma 4 | $0.16 | Open-weight LLMs |
| OpenAI GPT-5.6 Luna | $0.47 | LLMs |
| MiniMax M2.5 | $0.66 | LLMs |
| Mistral Large 3 | $0.85 | LLMs |
| Mistral Medium 3.1 | $1.07 | LLMs |
| DeepSeek V3.2 | $1.08 | LLMs |
| xAI Grok 4.20 (reasoning) | $1.29 | LLMs |
| xAI Grok 4.20 (non-reasoning) | $1.3 | LLMs |
| MoonshotAI Kimi K2.5 | $1.43 | LLMs |
| z.ai GLM-5 | $2.14 | LLMs |
| Anthropic Claude Haiku 4.5 | $2.47 | LLMs |
| OpenAI GPT-4.1 | $3.18 | LLMs |
| Google Gemini 3.5 Flash | $4.03 | LLMs |
| OpenAI GPT-5.6 Terra | $4.76 | LLMs |
| Google Gemini 3.1 Pro Preview | $5.34 | LLMs |
| Anthropic Claude Sonnet 5 | $6.83 | LLMs |
| Anthropic Claude Sonnet 4.6 | $7.75 | LLMs |
| OpenAI GPT-5.6 Sol | $8.2 | LLMs |
| Microsoft Translator | $10 | MT engines |
| OpenAI GPT-5.5 | $11.78 | LLMs |
| Amazon Translate | $15 | MT engines |
| Anthropic Claude Opus 4.8 | $16.88 | LLMs |
| Google Cloud Advanced Translation API | $20 | MT engines |
| Google Translation LLM | $22.55 | LLM-based MT engines |
| Translated Lara 2 | $24.99 | LLM-based MT engines |
| Anthropic Claude Fable 5 | $34.08 | LLMs |
| ALL-Lab | $100 | MT engines |
| Model | Tokens per character | Class |
|---|---|---|
| Anthropic Claude Sonnet 5 | 0.55 | LLMs |
| Anthropic Claude Fable 5 | 0.547 | LLMs |
| Anthropic Claude Opus 4.8 | 0.546 | LLMs |
| Mistral Medium 3.1 | 0.439 | LLMs |
| Mistral Large 3 | 0.435 | LLMs |
| Anthropic Claude Sonnet 4.6 | 0.424 | LLMs |
| Anthropic Claude Haiku 4.5 | 0.413 | LLMs |
| Cohere Command A Translate | 0.398 | LLMs |
| DeepSeek V3.2 | 0.393 | LLMs |
| z.ai GLM-5 | 0.381 | LLMs |
| MoonshotAI Kimi K2.5 | 0.381 | LLMs |
| xAI Grok 4.20 (non-reasoning) | 0.364 | LLMs |
| xAI Grok 4.20 (reasoning) | 0.362 | LLMs |
| Google Gemini 3.5 Flash | 0.357 | LLMs |
| Google Gemini 3.1 Pro Preview | 0.356 | LLMs |
| Google Gemma 4 | 0.355 | Open-weight LLMs |
| MiniMax M2.5 | 0.328 | LLMs |
| OpenAI GPT-5.6 Sol | 0.316 | LLMs |
| OpenAI GPT-5.6 Terra | 0.315 | LLMs |
| OpenAI GPT-4.1 | 0.314 | LLMs |
| OpenAI GPT-5.6 Luna | 0.313 | LLMs |
| OpenAI GPT-5.5 | 0.312 | LLMs |
Across the 22 models we counted tokens for, a translation into Somali has a median 0.37 tokens per character, 1.6 times that of English text. Translations from Google Gemma 4, Google Gemini 3.5 Flash and Google Gemini 3.1 Pro Preview have 0.355 to 0.357 tokens per character. Off the shelf, they cost $0.16, $4.03 and $5.34 per million characters. So the tokens alone do not set the cost. Swiss AI Apertus-8B-2509 and OpenAI gpt-oss-120b have no price, so they are not on the chart.
| Model | Seconds per 1,000 words | Class |
|---|---|---|
| Google Cloud Advanced Translation API | 0.1 | MT engines |
| Microsoft Translator | 0.2 | MT engines |
| ModernMT Enterprise Edition | 0.5 | MT engines |
| Google Translation LLM | 0.6 | LLM-based MT engines |
| Amazon Translate | 1.3 | MT engines |
| Translated Lara 2 | 2 | LLM-based MT engines |
| OpenAI gpt-oss-120b | 2.5 | Open-weight LLMs |
| OpenAI GPT-5.6 Luna | 20 | LLMs |
| Google Gemini 3.5 Flash | 22 | LLMs |
| xAI Grok 4.20 (non-reasoning) | 22 | LLMs |
| OpenAI GPT-4.1 | 29 | LLMs |
| ALL-Lab | 32 | MT engines |
| Anthropic Claude Haiku 4.5 | 49 | LLMs |
| OpenAI GPT-5.6 Terra | 51 | LLMs |
| xAI Grok 4.20 (reasoning) | 52 | LLMs |
| Swiss AI Apertus-8B-2509 | 57 | Open-weight LLMs |
| NiuTrans | 60 | MT engines |
| MiniMax M2.5 | 64 | LLMs |
| Mistral Medium 3.1 | 65 | LLMs |
| DeepSeek V3.2 | 85 | LLMs |
| Mistral Large 3 | 101 | LLMs |
| Cohere Command A Translate | 101 | LLMs |
| Anthropic Claude Sonnet 5 | 103 | LLMs |
| OpenAI GPT-5.5 | 104 | LLMs |
| Anthropic Claude Sonnet 4.6 | 104 | LLMs |
| Google Gemini 3.1 Pro Preview | 106 | LLMs |
| Anthropic Claude Fable 5 | 111 | LLMs |
| OpenAI GPT-5.6 Sol | 121 | LLMs |
| Anthropic Claude Opus 4.8 | 134 | LLMs |
| Google Gemma 4 | 174 | Open-weight LLMs |
| MoonshotAI Kimi K2.5 | 223 | LLMs |
| z.ai GLM-5 | 311 | LLMs |
Real-time models are those the shortlist counts as real-time, under 2 seconds per text across the languages we timed. The chart shows the time of each model into Somali.
The 7 real-time models in Somali are 4 MT engines, 2 LLM-based MT engines and the open-weight OpenAI gpt-oss-120b. Translations from the other two MT engines we got a translation from, ALL-Lab and NiuTrans, take 32 and 60 seconds per 1,000 words. Of the other LLMs, translations from OpenAI GPT-5.6 Luna are the fastest, at 20 seconds per 1,000 words. Translations from Google Gemma 4, also an open-weight LLM, take 174 seconds per 1,000 words.
Of the 11 shortlisted solutions, 6 meet tone of voice, with 100% of the sentences in the formal register. Translations from the other five have 95.7% to 97.9% of the sentences in the formal register. Translations from the 11 solutions have 91.8% to 95.5% of the sentences about people gender-neutral. In Somali these results are for information and do not meet or miss the Advanced requirements.
Leave feedback