The State of Translation Automation 2026 · Languages
What it takes to automate translation from English into Sesotho against 10 requirements in three levels: the solutions that meet them, what they cost and how fast they are, from the 2026 benchmark of 57 AI models and MT engines.
A solution is a model plus its harness: prompts, glossaries and other context, pre- and post-processing. It meets a quality level when it meets every requirement of the level.
| Requirement | Description | Met when |
|---|---|---|
| Basic requirements for any company | ||
| General translation quality | Translation errors of every type, such as mistranslations, omissions and grammar errors, classified by type and severity following MQM, the industry standard. | No critical error, at most 3 major errors in the four test texts (3,034 words) and at most 20 errors per 1,000 words, weighted by severity (minor 1, major 5, critical 10). |
| Standard requirements adds the rules companies set for their content | ||
| Terminology | The approved translation of every glossary term, in any of its correct forms. | 100% of the glossary terms. We test it in the 16 languages with a glossary, so it does not apply in Sesotho. |
| Tone of voice | The formal register in every sentence in the second person or the imperative. | 100% of those sentences. |
| Tags | Every HTML tag of the source, with its attributes, in the translation. | 100% of the tags. |
| Unicode character integrity | Every character intact, with no corrupted characters and no letter from another script that only looks like the right one. | 100% of the characters. |
| Locale conventions | The punctuation marks of the target language in place of the English ones. | No pass-or-fail bar. We report it as a share, because whether to localize punctuation is a matter of house style. |
| HTML entities | No HTML character code, such as the code for a quotation mark, that the source does not carry. | No added HTML entity. |
| Valid JSON | The translated JSON document still parses. | The document parses. |
| Spacing at tag boundaries | The spaces next to HTML tags stay as in the source. | 100% of the tag boundaries. |
| Advanced requirements adds a requirement for some companies and use cases | ||
| Gender neutrality | A gender-neutral translation wherever the English source leaves the gender of a person open, as in “your manager” or “the expert”. | 100% of the sentences that refer to a person. |
| Solution | Sentences on which the three judges agree (%) | Status |
|---|---|---|
| Google Gemini 3.5 Flash | 91.5% | Passes (90% or more) |
| Microsoft Translator | 86.4% | Does not pass |
| OpenAI GPT-5.6 Sol | 85.9% | Does not pass |
| DeepL Next Gen | 85.3% | Does not pass |
| Anthropic Claude Fable 5 | 84.2% | Does not pass |
| OpenAI GPT-5.5 | 84.2% | Does not pass |
| OpenAI GPT-4.1 | 84.2% | Does not pass |
| DeepSeek V3.2 | 82.5% | Does not pass |
| Anthropic Claude Opus 4.8 | 81.9% | Does not pass |
| xAI Grok 4.20 (non-reasoning) | 81.4% | Does not pass |
| Google Gemini 3.1 Pro Preview | 77.4% | Does not pass |
| Translated Lara 2 | 76.8% | Does not pass |
Pick a quality level, or your own requirements, bars and constraints, to see the shortlisted solutions that meet them in Sesotho, with a column for each requirement you select. Under Optimize for, choose which of them comes first: the one with the lowest cost or the shortest time. Click a column heading to sort by it.
The shortlisted solutions on 12 models, with their results on tone of voice, tags and formatting, which do not depend on the agreement of the three LQA judges, and on general translation quality only where the judges agree. We cannot assess translation quality in Sesotho: why. The models that did not make the shortlist have automatic scores only: the shortlist. Inconclusive: the three LQA judges agree on fewer than 90% of the sentences of the solution, so it has no general translation quality result (09-02). Gender neutrality: the Advanced requirements count the two best solutions of the language, with the gender-neutrality agent where they need it. The results of the other solutions are shown for information.
Of the five shortlisted solutions in Sesotho that meet tone of voice, tag handling and formatting, the one on OpenAI GPT-5.6 Sol is 1st on MetricX and costs $8.26 per million characters, but takes 126 seconds per 1,000 words. The one on Google Gemini 3.5 Flash is 2nd and takes 21 seconds per 1,000 words, at $4.06 per million characters. The one on DeepSeek V3.2 is 9th and costs $1.06 per million characters. MetricX ranks models within Sesotho and is not a requirement result.
A cost cap of $3.25 per million characters leaves three shortlisted solutions in Sesotho, on DeepSeek V3.2, xAI Grok 4.20 (non-reasoning) and OpenAI GPT-4.1. Only the one on DeepSeek V3.2 meets tone of voice, tag handling and formatting. Only the three real-time (<2 sec) solutions are faster than the one on Google Gemini 3.5 Flash, at 21 seconds per 1,000 words, and it is the only one of the four that meets all three requirements. All 12 shortlisted solutions are one-step solutions, so a limit on the steps leaves every one of them.
Of tone of voice, tag handling and formatting, tag handling is the one the fewest shortlisted solutions meet: 7 out of 12, against 8 for formatting and 11 for tone of voice. Translations from Anthropic Claude Opus 4.8 and OpenAI GPT-4.1 have 99.6% of the tags right, and those from xAI Grok 4.20 (non-reasoning) 99.4%, against a bar of 100%. The solutions on these three models meet tone of voice and formatting. The solutions on Google Gemini 3.1 Pro Preview and Translated Lara 2 meet tone of voice and tag handling, but not formatting.
We cannot assess general translation quality in Sesotho: how confident we are.
We cannot assess general translation quality in Sesotho: how confident we are.
We cannot assess general translation quality in Sesotho: how confident we are.
| Model | MetricX scores of the shortlisted solutions in Sesotho (lower is better) | Class |
|---|---|---|
| Microsoft Translator | 2.8 | MT engines |
| Translated Lara 2 | 2.69 | LLM-based MT engines |
| Anthropic Claude Opus 4.8 | 2.65 | LLMs |
| OpenAI GPT-4.1 | 2.57 | LLMs |
| DeepSeek V3.2 | 2.57 | LLMs |
| xAI Grok 4.20 (non-reasoning) | 2.48 | LLMs |
| DeepL Next Gen | 2.46 | LLM-based MT engines |
| OpenAI GPT-5.5 | 2.37 | LLMs |
| Anthropic Claude Fable 5 | 2.34 | LLMs |
| Google Gemini 3.1 Pro Preview | 2.34 | LLMs |
| Google Gemini 3.5 Flash | 2.29 | LLMs |
| OpenAI GPT-5.6 Sol | 2.13 | LLMs |
No translation into Sesotho: AI21 Jamba Large 1.7 (wrong language), ALL-Lab (far too short), Swiss AI Apertus-8B-2509 (garbage or near-empty, wrong language).
MetricX ranks models within Sesotho and is not a requirement result. On MetricX, where lower is better, the top 10 models score 2.13 to 2.65. DeepL Next Gen, an LLM-based MT engine, is 6th at 2.46, and Translated Lara 2, also an LLM-based MT engine, is 11th at 2.69. The three MT engines in the ranking score 2.8 to 3.38, from Microsoft Translator in 12th place to Google Cloud Advanced Translation API in 18th. The open-weight Google Gemma 4 is 15th at 3.2 and the open-weight OpenAI gpt-oss-120b 17th at 3.28.
| Rank | Model | MetricX (lower is better) | 83% confidence interval, from | to | Status |
|---|---|---|---|---|---|
| 1 | OpenAI GPT-5.6 Sol | 2.13 | 1.90 | 2.36 | Shortlisted |
| 2 | Google Gemini 3.5 Flash | 2.29 | 2.07 | 2.51 | Shortlisted |
| 3 | Anthropic Claude Fable 5 | 2.34 | 2.12 | 2.56 | Shortlisted |
| 4 | Google Gemini 3.1 Pro Preview | 2.34 | 2.12 | 2.55 | Shortlisted |
| 5 | OpenAI GPT-5.5 | 2.37 | 2.13 | 2.61 | Shortlisted |
| 6 | DeepL Next Gen | 2.46 | 2.23 | 2.69 | Shortlisted |
| 7 | xAI Grok 4.20 (non-reasoning) | 2.48 | 2.27 | 2.69 | Shortlisted |
| 8 | OpenAI GPT-4.1 | 2.57 | 2.31 | 2.82 | Shortlisted |
| 9 | DeepSeek V3.2 | 2.57 | 2.35 | 2.79 | Shortlisted |
| 10 | Anthropic Claude Opus 4.8 | 2.65 | 2.38 | 2.91 | Shortlisted |
| 11 | Translated Lara 2 | 2.69 | 2.41 | 2.98 | Shortlisted |
| 12 | Microsoft Translator | 2.80 | 2.51 | 3.08 | Shortlisted |
| 13 | Anthropic Claude Sonnet 4.6 | 3.12 | 2.85 | 3.39 | Not shortlisted |
| 14 | ModernMT Enterprise Edition | 3.13 | 2.84 | 3.42 | Not shortlisted |
| 15 | Google Gemma 4 | 3.20 | 2.93 | 3.47 | Not shortlisted |
| 16 | Anthropic Claude Sonnet 5 | 3.20 | 2.95 | 3.46 | Not shortlisted |
| 17 | OpenAI gpt-oss-120b | 3.28 | 2.95 | 3.60 | Not shortlisted |
| 18 | Google Cloud Advanced Translation API | 3.38 | 2.89 | 3.87 | Not shortlisted |
| 19 | z.ai GLM-5 | 3.42 | 3.16 | 3.68 | Not shortlisted |
| 20 | MoonshotAI Kimi K2.5 | 3.91 | 3.60 | 4.22 | Not shortlisted |
| 21 | Anthropic Claude Haiku 4.5 | 4.29 | 4.00 | 4.58 | Not shortlisted |
| 22 | Mistral Large 3 | 4.47 | 4.17 | 4.77 | Not shortlisted |
| 23 | MiniMax M2.5 | 6.14 | 5.77 | 6.51 | Not shortlisted |
| 24 | Cohere Command A Translate | 6.53 | 6.14 | 6.91 | Not shortlisted |
| Model | Solution complexity | MetricX rank | MetricX | Shortlist |
|---|---|---|---|---|
| OpenAI GPT-5.6 Sol | Adjusted LLM | 1 | 2.13 | Top 10 by MetricX |
| Google Gemini 3.5 Flash | Adjusted LLM | 2 | 2.29 | Top 10 by MetricX |
| Anthropic Claude Fable 5 | Adjusted LLM | 3 | 2.34 | Top 10 by MetricX |
| Google Gemini 3.1 Pro Preview | Adjusted LLM | 4 | 2.34 | Top 10 by MetricX |
| OpenAI GPT-5.5 | Adjusted LLM | 5 | 2.37 | Top 10 by MetricX |
| DeepL Next Gen | Off-the-shelf NMT | 6 | 2.46 | Top 10 by MetricX |
| xAI Grok 4.20 (non-reasoning) | Adjusted LLM | 7 | 2.48 | Top 10 by MetricX |
| OpenAI GPT-4.1 | Adjusted LLM | 8 | 2.57 | Top 10 by MetricX |
| DeepSeek V3.2 | Adjusted LLM | 9 | 2.57 | Top 10 by MetricX |
| Anthropic Claude Opus 4.8 | Adjusted LLM | 10 | 2.65 | Top 10 by MetricX |
| Translated Lara 2 | Adjusted NMT | 11 | 2.69 | Real-time place |
| Microsoft Translator | Off-the-shelf NMT | 12 | 2.80 | Real-time place |
| Anthropic Claude Sonnet 4.6 | Adjusted LLM | 13 | 3.12 | Not shortlisted |
| ModernMT Enterprise Edition | Off-the-shelf NMT | 14 | 3.13 | Not shortlisted |
| Google Gemma 4 | Adjusted LLM | 15 | 3.20 | Not shortlisted |
| Anthropic Claude Sonnet 5 | Adjusted LLM | 16 | 3.20 | Not shortlisted |
| OpenAI gpt-oss-120b | Adjusted LLM | 17 | 3.28 | Not shortlisted |
| Google Cloud Advanced Translation API | Off-the-shelf NMT | 18 | 3.38 | Not shortlisted |
| z.ai GLM-5 | Adjusted LLM | 19 | 3.42 | Not shortlisted |
| MoonshotAI Kimi K2.5 | Adjusted LLM | 20 | 3.91 | Not shortlisted |
| Anthropic Claude Haiku 4.5 | Adjusted LLM | 21 | 4.29 | Not shortlisted |
| Mistral Large 3 | Adjusted LLM | 22 | 4.47 | Not shortlisted |
| MiniMax M2.5 | Adjusted LLM | 23 | 6.14 | Not shortlisted |
| Cohere Command A Translate | Adjusted LLM | 24 | 6.53 | Not shortlisted |
| OpenAI GPT-5.6 Luna | Adjusted LLM | — | 2.33 | Same model family as OpenAI GPT-5.6 Sol |
| OpenAI GPT-5.6 Terra | Adjusted LLM | — | 2.46 | Same model family as OpenAI GPT-5.6 Sol |
| xAI Grok 4.20 (reasoning) | Adjusted LLM | — | 2.71 | Same model family as xAI Grok 4.20 (non-reasoning) |
Every model we got a translation from, as the shortlist compared it: off the shelf, without context in the request, or adjusted, with it. Ranks among 24 models, with each model family counted once, and the top 10 in bold. Lower is better on MetricX. The scores rank models within this language and are not a requirement result.
The two real-time places go to Translated Lara 2 and Microsoft Translator, 11th and 12th on MetricX at 2.69 and 2.8. The next real-time model, ModernMT Enterprise Edition, is 14th at 3.13 and not shortlisted. Nearest the cut is Anthropic Claude Sonnet 4.6, 13th at 3.12. Of the models counted once with their family, OpenAI GPT-5.6 Luna scores 2.33 on MetricX, OpenAI GPT-5.6 Terra 2.46 and xAI Grok 4.20 (reasoning) 2.71.
| Model | US dollars per million characters | Class |
|---|---|---|
| Google Gemma 4 | $0.15 | Open-weight LLMs |
| OpenAI GPT-5.6 Luna | $0.47 | LLMs |
| MiniMax M2.5 | $0.65 | LLMs |
| Mistral Medium 3.1 | $0.67 | LLMs |
| Mistral Large 3 | $0.82 | LLMs |
| DeepSeek V3.2 | $1.06 | LLMs |
| xAI Grok 4.20 (reasoning) | $1.25 | LLMs |
| xAI Grok 4.20 (non-reasoning) | $1.26 | LLMs |
| MoonshotAI Kimi K2.5 | $1.36 | LLMs |
| z.ai GLM-5 | $2.07 | LLMs |
| Anthropic Claude Haiku 4.5 | $2.29 | LLMs |
| OpenAI GPT-4.1 | $3.14 | LLMs |
| Google Gemini 3.5 Flash | $4 | LLMs |
| OpenAI GPT-5.6 Terra | $4.72 | LLMs |
| Google Gemini 3.1 Pro Preview | $5.22 | LLMs |
| Anthropic Claude Sonnet 5 | $6.37 | LLMs |
| Youdao Translation Service API | $6.71 | MT engines |
| Anthropic Claude Sonnet 4.6 | $7.16 | LLMs |
| OpenAI GPT-5.6 Sol | $8.12 | LLMs |
| Microsoft Translator | $10 | MT engines |
| OpenAI GPT-5.5 | $11.88 | LLMs |
| Anthropic Claude Opus 4.8 | $16.24 | LLMs |
| Google Cloud Advanced Translation API | $20 | MT engines |
| Translated Lara 2 | $24.99 | LLM-based MT engines |
| DeepL Next Gen | $27.5 | LLM-based MT engines |
| Anthropic Claude Fable 5 | $33.29 | LLMs |
| Model | Tokens per character | Class |
|---|---|---|
| Anthropic Claude Fable 5 | 0.527 | LLMs |
| Anthropic Claude Sonnet 5 | 0.524 | LLMs |
| Anthropic Claude Opus 4.8 | 0.523 | LLMs |
| Mistral Medium 3.1 | 0.434 | LLMs |
| Mistral Large 3 | 0.427 | LLMs |
| Anthropic Claude Sonnet 4.6 | 0.395 | LLMs |
| Anthropic Claude Haiku 4.5 | 0.389 | LLMs |
| DeepSeek V3.2 | 0.382 | LLMs |
| MoonshotAI Kimi K2.5 | 0.375 | LLMs |
| z.ai GLM-5 | 0.37 | LLMs |
| Cohere Command A Translate | 0.367 | LLMs |
| Google Gemini 3.5 Flash | 0.357 | LLMs |
| xAI Grok 4.20 (reasoning) | 0.357 | LLMs |
| xAI Grok 4.20 (non-reasoning) | 0.356 | LLMs |
| Google Gemini 3.1 Pro Preview | 0.355 | LLMs |
| Google Gemma 4 | 0.354 | Open-weight LLMs |
| OpenAI GPT-5.6 Terra | 0.318 | LLMs |
| OpenAI GPT-5.5 | 0.317 | LLMs |
| OpenAI GPT-5.6 Sol | 0.317 | LLMs |
| OpenAI GPT-4.1 | 0.316 | LLMs |
| OpenAI GPT-5.6 Luna | 0.314 | LLMs |
| MiniMax M2.5 | 0.301 | LLMs |
Across 22 models, a translation into Sesotho has a median 0.36 tokens per character, 1.5 times that of English text. The tokenizers of OpenAI GPT-5.6 Sol and OpenAI GPT-5.5 both produce 0.317 tokens per character, but off the shelf translations from the two cost at least $8.12 and $11.88 per million characters. Off the shelf, translations from the LLM-based MT engines, Translated Lara 2 and DeepL Next Gen, cost $24.99 and $27.50 per million characters, and those from the three MT engines with a cost $6.71 to $20.
| Model | Seconds per 1,000 words | Class |
|---|---|---|
| Google Cloud Advanced Translation API | 0.1 | MT engines |
| Microsoft Translator | 0.2 | MT engines |
| ModernMT Enterprise Edition | 0.4 | MT engines |
| DeepL Next Gen | 0.6 | LLM-based MT engines |
| Translated Lara 2 | 2.2 | LLM-based MT engines |
| OpenAI gpt-oss-120b | 2.3 | Open-weight LLMs |
| OpenAI GPT-5.6 Luna | 17 | LLMs |
| Google Gemini 3.5 Flash | 21 | LLMs |
| xAI Grok 4.20 (non-reasoning) | 23 | LLMs |
| OpenAI GPT-4.1 | 35 | LLMs |
| Anthropic Claude Haiku 4.5 | 48 | LLMs |
| OpenAI GPT-5.6 Terra | 51 | LLMs |
| xAI Grok 4.20 (reasoning) | 51 | LLMs |
| DeepSeek V3.2 | 62 | LLMs |
| MiniMax M2.5 | 70 | LLMs |
| Cohere Command A Translate | 76 | LLMs |
| Google Gemini 3.1 Pro Preview | 89 | LLMs |
| Anthropic Claude Sonnet 5 | 89 | LLMs |
| Mistral Large 3 | 93 | LLMs |
| Anthropic Claude Sonnet 4.6 | 94 | LLMs |
| Anthropic Claude Fable 5 | 101 | LLMs |
| OpenAI GPT-5.5 | 111 | LLMs |
| Anthropic Claude Opus 4.8 | 119 | LLMs |
| OpenAI GPT-5.6 Sol | 126 | LLMs |
| MoonshotAI Kimi K2.5 | 203 | LLMs |
| Google Gemma 4 | 205 | Open-weight LLMs |
| z.ai GLM-5 | 280 | LLMs |
Real-time models are those the shortlist counts as real-time, under 2 seconds per text across the languages we timed. The chart shows the time of each model into Sesotho.
The six real-time models are three MT engines, two LLM-based MT engines, DeepL Next Gen and Translated Lara 2, and the open-weight OpenAI gpt-oss-120b, the slowest of them at 2.3 seconds per 1,000 words. Of the other LLMs, translations from OpenAI GPT-5.6 Luna are the fastest, at 17 seconds per 1,000 words. None of the three real-time solutions on the shortlist meet formatting.
Leave feedback