The State of Translation Automation 2026 · Languages
What it takes to automate translation from English into Wolof against 10 requirements in three levels: the solutions that meet them, what they cost and how fast they are, from the 2026 benchmark of 57 AI models and MT engines.
A solution is a model plus its harness: prompts, glossaries and other context, pre- and post-processing. It meets a quality level when it meets every requirement of the level.
| Requirement | Description | Met when |
|---|---|---|
| Basic requirements for any company | ||
| General translation quality | Translation errors of every type, such as mistranslations, omissions and grammar errors, classified by type and severity following MQM, the industry standard. | No critical error, at most 3 major errors in the four test texts (3,034 words) and at most 20 errors per 1,000 words, weighted by severity (minor 1, major 5, critical 10). |
| Standard requirements adds the rules companies set for their content | ||
| Terminology | The approved translation of every glossary term, in any of its correct forms. | 100% of the glossary terms. We test it in the 16 languages with a glossary, so it does not apply in Wolof. |
| Tone of voice | The formal register in every sentence in the second person or the imperative. | 100% of those sentences. |
| Tags | Every HTML tag of the source, with its attributes, in the translation. | 100% of the tags. |
| Unicode character integrity | Every character intact, with no corrupted characters and no letter from another script that only looks like the right one. | 100% of the characters. |
| Locale conventions | The punctuation marks of the target language in place of the English ones. | No pass-or-fail bar. We report it as a share, because whether to localize punctuation is a matter of house style. |
| HTML entities | No HTML character code, such as the code for a quotation mark, that the source does not carry. | No added HTML entity. |
| Valid JSON | The translated JSON document still parses. | The document parses. |
| Spacing at tag boundaries | The spaces next to HTML tags stay as in the source. | 100% of the tag boundaries. |
| Advanced requirements adds a requirement for some companies and use cases | ||
| Gender neutrality | A gender-neutral translation wherever the English source leaves the gender of a person open, as in “your manager” or “the expert”. | 100% of the sentences that refer to a person. |
| Solution | Sentences on which the three judges agree (%) | Status |
|---|---|---|
| OpenAI GPT-4.1 | 91% | Passes (90% or more) |
| Google Gemma 4 | 89.3% | Does not pass |
| Anthropic Claude Sonnet 5 | 88.7% | Does not pass |
| Anthropic Claude Sonnet 4.6 | 86.4% | Does not pass |
| xAI Grok 4.20 (reasoning) | 85.3% | Does not pass |
| xAI Grok 4.20 (non-reasoning) | 85.3% | Does not pass |
| ModernMT Enterprise Edition | 85.3% | Does not pass |
| Translated Lara 2 | 83.6% | Does not pass |
| DeepSeek V3.2 | 83.1% | Does not pass |
| DeepL Next Gen | 79.1% | Does not pass |
| OpenAI GPT-5.6 Terra | 79.1% | Does not pass |
| Anthropic Claude Haiku 4.5 | 68.9% | Does not pass |
Only the solution on OpenAI GPT-4.1 passes the check: the three LQA judges agree on 91.0% of the sentences of its translations. The agreement of the similarity pair, 0.59, stays above the minimum of 0.461 across its whole band, 0.574 to 0.604. So the similarity score can rank models within Wolof, but it does not measure fluency and is not a requirement result.
Pick a quality level, or your own requirements, bars and constraints, to see the shortlisted solutions that meet them in Wolof, with a column for each requirement you select. Under Optimize for, choose which of them comes first: the one with the lowest cost or the shortest time. Click a column heading to sort by it.
The shortlisted solutions on 12 models, with their results on tone of voice, tags and formatting, which do not depend on the agreement of the three LQA judges, and on general translation quality only where the judges agree. We cannot assess translation quality in Wolof: why. The models that did not make the shortlist have automatic scores only: the shortlist. Inconclusive: the three LQA judges agree on fewer than 90% of the sentences of the solution, so it has no general translation quality result (09-02). Gender neutrality: the Advanced requirements count the two best solutions of the language, with the gender-neutrality agent where they need it. The results of the other solutions are shown for information.
Of the four shortlisted solutions that meet tag handling and formatting, the one on DeepSeek V3.2 costs the least, $0.93 per million characters, and takes the longest, 84 seconds per 1,000 words. The one on OpenAI GPT-4.1 is the fastest, at 47 seconds per 1,000 words, and costs $3.24 per million characters. It also has the highest tone of voice result of the four, 83.0%. The solution on Anthropic Claude Sonnet 4.6 costs the most of the four, $6.84 per million characters.
A time limit under the 47 seconds per 1,000 words of OpenAI GPT-4.1 leaves six shortlisted solutions: the three real-time (<2 sec) ones and those on xAI Grok 4.20 (non-reasoning), Anthropic Claude Haiku 4.5 and OpenAI GPT-5.6 Terra. None of these six meet formatting. A cost cap under $24.99 per million characters leaves out both solutions that meet tone of voice. A cap under $0.93 per million characters leaves one, on the open-weight Google Gemma 4 at $0.14, which meets tag handling and misses tone of voice and formatting.
The solution on Translated Lara 2 meets tone of voice, and its translations have 99.8% of the tags. The shortlisted solution on xAI Grok 4.20 (non-reasoning) costs $1.19 per million characters and takes 21 seconds per 1,000 words. It has a tone of voice result of 97.9%, and its translations have 99.2% of the tags. Neither of these two solutions meets formatting.
We cannot assess general translation quality in Wolof: how confident we are.
We cannot assess general translation quality in Wolof: how confident we are.
We cannot assess general translation quality in Wolof: how confident we are.
No translation into Wolof: AI21 Jamba Large 1.7 (garbage or near-empty), Swiss AI Apertus-8B-2509 (wrong language, garbage or near-empty), Cohere Command A Translate (garbage or near-empty), Cohere Tiny Aya (garbage or near-empty, wrong language), OpenAI gpt-oss-120b (garbage or near-empty).
| Rank | Model | Similarity score (higher is better) | LaBSE | Cohere embed v3 | Status |
|---|---|---|---|---|---|
| 1 | xAI Grok 4.20 (reasoning) | 2.56 | 0.677 | 0.694 | Shortlisted |
| 2 | Google Gemma 4 | 1.44 | 0.579 | 0.651 | Shortlisted |
| 3 | xAI Grok 4.20 (non-reasoning) | 1.17 | 0.538 | 0.653 | Shortlisted |
| 4 | Anthropic Claude Sonnet 4.6 | 0.95 | 0.552 | 0.622 | Shortlisted |
| 5 | DeepSeek V3.2 | 0.40 | 0.507 | 0.597 | Shortlisted |
| 6 | OpenAI GPT-4.1 | 0.39 | 0.493 | 0.606 | Shortlisted |
| 7 | Anthropic Claude Haiku 4.5 | 0.23 | 0.507 | 0.581 | Shortlisted |
| 8 | Anthropic Claude Sonnet 5 | 0.23 | 0.472 | 0.605 | Shortlisted |
| 9 | Translated Lara 2 | 0.23 | 0.473 | 0.605 | Shortlisted |
| 10 | OpenAI GPT-5.6 Terra | 0.18 | 0.471 | 0.601 | Shortlisted |
| 11 | OpenAI GPT-5.5 | -0.02 | 0.454 | 0.593 | Not shortlisted |
| 12 | Anthropic Claude Opus 4.8 | -0.09 | 0.450 | 0.589 | Not shortlisted |
| 13 | MoonshotAI Kimi K2.5 | -0.10 | 0.450 | 0.588 | Not shortlisted |
| 14 | Anthropic Claude Fable 5 | -0.13 | 0.442 | 0.591 | Not shortlisted |
| 15 | z.ai GLM-5 | -0.17 | 0.443 | 0.585 | Not shortlisted |
| 16 | Mistral Medium 3.1 | -0.22 | 0.493 | 0.545 | Not shortlisted |
| 17 | DeepL Next Gen | -0.26 | 0.436 | 0.581 | Shortlisted |
| 18 | Google Gemini 3.5 Flash | -0.42 | 0.428 | 0.571 | Not shortlisted |
| 19 | OpenAI GPT-5.6 Sol | -0.54 | 0.417 | 0.567 | Not shortlisted |
| 20 | Google Gemini 3.1 Pro Preview | -0.55 | 0.419 | 0.564 | Not shortlisted |
| 21 | Alibaba Machine Translation API | -0.60 | 0.405 | 0.569 | Not shortlisted |
| 22 | Mistral Large 3 | -0.65 | 0.425 | 0.550 | Not shortlisted |
| 23 | ModernMT Enterprise Edition | -1.49 | 0.323 | 0.538 | Shortlisted |
| 24 | MiniMax M2.5 | -2.51 | 0.358 | 0.410 | Not shortlisted |
| Model | Solution complexity | Similarity rank | LaBSE | Cohere embed v3 | Shortlist |
|---|---|---|---|---|---|
| xAI Grok 4.20 (reasoning) | Off-the-shelf LLM | 1 | 0.677 | 0.694 | Top 10 by the similarity score |
| Google Gemma 4 | Adjusted LLM | 2 | 0.579 | 0.651 | Top 10 by the similarity score |
| xAI Grok 4.20 (non-reasoning) | Adjusted LLM | 3 | 0.538 | 0.653 | Top 10 by the similarity score |
| Anthropic Claude Sonnet 4.6 | Off-the-shelf LLM | 4 | 0.552 | 0.622 | Top 10 by the similarity score |
| DeepSeek V3.2 | Adjusted LLM | 5 | 0.507 | 0.597 | Top 10 by the similarity score |
| OpenAI GPT-4.1 | Adjusted LLM | 6 | 0.493 | 0.606 | Top 10 by the similarity score |
| Anthropic Claude Haiku 4.5 | Adjusted LLM | 7 | 0.507 | 0.581 | Top 10 by the similarity score |
| Anthropic Claude Sonnet 5 | Adjusted LLM | 8 | 0.472 | 0.605 | Top 10 by the similarity score |
| Translated Lara 2 | Adjusted NMT | 9 | 0.473 | 0.605 | Top 10 by the similarity score |
| OpenAI GPT-5.6 Terra | Adjusted LLM | 10 | 0.471 | 0.601 | Top 10 by the similarity score |
| OpenAI GPT-5.5 | Adjusted LLM | 11 | 0.454 | 0.593 | Not shortlisted |
| Anthropic Claude Opus 4.8 | Adjusted LLM | 12 | 0.450 | 0.589 | Not shortlisted |
| MoonshotAI Kimi K2.5 | Adjusted LLM | 13 | 0.450 | 0.588 | Not shortlisted |
| Anthropic Claude Fable 5 | Adjusted LLM | 14 | 0.442 | 0.591 | Not shortlisted |
| z.ai GLM-5 | Adjusted LLM | 15 | 0.443 | 0.585 | Not shortlisted |
| Mistral Medium 3.1 | Adjusted LLM | 16 | 0.493 | 0.545 | Not shortlisted |
| DeepL Next Gen | Off-the-shelf NMT | 17 | 0.436 | 0.581 | Real-time place |
| Google Gemini 3.5 Flash | Adjusted LLM | 18 | 0.428 | 0.571 | Not shortlisted |
| OpenAI GPT-5.6 Sol | Adjusted LLM | 19 | 0.417 | 0.567 | Not shortlisted |
| Google Gemini 3.1 Pro Preview | Adjusted LLM | 20 | 0.419 | 0.564 | Not shortlisted |
| Alibaba Machine Translation API | Off-the-shelf NMT | 21 | 0.405 | 0.569 | Not shortlisted |
| Mistral Large 3 | Adjusted LLM | 22 | 0.425 | 0.550 | Not shortlisted |
| ModernMT Enterprise Edition | Off-the-shelf NMT | 23 | 0.323 | 0.538 | Real-time place |
| MiniMax M2.5 | Off-the-shelf LLM | 24 | 0.358 | 0.410 | Not shortlisted |
| OpenAI GPT-5.6 Luna | Adjusted LLM | — | 0.471 | 0.594 | Same model family as OpenAI GPT-5.6 Terra |
Every model we got a translation from, as the shortlist compared it: off the shelf, without context in the request, or adjusted, with it. Ranks among 24 models, with each model family counted once, and the top 10 in bold. The similarity rank combines LaBSE and Cohere embed v3, higher is better on both. The scores rank models within this language and are not a requirement result.
| Model | US dollars per million characters | Class |
|---|---|---|
| Google Gemma 4 | $0.13 | Open-weight LLMs |
| OpenAI GPT-5.6 Luna | $0.48 | LLMs |
| Mistral Large 3 | $0.8 | LLMs |
| DeepSeek V3.2 | $0.88 | LLMs |
| Mistral Medium 3.1 | $1 | LLMs |
| xAI Grok 4.20 (reasoning) | $1.05 | LLMs |
| xAI Grok 4.20 (non-reasoning) | $1.13 | LLMs |
| MoonshotAI Kimi K2.5 | $1.26 | LLMs |
| z.ai GLM-5 | $1.92 | LLMs |
| Anthropic Claude Haiku 4.5 | $3.03 | LLMs |
| OpenAI GPT-4.1 | $3.03 | LLMs |
| MiniMax M2.5 | $3.45 | LLMs |
| Google Gemini 3.5 Flash | $3.9 | LLMs |
| OpenAI GPT-5.6 Terra | $4.73 | LLMs |
| Google Gemini 3.1 Pro Preview | $5.02 | LLMs |
| Anthropic Claude Sonnet 5 | $5.95 | LLMs |
| Anthropic Claude Sonnet 4.6 | $6.84 | LLMs |
| Alibaba Machine Translation API | $7 | MT engines |
| OpenAI GPT-5.6 Sol | $8.39 | LLMs |
| OpenAI GPT-5.5 | $12.07 | LLMs |
| Anthropic Claude Opus 4.8 | $14.77 | LLMs |
| Translated Lara 2 | $24.99 | LLM-based MT engines |
| DeepL Next Gen | $27.5 | LLM-based MT engines |
| Anthropic Claude Fable 5 | $30.17 | LLMs |
| Model | Tokens per character | Class |
|---|---|---|
| Anthropic Claude Fable 5 | 0.561 | LLMs |
| Anthropic Claude Opus 4.8 | 0.56 | LLMs |
| Anthropic Claude Sonnet 5 | 0.549 | LLMs |
| Mistral Large 3 | 0.461 | LLMs |
| Mistral Medium 3.1 | 0.458 | LLMs |
| Anthropic Claude Haiku 4.5 | 0.451 | LLMs |
| MiniMax M2.5 | 0.44 | LLMs |
| Anthropic Claude Sonnet 4.6 | 0.423 | LLMs |
| MoonshotAI Kimi K2.5 | 0.4 | LLMs |
| Google Gemini 3.1 Pro Preview | 0.398 | LLMs |
| z.ai GLM-5 | 0.397 | LLMs |
| Google Gemini 3.5 Flash | 0.396 | LLMs |
| DeepSeek V3.2 | 0.381 | LLMs |
| OpenAI GPT-5.6 Sol | 0.372 | LLMs |
| OpenAI GPT-5.5 | 0.368 | LLMs |
| OpenAI GPT-5.6 Terra | 0.364 | LLMs |
| OpenAI GPT-4.1 | 0.36 | LLMs |
| OpenAI GPT-5.6 Luna | 0.36 | LLMs |
| xAI Grok 4.20 (non-reasoning) | 0.358 | LLMs |
| Google Gemma 4 | 0.348 | Open-weight LLMs |
| xAI Grok 4.20 (reasoning) | 0.317 | LLMs |
Across the 21 models with a token count, the median is 0.4 tokens per character, 1.6 times as many as English text. Translations from OpenAI GPT-4.1 and xAI Grok 4.20 (non-reasoning) have about the same tokens per character, 0.36 and 0.358. Yet off the shelf, they cost $3.03 and $1.13 per million characters. So tokens per character set only part of the cost.
| Model | Seconds per 1,000 words | Class |
|---|---|---|
| ModernMT Enterprise Edition | 0.4 | MT engines |
| DeepL Next Gen | 0.6 | LLM-based MT engines |
| Translated Lara 2 | 2.1 | LLM-based MT engines |
| xAI Grok 4.20 (non-reasoning) | 21 | LLMs |
| Alibaba Machine Translation API | 25 | MT engines |
| Google Gemini 3.5 Flash | 26 | LLMs |
| OpenAI GPT-5.6 Luna | 26 | LLMs |
| Anthropic Claude Haiku 4.5 | 28 | LLMs |
| OpenAI GPT-5.6 Terra | 45 | LLMs |
| OpenAI GPT-4.1 | 47 | LLMs |
| Mistral Medium 3.1 | 49 | LLMs |
| xAI Grok 4.20 (reasoning) | 60 | LLMs |
| Anthropic Claude Sonnet 4.6 | 72 | LLMs |
| Anthropic Claude Sonnet 5 | 78 | LLMs |
| DeepSeek V3.2 | 84 | LLMs |
| Anthropic Claude Fable 5 | 87 | LLMs |
| Mistral Large 3 | 90 | LLMs |
| Anthropic Claude Opus 4.8 | 98 | LLMs |
| OpenAI GPT-5.5 | 106 | LLMs |
| Google Gemini 3.1 Pro Preview | 106 | LLMs |
| OpenAI GPT-5.6 Sol | 115 | LLMs |
| Google Gemma 4 | 183 | Open-weight LLMs |
| MiniMax M2.5 | 183 | LLMs |
| z.ai GLM-5 | 249 | LLMs |
| MoonshotAI Kimi K2.5 | 274 | LLMs |
Real-time models are those the shortlist counts as real-time, under 2 seconds per text across the languages we timed. The chart shows the time of each model into Wolof.
The three real-time (<2 sec) models in Wolof are the MT engine ModernMT Enterprise Edition and the LLM-based MT engines DeepL Next Gen and Translated Lara 2. A translation takes 21 to 183 seconds per 1,000 words with the nine shortlisted LLMs, from xAI Grok 4.20 (non-reasoning), the fastest LLM we timed, to the open-weight Google Gemma 4.
Leave feedback