The State of Translation Automation 2026 · Languages
What it takes to automate translation from English into Kinyarwanda against 10 requirements in three levels: the solutions that meet them, what they cost and how fast they are, from the 2026 benchmark of 57 AI models and MT engines.
A solution is a model plus its harness: prompts, glossaries and other context, pre- and post-processing. It meets a quality level when it meets every requirement of the level.
| Requirement | Description | Met when |
|---|---|---|
| Basic requirements for any company | ||
| General translation quality | Translation errors of every type, such as mistranslations, omissions and grammar errors, classified by type and severity following MQM, the industry standard. | No critical error, at most 3 major errors in the four test texts (3,034 words) and at most 20 errors per 1,000 words, weighted by severity (minor 1, major 5, critical 10). |
| Standard requirements adds the rules companies set for their content | ||
| Terminology | The approved translation of every glossary term, in any of its correct forms. | 100% of the glossary terms. We test it in the 16 languages with a glossary, so it does not apply in Kinyarwanda. |
| Tone of voice | The formal register in every sentence in the second person or the imperative. | 100% of those sentences. |
| Tags | Every HTML tag of the source, with its attributes, in the translation. | 100% of the tags. |
| Unicode character integrity | Every character intact, with no corrupted characters and no letter from another script that only looks like the right one. | 100% of the characters. |
| Locale conventions | The punctuation marks of the target language in place of the English ones. | No pass-or-fail bar. We report it as a share, because whether to localize punctuation is a matter of house style. |
| HTML entities | No HTML character code, such as the code for a quotation mark, that the source does not carry. | No added HTML entity. |
| Valid JSON | The translated JSON document still parses. | The document parses. |
| Spacing at tag boundaries | The spaces next to HTML tags stay as in the source. | 100% of the tag boundaries. |
| Advanced requirements adds a requirement for some companies and use cases | ||
| Gender neutrality | A gender-neutral translation wherever the English source leaves the gender of a person open, as in “your manager” or “the expert”. | 100% of the sentences that refer to a person. |
| Solution | Sentences on which the three judges agree (%) | Status |
|---|---|---|
| xAI Grok 4.20 (reasoning) | 90.4% | Passes (90% or more) |
| Anthropic Claude Sonnet 5 | 87% | Does not pass |
| DeepSeek V3.2 | 87% | Does not pass |
| OpenAI GPT-5.6 Terra | 85.9% | Does not pass |
| Anthropic Claude Opus 4.8 | 84.2% | Does not pass |
| Microsoft Translator | 84.2% | Does not pass |
| OpenAI gpt-oss-120b | 83.6% | Does not pass |
| Anthropic Claude Fable 5 | 81.4% | Does not pass |
| OpenAI GPT-4.1 | 81.4% | Does not pass |
| ALL-Lab | 79.7% | Does not pass |
| Translated Lara 2 | 78% | Does not pass |
In Kinyarwanda the similarity pair agrees at 0.526, and its 90% band, 0.510 to 0.541, lies above the minimum of 0.461, so the similarity score can rank the models within Kinyarwanda. It does not measure fluency. The judges agree on 90.4% of the sentences for the solution on xAI Grok 4.20 (reasoning), the only one above the bar. The agreement of a solution is not a quality result.
Pick a quality level, or your own requirements, bars and constraints, to see the shortlisted solutions that meet them in Kinyarwanda, with a column for each requirement you select. Under Optimize for, choose which of them comes first: the one with the lowest cost or the shortest time. Click a column heading to sort by it.
The shortlisted solutions on 11 models, with their results on tone of voice, tags and formatting, which do not depend on the agreement of the three LQA judges, and on general translation quality only where the judges agree. We cannot assess translation quality in Kinyarwanda: why. The models that did not make the shortlist have automatic scores only: the shortlist. Inconclusive: the three LQA judges agree on fewer than 90% of the sentences of the solution, so it has no general translation quality result (09-02). Gender neutrality: the Advanced requirements count the two best solutions of the language, with the gender-neutrality agent where they need it. The results of the other solutions are shown for information.
Of the three solutions that meet tone of voice, the one on Anthropic Claude Sonnet 5 also meets tag handling and formatting, at $6.63 per million characters and 88 seconds per 1,000 words. The solution on OpenAI GPT-5.6 Terra is cheaper and faster, at $4.65 per million characters and 52 seconds per 1,000 words, and meets formatting but not tag handling. The solution on Translated Lara 2 takes 2.3 seconds per 1,000 words, costs $24.99 per million characters and meets tag handling but not formatting.
A cost cap of $4.64 per million characters or less leaves three of the shortlisted solutions with a price in Kinyarwanda, on DeepSeek V3.2, xAI Grok 4.20 (reasoning) and OpenAI GPT-4.1. None of them meet tone of voice or tag handling. A time limit under 31 seconds per 1,000 words leaves the solutions on Microsoft Translator, Translated Lara 2 and OpenAI gpt-oss-120b.
Tag handling asks for 100% of the tags. Translations from OpenAI GPT-5.6 Terra, xAI Grok 4.20 (reasoning) and OpenAI gpt-oss-120b have 99.6% of the tags, and translations from Anthropic Claude Opus 4.8 and DeepSeek V3.2 have 99.2%. The solutions on Anthropic Claude Fable 5 and ALL-Lab meet tag handling and formatting. Their translations have 95.7% and 97.9% of the sentences in the formal register, against a bar of 100%.
We cannot assess general translation quality in Kinyarwanda: how confident we are.
We cannot assess general translation quality in Kinyarwanda: how confident we are.
We cannot assess general translation quality in Kinyarwanda: how confident we are.
No translation into Kinyarwanda: AI21 Jamba Large 1.7 (garbage or near-empty).
The similarity score ranks models within Kinyarwanda. It does not measure fluency and is not a requirement result. Of the five MT engines we got a translation from, Microsoft Translator and ALL-Lab rank 1st and 2nd of the 27 models in the ranking, and Alibaba Machine Translation API, Google Cloud Advanced Translation API and ModernMT Enterprise Edition 18th, 20th and 23rd. DeepSeek V3.2 is the highest-ranked LLM, 3rd. Of the three open-weight LLMs, OpenAI gpt-oss-120b is 11th, Google Gemma 4 17th and Swiss AI Apertus-8B-2509 24th.
| Rank | Model | Similarity score (higher is better) | LaBSE | Cohere embed v3 | Status |
|---|---|---|---|---|---|
| 1 | Microsoft Translator | 1.42 | 0.865 | 0.723 | Shortlisted |
| 2 | ALL-Lab | 0.70 | 0.841 | 0.679 | Shortlisted |
| 3 | DeepSeek V3.2 | 0.67 | 0.841 | 0.676 | Shortlisted |
| 4 | Anthropic Claude Opus 4.8 | 0.67 | 0.856 | 0.668 | Shortlisted |
| 5 | xAI Grok 4.20 (reasoning) | 0.66 | 0.841 | 0.675 | Shortlisted |
| 6 | Anthropic Claude Fable 5 | 0.63 | 0.856 | 0.665 | Shortlisted |
| 7 | OpenAI GPT-5.6 Terra | 0.61 | 0.845 | 0.669 | Shortlisted |
| 8 | OpenAI GPT-4.1 | 0.58 | 0.839 | 0.670 | Shortlisted |
| 9 | Anthropic Claude Sonnet 5 | 0.56 | 0.847 | 0.664 | Shortlisted |
| 10 | Translated Lara 2 | 0.52 | 0.842 | 0.663 | Shortlisted |
| 11 | OpenAI gpt-oss-120b | 0.50 | 0.843 | 0.662 | Shortlisted |
| 12 | OpenAI GPT-5.5 | 0.50 | 0.848 | 0.659 | Not shortlisted |
| 13 | Anthropic Claude Sonnet 4.6 | 0.42 | 0.835 | 0.660 | Not shortlisted |
| 14 | Google Gemini 3.1 Pro Preview | 0.42 | 0.847 | 0.653 | Not shortlisted |
| 15 | Google Gemini 3.5 Flash | 0.40 | 0.837 | 0.656 | Not shortlisted |
| 16 | z.ai GLM-5 | 0.32 | 0.826 | 0.656 | Not shortlisted |
| 17 | Google Gemma 4 | 0.30 | 0.822 | 0.657 | Not shortlisted |
| 18 | Alibaba Machine Translation API | 0.21 | 0.834 | 0.643 | Not shortlisted |
| 19 | MoonshotAI Kimi K2.5 | 0.06 | 0.809 | 0.645 | Not shortlisted |
| 20 | Google Cloud Advanced Translation API | -0.08 | 0.816 | 0.630 | Not shortlisted |
| 21 | Mistral Large 3 | -0.56 | 0.730 | 0.638 | Not shortlisted |
| 22 | Anthropic Claude Haiku 4.5 | -0.70 | 0.746 | 0.619 | Not shortlisted |
| 23 | ModernMT Enterprise Edition | -0.98 | 0.737 | 0.601 | Not shortlisted |
| 24 | Swiss AI Apertus-8B-2509 | -1.07 | 0.714 | 0.606 | Not shortlisted |
| 25 | Mistral Medium 3.1 | -1.47 | 0.663 | 0.602 | Not shortlisted |
| 26 | Cohere Command A Translate | -2.49 | 0.607 | 0.551 | Not shortlisted |
| 27 | MiniMax M2.5 | -2.81 | 0.614 | 0.521 | Not shortlisted |
| Model | Solution complexity | Similarity rank | LaBSE | Cohere embed v3 | Shortlist |
|---|---|---|---|---|---|
| Microsoft Translator | Off-the-shelf NMT | 1 | 0.865 | 0.723 | Top 10 by the similarity score |
| ALL-Lab | Off-the-shelf NMT | 2 | 0.841 | 0.679 | Top 10 by the similarity score |
| DeepSeek V3.2 | Adjusted LLM | 3 | 0.841 | 0.676 | Top 10 by the similarity score |
| Anthropic Claude Opus 4.8 | Adjusted LLM | 4 | 0.856 | 0.668 | Top 10 by the similarity score |
| xAI Grok 4.20 (reasoning) | Adjusted LLM | 5 | 0.841 | 0.675 | Top 10 by the similarity score |
| Anthropic Claude Fable 5 | Adjusted LLM | 6 | 0.856 | 0.665 | Top 10 by the similarity score |
| OpenAI GPT-5.6 Terra | Adjusted LLM | 7 | 0.845 | 0.669 | Top 10 by the similarity score |
| OpenAI GPT-4.1 | Adjusted LLM | 8 | 0.839 | 0.670 | Top 10 by the similarity score |
| Anthropic Claude Sonnet 5 | Adjusted LLM | 9 | 0.847 | 0.664 | Top 10 by the similarity score |
| Translated Lara 2 | Adjusted NMT | 10 | 0.842 | 0.663 | Top 10 by the similarity score |
| OpenAI gpt-oss-120b | Adjusted LLM | 11 | 0.843 | 0.662 | Real-time place |
| OpenAI GPT-5.5 | Adjusted LLM | 12 | 0.848 | 0.659 | Not shortlisted |
| Anthropic Claude Sonnet 4.6 | Adjusted LLM | 13 | 0.835 | 0.660 | Not shortlisted |
| Google Gemini 3.1 Pro Preview | Adjusted LLM | 14 | 0.847 | 0.653 | Not shortlisted |
| Google Gemini 3.5 Flash | Adjusted LLM | 15 | 0.837 | 0.656 | Not shortlisted |
| z.ai GLM-5 | Adjusted LLM | 16 | 0.826 | 0.656 | Not shortlisted |
| Google Gemma 4 | Adjusted LLM | 17 | 0.822 | 0.657 | Not shortlisted |
| Alibaba Machine Translation API | Off-the-shelf NMT | 18 | 0.834 | 0.643 | Not shortlisted |
| MoonshotAI Kimi K2.5 | Adjusted LLM | 19 | 0.809 | 0.645 | Not shortlisted |
| Google Cloud Advanced Translation API | Off-the-shelf NMT | 20 | 0.816 | 0.630 | Not shortlisted |
| Mistral Large 3 | Adjusted LLM | 21 | 0.730 | 0.638 | Not shortlisted |
| Anthropic Claude Haiku 4.5 | Adjusted LLM | 22 | 0.746 | 0.619 | Not shortlisted |
| ModernMT Enterprise Edition | Off-the-shelf NMT | 23 | 0.737 | 0.601 | Not shortlisted |
| Swiss AI Apertus-8B-2509 | Adjusted LLM | 24 | 0.714 | 0.606 | Not shortlisted |
| Mistral Medium 3.1 | Adjusted LLM | 25 | 0.663 | 0.602 | Not shortlisted |
| Cohere Command A Translate | Adjusted LLM | 26 | 0.607 | 0.551 | Not shortlisted |
| MiniMax M2.5 | Adjusted LLM | 27 | 0.614 | 0.521 | Not shortlisted |
| OpenAI GPT-5.6 Luna | Adjusted LLM | — | 0.832 | 0.665 | Same model family as OpenAI GPT-5.6 Terra |
| OpenAI GPT-5.6 Sol | Adjusted LLM | — | 0.838 | 0.657 | Same model family as OpenAI GPT-5.6 Terra |
| xAI Grok 4.20 (non-reasoning) | Adjusted LLM | — | 0.817 | 0.663 | Same model family as xAI Grok 4.20 (reasoning) |
Every model we got a translation from, as the shortlist compared it: off the shelf, without context in the request, or adjusted, with it. Ranks among 27 models, with each model family counted once, and the top 10 in bold. The similarity rank combines LaBSE and Cohere embed v3, higher is better on both. The scores rank models within this language and are not a requirement result.
OpenAI gpt-oss-120b, 11th on the similarity score, took the real-time place on the list. Of the models that did not make the shortlist, OpenAI GPT-5.5, 12th, and Anthropic Claude Sonnet 4.6, 13th, sit nearest the cut. Translations from OpenAI GPT-5.5 have a LaBSE score of 0.848, above the scores of eight of the 11 shortlisted models, and a Cohere embed v3 score of 0.659, below all of theirs.
| Model | US dollars per million characters | Class |
|---|---|---|
| Google Gemma 4 | $0.15 | Open-weight LLMs |
| OpenAI GPT-5.6 Luna | $0.45 | LLMs |
| MiniMax M2.5 | $0.6 | LLMs |
| Mistral Large 3 | $0.85 | LLMs |
| DeepSeek V3.2 | $1 | LLMs |
| Mistral Medium 3.1 | $1.04 | LLMs |
| xAI Grok 4.20 (reasoning) | $1.21 | LLMs |
| xAI Grok 4.20 (non-reasoning) | $1.22 | LLMs |
| MoonshotAI Kimi K2.5 | $1.31 | LLMs |
| z.ai GLM-5 | $2.1 | LLMs |
| Anthropic Claude Haiku 4.5 | $2.44 | LLMs |
| OpenAI GPT-4.1 | $3.11 | LLMs |
| Google Gemini 3.5 Flash | $3.97 | LLMs |
| OpenAI GPT-5.6 Terra | $4.59 | LLMs |
| Google Gemini 3.1 Pro Preview | $5.11 | LLMs |
| Anthropic Claude Sonnet 5 | $6.5 | LLMs |
| Alibaba Machine Translation API | $7 | MT engines |
| Anthropic Claude Sonnet 4.6 | $7.19 | LLMs |
| OpenAI GPT-5.6 Sol | $7.9 | LLMs |
| Microsoft Translator | $10 | MT engines |
| OpenAI GPT-5.5 | $11.49 | LLMs |
| Anthropic Claude Opus 4.8 | $15.84 | LLMs |
| Google Cloud Advanced Translation API | $20 | MT engines |
| Translated Lara 2 | $24.99 | LLM-based MT engines |
| Anthropic Claude Fable 5 | $32.76 | LLMs |
| ALL-Lab | $100 | MT engines |
| Model | Tokens per character | Class |
|---|---|---|
| Anthropic Claude Sonnet 5 | 0.547 | LLMs |
| Anthropic Claude Fable 5 | 0.545 | LLMs |
| Anthropic Claude Opus 4.8 | 0.542 | LLMs |
| Mistral Medium 3.1 | 0.454 | LLMs |
| Mistral Large 3 | 0.444 | LLMs |
| Anthropic Claude Haiku 4.5 | 0.433 | LLMs |
| Anthropic Claude Sonnet 4.6 | 0.418 | LLMs |
| DeepSeek V3.2 | 0.385 | LLMs |
| MoonshotAI Kimi K2.5 | 0.38 | LLMs |
| z.ai GLM-5 | 0.378 | LLMs |
| Google Gemini 3.5 Flash | 0.366 | LLMs |
| Google Gemma 4 | 0.366 | Open-weight LLMs |
| xAI Grok 4.20 (non-reasoning) | 0.366 | LLMs |
| Google Gemini 3.1 Pro Preview | 0.365 | LLMs |
| xAI Grok 4.20 (reasoning) | 0.363 | LLMs |
| Cohere Command A Translate | 0.333 | LLMs |
| OpenAI GPT-5.5 | 0.319 | LLMs |
| OpenAI GPT-5.6 Sol | 0.319 | LLMs |
| OpenAI GPT-5.6 Terra | 0.319 | LLMs |
| OpenAI GPT-4.1 | 0.318 | LLMs |
| MiniMax M2.5 | 0.317 | LLMs |
| OpenAI GPT-5.6 Luna | 0.315 | LLMs |
A translation into Kinyarwanda has a median 0.37 tokens per character, 1.5 times as many as English text. Tokens alone do not set the cost. Translations from OpenAI GPT-4.1 and Mistral Large 3 have 0.318 and 0.444 tokens per character, and cost $3.11 and $0.85 per million characters off the shelf. Those from Anthropic Claude Opus 4.8 and Anthropic Claude Fable 5 have 0.542 and 0.545 tokens per character, and cost $15.84 and at least $32.76 per million characters off the shelf.
| Model | Seconds per 1,000 words | Class |
|---|---|---|
| Google Cloud Advanced Translation API | 0.1 | MT engines |
| Microsoft Translator | 0.2 | MT engines |
| ModernMT Enterprise Edition | 0.3 | MT engines |
| Translated Lara 2 | 2.3 | LLM-based MT engines |
| OpenAI gpt-oss-120b | 2.4 | Open-weight LLMs |
| OpenAI GPT-5.6 Luna | 20 | LLMs |
| xAI Grok 4.20 (non-reasoning) | 21 | LLMs |
| Alibaba Machine Translation API | 23 | MT engines |
| OpenAI GPT-4.1 | 31 | LLMs |
| Anthropic Claude Haiku 4.5 | 44 | LLMs |
| ALL-Lab | 45 | MT engines |
| xAI Grok 4.20 (reasoning) | 52 | LLMs |
| OpenAI GPT-5.6 Terra | 52 | LLMs |
| Swiss AI Apertus-8B-2509 | 56 | Open-weight LLMs |
| Cohere Command A Translate | 77 | LLMs |
| Google Gemini 3.1 Pro Preview | 81 | LLMs |
| Google Gemini 3.5 Flash | 81 | LLMs |
| DeepSeek V3.2 | 84 | LLMs |
| Anthropic Claude Sonnet 5 | 88 | LLMs |
| Anthropic Claude Sonnet 4.6 | 91 | LLMs |
| Mistral Large 3 | 92 | LLMs |
| Anthropic Claude Fable 5 | 96 | LLMs |
| OpenAI GPT-5.5 | 105 | LLMs |
| Anthropic Claude Opus 4.8 | 110 | LLMs |
| OpenAI GPT-5.6 Sol | 115 | LLMs |
| MoonshotAI Kimi K2.5 | 166 | LLMs |
| Google Gemma 4 | 181 | Open-weight LLMs |
| MiniMax M2.5 | 202 | LLMs |
| Mistral Medium 3.1 | 251 | LLMs |
| z.ai GLM-5 | 292 | LLMs |
Real-time models are those the shortlist counts as real-time, under 2 seconds per text across the languages we timed. The chart shows the time of each model into Kinyarwanda.
Three of the five real-time models are MT engines. The others are Translated Lara 2, an LLM-based MT engine, and the open-weight OpenAI gpt-oss-120b, the fastest LLM at 2.4 seconds per 1,000 words. On the shortlist, translations from Microsoft Translator have 89.4% of the sentences in the formal register and 95.7% of the tags. Not every MT engine is real-time. Translations from Alibaba Machine Translation API and ALL-Lab take 23 and 45 seconds per 1,000 words. The next LLM, OpenAI GPT-5.6 Luna, takes 20 seconds per 1,000 words.
For tone of voice in Kinyarwanda we check the formal register, with a bar of 100% of the sentences. Of the shortlisted solutions, 3 out of 11 meet it, on OpenAI GPT-5.6 Terra, Translated Lara 2 and Anthropic Claude Sonnet 5. Translations from Anthropic Claude Opus 4.8, OpenAI GPT-4.1, ALL-Lab and DeepSeek V3.2 have 97.9% of the sentences in the formal register.
Leave feedback