The State of Translation Automation 2026 · Languages
What it takes to automate translation from English into Malagasy against 10 requirements in three levels: the solutions that meet them, what they cost and how fast they are, from the 2026 benchmark of 57 AI models and MT engines.
A solution is a model plus its harness: prompts, glossaries and other context, pre- and post-processing. It meets a quality level when it meets every requirement of the level.
| Requirement | Description | Met when |
|---|---|---|
| Basic requirements for any company | ||
| General translation quality | Translation errors of every type, such as mistranslations, omissions and grammar errors, classified by type and severity following MQM, the industry standard. | No critical error, at most 3 major errors in the four test texts (3,034 words) and at most 20 errors per 1,000 words, weighted by severity (minor 1, major 5, critical 10). |
| Standard requirements adds the rules companies set for their content | ||
| Terminology | The approved translation of every glossary term, in any of its correct forms. | 100% of the glossary terms. We test it in the 16 languages with a glossary, so it does not apply in Malagasy. |
| Tone of voice | The formal register (respectful forms) in every sentence in the second person or the imperative. | 100% of those sentences. |
| Tags | Every HTML tag of the source, with its attributes, in the translation. | 100% of the tags. |
| Unicode character integrity | Every character intact, with no corrupted characters and no letter from another script that only looks like the right one. | 100% of the characters. |
| Locale conventions | The punctuation marks of the target language in place of the English ones. | No pass-or-fail bar. We report it as a share, because whether to localize punctuation is a matter of house style. |
| HTML entities | No HTML character code, such as the code for a quotation mark, that the source does not carry. | No added HTML entity. |
| Valid JSON | The translated JSON document still parses. | The document parses. |
| Spacing at tag boundaries | The spaces next to HTML tags stay as in the source. | 100% of the tag boundaries. |
| Advanced requirements adds a requirement for some companies and use cases | ||
| Gender neutrality | A gender-neutral translation wherever the English source leaves the gender of a person open, as in “your manager” or “the expert”. | 100% of the sentences that refer to a person. |
| Solution | Sentences on which the three judges agree (%) | Status |
|---|---|---|
| Anthropic Claude Fable 5 | 87% | Does not pass |
| OpenAI GPT-5.6 Luna | 86.4% | Does not pass |
| OpenAI GPT-5.5 | 85.9% | Does not pass |
| DeepSeek V3.2 | 85.9% | Does not pass |
| Google Gemini 3.1 Pro Preview | 84.7% | Does not pass |
| DeepL Next Gen | 84.7% | Does not pass |
| OpenAI gpt-oss-120b | 84.2% | Does not pass |
| Translated Lara 2 | 83.6% | Does not pass |
| Google Gemini 3.5 Flash | 83.1% | Does not pass |
| Anthropic Claude Opus 4.8 | 81.9% | Does not pass |
| Google Gemma 4 | 80.8% | Does not pass |
| OpenAI GPT-4.1 | 78% | Does not pass |
In Malagasy the agreement of the three LQA judges runs from 78.0% of the sentences, for the solution on OpenAI GPT-4.1, to 87.0%, for the one on Anthropic Claude Fable 5. The agreement is not an Intento LQA error result. The similarity pair agrees at 0.56 against a threshold of 0.46, and the quality-estimation pair at 0.57 against 0.53. So the automatic scores rank models within Malagasy, but they are not a requirement result.
Pick a quality level, or your own requirements, bars and constraints, to see the shortlisted solutions that meet them in Malagasy, with a column for each requirement you select. Under Optimize for, choose which of them comes first: the one with the lowest cost or the shortest time. Click a column heading to sort by it.
The shortlisted solutions on 12 models, with their results on tone of voice, tags and formatting, which do not depend on the agreement of the three LQA judges, and on general translation quality only where the judges agree. We cannot assess translation quality in Malagasy: why. The models that did not make the shortlist have automatic scores only: the shortlist. Inconclusive: the three LQA judges agree on fewer than 90% of the sentences of the solution, so it has no general translation quality result (09-02). Gender neutrality: the Advanced requirements count the two best solutions of the language, with the gender-neutrality agent where they need it. The results of the other solutions are shown for information.
Of the seven shortlisted solutions that meet tone of voice, tag handling and formatting, the one on OpenAI GPT-5.6 Luna is the fastest. It takes 19 seconds per 1,000 words and costs $0.50 per million characters. Only the one on Google Gemma 4 costs less, $0.17 per million characters, and it takes 181 seconds per 1,000 words. The one on OpenAI GPT-4.1 costs $3.36 per million characters and takes 23 seconds per 1,000 words. The other four cost $4.34 to $34.67 per million characters and take 79 to 124 seconds per 1,000 words.
A real-time (<2 sec) limit leaves three shortlisted solutions in Malagasy, on DeepL Next Gen, Translated Lara 2 and OpenAI gpt-oss-120b. None of them meet tone of voice, tag handling and formatting together. A time limit of 24 seconds per 1,000 words adds the solutions on OpenAI GPT-5.6 Luna and OpenAI GPT-4.1, and both meet all three. Of the shortlisted solutions with a known cost, only the ones on Google Gemma 4 and OpenAI GPT-5.6 Luna cost $0.50 per million characters or less. Both meet all three as well.
Of the requirements we can assess in Malagasy, formatting is the one the fewest shortlisted solutions meet: 8 out of 12, against 9 each for tone of voice and tag handling. The four solutions that miss formatting, on OpenAI GPT-5.5 and the three real-time ones, also miss tone of voice or tag handling. The solution on DeepSeek V3.2 misses tag handling alone, with 99.6% of the tags right. It costs $1.05 per million characters.
We cannot assess general translation quality in Malagasy: how confident we are.
We cannot assess general translation quality in Malagasy: how confident we are.
We cannot assess general translation quality in Malagasy: how confident we are.
| Model | COMET-Kiwi scores of the shortlisted solutions in Malagasy (higher is better) | Class |
|---|---|---|
| Google Gemini 3.5 Flash | 0.628 | LLMs |
| OpenAI GPT-4.1 | 0.625 | LLMs |
| Google Gemma 4 | 0.625 | Open-weight LLMs |
| Anthropic Claude Opus 4.8 | 0.624 | LLMs |
| DeepL Next Gen | 0.623 | LLM-based MT engines |
| OpenAI GPT-5.6 Luna | 0.622 | LLMs |
| DeepSeek V3.2 | 0.621 | LLMs |
| OpenAI GPT-5.5 | 0.619 | LLMs |
| Anthropic Claude Fable 5 | 0.618 | LLMs |
| Translated Lara 2 | 0.614 | LLM-based MT engines |
| Google Gemini 3.1 Pro Preview | 0.61 | LLMs |
| OpenAI gpt-oss-120b | 0.61 | Open-weight LLMs |
No translation into Malagasy: AI21 Jamba Large 1.7 (garbage or near-empty, wrong language), ALL-Lab (far too short, wrong language).
The automatic scores rank models within Malagasy and are not a requirement result. The top 8 on COMET-Kiwi are seven LLMs, one of them the open-weight Google Gemma 4, and the LLM-based MT engine DeepL Next Gen. The other LLM-based MT engine, Translated Lara 2, is 10th on COMET-Kiwi and 11th on MetricX. The MT engine ranked highest, Microsoft Translator, is 18th and 13th. The last three on both metrics are LLMs compared off the shelf: Swiss AI Apertus-8B-2509, Cohere Tiny Aya and Cohere Command A Translate.
| Model | COMET-Kiwi | Rank on COMET-Kiwi | MetricX | Rank on MetricX | Shortlisted | Shortlist place |
|---|---|---|---|---|---|---|
| z.ai GLM-5 | 0.611 | 11 | 3.09 | 17 | No | Not shortlisted |
| Anthropic Claude Sonnet 4.6 | 0.609 | 14 | 2.95 | 14 | No | Not shortlisted |
| Anthropic Claude Sonnet 5 | 0.609 | 15 | 2.65 | 10 | No | Not shortlisted |
| xAI Grok 4.20 (non-reasoning) | 0.607 | 16 | 3.07 | 16 | No | Not shortlisted |
| MoonshotAI Kimi K2.5 | 0.607 | 17 | 3.27 | 19 | No | Not shortlisted |
| Microsoft Translator | 0.605 | 18 | 2.80 | 13 | No | Not shortlisted |
| Google Cloud Advanced Translation API | 0.600 | 19 | 3.34 | 21 | No | Not shortlisted |
| Yandex Cloud Translate v2 | 0.593 | 20 | 3.19 | 18 | No | Not shortlisted |
| Mistral Large 3 | 0.586 | 21 | 3.33 | 20 | No | Not shortlisted |
| Mistral Medium 3.1 | 0.559 | 22 | 5.71 | 26 | No | Not shortlisted |
| Anthropic Claude Haiku 4.5 | 0.557 | 23 | 5.18 | 24 | No | Not shortlisted |
| Alibaba Machine Translation API | 0.545 | 24 | 4.84 | 22 | No | Not shortlisted |
| ModernMT Enterprise Edition | 0.534 | 25 | 5.17 | 23 | No | Not shortlisted |
| MiniMax M2.5 | 0.533 | 26 | 5.61 | 25 | No | Not shortlisted |
| Swiss AI Apertus-8B-2509 | 0.488 | 27 | 7.61 | 27 | No | Not shortlisted |
| Cohere Tiny Aya | 0.444 | 28 | 8.57 | 28 | No | Not shortlisted |
| Cohere Command A Translate | 0.438 | 29 | 9.83 | 29 | No | Not shortlisted |
| Google Gemini 3.5 Flash | 0.628 | 1 | 2.21 | 1 | Yes | Top 8 by both metrics |
| OpenAI GPT-4.1 | 0.625 | 2 | 2.58 | 8 | Yes | Top 8 by both metrics |
| Google Gemma 4 | 0.625 | 3 | 2.35 | 6 | Yes | Top 8 by both metrics |
| Anthropic Claude Opus 4.8 | 0.624 | 4 | 2.59 | 9 | Yes | Top 8 by COMET-Kiwi |
| DeepL Next Gen | 0.623 | 5 | 2.34 | 5 | Yes | Top 8 by both metrics |
| OpenAI GPT-5.6 Luna | 0.622 | 6 | 2.26 | 3 | Yes | Top 8 by both metrics |
| DeepSeek V3.2 | 0.621 | 7 | 2.73 | 12 | Yes | Top 8 by COMET-Kiwi |
| OpenAI GPT-5.5 | 0.619 | 8 | 2.30 | 4 | Yes | Top 8 by both metrics |
| Anthropic Claude Fable 5 | 0.618 | 9 | 2.44 | 7 | Yes | Top 8 by MetricX |
| Translated Lara 2 | 0.614 | 10 | 2.72 | 11 | Yes | Real-time place |
| Google Gemini 3.1 Pro Preview | 0.610 | 12 | 2.24 | 2 | Yes | Top 8 by MetricX |
| OpenAI gpt-oss-120b | 0.610 | 13 | 3.03 | 15 | Yes | Real-time place |
| Model | Solution complexity | COMET-Kiwi rank | COMET-Kiwi | MetricX rank | MetricX | Shortlist |
|---|---|---|---|---|---|---|
| Google Gemini 3.5 Flash | Adjusted LLM | 1 | 0.628 | 1 | 2.21 | Top 8 by both metrics |
| OpenAI GPT-4.1 | Adjusted LLM | 2 | 0.625 | 8 | 2.58 | Top 8 by both metrics |
| Google Gemma 4 | Adjusted LLM | 3 | 0.625 | 6 | 2.35 | Top 8 by both metrics |
| Anthropic Claude Opus 4.8 | Adjusted LLM | 4 | 0.624 | 9 | 2.59 | Top 8 by COMET-Kiwi |
| DeepL Next Gen | Off-the-shelf NMT | 5 | 0.623 | 5 | 2.34 | Top 8 by both metrics |
| OpenAI GPT-5.6 Luna | Adjusted LLM | 6 | 0.622 | 3 | 2.26 | Top 8 by both metrics |
| DeepSeek V3.2 | Adjusted LLM | 7 | 0.621 | 12 | 2.73 | Top 8 by COMET-Kiwi |
| OpenAI GPT-5.5 | Adjusted LLM | 8 | 0.619 | 4 | 2.30 | Top 8 by both metrics |
| Anthropic Claude Fable 5 | Adjusted LLM | 9 | 0.618 | 7 | 2.44 | Top 8 by MetricX |
| Translated Lara 2 | Adjusted NMT | 10 | 0.614 | 11 | 2.72 | Real-time place |
| z.ai GLM-5 | Adjusted LLM | 11 | 0.611 | 17 | 3.09 | Not shortlisted |
| Google Gemini 3.1 Pro Preview | Adjusted LLM | 12 | 0.610 | 2 | 2.24 | Top 8 by MetricX |
| OpenAI gpt-oss-120b | Adjusted LLM | 13 | 0.610 | 15 | 3.03 | Real-time place |
| Anthropic Claude Sonnet 4.6 | Adjusted LLM | 14 | 0.609 | 14 | 2.95 | Not shortlisted |
| Anthropic Claude Sonnet 5 | Adjusted LLM | 15 | 0.609 | 10 | 2.65 | Not shortlisted |
| xAI Grok 4.20 (non-reasoning) | Adjusted LLM | 16 | 0.607 | 16 | 3.07 | Not shortlisted |
| MoonshotAI Kimi K2.5 | Adjusted LLM | 17 | 0.607 | 19 | 3.27 | Not shortlisted |
| Microsoft Translator | Off-the-shelf NMT | 18 | 0.605 | 13 | 2.80 | Not shortlisted |
| Google Cloud Advanced Translation API | Off-the-shelf NMT | 19 | 0.600 | 21 | 3.34 | Not shortlisted |
| Yandex Cloud Translate v2 | Off-the-shelf NMT | 20 | 0.593 | 18 | 3.19 | Not shortlisted |
| Mistral Large 3 | Adjusted LLM | 21 | 0.586 | 20 | 3.33 | Not shortlisted |
| Mistral Medium 3.1 | Adjusted LLM | 22 | 0.559 | 26 | 5.71 | Not shortlisted |
| Anthropic Claude Haiku 4.5 | Adjusted LLM | 23 | 0.557 | 24 | 5.18 | Not shortlisted |
| Alibaba Machine Translation API | Off-the-shelf NMT | 24 | 0.545 | 22 | 4.84 | Not shortlisted |
| ModernMT Enterprise Edition | Off-the-shelf NMT | 25 | 0.534 | 23 | 5.17 | Not shortlisted |
| MiniMax M2.5 | Adjusted LLM | 26 | 0.533 | 25 | 5.61 | Not shortlisted |
| Swiss AI Apertus-8B-2509 | Off-the-shelf LLM | 27 | 0.488 | 27 | 7.61 | Not shortlisted |
| Cohere Tiny Aya | Off-the-shelf LLM | 28 | 0.444 | 28 | 8.57 | Not shortlisted |
| Cohere Command A Translate | Off-the-shelf LLM | 29 | 0.438 | 29 | 9.83 | Not shortlisted |
| OpenAI GPT-5.6 Sol | Adjusted LLM | — | 0.620 | — | 2.30 | Same model family as OpenAI GPT-5.6 Luna |
| OpenAI GPT-5.6 Terra | Adjusted LLM | — | 0.618 | — | 2.29 | Same model family as OpenAI GPT-5.6 Luna |
| xAI Grok 4.20 (reasoning) | Adjusted LLM | — | 0.606 | — | 3.10 | Same model family as xAI Grok 4.20 (non-reasoning) |
Every model we got a translation from, as the shortlist compared it: off the shelf, without context in the request, or adjusted, with it. A model that did not make the shortlist is shown adjusted, unless off the shelf it scores higher by 0.03 or more on COMET-Kiwi. Ranks among 29 models, with each model family counted once, and the top 8 of each metric in bold. Higher is better on COMET-Kiwi, lower on MetricX. The scores rank models within this language and are not a requirement result.
The automatic scores rank models within Malagasy and are not a requirement result. COMET-Kiwi alone brought in Anthropic Claude Opus 4.8 and DeepSeek V3.2, 4th and 7th there and 9th and 12th on MetricX. MetricX alone brought in Google Gemini 3.1 Pro Preview, 2nd there and 12th on COMET-Kiwi, and Anthropic Claude Fable 5, 7th and 9th. The real-time places went to Translated Lara 2 and OpenAI gpt-oss-120b. Nearest the cut are z.ai GLM-5, 11th on COMET-Kiwi, and Anthropic Claude Sonnet 5, 10th on MetricX.
| Model | US dollars per million characters | Class |
|---|---|---|
| Google Gemma 4 | $0.16 | Open-weight LLMs |
| OpenAI GPT-5.6 Luna | $0.49 | LLMs |
| MiniMax M2.5 | $0.65 | LLMs |
| Mistral Large 3 | $0.85 | LLMs |
| DeepSeek V3.2 | $1.02 | LLMs |
| Mistral Medium 3.1 | $1.06 | LLMs |
| xAI Grok 4.20 (reasoning) | $1.27 | LLMs |
| xAI Grok 4.20 (non-reasoning) | $1.29 | LLMs |
| MoonshotAI Kimi K2.5 | $1.37 | LLMs |
| z.ai GLM-5 | $2.15 | LLMs |
| Anthropic Claude Haiku 4.5 | $2.38 | LLMs |
| OpenAI GPT-4.1 | $3.26 | LLMs |
| Yandex Cloud Translate v2 | $3.94 | MT engines |
| Google Gemini 3.5 Flash | $4.19 | LLMs |
| OpenAI GPT-5.6 Terra | $4.9 | LLMs |
| Google Gemini 3.1 Pro Preview | $5.53 | LLMs |
| Anthropic Claude Sonnet 5 | $6.77 | LLMs |
| Alibaba Machine Translation API | $7 | MT engines |
| Anthropic Claude Sonnet 4.6 | $7.56 | LLMs |
| OpenAI GPT-5.6 Sol | $8.44 | LLMs |
| Microsoft Translator | $10 | MT engines |
| OpenAI GPT-5.5 | $12.1 | LLMs |
| Anthropic Claude Opus 4.8 | $16.72 | LLMs |
| Google Cloud Advanced Translation API | $20 | MT engines |
| Translated Lara 2 | $24.99 | LLM-based MT engines |
| DeepL Next Gen | $27.5 | LLM-based MT engines |
| Anthropic Claude Fable 5 | $33.96 | LLMs |
| Model | Tokens per character | Class |
|---|---|---|
| Anthropic Claude Fable 5 | 0.514 | LLMs |
| Anthropic Claude Opus 4.8 | 0.512 | LLMs |
| Anthropic Claude Sonnet 5 | 0.51 | LLMs |
| Mistral Medium 3.1 | 0.421 | LLMs |
| Mistral Large 3 | 0.413 | LLMs |
| Anthropic Claude Sonnet 4.6 | 0.397 | LLMs |
| Anthropic Claude Haiku 4.5 | 0.369 | LLMs |
| z.ai GLM-5 | 0.363 | LLMs |
| DeepSeek V3.2 | 0.355 | LLMs |
| Google Gemini 3.1 Pro Preview | 0.351 | LLMs |
| MoonshotAI Kimi K2.5 | 0.35 | LLMs |
| Google Gemini 3.5 Flash | 0.349 | LLMs |
| Google Gemma 4 | 0.348 | Open-weight LLMs |
| xAI Grok 4.20 (non-reasoning) | 0.345 | LLMs |
| xAI Grok 4.20 (reasoning) | 0.344 | LLMs |
| Cohere Command A Translate | 0.341 | LLMs |
| Cohere Tiny Aya | 0.321 | LLMs |
| OpenAI GPT-5.6 Luna | 0.308 | LLMs |
| OpenAI GPT-5.6 Sol | 0.308 | LLMs |
| OpenAI GPT-5.5 | 0.307 | LLMs |
| OpenAI GPT-5.6 Terra | 0.307 | LLMs |
| OpenAI GPT-4.1 | 0.305 | LLMs |
| MiniMax M2.5 | 0.302 | LLMs |
Over the tokenizers of 23 models, a translation into Malagasy has a median 0.35 tokens per character, 1.5 times as many as English text. Tokens per character do not set the cost alone. The tokenizers of Google Gemma 4 and Google Gemini 3.5 Flash produce about the same tokens per character, 0.348 and 0.349. Yet off the shelf, translations from the two cost $0.16 and $4.19 per million characters.
| Model | Seconds per 1,000 words | Class |
|---|---|---|
| Google Cloud Advanced Translation API | 0.1 | MT engines |
| Microsoft Translator | 0.2 | MT engines |
| ModernMT Enterprise Edition | 0.3 | MT engines |
| DeepL Next Gen | 0.7 | LLM-based MT engines |
| Translated Lara 2 | 2 | LLM-based MT engines |
| OpenAI gpt-oss-120b | 2.4 | Open-weight LLMs |
| Yandex Cloud Translate v2 | 3 | MT engines |
| Cohere Tiny Aya | 18 | LLMs |
| OpenAI GPT-5.6 Luna | 19 | LLMs |
| OpenAI GPT-4.1 | 23 | LLMs |
| Alibaba Machine Translation API | 24 | MT engines |
| xAI Grok 4.20 (non-reasoning) | 27 | LLMs |
| Swiss AI Apertus-8B-2509 | 46 | Open-weight LLMs |
| Mistral Medium 3.1 | 52 | LLMs |
| Anthropic Claude Haiku 4.5 | 54 | LLMs |
| OpenAI GPT-5.6 Terra | 55 | LLMs |
| MiniMax M2.5 | 62 | LLMs |
| DeepSeek V3.2 | 67 | LLMs |
| xAI Grok 4.20 (reasoning) | 67 | LLMs |
| Cohere Command A Translate | 73 | LLMs |
| Google Gemini 3.5 Flash | 79 | LLMs |
| Mistral Large 3 | 85 | LLMs |
| Anthropic Claude Sonnet 5 | 92 | LLMs |
| Anthropic Claude Sonnet 4.6 | 94 | LLMs |
| Google Gemini 3.1 Pro Preview | 96 | LLMs |
| Anthropic Claude Fable 5 | 98 | LLMs |
| OpenAI GPT-5.6 Sol | 98 | LLMs |
| OpenAI GPT-5.5 | 104 | LLMs |
| Anthropic Claude Opus 4.8 | 124 | LLMs |
| Google Gemma 4 | 181 | Open-weight LLMs |
| z.ai GLM-5 | 218 | LLMs |
| MoonshotAI Kimi K2.5 | 235 | LLMs |
Real-time models are those the shortlist counts as real-time, under 2 seconds per text across the languages we timed. The chart shows the time of each model into Malagasy.
Five of the six real-time (<2 sec) models in Malagasy are MT engines or LLM-based MT engines. The sixth is the open-weight LLM OpenAI gpt-oss-120b. Of the other LLMs, translations from Cohere Tiny Aya are the fastest, at 18 seconds per 1,000 words. Of the six real-time models, only DeepL Next Gen, Translated Lara 2 and OpenAI gpt-oss-120b are on the shortlist. The solution on DeepL Next Gen meets tone of voice, and the one on Translated Lara 2 meets tag handling.
For tone of voice in Malagasy we check the formal register, with respectful forms of address. Of the 12 shortlisted solutions, 9 meet it. Translations from OpenAI GPT-5.5, Translated Lara 2 and OpenAI gpt-oss-120b have 97.9% of the sentences in the formal register. For gender neutrality, a translation does not add a gender the source does not specify. Malagasy has no grammatical gender for persons, so a translation is neutral by default and must avoid gendered terms. A default gender is never allowed.
Leave feedback