The State of Translation Automation 2026 · Languages
What it takes to automate translation from English into Mongolian against 10 requirements in three levels: the solutions that meet them, what they cost and how fast they are, from the 2026 benchmark of 57 AI models and MT engines.
A solution is a model plus its harness: prompts, glossaries and other context, pre- and post-processing. It meets a quality level when it meets every requirement of the level.
| Requirement | Description | Met when |
|---|---|---|
| Basic requirements for any company | ||
| General translation quality | Translation errors of every type, such as mistranslations, omissions and grammar errors, classified by type and severity following MQM, the industry standard. | No critical error, at most 3 major errors in the four test texts (3,034 words) and at most 20 errors per 1,000 words, weighted by severity (minor 1, major 5, critical 10). |
| Standard requirements adds the rules companies set for their content | ||
| Terminology | The approved translation of every glossary term, in any of its correct forms. | 100% of the glossary terms. We test it in the 16 languages with a glossary, so it does not apply in Mongolian. |
| Tone of voice | The formal register (Та) in every sentence in the second person or the imperative. | 100% of those sentences. |
| Tags | Every HTML tag of the source, with its attributes, in the translation. | 100% of the tags. |
| Unicode character integrity | Every character intact, with no corrupted characters and no letter from another script that only looks like the right one. | 100% of the characters. |
| Locale conventions | The punctuation marks of the target language in place of the English ones. | No pass-or-fail bar. We report it as a share, because whether to localize punctuation is a matter of house style. |
| HTML entities | No HTML character code, such as the code for a quotation mark, that the source does not carry. | No added HTML entity. |
| Valid JSON | The translated JSON document still parses. | The document parses. |
| Spacing at tag boundaries | The spaces next to HTML tags stay as in the source. | 100% of the tag boundaries. |
| Advanced requirements adds a requirement for some companies and use cases | ||
| Gender neutrality | A gender-neutral translation wherever the English source leaves the gender of a person open, as in “your manager” or “the expert”. | 100% of the sentences that refer to a person. |
Pick a quality level, or your own requirements, bars and constraints, to see the shortlisted solutions that meet them in Mongolian, with a column for each requirement you select. Under Optimize for, choose which of them comes first: the one with the fewest total errors, the lowest cost or the shortest time. Click a column heading to sort by it.
The shortlisted solutions on 10 models, with their Intento LQA results. Total errors per 1,000 words add the weighted general translation quality errors (minor 1, major 5, critical 10) to the missed glossary terms, sentences in the wrong register, tag errors and sentences that fail gender neutrality, each counted once. Formatting counts as met or missed and does not add to the total errors. The models that did not make the shortlist have automatic scores only: the shortlist. Inconclusive: the three LQA judges agree on fewer than 90% of the sentences of the solution, so it has no general translation quality result (09-02). Gender neutrality: the Advanced requirements count the two best solutions of the language, with the gender-neutrality agent where they need it. The results of the other solutions are shown for information.
Two solutions meet every Standard requirement in Mongolian, both three-step solutions (translation + LQA + repair). The one on Anthropic Claude Fable 5 has 12.5 total errors per 1,000 words, against 14.2 for the one on Google Gemini 3.1 Pro Preview. The one on Google Gemini 3.1 Pro Preview costs $609 against $666 per million characters, and takes 325 against 355 seconds per 1,000 words. Optimizing for quality selects Anthropic Claude Fable 5, and optimizing for cost or latency Google Gemini 3.1 Pro Preview.
None of the one-step or two-step solutions (translation + post-edit) in Mongolian meet even the Basic requirements. None of the real-time solutions meet them either. The four three-step solutions (translation + LQA + repair) that meet them cost $609 to $666 per million characters and take 316 to 372 seconds per 1,000 words. Under the Standard requirements, a cost cap below the $666 per million characters of the one on Anthropic Claude Fable 5 leaves only the one on Google Gemini 3.1 Pro Preview, and a cap below its $609 leaves none.
The Advanced requirements add gender neutrality, and in Mongolian they remove neither of the two solutions that meet every Standard requirement. Translations from both have 100% of the sentences about people gender-neutral without the gender-neutrality agent. So meeting the Advanced requirements takes the same three steps (translation + LQA + repair) and the same cost and time: $609 per million characters for the cheapest solution and 325 seconds per 1,000 words for the fastest.
OpenAI GPT-5.5
three-step
Google Gemini 3.1 Pro Preview
three-step
OpenAI GPT-5.5
three-step
Anthropic Claude Fable 5
three-step
Google Gemini 3.1 Pro Preview
three-step
Google Gemini 3.1 Pro Preview
three-step
Anthropic Claude Fable 5
three-step
Google Gemini 3.1 Pro Preview
three-step
Google Gemini 3.1 Pro Preview
three-step
Optimize for selects one of the solutions that meet the level: the one with the fewest total errors, the lowest cost or the shortest time. Where no solution meets the level, the card shows the available solution with the fewest total errors and what it misses. Total errors count every requirement with countable errors.
| Quality level | Tier | Solutions | Simplest solution | Model | Total errors per 1,000 words | Cost per million characters | Seconds per 1,000 words |
|---|---|---|---|---|---|---|---|
| Basic | Tier 2: two or more solutions, none simple | ☆×4 | three-step | Google Gemini 3.1 Pro Preview | 14.2 | $609 | 325 |
| Standard | Tier 2: two or more solutions, none simple | ☆☆ | three-step | Google Gemini 3.1 Pro Preview | 14.2 | $609 | 325 |
| Advanced | Tier 2: two or more solutions, none simple | ☆☆ | three-step | Google Gemini 3.1 Pro Preview | 14.2 | $609 | 325 |
One solution per level: the simplest that meets the level, the cheapest of them where several are equally simple. ★ a one-step solution, ☆ a solution with more steps or the gender-neutrality agent.
Four three-step solutions (translation + LQA + repair) meet every Basic requirement in Mongolian, and the Standard ones remove two. The one on OpenAI GPT-5.5 has 97.9% of the sentences in the formal register, and the one on Anthropic Claude Opus 4.8 99.6% of the tags right. For the Basic requirements, optimizing for quality or latency selects OpenAI GPT-5.5, and for cost Google Gemini 3.1 Pro Preview. For the Standard ones, optimizing for quality selects Anthropic Claude Fable 5, and for cost or latency Google Gemini 3.1 Pro Preview.
| Model | US dollars per million characters | Weighted errors per 1,000 words | Group |
|---|---|---|---|
| OpenAI GPT-5.6 Luna | 0.51 | 41.5 | Misses a Standard requirement |
| Google Gemini 3.1 Pro Preview | 6.53 | 30.7 | Misses a Standard requirement |
| OpenAI GPT-5.5 | 12.65 | 36.3 | Misses a Standard requirement |
| Anthropic Claude Opus 4.8 | 18.37 | 49.8 | Misses a Standard requirement |
| Google Translation LLM | 21.7 | 91 | Misses a Standard requirement |
| Translated Lara 2 | 24.99 | 115 | Misses a Standard requirement |
| DeepL Next Gen | 27.5 | 90.6 | Misses a Standard requirement |
| Anthropic Claude Fable 5 | 37.14 | 32.3 | Misses a Standard requirement |
Of the shortlisted one-step solutions in Mongolian, the dearest does not have the fewest weighted errors. The one on Anthropic Claude Fable 5 costs $37.14 per million characters and has 32.3 weighted errors per 1,000 words, against 30.7 for the one on Google Gemini 3.1 Pro Preview at $6.53. The cheapest, on OpenAI GPT-5.6 Luna, costs $0.51 per million characters and has 41.5 weighted errors per 1,000 words. The three LLM-based MT engines cost $21.70 to $27.50 per million characters and have the most weighted errors.
| Requirement | Median errors per 1,000 words |
|---|---|
| General translation quality | 45.6 |
| Tag handling | 1.3 |
| Tone of voice | 0 |
| Gender neutrality | 0 |
The requirements with countable errors. Every solution with its result on each requirement: the explorer.
Of the six three-step solutions (translation + LQA + repair) in Mongolian, four meet general translation quality. The one on Translated Lara 2 has 17.5 weighted errors per 1,000 words but 4 major errors, one major error more than the requirement allows, and the one on OpenAI GPT-5.6 Luna has 31.0 weighted errors per 1,000 words and 5 major errors. On OpenAI GPT-5.5, the second and third steps take the weighted errors from 36.3 to 25.4 and 11.2 per 1,000 words. In one step, the three LLM-based MT engines miss formatting.
| Model | Weighted errors per 1,000 words of the shortlisted one-step solutions in Mongolian | Class |
|---|---|---|
| Translated Lara 2 | 115 | LLM-based MT engines |
| Google Translation LLM | 91 | LLM-based MT engines |
| DeepL Next Gen | 90.6 | LLM-based MT engines |
| Anthropic Claude Opus 4.8 | 49.8 | LLMs |
| OpenAI GPT-5.6 Luna | 41.5 | LLMs |
| OpenAI GPT-5.5 | 36.3 | LLMs |
| Anthropic Claude Fable 5 | 32.3 | LLMs |
| Google Gemini 3.1 Pro Preview | 30.7 | LLMs |
No translation into Mongolian: AI21 Jamba Large 1.7 (garbage or near-empty).
In one step, translations into Mongolian from the five shortlisted LLMs with a conclusive Intento LQA result have 30.7 to 49.8 weighted errors per 1,000 words, and all five support Mongolian by our measured criteria. Those from the three LLM-based MT engines have 90.6 to 115.0 weighted errors per 1,000 words. Translations from Google Translation LLM have 9 critical errors, the only ones among the shortlisted one-step solutions with a conclusive result. The texts we got from AI21 Jamba Large 1.7 in Mongolian were garbage or near-empty.
| Model | COMET-Kiwi | Rank on COMET-Kiwi | MetricX | Rank on MetricX | Shortlisted | Shortlist place |
|---|---|---|---|---|---|---|
| xAI Grok 4.20 (non-reasoning) | 0.762 | 9 | 1.99 | 12 | No | Not shortlisted |
| Google Gemma 4 | 0.761 | 11 | 1.81 | 10 | No | Not shortlisted |
| OpenAI GPT-4.1 | 0.760 | 12 | 1.76 | 9 | No | Not shortlisted |
| Anthropic Claude Sonnet 4.6 | 0.757 | 13 | 1.85 | 11 | No | Not shortlisted |
| DeepSeek V3.2 | 0.754 | 15 | 2.00 | 13 | No | Not shortlisted |
| Anthropic Claude Sonnet 5 | 0.754 | 16 | 2.09 | 16 | No | Not shortlisted |
| Mistral Large 3 | 0.749 | 17 | 2.26 | 18 | No | Not shortlisted |
| MoonshotAI Kimi K2.5 | 0.749 | 18 | 2.33 | 19 | No | Not shortlisted |
| Microsoft Translator | 0.741 | 19 | 2.07 | 15 | No | Not shortlisted |
| Google Cloud Advanced Translation API | 0.733 | 20 | 2.68 | 22 | No | Not shortlisted |
| Anthropic Claude Haiku 4.5 | 0.729 | 21 | 2.53 | 21 | No | Not shortlisted |
| OpenAI gpt-oss-120b | 0.727 | 22 | 2.47 | 20 | No | Not shortlisted |
| MiniMax M2.5 | 0.718 | 23 | 2.90 | 23 | No | Not shortlisted |
| Amazon Translate | 0.712 | 24 | 3.27 | 25 | No | Not shortlisted |
| Swiss AI Apertus-8B-2509 | 0.703 | 25 | 3.24 | 24 | No | Not shortlisted |
| Cohere Command A Translate | 0.684 | 26 | 4.02 | 27 | No | Not shortlisted |
| Yandex Cloud Translate v2 | 0.681 | 27 | 4.23 | 29 | No | Not shortlisted |
| Mistral Medium 3.1 | 0.679 | 28 | 3.50 | 26 | No | Not shortlisted |
| ModernMT Enterprise Edition | 0.641 | 29 | 4.23 | 28 | No | Not shortlisted |
| Google Gemini 3.1 Pro Preview | 0.776 | 1 | 1.57 | 2 | Yes | Top 8 by both metrics |
| OpenAI GPT-5.6 Luna | 0.774 | 2 | 1.75 | 6 | Yes | Top 8 by both metrics |
| Anthropic Claude Fable 5 | 0.772 | 3 | 1.61 | 4 | Yes | Top 8 by both metrics |
| Google Gemini 3.5 Flash | 0.770 | 4 | 1.49 | 1 | Yes | Top 8 by both metrics |
| OpenAI GPT-5.5 | 0.770 | 5 | 1.71 | 5 | Yes | Top 8 by both metrics |
| Anthropic Claude Opus 4.8 | 0.769 | 6 | 1.75 | 7 | Yes | Top 8 by both metrics |
| z.ai GLM-5 | 0.768 | 7 | 1.76 | 8 | Yes | Top 8 by both metrics |
| DeepL Next Gen | 0.764 | 8 | 1.60 | 3 | Yes | Top 8 by both metrics |
| Google Translation LLM | 0.762 | 10 | 2.18 | 17 | Yes | Real-time place |
| Translated Lara 2 | 0.756 | 14 | 2.02 | 14 | Yes | Real-time place |
| Model | Solution complexity | COMET-Kiwi rank | COMET-Kiwi | MetricX rank | MetricX | Shortlist |
|---|---|---|---|---|---|---|
| Google Gemini 3.1 Pro Preview | Adjusted LLM | 1 | 0.776 | 2 | 1.57 | Top 8 by both metrics |
| OpenAI GPT-5.6 Luna | Adjusted LLM | 2 | 0.774 | 6 | 1.75 | Top 8 by both metrics |
| Anthropic Claude Fable 5 | Adjusted LLM | 3 | 0.772 | 4 | 1.61 | Top 8 by both metrics |
| Google Gemini 3.5 Flash | Adjusted LLM | 4 | 0.770 | 1 | 1.49 | Top 8 by both metrics |
| OpenAI GPT-5.5 | Adjusted LLM | 5 | 0.770 | 5 | 1.71 | Top 8 by both metrics |
| Anthropic Claude Opus 4.8 | Adjusted LLM | 6 | 0.769 | 7 | 1.75 | Top 8 by both metrics |
| z.ai GLM-5 | Adjusted LLM | 7 | 0.768 | 8 | 1.76 | Top 8 by both metrics |
| DeepL Next Gen | Off-the-shelf NMT | 8 | 0.764 | 3 | 1.60 | Top 8 by both metrics |
| xAI Grok 4.20 (non-reasoning) | Adjusted LLM | 9 | 0.762 | 12 | 1.99 | Not shortlisted |
| Google Translation LLM | Adjusted NMT | 10 | 0.762 | 17 | 2.18 | Real-time place |
| Google Gemma 4 | Adjusted LLM | 11 | 0.761 | 10 | 1.81 | Not shortlisted |
| OpenAI GPT-4.1 | Adjusted LLM | 12 | 0.760 | 9 | 1.76 | Not shortlisted |
| Anthropic Claude Sonnet 4.6 | Adjusted LLM | 13 | 0.757 | 11 | 1.85 | Not shortlisted |
| Translated Lara 2 | Adjusted NMT | 14 | 0.756 | 14 | 2.02 | Real-time place |
| DeepSeek V3.2 | Adjusted LLM | 15 | 0.754 | 13 | 2.00 | Not shortlisted |
| Anthropic Claude Sonnet 5 | Adjusted LLM | 16 | 0.754 | 16 | 2.09 | Not shortlisted |
| Mistral Large 3 | Adjusted LLM | 17 | 0.749 | 18 | 2.26 | Not shortlisted |
| MoonshotAI Kimi K2.5 | Adjusted LLM | 18 | 0.749 | 19 | 2.33 | Not shortlisted |
| Microsoft Translator | Off-the-shelf NMT | 19 | 0.741 | 15 | 2.07 | Not shortlisted |
| Google Cloud Advanced Translation API | Off-the-shelf NMT | 20 | 0.733 | 22 | 2.68 | Not shortlisted |
| Anthropic Claude Haiku 4.5 | Adjusted LLM | 21 | 0.729 | 21 | 2.53 | Not shortlisted |
| OpenAI gpt-oss-120b | Adjusted LLM | 22 | 0.727 | 20 | 2.47 | Not shortlisted |
| MiniMax M2.5 | Adjusted LLM | 23 | 0.718 | 23 | 2.90 | Not shortlisted |
| Amazon Translate | Off-the-shelf NMT | 24 | 0.712 | 25 | 3.27 | Not shortlisted |
| Swiss AI Apertus-8B-2509 | Adjusted LLM | 25 | 0.703 | 24 | 3.24 | Not shortlisted |
| Cohere Command A Translate | Adjusted LLM | 26 | 0.684 | 27 | 4.02 | Not shortlisted |
| Yandex Cloud Translate v2 | Off-the-shelf NMT | 27 | 0.681 | 29 | 4.23 | Not shortlisted |
| Mistral Medium 3.1 | Adjusted LLM | 28 | 0.679 | 26 | 3.50 | Not shortlisted |
| ModernMT Enterprise Edition | Off-the-shelf NMT | 29 | 0.641 | 28 | 4.23 | Not shortlisted |
| OpenAI GPT-5.6 Sol | Adjusted LLM | — | 0.770 | — | 1.86 | Same model family as OpenAI GPT-5.6 Luna |
| OpenAI GPT-5.6 Terra | Adjusted LLM | — | 0.772 | — | 1.63 | Same model family as OpenAI GPT-5.6 Luna |
| xAI Grok 4.20 (reasoning) | Adjusted LLM | — | 0.756 | — | 2.01 | Same model family as xAI Grok 4.20 (non-reasoning) |
Every model we got a translation from, as the shortlist compared it: off the shelf, without context in the request, or adjusted, with it. A model that did not make the shortlist is shown adjusted, unless off the shelf it scores higher by 0.03 or more on COMET-Kiwi. Ranks among 29 models, with each model family counted once, and the top 8 of each metric in bold. Higher is better on COMET-Kiwi, lower on MetricX. The scores rank models within this language and are not a requirement result.
MetricX ranks DeepL Next Gen third in Mongolian, ahead of OpenAI GPT-5.5, fifth on both metrics. By Intento LQA, the one-step solution on DeepL Next Gen has 90.6 weighted errors per 1,000 words, against 36.3 for the one on OpenAI GPT-5.5. The one-step solution on Google Gemini 3.5 Flash, first on MetricX, has no conclusive Intento LQA result. Nearest the cut in the tail are xAI Grok 4.20 (non-reasoning), ninth on COMET-Kiwi and 12th on MetricX, and OpenAI GPT-4.1, 12th on COMET-Kiwi and ninth on MetricX.
| Quality level | Cheapest solution that meets every requirement of the level | Cost per million characters |
|---|---|---|
| Basic | Google Gemini 3.1 Pro Preview, three-step | $609 |
| Standard | Google Gemini 3.1 Pro Preview, three-step | $609 |
| Advanced | Google Gemini 3.1 Pro Preview, three-step | $609 |
| Model | US dollars per million characters | Class |
|---|---|---|
| OpenAI GPT-5.6 Luna | $0.51 | LLMs |
| z.ai GLM-5 | $2.81 | LLMs |
| Google Gemini 3.5 Flash | $4.84 | LLMs |
| Google Gemini 3.1 Pro Preview | $6.53 | LLMs |
| OpenAI GPT-5.5 | $12.65 | LLMs |
| Anthropic Claude Opus 4.8 | $18.37 | LLMs |
| Google Translation LLM | $21.7 | LLM-based MT engines |
| Translated Lara 2 | $24.99 | LLM-based MT engines |
| DeepL Next Gen | $27.5 | LLM-based MT engines |
| Anthropic Claude Fable 5 | $37.14 | LLMs |
| Model | Tokens per character | Class |
|---|---|---|
| Anthropic Claude Fable 5 | 0.616 | LLMs |
| Anthropic Claude Sonnet 5 | 0.616 | LLMs |
| Anthropic Claude Opus 4.8 | 0.61 | LLMs |
| Mistral Medium 3.1 | 0.608 | LLMs |
| Mistral Large 3 | 0.606 | LLMs |
| Anthropic Claude Sonnet 4.6 | 0.586 | LLMs |
| Anthropic Claude Haiku 4.5 | 0.585 | LLMs |
| DeepSeek V3.2 | 0.536 | LLMs |
| z.ai GLM-5 | 0.532 | LLMs |
| MoonshotAI Kimi K2.5 | 0.507 | LLMs |
| Cohere Command A Translate | 0.506 | LLMs |
| xAI Grok 4.20 (non-reasoning) | 0.449 | LLMs |
| xAI Grok 4.20 (reasoning) | 0.447 | LLMs |
| MiniMax M2.5 | 0.447 | LLMs |
| Google Gemini 3.1 Pro Preview | 0.445 | LLMs |
| Google Gemma 4 | 0.444 | Open-weight LLMs |
| Google Gemini 3.5 Flash | 0.443 | LLMs |
| OpenAI GPT-4.1 | 0.359 | LLMs |
| OpenAI GPT-5.6 Terra | 0.358 | LLMs |
| OpenAI GPT-5.6 Sol | 0.355 | LLMs |
| OpenAI GPT-5.5 | 0.354 | LLMs |
| OpenAI GPT-5.6 Luna | 0.354 | LLMs |
| Quality level | Fastest solution that meets every requirement of the level | Seconds per 1,000 words |
|---|---|---|
| Basic | OpenAI GPT-5.5, three-step | 316 |
| Standard | Google Gemini 3.1 Pro Preview, three-step | 325 |
| Advanced | Google Gemini 3.1 Pro Preview, three-step | 325 |
| Model | Seconds per 1,000 words | Class |
|---|---|---|
| Google Cloud Advanced Translation API | 0.2 | MT engines |
| Microsoft Translator | 0.3 | MT engines |
| ModernMT Enterprise Edition | 0.5 | MT engines |
| Google Translation LLM | 0.9 | LLM-based MT engines |
| DeepL Next Gen | 1.2 | LLM-based MT engines |
| Amazon Translate | 1.6 | MT engines |
| Translated Lara 2 | 2.2 | LLM-based MT engines |
| OpenAI gpt-oss-120b | 3.2 | Open-weight LLMs |
| Yandex Cloud Translate v2 | 3.9 | MT engines |
| OpenAI GPT-5.6 Luna | 18 | LLMs |
| Google Gemini 3.5 Flash | 38 | LLMs |
| OpenAI GPT-4.1 | 39 | LLMs |
| xAI Grok 4.20 (non-reasoning) | 43 | LLMs |
| OpenAI GPT-5.6 Terra | 49 | LLMs |
| DeepSeek V3.2 | 55 | LLMs |
| Swiss AI Apertus-8B-2509 | 60 | Open-weight LLMs |
| Anthropic Claude Haiku 4.5 | 64 | LLMs |
| MiniMax M2.5 | 68 | LLMs |
| xAI Grok 4.20 (reasoning) | 68 | LLMs |
| Mistral Medium 3.1 | 71 | LLMs |
| OpenAI GPT-5.5 | 88 | LLMs |
| Mistral Large 3 | 88 | LLMs |
| Cohere Command A Translate | 95 | LLMs |
| Google Gemini 3.1 Pro Preview | 98 | LLMs |
| OpenAI GPT-5.6 Sol | 101 | LLMs |
| Anthropic Claude Sonnet 4.6 | 117 | LLMs |
| Anthropic Claude Sonnet 5 | 124 | LLMs |
| Anthropic Claude Fable 5 | 141 | LLMs |
| Anthropic Claude Opus 4.8 | 166 | LLMs |
| Google Gemma 4 | 198 | Open-weight LLMs |
| z.ai GLM-5 | 296 | LLMs |
| MoonshotAI Kimi K2.5 | 331 | LLMs |
Real-time models are those the shortlist counts as real-time, under 2 seconds per text across the languages we timed. The chart shows the time of each model into Mongolian.
| The two best solutions | Solution | Sentences about people that are gender-neutral | Meets every Advanced requirement | Cost per million characters | Seconds per 1,000 words |
|---|---|---|---|---|---|
| Anthropic Claude Fable 5 | three-step | 100% | ✓ | $666 | 355 |
| Google Gemini 3.1 Pro Preview | three-step | 100% | ✓ | $609 | 325 |
For Mongolian we check the formal register, with Та as the form of address. Translations from all ten shortlisted one-step solutions have 100% of the sentences in the formal register. Those from the three-step solutions (translation + LQA + repair) on OpenAI GPT-5.5 and Translated Lara 2 have 97.9% of the sentences in the formal register. Mongolian has no grammatical gender for persons, so a gender-neutral translation into it has no gendered terms and never a default gender.
Leave feedback