The State of Translation Automation 2026 · Languages
What it takes to automate translation from English into Scottish Gaelic against 10 requirements in three levels: the solutions that meet them, what they cost and how fast they are, from the 2026 benchmark of 57 AI models and MT engines.
A solution is a model plus its harness: prompts, glossaries and other context, pre- and post-processing. It meets a quality level when it meets every requirement of the level.
| Requirement | Description | Met when |
|---|---|---|
| Basic requirements for any company | ||
| General translation quality | Translation errors of every type, such as mistranslations, omissions and grammar errors, classified by type and severity following MQM, the industry standard. | No critical error, at most 3 major errors in the four test texts (3,034 words) and at most 20 errors per 1,000 words, weighted by severity (minor 1, major 5, critical 10). |
| Standard requirements adds the rules companies set for their content | ||
| Terminology | The approved translation of every glossary term, in any of its correct forms. | 100% of the glossary terms. We test it in the 16 languages with a glossary, so it does not apply in Scottish Gaelic. |
| Tone of voice | The formal register (sibh) in every sentence in the second person or the imperative. | 100% of those sentences. |
| Tags | Every HTML tag of the source, with its attributes, in the translation. | 100% of the tags. |
| Unicode character integrity | Every character intact, with no corrupted characters and no letter from another script that only looks like the right one. | 100% of the characters. |
| Locale conventions | The punctuation marks of the target language in place of the English ones. | No pass-or-fail bar. We report it as a share, because whether to localize punctuation is a matter of house style. |
| HTML entities | No HTML character code, such as the code for a quotation mark, that the source does not carry. | No added HTML entity. |
| Valid JSON | The translated JSON document still parses. | The document parses. |
| Spacing at tag boundaries | The spaces next to HTML tags stay as in the source. | 100% of the tag boundaries. |
| Advanced requirements adds a requirement for some companies and use cases | ||
| Gender neutrality | A gender-neutral translation wherever the English source leaves the gender of a person open, as in “your manager” or “the expert”. | 100% of the sentences that refer to a person. |
| Solution | Sentences on which the three judges agree (%) | Status |
|---|---|---|
| Google Cloud Advanced Translation API | 100% | Passes (90% or more) |
| OpenAI GPT-5.5 | 92.1% | Passes (90% or more) |
| Google Gemini 3.1 Pro Preview | 91% | Passes (90% or more) |
| OpenAI GPT-4.1 | 89.8% | Does not pass |
| Google Gemini 3.5 Flash | 89.3% | Does not pass |
| OpenAI GPT-5.6 Terra | 88.1% | Does not pass |
| Anthropic Claude Fable 5 | 87.6% | Does not pass |
| Google Translation LLM | 87.6% | Does not pass |
| DeepSeek V3.2 | 87% | Does not pass |
| Anthropic Claude Sonnet 5 | 85.9% | Does not pass |
| Translated Lara 2 | 85.3% | Does not pass |
| Anthropic Claude Sonnet 4.6 | 84.2% | Does not pass |
The quality-estimation pair agrees above its minimum of 0.534, so COMET-Kiwi and MetricX can order the models within Scottish Gaelic. That order is not a requirement result. The LQA judges agree on 90% or more of the sentences for the solutions on Google Cloud Advanced Translation API, OpenAI GPT-5.5 and Google Gemini 3.1 Pro Preview. Of the other nine, the three judges agree most often on the translations from OpenAI GPT-4.1, on 89.8% of the sentences, short of 90%.
Pick a quality level, or your own requirements, bars and constraints, to see the shortlisted solutions that meet them in Scottish Gaelic, with a column for each requirement you select. Under Optimize for, choose which of them comes first: the one with the lowest cost or the shortest time. Click a column heading to sort by it.
The shortlisted solutions on 12 models, with their results on tone of voice, tags and formatting, which do not depend on the agreement of the three LQA judges, and on general translation quality only where the judges agree. We cannot assess translation quality in Scottish Gaelic: why. The models that did not make the shortlist have automatic scores only: the shortlist. Inconclusive: the three LQA judges agree on fewer than 90% of the sentences of the solution, so it has no general translation quality result (09-02). Gender neutrality: the Advanced requirements count the two best solutions of the language, with the gender-neutrality agent where they need it. The results of the other solutions are shown for information.
Of the five solutions that meet tag handling and formatting, translations from OpenAI GPT-4.1 have 87.2% of the sentences in the formal register, and those from the other four 97.9%. The solution on Google Gemini 3.5 Flash costs $4.41 per million characters, against $3.55 on OpenAI GPT-4.1. It takes 80 seconds per 1,000 words, against 66. The solutions on Google Gemini 3.1 Pro Preview, Anthropic Claude Sonnet 4.6 and Anthropic Claude Fable 5 cost $5.97 to $35.67 per million characters and take 102 to 136 seconds per 1,000 words.
A cost cap under $3.55 per million characters leaves one shortlisted solution, on DeepSeek V3.2 at $1.17. It meets formatting, and its translations have 99.2% of the tags right and 25.5% of the sentences in the formal register. A time limit of 60 seconds per 1,000 words leaves four: the solution on OpenAI GPT-5.6 Terra and the three real-time (<2 sec) solutions. Only the one on OpenAI GPT-5.6 Terra meets formatting, and only the one on Translated Lara 2 meets tag handling.
Of the 12 shortlisted solutions, 7 meet tag handling, with a bar of 100% of the tags right. Translations from OpenAI GPT-5.6 Terra, DeepSeek V3.2 and Anthropic Claude Sonnet 5 have 99.6%, 99.2% and 98.8% of the tags right, and all three solutions meet formatting. The solution on OpenAI GPT-5.6 Terra costs $5.18 per million characters and takes 60 seconds per 1,000 words. The solution on Anthropic Claude Sonnet 5 costs $7.15 per million characters and takes 96 seconds per 1,000 words.
We cannot assess general translation quality in Scottish Gaelic: how confident we are.
We cannot assess general translation quality in Scottish Gaelic: how confident we are.
We cannot assess general translation quality in Scottish Gaelic: how confident we are.
| Model | COMET-Kiwi scores of the shortlisted solutions in Scottish Gaelic (higher is better) | Class |
|---|---|---|
| OpenAI GPT-5.6 Terra | 0.684 | LLMs |
| OpenAI GPT-4.1 | 0.681 | LLMs |
| Anthropic Claude Sonnet 4.6 | 0.681 | LLMs |
| OpenAI GPT-5.5 | 0.679 | LLMs |
| Google Gemini 3.1 Pro Preview | 0.678 | LLMs |
| Google Gemini 3.5 Flash | 0.677 | LLMs |
| Anthropic Claude Fable 5 | 0.675 | LLMs |
| DeepSeek V3.2 | 0.674 | LLMs |
| Anthropic Claude Sonnet 5 | 0.673 | LLMs |
| Google Translation LLM | 0.67 | LLM-based MT engines |
| Translated Lara 2 | 0.667 | LLM-based MT engines |
| Google Cloud Advanced Translation API | 0.654 | MT engines |
No translation into Scottish Gaelic: Swiss AI Apertus-8B-2509 (garbage or near-empty).
The automatic scores rank models within Scottish Gaelic and are not a requirement result. The top 11 models on COMET-Kiwi are all LLMs. The two LLM-based MT engines, Google Translation LLM and Translated Lara 2, are 12th and 13th of 26. The MT engines come later: Google Cloud Advanced Translation API is 16th, Yandex Cloud Translate v2 19th and ModernMT Enterprise Edition 23rd. The open-weight OpenAI gpt-oss-120b is 20th, and the open-weight Google Gemma 4 25th.
| Model | COMET-Kiwi | Rank on COMET-Kiwi | MetricX | Rank on MetricX | Shortlisted | Shortlist place |
|---|---|---|---|---|---|---|
| xAI Grok 4.20 (reasoning) | 0.672 | 10 | 2.68 | 12 | No | Not shortlisted |
| Anthropic Claude Opus 4.8 | 0.672 | 11 | 2.61 | 10 | No | Not shortlisted |
| Anthropic Claude Haiku 4.5 | 0.665 | 14 | 2.80 | 13 | No | Not shortlisted |
| z.ai GLM-5 | 0.663 | 15 | 2.92 | 15 | No | Not shortlisted |
| MoonshotAI Kimi K2.5 | 0.652 | 17 | 3.11 | 17 | No | Not shortlisted |
| Mistral Large 3 | 0.647 | 18 | 2.99 | 16 | No | Not shortlisted |
| Yandex Cloud Translate v2 | 0.623 | 19 | 3.43 | 19 | No | Not shortlisted |
| OpenAI gpt-oss-120b | 0.617 | 20 | 4.06 | 21 | No | Not shortlisted |
| MiniMax M2.5 | 0.615 | 21 | 4.17 | 22 | No | Not shortlisted |
| Cohere Command A Translate | 0.607 | 22 | 4.05 | 20 | No | Not shortlisted |
| ModernMT Enterprise Edition | 0.607 | 23 | 4.78 | 24 | No | Not shortlisted |
| Mistral Medium 3.1 | 0.602 | 24 | 4.22 | 23 | No | Not shortlisted |
| Google Gemma 4 | 0.599 | 25 | 4.80 | 25 | No | Not shortlisted |
| AI21 Jamba Large 1.7 | 0.445 | 26 | 9.47 | 26 | No | Not shortlisted |
| OpenAI GPT-5.6 Terra | 0.684 | 1 | 2.35 | 4 | Yes | Top 8 by both metrics |
| OpenAI GPT-4.1 | 0.681 | 2 | 2.40 | 5 | Yes | Top 8 by both metrics |
| Anthropic Claude Sonnet 4.6 | 0.681 | 3 | 2.63 | 11 | Yes | Top 8 by COMET-Kiwi |
| OpenAI GPT-5.5 | 0.679 | 4 | 2.30 | 3 | Yes | Top 8 by both metrics |
| Google Gemini 3.1 Pro Preview | 0.678 | 5 | 2.15 | 1 | Yes | Top 8 by both metrics |
| Google Gemini 3.5 Flash | 0.677 | 6 | 2.26 | 2 | Yes | Top 8 by both metrics |
| Anthropic Claude Fable 5 | 0.675 | 7 | 2.47 | 7 | Yes | Top 8 by both metrics |
| DeepSeek V3.2 | 0.674 | 8 | 2.46 | 6 | Yes | Top 8 by both metrics |
| Anthropic Claude Sonnet 5 | 0.673 | 9 | 2.52 | 8 | Yes | Top 8 by MetricX |
| Google Translation LLM | 0.670 | 12 | 2.53 | 9 | Yes | Real-time place |
| Translated Lara 2 | 0.667 | 13 | 2.90 | 14 | Yes | Real-time place |
| Google Cloud Advanced Translation API | 0.654 | 16 | 3.20 | 18 | Yes | Real-time place |
| Model | Solution complexity | COMET-Kiwi rank | COMET-Kiwi | MetricX rank | MetricX | Shortlist |
|---|---|---|---|---|---|---|
| OpenAI GPT-5.6 Terra | Adjusted LLM | 1 | 0.684 | 4 | 2.35 | Top 8 by both metrics |
| OpenAI GPT-4.1 | Adjusted LLM | 2 | 0.681 | 5 | 2.40 | Top 8 by both metrics |
| Anthropic Claude Sonnet 4.6 | Adjusted LLM | 3 | 0.681 | 11 | 2.63 | Top 8 by COMET-Kiwi |
| OpenAI GPT-5.5 | Adjusted LLM | 4 | 0.679 | 3 | 2.30 | Top 8 by both metrics |
| Google Gemini 3.1 Pro Preview | Adjusted LLM | 5 | 0.678 | 1 | 2.15 | Top 8 by both metrics |
| Google Gemini 3.5 Flash | Adjusted LLM | 6 | 0.677 | 2 | 2.26 | Top 8 by both metrics |
| Anthropic Claude Fable 5 | Adjusted LLM | 7 | 0.675 | 7 | 2.47 | Top 8 by both metrics |
| DeepSeek V3.2 | Adjusted LLM | 8 | 0.674 | 6 | 2.46 | Top 8 by both metrics |
| Anthropic Claude Sonnet 5 | Adjusted LLM | 9 | 0.673 | 8 | 2.52 | Top 8 by MetricX |
| xAI Grok 4.20 (reasoning) | Adjusted LLM | 10 | 0.672 | 12 | 2.68 | Not shortlisted |
| Anthropic Claude Opus 4.8 | Adjusted LLM | 11 | 0.672 | 10 | 2.61 | Not shortlisted |
| Google Translation LLM | Adjusted NMT | 12 | 0.670 | 9 | 2.53 | Real-time place |
| Translated Lara 2 | Adjusted NMT | 13 | 0.667 | 14 | 2.90 | Real-time place |
| Anthropic Claude Haiku 4.5 | Adjusted LLM | 14 | 0.665 | 13 | 2.80 | Not shortlisted |
| z.ai GLM-5 | Adjusted LLM | 15 | 0.663 | 15 | 2.92 | Not shortlisted |
| Google Cloud Advanced Translation API | Off-the-shelf NMT | 16 | 0.654 | 18 | 3.20 | Real-time place |
| MoonshotAI Kimi K2.5 | Adjusted LLM | 17 | 0.652 | 17 | 3.11 | Not shortlisted |
| Mistral Large 3 | Adjusted LLM | 18 | 0.647 | 16 | 2.99 | Not shortlisted |
| Yandex Cloud Translate v2 | Off-the-shelf NMT | 19 | 0.623 | 19 | 3.43 | Not shortlisted |
| OpenAI gpt-oss-120b | Adjusted LLM | 20 | 0.617 | 21 | 4.06 | Not shortlisted |
| MiniMax M2.5 | Adjusted LLM | 21 | 0.615 | 22 | 4.17 | Not shortlisted |
| Cohere Command A Translate | Off-the-shelf LLM | 22 | 0.607 | 20 | 4.05 | Not shortlisted |
| ModernMT Enterprise Edition | Off-the-shelf NMT | 23 | 0.607 | 24 | 4.78 | Not shortlisted |
| Mistral Medium 3.1 | Adjusted LLM | 24 | 0.602 | 23 | 4.22 | Not shortlisted |
| Google Gemma 4 | Off-the-shelf LLM | 25 | 0.599 | 25 | 4.80 | Not shortlisted |
| AI21 Jamba Large 1.7 | Adjusted LLM | 26 | 0.445 | 26 | 9.47 | Not shortlisted |
| OpenAI GPT-5.6 Luna | Adjusted LLM | — | 0.683 | — | 2.21 | Same model family as OpenAI GPT-5.6 Terra |
| OpenAI GPT-5.6 Sol | Adjusted LLM | — | 0.676 | — | 2.21 | Same model family as OpenAI GPT-5.6 Terra |
| xAI Grok 4.20 (non-reasoning) | Adjusted LLM | — | 0.670 | — | 2.71 | Same model family as xAI Grok 4.20 (reasoning) |
Every model we got a translation from, as the shortlist compared it: off the shelf, without context in the request, or adjusted, with it. A model that did not make the shortlist is shown adjusted, unless off the shelf it scores higher by 0.03 or more on COMET-Kiwi. Ranks among 26 models, with each model family counted once, and the top 8 of each metric in bold. Higher is better on COMET-Kiwi, lower on MetricX. The scores rank models within this language and are not a requirement result.
Anthropic Claude Sonnet 4.6 is on the list by COMET-Kiwi alone, 3rd there and 11th on MetricX. Anthropic Claude Sonnet 5 is on it by MetricX alone, 8th there and 9th on COMET-Kiwi. Google Translation LLM, 9th on MetricX, is on it for a real-time place. Nearest the cut in the tail are xAI Grok 4.20 (reasoning), 10th on COMET-Kiwi, and Anthropic Claude Opus 4.8, 11th on COMET-Kiwi and 10th on MetricX.
| Model | US dollars per million characters | Class |
|---|---|---|
| Google Gemma 4 | $0.17 | Open-weight LLMs |
| OpenAI GPT-5.6 Luna | $0.51 | LLMs |
| MiniMax M2.5 | $0.72 | LLMs |
| Mistral Large 3 | $0.9 | LLMs |
| Mistral Medium 3.1 | $1.15 | LLMs |
| DeepSeek V3.2 | $1.16 | LLMs |
| xAI Grok 4.20 (non-reasoning) | $1.39 | LLMs |
| xAI Grok 4.20 (reasoning) | $1.39 | LLMs |
| MoonshotAI Kimi K2.5 | $1.54 | LLMs |
| z.ai GLM-5 | $2.35 | LLMs |
| Anthropic Claude Haiku 4.5 | $2.86 | LLMs |
| OpenAI GPT-4.1 | $3.5 | LLMs |
| Yandex Cloud Translate v2 | $3.94 | MT engines |
| AI21 Jamba Large 1.7 | $4.07 | LLMs |
| Google Gemini 3.5 Flash | $4.34 | LLMs |
| OpenAI GPT-5.6 Terra | $5.06 | LLMs |
| Google Gemini 3.1 Pro Preview | $5.79 | LLMs |
| Anthropic Claude Sonnet 5 | $7.04 | LLMs |
| Anthropic Claude Sonnet 4.6 | $8.65 | LLMs |
| OpenAI GPT-5.6 Sol | $8.74 | LLMs |
| OpenAI GPT-5.5 | $12.81 | LLMs |
| Anthropic Claude Opus 4.8 | $17.61 | LLMs |
| Google Cloud Advanced Translation API | $20 | MT engines |
| Google Translation LLM | $23.09 | LLM-based MT engines |
| Translated Lara 2 | $24.99 | LLM-based MT engines |
| Anthropic Claude Fable 5 | $35.25 | LLMs |
| Model | Tokens per character | Class |
|---|---|---|
| Anthropic Claude Fable 5 | 0.527 | LLMs |
| Anthropic Claude Sonnet 5 | 0.526 | LLMs |
| Anthropic Claude Opus 4.8 | 0.524 | LLMs |
| Mistral Medium 3.1 | 0.449 | LLMs |
| Anthropic Claude Sonnet 4.6 | 0.44 | LLMs |
| Mistral Large 3 | 0.439 | LLMs |
| Anthropic Claude Haiku 4.5 | 0.433 | LLMs |
| DeepSeek V3.2 | 0.393 | LLMs |
| Cohere Command A Translate | 0.39 | LLMs |
| MoonshotAI Kimi K2.5 | 0.39 | LLMs |
| z.ai GLM-5 | 0.386 | LLMs |
| AI21 Jamba Large 1.7 | 0.381 | LLMs |
| xAI Grok 4.20 (non-reasoning) | 0.368 | LLMs |
| xAI Grok 4.20 (reasoning) | 0.367 | LLMs |
| Google Gemini 3.1 Pro Preview | 0.363 | LLMs |
| Google Gemini 3.5 Flash | 0.361 | LLMs |
| Google Gemma 4 | 0.357 | Open-weight LLMs |
| OpenAI GPT-5.5 | 0.321 | LLMs |
| OpenAI GPT-5.6 Sol | 0.321 | LLMs |
| OpenAI GPT-5.6 Luna | 0.32 | LLMs |
| OpenAI GPT-5.6 Terra | 0.319 | LLMs |
| OpenAI GPT-4.1 | 0.318 | LLMs |
| MiniMax M2.5 | 0.317 | LLMs |
Over the 23 models we counted tokens for, a translation into Scottish Gaelic has a median 0.38 tokens per character, 1.5 times the figure for English. Translations from Anthropic Claude Haiku 4.5 and Anthropic Claude Sonnet 4.6 have 0.433 and 0.44 tokens per character. Off the shelf, they cost $2.86 and $8.65 per million characters. So the token count sets only part of the cost. The shortlisted solutions on the four LLMs without reasoning cost a little more, $3.55 per million characters on OpenAI GPT-4.1 against $3.50 off the shelf.
| Model | Seconds per 1,000 words | Class |
|---|---|---|
| Google Cloud Advanced Translation API | 0.2 | MT engines |
| ModernMT Enterprise Edition | 0.6 | MT engines |
| Google Translation LLM | 1 | LLM-based MT engines |
| Translated Lara 2 | 2.3 | LLM-based MT engines |
| Yandex Cloud Translate v2 | 2.3 | MT engines |
| OpenAI gpt-oss-120b | 2.6 | Open-weight LLMs |
| OpenAI GPT-5.6 Luna | 20 | LLMs |
| xAI Grok 4.20 (non-reasoning) | 29 | LLMs |
| Anthropic Claude Haiku 4.5 | 52 | LLMs |
| Mistral Medium 3.1 | 52 | LLMs |
| OpenAI GPT-5.6 Terra | 60 | LLMs |
| OpenAI GPT-4.1 | 66 | LLMs |
| MiniMax M2.5 | 70 | LLMs |
| AI21 Jamba Large 1.7 | 71 | LLMs |
| xAI Grok 4.20 (reasoning) | 79 | LLMs |
| Google Gemini 3.5 Flash | 80 | LLMs |
| DeepSeek V3.2 | 81 | LLMs |
| Cohere Command A Translate | 86 | LLMs |
| Mistral Large 3 | 93 | LLMs |
| Anthropic Claude Sonnet 5 | 96 | LLMs |
| Anthropic Claude Fable 5 | 102 | LLMs |
| Anthropic Claude Sonnet 4.6 | 103 | LLMs |
| OpenAI GPT-5.5 | 120 | LLMs |
| Anthropic Claude Opus 4.8 | 127 | LLMs |
| OpenAI GPT-5.6 Sol | 129 | LLMs |
| Google Gemini 3.1 Pro Preview | 136 | LLMs |
| Google Gemma 4 | 241 | Open-weight LLMs |
| MoonshotAI Kimi K2.5 | 265 | LLMs |
| z.ai GLM-5 | 287 | LLMs |
Real-time models are those the shortlist counts as real-time, under 2 seconds per text across the languages we timed. The chart shows the time of each model into Scottish Gaelic.
The real-time cut is 2 seconds per text, across the languages we timed. In Scottish Gaelic, Yandex Cloud Translate v2 and the open-weight OpenAI gpt-oss-120b take 2.3 and 2.6 seconds per 1,000 words, or 1.72 and 1.96 seconds per text. Yet OpenAI gpt-oss-120b counts as real-time and Yandex Cloud Translate v2 does not. Among the other LLMs, translations from OpenAI GPT-5.6 Luna are the fastest, at 20 seconds per 1,000 words. With the nine shortlisted solutions on LLMs, a translation takes 60 to 136 seconds per 1,000 words.
Leave feedback