The State of Translation Automation 2026 · Languages
What it takes to automate translation from English into Frisian against 10 requirements in three levels: the solutions that meet them, what they cost and how fast they are, from the 2026 benchmark of 57 AI models and MT engines.
A solution is a model plus its harness: prompts, glossaries and other context, pre- and post-processing. It meets a quality level when it meets every requirement of the level.
| Requirement | Description | Met when |
|---|---|---|
| Basic requirements for any company | ||
| General translation quality | Translation errors of every type, such as mistranslations, omissions and grammar errors, classified by type and severity following MQM, the industry standard. | No critical error, at most 3 major errors in the four test texts (3,034 words) and at most 20 errors per 1,000 words, weighted by severity (minor 1, major 5, critical 10). |
| Standard requirements adds the rules companies set for their content | ||
| Terminology | The approved translation of every glossary term, in any of its correct forms. | 100% of the glossary terms. We test it in the 16 languages with a glossary, so it does not apply in Frisian. |
| Tone of voice | The formal register (jo) in every sentence in the second person or the imperative. | 100% of those sentences. |
| Tags | Every HTML tag of the source, with its attributes, in the translation. | 100% of the tags. |
| Unicode character integrity | Every character intact, with no corrupted characters and no letter from another script that only looks like the right one. | 100% of the characters. |
| Locale conventions | The punctuation marks of the target language in place of the English ones. | No pass-or-fail bar. We report it as a share, because whether to localize punctuation is a matter of house style. |
| HTML entities | No HTML character code, such as the code for a quotation mark, that the source does not carry. | No added HTML entity. |
| Valid JSON | The translated JSON document still parses. | The document parses. |
| Spacing at tag boundaries | The spaces next to HTML tags stay as in the source. | 100% of the tag boundaries. |
| Advanced requirements adds a requirement for some companies and use cases | ||
| Gender neutrality | A gender-neutral translation wherever the English source leaves the gender of a person open, as in “your manager” or “the expert”. | 100% of the sentences that refer to a person. |
Pick a quality level, or your own requirements, bars and constraints, to see the shortlisted solutions that meet them in Frisian, with a column for each requirement you select. Under Optimize for, choose which of them comes first: the one with the fewest total errors, the lowest cost or the shortest time. Click a column heading to sort by it.
The shortlisted solutions on 13 models, with their Intento LQA results. Total errors per 1,000 words add the weighted general translation quality errors (minor 1, major 5, critical 10) to the missed glossary terms, sentences in the wrong register, tag errors and sentences that fail gender neutrality, each counted once. Formatting counts as met or missed and does not add to the total errors. The models that did not make the shortlist have automatic scores only: the shortlist. Inconclusive: the three LQA judges agree on fewer than 90% of the sentences of the solution, so it has no general translation quality result (09-02). Gender neutrality: the Advanced requirements count the two best solutions of the language, with the gender-neutrality agent where they need it. The results of the other solutions are shown for information.
For the Basic requirements, Optimize for selects the three-step solution (translation + LQA + repair) on OpenAI GPT-5.5 for quality, with 10.6 total errors per 1,000 words. For cost and latency it selects the one on Google Gemini 3.1 Pro Preview. It has 16.8 total errors per 1,000 words, but costs $560 instead of $581 per million characters and takes 281 instead of 320 seconds per 1,000 words.
None of the one-step or two-step solutions meet general translation quality in Frisian, and every level includes it. So meeting any level takes three steps. Every solution that meets the Basic requirements costs $560 per million characters or more and takes 280 seconds per 1,000 words or more. None of the three real-time solutions (<2 sec) on the shortlist have a conclusive Intento LQA result.
For the Advanced requirements, the gender-neutrality agent raises the cost of the three-step solution on Google Gemini 3.1 Pro Preview from $560 to $964 per million characters. The time rises from 281 to 301 seconds per 1,000 words. Weighted errors fall from 16.5 to 14.2 per 1,000 words. With the agent the solution has no major error, against two without it. On OpenAI GPT-5.5, the solution with the agent does not meet every Advanced requirement. It has 12.2 total errors per 1,000 words, against 10.6 without it.
OpenAI GPT-5.5
three-step
Google Gemini 3.1 Pro Preview
three-step
Google Gemini 3.1 Pro Preview
three-step
Google Gemini 3.1 Pro Preview
three-step
Google Gemini 3.1 Pro Preview
three-step
Google Gemini 3.1 Pro Preview
three-step
Google Gemini 3.1 Pro Preview
three-step with the gender-neutrality agent
Google Gemini 3.1 Pro Preview
three-step with the gender-neutrality agent
Google Gemini 3.1 Pro Preview
three-step with the gender-neutrality agent
Optimize for selects one of the solutions that meet the level: the one with the fewest total errors, the lowest cost or the shortest time. Where no solution meets the level, the card shows the available solution with the fewest total errors and what it misses. Total errors count every requirement with countable errors.
| Quality level | Tier | Solutions | Simplest solution | Model | Total errors per 1,000 words | Cost per million characters | Seconds per 1,000 words |
|---|---|---|---|---|---|---|---|
| Basic | Tier 2: two or more solutions, none simple | ☆☆ | three-step | Google Gemini 3.1 Pro Preview | 16.8 | $560 | 281 |
| Standard | Tier 3: one solution | ☆ | three-step | Google Gemini 3.1 Pro Preview | 16.8 | $560 | 281 |
| Advanced | Tier 3: one solution | ☆ | with the gender-neutrality agent | Google Gemini 3.1 Pro Preview | 14.2 | $964 | 301 |
One solution per level: the simplest that meets the level, the cheapest of them where several are equally simple. ★ a one-step solution, ☆ a solution with more steps or the gender-neutrality agent.
Every level takes three steps (translation + LQA + repair) in Frisian. Here the Standard requirements add tone of voice, tag handling and formatting to general translation quality. Of the two solutions that meet the Basic requirements, only the one on Google Gemini 3.1 Pro Preview meets these too, with 100% of the sentences in the tone of voice and 100% of the tags in place. The Advanced requirements add gender neutrality. With the gender-neutrality agent, its translations have 100% of the sentences about people gender-neutral.
| Model | US dollars per million characters | Weighted errors per 1,000 words | Group |
|---|---|---|---|
| OpenAI GPT-5.6 Luna | 0.44 | 35.9 | Misses a Standard requirement |
| DeepSeek V3.2 | 0.93 | 80.8 | Misses a Standard requirement |
| xAI Grok 4.20 (non-reasoning) | 1.14 | 90.3 | Misses a Standard requirement |
| MoonshotAI Kimi K2.5 | 1.25 | 141.7 | Misses a Standard requirement |
| z.ai GLM-5 | 1.96 | 80.8 | Misses a Standard requirement |
| Google Gemini 3.5 Flash | 3.59 | 48.1 | Misses a Standard requirement |
| Google Gemini 3.1 Pro Preview | 4.81 | 29 | Misses a Standard requirement |
| OpenAI GPT-5.5 | 10.96 | 28.3 | Misses a Standard requirement |
| Anthropic Claude Opus 4.8 | 14.73 | 61.3 | Misses a Standard requirement |
The cheapest one-step solution, on OpenAI GPT-5.6 Luna, costs $0.44 per million characters and has 35.9 weighted errors per 1,000 words. Only two have fewer: the solutions on Google Gemini 3.1 Pro Preview, with 29.0 weighted errors per 1,000 words at $4.81 per million characters, and on OpenAI GPT-5.5, with 28.3 at $10.96. The dearest with a conclusive Intento LQA result, on Anthropic Claude Opus 4.8, costs $14.73 per million characters and has 61.3 weighted errors per 1,000 words.
| Requirement | Median errors per 1,000 words |
|---|---|
| General translation quality | 61.3 |
| Tone of voice | 0.3 |
| Gender neutrality | 0.3 |
| Tag handling | 0 |
The requirements with countable errors. Every solution with its result on each requirement: the explorer.
Every one-step solution with a conclusive Intento LQA result is over the bar for weighted errors. Only the ones on OpenAI GPT-5.5 and Google Gemini 3.1 Pro Preview meet the bars for critical and major errors, with 28.3 and 29.0 weighted errors per 1,000 words. With three steps (translation + LQA + repair), they come to 9.6 and 16.5 weighted errors per 1,000 words, under the bar. The three-step solutions on Anthropic Claude Opus 4.8, DeepSeek V3.2 and z.ai GLM-5 have 25.0 to 38.9 weighted errors per 1,000 words.
| Model | Weighted errors per 1,000 words of the shortlisted one-step solutions in Frisian | Class |
|---|---|---|
| MoonshotAI Kimi K2.5 | 141.7 | LLMs |
| xAI Grok 4.20 (non-reasoning) | 90.3 | LLMs |
| z.ai GLM-5 | 80.8 | LLMs |
| DeepSeek V3.2 | 80.8 | LLMs |
| Anthropic Claude Opus 4.8 | 61.3 | LLMs |
| Google Gemini 3.5 Flash | 48.1 | LLMs |
| OpenAI GPT-5.6 Luna | 35.9 | LLMs |
| Google Gemini 3.1 Pro Preview | 29 | LLMs |
| OpenAI GPT-5.5 | 28.3 | LLMs |
The nine one-step solutions with a conclusive Intento LQA result in Frisian are all on LLMs. Four of these LLMs meet our language-support criteria: OpenAI GPT-5.5, Google Gemini 3.1 Pro Preview, OpenAI GPT-5.6 Luna and Google Gemini 3.5 Flash. Their translations have 28.3 to 48.1 weighted errors per 1,000 words. Translations from Anthropic Claude Opus 4.8 have 61.3 weighted errors per 1,000 words, just over the limit of 60. The solutions on the MT engine, the LLM-based MT engine and the open-weight LLM have no conclusive result.
| Model | COMET-Kiwi | Rank on COMET-Kiwi | MetricX | Rank on MetricX | Shortlisted | Shortlist place |
|---|---|---|---|---|---|---|
| Anthropic Claude Fable 5 | 0.681 | 10 | 2.28 | 11 | No | Not shortlisted |
| OpenAI GPT-4.1 | 0.679 | 11 | 2.22 | 9 | No | Not shortlisted |
| Anthropic Claude Sonnet 5 | 0.673 | 14 | 2.39 | 13 | No | Not shortlisted |
| Anthropic Claude Haiku 4.5 | 0.671 | 15 | 2.79 | 18 | No | Not shortlisted |
| Anthropic Claude Sonnet 4.6 | 0.668 | 16 | 2.40 | 14 | No | Not shortlisted |
| Mistral Large 3 | 0.661 | 17 | 2.59 | 16 | No | Not shortlisted |
| Google Gemma 4 | 0.653 | 19 | 3.67 | 23 | No | Not shortlisted |
| Swiss AI Apertus-8B-2509 | 0.642 | 21 | 3.35 | 20 | No | Not shortlisted |
| Mistral Medium 3.1 | 0.642 | 22 | 3.97 | 24 | No | Not shortlisted |
| MiniMax M2.5 | 0.640 | 23 | 3.46 | 21 | No | Not shortlisted |
| AI21 Jamba Large 1.7 | 0.638 | 24 | 3.66 | 22 | No | Not shortlisted |
| Alibaba Machine Translation API | 0.376 | 25 | 10.69 | 25 | No | Not shortlisted |
| Cohere Command A Translate | 0.726 | 1 | 2.54 | 15 | Yes | Top 8 by COMET-Kiwi |
| OpenAI GPT-5.6 Luna | 0.702 | 2 | 1.91 | 1 | Yes | Top 8 by both metrics |
| OpenAI GPT-5.5 | 0.696 | 3 | 1.94 | 2 | Yes | Top 8 by both metrics |
| Google Gemini 3.5 Flash | 0.694 | 4 | 2.04 | 4 | Yes | Top 8 by both metrics |
| Google Translation LLM | 0.688 | 5 | 2.32 | 12 | Yes | Top 8 by COMET-Kiwi |
| z.ai GLM-5 | 0.686 | 6 | 2.11 | 6 | Yes | Top 8 by both metrics |
| xAI Grok 4.20 (non-reasoning) | 0.685 | 7 | 2.09 | 5 | Yes | Top 8 by both metrics |
| MoonshotAI Kimi K2.5 | 0.684 | 8 | 2.27 | 10 | Yes | Top 8 by COMET-Kiwi |
| Anthropic Claude Opus 4.8 | 0.683 | 9 | 2.16 | 7 | Yes | Top 8 by MetricX |
| Google Gemini 3.1 Pro Preview | 0.678 | 12 | 1.96 | 3 | Yes | Top 8 by MetricX |
| DeepSeek V3.2 | 0.675 | 13 | 2.17 | 8 | Yes | Top 8 by MetricX |
| Google Cloud Advanced Translation API | 0.657 | 18 | 2.91 | 19 | Yes | Real-time place |
| OpenAI gpt-oss-120b | 0.648 | 20 | 2.66 | 17 | Yes | Real-time place |
| Model | Solution complexity | COMET-Kiwi rank | COMET-Kiwi | MetricX rank | MetricX | Shortlist |
|---|---|---|---|---|---|---|
| Cohere Command A Translate | Off-the-shelf LLM | 1 | 0.726 | 15 | 2.54 | Top 8 by COMET-Kiwi |
| OpenAI GPT-5.6 Luna | Adjusted LLM | 2 | 0.702 | 1 | 1.91 | Top 8 by both metrics |
| OpenAI GPT-5.5 | Adjusted LLM | 3 | 0.696 | 2 | 1.94 | Top 8 by both metrics |
| Google Gemini 3.5 Flash | Adjusted LLM | 4 | 0.694 | 4 | 2.04 | Top 8 by both metrics |
| Google Translation LLM | Adjusted NMT | 5 | 0.688 | 12 | 2.32 | Top 8 by COMET-Kiwi |
| z.ai GLM-5 | Adjusted LLM | 6 | 0.686 | 6 | 2.11 | Top 8 by both metrics |
| xAI Grok 4.20 (non-reasoning) | Adjusted LLM | 7 | 0.685 | 5 | 2.09 | Top 8 by both metrics |
| MoonshotAI Kimi K2.5 | Adjusted LLM | 8 | 0.684 | 10 | 2.27 | Top 8 by COMET-Kiwi |
| Anthropic Claude Opus 4.8 | Adjusted LLM | 9 | 0.683 | 7 | 2.16 | Top 8 by MetricX |
| Anthropic Claude Fable 5 | Adjusted LLM | 10 | 0.681 | 11 | 2.28 | Not shortlisted |
| OpenAI GPT-4.1 | Adjusted LLM | 11 | 0.679 | 9 | 2.22 | Not shortlisted |
| Google Gemini 3.1 Pro Preview | Adjusted LLM | 12 | 0.678 | 3 | 1.96 | Top 8 by MetricX |
| DeepSeek V3.2 | Adjusted LLM | 13 | 0.675 | 8 | 2.17 | Top 8 by MetricX |
| Anthropic Claude Sonnet 5 | Adjusted LLM | 14 | 0.673 | 13 | 2.39 | Not shortlisted |
| Anthropic Claude Haiku 4.5 | Adjusted LLM | 15 | 0.671 | 18 | 2.79 | Not shortlisted |
| Anthropic Claude Sonnet 4.6 | Adjusted LLM | 16 | 0.668 | 14 | 2.40 | Not shortlisted |
| Mistral Large 3 | Adjusted LLM | 17 | 0.661 | 16 | 2.59 | Not shortlisted |
| Google Cloud Advanced Translation API | Off-the-shelf NMT | 18 | 0.657 | 19 | 2.91 | Real-time place |
| Google Gemma 4 | Adjusted LLM | 19 | 0.653 | 23 | 3.67 | Not shortlisted |
| OpenAI gpt-oss-120b | Adjusted LLM | 20 | 0.648 | 17 | 2.66 | Real-time place |
| Swiss AI Apertus-8B-2509 | Off-the-shelf LLM | 21 | 0.642 | 20 | 3.35 | Not shortlisted |
| Mistral Medium 3.1 | Adjusted LLM | 22 | 0.642 | 24 | 3.97 | Not shortlisted |
| MiniMax M2.5 | Adjusted LLM | 23 | 0.640 | 21 | 3.46 | Not shortlisted |
| AI21 Jamba Large 1.7 | Adjusted LLM | 24 | 0.638 | 22 | 3.66 | Not shortlisted |
| Alibaba Machine Translation API | Off-the-shelf NMT | 25 | 0.376 | 25 | 10.69 | Not shortlisted |
| OpenAI GPT-5.6 Sol | Adjusted LLM | — | 0.687 | — | 1.95 | Same model family as OpenAI GPT-5.6 Luna |
| OpenAI GPT-5.6 Terra | Adjusted LLM | — | 0.701 | — | 1.82 | Same model family as OpenAI GPT-5.6 Luna |
| xAI Grok 4.20 (reasoning) | Adjusted LLM | — | 0.681 | — | 2.31 | Same model family as xAI Grok 4.20 (non-reasoning) |
Every model we got a translation from, as the shortlist compared it: off the shelf, without context in the request, or adjusted, with it. A model that did not make the shortlist is shown adjusted, unless off the shelf it scores higher by 0.03 or more on COMET-Kiwi. Ranks among 25 models, with each model family counted once, and the top 8 of each metric in bold. Higher is better on COMET-Kiwi, lower on MetricX. The scores rank models within this language and are not a requirement result.
Google Gemini 3.1 Pro Preview, the model of the only solution that meets every Standard requirement, is on the shortlist through MetricX, 3rd there and 12th on COMET-Kiwi. Cohere Command A Translate, first on COMET-Kiwi and 15th on MetricX, has no conclusive result. Translations from MoonshotAI Kimi K2.5, 8th on COMET-Kiwi, have 141.7 weighted errors per 1,000 words, the most of the solutions with a conclusive result. Nearest the cut in the tail are OpenAI GPT-4.1, 9th on MetricX, and Anthropic Claude Fable 5, 10th on COMET-Kiwi.
| Quality level | Cheapest solution that meets every requirement of the level | Cost per million characters |
|---|---|---|
| Basic | Google Gemini 3.1 Pro Preview, three-step | $560 |
| Standard | Google Gemini 3.1 Pro Preview, three-step | $560 |
| Advanced | Google Gemini 3.1 Pro Preview, three-step with the gender-neutrality agent | $964 |
| Model | US dollars per million characters | Class |
|---|---|---|
| OpenAI GPT-5.6 Luna | $0.44 | LLMs |
| DeepSeek V3.2 | $0.93 | LLMs |
| xAI Grok 4.20 (non-reasoning) | $1.14 | LLMs |
| MoonshotAI Kimi K2.5 | $1.25 | LLMs |
| z.ai GLM-5 | $1.96 | LLMs |
| Google Gemini 3.5 Flash | $3.59 | LLMs |
| Google Gemini 3.1 Pro Preview | $4.81 | LLMs |
| OpenAI GPT-5.5 | $10.96 | LLMs |
| Anthropic Claude Opus 4.8 | $14.73 | LLMs |
| Google Cloud Advanced Translation API | $20 | MT engines |
| Google Translation LLM | $20.9 | LLM-based MT engines |
| Model | Tokens per character | Class |
|---|---|---|
| Anthropic Claude Fable 5 | 0.486 | LLMs |
| Anthropic Claude Opus 4.8 | 0.485 | LLMs |
| Anthropic Claude Sonnet 5 | 0.483 | LLMs |
| Mistral Medium 3.1 | 0.394 | LLMs |
| Mistral Large 3 | 0.39 | LLMs |
| Anthropic Claude Sonnet 4.6 | 0.383 | LLMs |
| AI21 Jamba Large 1.7 | 0.38 | LLMs |
| Anthropic Claude Haiku 4.5 | 0.378 | LLMs |
| MoonshotAI Kimi K2.5 | 0.349 | LLMs |
| DeepSeek V3.2 | 0.347 | LLMs |
| z.ai GLM-5 | 0.344 | LLMs |
| Google Gemini 3.1 Pro Preview | 0.333 | LLMs |
| Google Gemini 3.5 Flash | 0.33 | LLMs |
| Google Gemma 4 | 0.326 | Open-weight LLMs |
| xAI Grok 4.20 (reasoning) | 0.314 | LLMs |
| xAI Grok 4.20 (non-reasoning) | 0.313 | LLMs |
| Cohere Command A Translate | 0.311 | LLMs |
| OpenAI GPT-5.5 | 0.305 | LLMs |
| OpenAI GPT-5.6 Sol | 0.304 | LLMs |
| OpenAI GPT-5.6 Terra | 0.304 | LLMs |
| OpenAI GPT-4.1 | 0.303 | LLMs |
| OpenAI GPT-5.6 Luna | 0.303 | LLMs |
| MiniMax M2.5 | 0.302 | LLMs |
| Quality level | Fastest solution that meets every requirement of the level | Seconds per 1,000 words |
|---|---|---|
| Basic | Google Gemini 3.1 Pro Preview, three-step | 281 |
| Standard | Google Gemini 3.1 Pro Preview, three-step | 281 |
| Advanced | Google Gemini 3.1 Pro Preview, three-step with the gender-neutrality agent | 301 |
| Model | Seconds per 1,000 words | Class |
|---|---|---|
| Google Cloud Advanced Translation API | 0.2 | MT engines |
| Google Translation LLM | 1.4 | LLM-based MT engines |
| OpenAI gpt-oss-120b | 2.3 | Open-weight LLMs |
| OpenAI GPT-5.6 Luna | 17 | LLMs |
| Google Gemini 3.5 Flash | 18 | LLMs |
| xAI Grok 4.20 (non-reasoning) | 21 | LLMs |
| Alibaba Machine Translation API | 26 | MT engines |
| OpenAI GPT-5.6 Terra | 41 | LLMs |
| xAI Grok 4.20 (reasoning) | 43 | LLMs |
| Anthropic Claude Haiku 4.5 | 45 | LLMs |
| Swiss AI Apertus-8B-2509 | 46 | Open-weight LLMs |
| Cohere Command A Translate | 48 | LLMs |
| Mistral Medium 3.1 | 51 | LLMs |
| AI21 Jamba Large 1.7 | 52 | LLMs |
| MiniMax M2.5 | 53 | LLMs |
| OpenAI GPT-4.1 | 56 | LLMs |
| Mistral Large 3 | 68 | LLMs |
| Anthropic Claude Sonnet 5 | 74 | LLMs |
| Anthropic Claude Fable 5 | 81 | LLMs |
| Anthropic Claude Sonnet 4.6 | 84 | LLMs |
| Google Gemini 3.1 Pro Preview | 88 | LLMs |
| DeepSeek V3.2 | 88 | LLMs |
| Anthropic Claude Opus 4.8 | 100 | LLMs |
| OpenAI GPT-5.6 Sol | 100 | LLMs |
| OpenAI GPT-5.5 | 128 | LLMs |
| Google Gemma 4 | 149 | Open-weight LLMs |
| z.ai GLM-5 | 245 | LLMs |
| MoonshotAI Kimi K2.5 | 248 | LLMs |
Real-time models are those the shortlist counts as real-time, under 2 seconds per text across the languages we timed. The chart shows the time of each model into Frisian.
The three real-time models in Frisian are an MT engine, Google Cloud Advanced Translation API, an LLM-based MT engine, Google Translation LLM, and an open-weight LLM, OpenAI gpt-oss-120b. Their translations have 94.3%, 95.1% and 99.6% of the tags in place. Next come translations from OpenAI GPT-5.6 Luna, at 17 seconds per 1,000 words. The fastest solution that meets the Basic or Standard requirements takes 281 seconds per 1,000 words, and the one that meets the Advanced requirements 301, with the gender-neutrality agent.
| The two best solutions | Solution | Sentences about people that are gender-neutral | Meets every Advanced requirement | Cost per million characters | Seconds per 1,000 words |
|---|---|---|---|---|---|
| Google Gemini 3.1 Pro Preview | three-step, with the gender-neutrality agent | 100% | ✓ | $964 | 301 |
| OpenAI GPT-5.5 | three-step, with the gender-neutrality agent | 99.1% | ✗ | $978 | 313 |
The tone of voice we check in Frisian is the formal register, with "jo" as the form of address. For gender neutrality we check neutral nouns and rephrasing, with no default gender. The other options in the language are generic "hy" and "hy of sy". With the gender-neutrality agent, translations from the three-step solutions (translation + LQA + repair) on Google Gemini 3.1 Pro Preview and OpenAI GPT-5.5 have 100% and 97.9% of the sentences in the tone of voice.
We shortlist the models on COMET-Kiwi and MetricX, the quality-estimation pair that agrees in Frisian, and the result is not borderline. The three LQA judges agree on a median of 92.1% of the sentences across the 13 shortlisted solutions. So the ranking of the shortlist and the Intento LQA results on this page carry weight, even with the similarity pair short of its threshold.
Leave feedback