The State of Translation Automation 2026 · Languages
What it takes to automate translation from English into Kurdish (Kurmanji) against 10 requirements in three levels: the solutions that meet them, what they cost and how fast they are, from the 2026 benchmark of 57 AI models and MT engines.
A solution is a model plus its harness: prompts, glossaries and other context, pre- and post-processing. It meets a quality level when it meets every requirement of the level.
| Requirement | Description | Met when |
|---|---|---|
| Basic requirements for any company | ||
| General translation quality | Translation errors of every type, such as mistranslations, omissions and grammar errors, classified by type and severity following MQM, the industry standard. | No critical error, at most 3 major errors in the four test texts (3,034 words) and at most 20 errors per 1,000 words, weighted by severity (minor 1, major 5, critical 10). |
| Standard requirements adds the rules companies set for their content | ||
| Terminology | The approved translation of every glossary term, in any of its correct forms. | 100% of the glossary terms. We test it in the 16 languages with a glossary, so it does not apply in Kurdish (Kurmanji). |
| Tone of voice | The formal register (hûn) in every sentence in the second person or the imperative. | 100% of those sentences. |
| Tags | Every HTML tag of the source, with its attributes, in the translation. | 100% of the tags. |
| Unicode character integrity | Every character intact, with no corrupted characters and no letter from another script that only looks like the right one. | 100% of the characters. |
| Locale conventions | The punctuation marks of the target language in place of the English ones. | No pass-or-fail bar. We report it as a share, because whether to localize punctuation is a matter of house style. |
| HTML entities | No HTML character code, such as the code for a quotation mark, that the source does not carry. | No added HTML entity. |
| Valid JSON | The translated JSON document still parses. | The document parses. |
| Spacing at tag boundaries | The spaces next to HTML tags stay as in the source. | 100% of the tag boundaries. |
| Advanced requirements adds a requirement for some companies and use cases | ||
| Gender neutrality | A gender-neutral translation wherever the English source leaves the gender of a person open, as in “your manager” or “the expert”. | 100% of the sentences that refer to a person. |
| Solution | Sentences on which the three judges agree (%) | Status |
|---|---|---|
| Google Cloud Advanced Translation API | 100% | Passes (90% or more) |
| OpenAI gpt-oss-120b | 94.4% | Passes (90% or more) |
| Anthropic Claude Opus 4.8 | 92.1% | Passes (90% or more) |
| xAI Grok 4.20 (non-reasoning) | 91% | Passes (90% or more) |
| Google Gemini 3.5 Flash | 89.8% | Does not pass |
| Mistral Large 3 | 89.8% | Does not pass |
| OpenAI GPT-5.6 Sol | 89.3% | Does not pass |
| Anthropic Claude Fable 5 | 89.3% | Does not pass |
| Google Gemma 4 | 89.3% | Does not pass |
| Google Gemini 3.1 Pro Preview | 88.1% | Does not pass |
| OpenAI GPT-5.5 | 87.6% | Does not pass |
| Anthropic Claude Sonnet 4.6 | 86.4% | Does not pass |
The similarity pair agrees at 0.586, against a minimum of 0.461, and the quality-estimation pair at 0.655, against 0.534. So the automatic scores can rank models within Kurdish (Kurmanji). The four solutions that pass the check are on Google Cloud Advanced Translation API, OpenAI gpt-oss-120b, Anthropic Claude Opus 4.8 and xAI Grok 4.20 (non-reasoning). None of them meets tone of voice, tag handling and formatting together.
Pick a quality level, or your own requirements, bars and constraints, to see the shortlisted solutions that meet them in Kurdish (Kurmanji), with a column for each requirement you select. Under Optimize for, choose which of them comes first: the one with the lowest cost or the shortest time. Click a column heading to sort by it.
The shortlisted solutions on 12 models, with their results on tone of voice, tags and formatting, which do not depend on the agreement of the three LQA judges, and on general translation quality only where the judges agree. We cannot assess translation quality in Kurdish (Kurmanji): why. The models that did not make the shortlist have automatic scores only: the shortlist. Inconclusive: the three LQA judges agree on fewer than 90% of the sentences of the solution, so it has no general translation quality result (09-02). Gender neutrality: the Advanced requirements count the two best solutions of the language, with the gender-neutrality agent where they need it. The results of the other solutions are shown for information.
With tone of voice, tag handling and formatting selected, four shortlisted solutions meet them. Optimize for Cost puts the one on Google Gemini 3.5 Flash first, at $4.02 per million characters and 80 seconds per 1,000 words. Optimize for Latency puts the one on Google Gemini 3.1 Pro Preview first, at 79 seconds per 1,000 words and $5.37 per million characters. The solutions on OpenAI GPT-5.6 Sol and Anthropic Claude Fable 5 cost $9.04 and $31.78 per million characters and take 113 and 92 seconds per 1,000 words.
Of the 11 solutions with a cost, the three cheapest, on Google Gemma 4, Mistral Large 3 and xAI Grok 4.20 (non-reasoning), cost $0.16 to $1.28 per million characters. Of all 12, the three fastest, on Google Cloud Advanced Translation API, OpenAI gpt-oss-120b and xAI Grok 4.20 (non-reasoning), take 0.1 to 24 seconds per 1,000 words. None of these five meet tone of voice, tag handling and formatting together. The next cheapest and next fastest, on Google Gemini 3.5 Flash and Google Gemini 3.1 Pro Preview, meet all three.
Four more shortlisted solutions meet two of tone of voice, tag handling and formatting. The solutions on Anthropic Claude Opus 4.8, Anthropic Claude Sonnet 4.6 and xAI Grok 4.20 (non-reasoning) miss tag handling alone: their translations have 99.2%, 99.2% and 98% of the tags. The solution on Google Gemma 4, at $0.16 per million characters, misses formatting alone. The solution on OpenAI GPT-5.5 meets tone of voice alone, and translations from it have 99.8% of the tags.
We cannot assess general translation quality in Kurdish (Kurmanji): how confident we are.
We cannot assess general translation quality in Kurdish (Kurmanji): how confident we are.
We cannot assess general translation quality in Kurdish (Kurmanji): how confident we are.
| Model | COMET-Kiwi scores of the shortlisted solutions in Kurdish (Kurmanji) (higher is better) | Class |
|---|---|---|
| Google Gemini 3.1 Pro Preview | 0.691 | LLMs |
| OpenAI GPT-5.6 Sol | 0.69 | LLMs |
| Anthropic Claude Fable 5 | 0.682 | LLMs |
| Google Gemini 3.5 Flash | 0.675 | LLMs |
| OpenAI GPT-5.5 | 0.672 | LLMs |
| Anthropic Claude Sonnet 4.6 | 0.67 | LLMs |
| Google Gemma 4 | 0.67 | Open-weight LLMs |
| xAI Grok 4.20 (non-reasoning) | 0.669 | LLMs |
| Anthropic Claude Opus 4.8 | 0.665 | LLMs |
| Mistral Large 3 | 0.664 | LLMs |
| Google Cloud Advanced Translation API | 0.65 | MT engines |
| OpenAI gpt-oss-120b | 0.532 | Open-weight LLMs |
No translation into Kurdish (Kurmanji): AI21 Jamba Large 1.7 (garbage or near-empty, wrong language), Swiss AI Apertus-8B-2509 (garbage or near-empty), Microsoft Translator (wrong language), Youdao Translation Service API (wrong language).
The automatic scores rank models within Kurdish (Kurmanji) and are not a requirement result. The 14 models ranked highest on COMET-Kiwi are all adjusted LLMs, among them the open-weight Google Gemma 4 in 7th place. The highest-ranked MT engine, Google Cloud Advanced Translation API, is 15th on both metrics. The two other ranked MT engines, NiuTrans and Alibaba Machine Translation API, are 21st and 23rd of 23 on COMET-Kiwi. The open-weight OpenAI gpt-oss-120b, used off the shelf, is 19th on COMET-Kiwi and 20th on MetricX.
| Model | COMET-Kiwi | Rank on COMET-Kiwi | MetricX | Rank on MetricX | Shortlisted | Shortlist place |
|---|---|---|---|---|---|---|
| Anthropic Claude Sonnet 5 | 0.669 | 9 | 2.82 | 9 | No | Not shortlisted |
| z.ai GLM-5 | 0.668 | 10 | 3.29 | 14 | No | Not shortlisted |
| DeepSeek V3.2 | 0.666 | 11 | 2.99 | 11 | No | Not shortlisted |
| MoonshotAI Kimi K2.5 | 0.658 | 14 | 3.16 | 12 | No | Not shortlisted |
| Mistral Medium 3.1 | 0.623 | 16 | 3.83 | 16 | No | Not shortlisted |
| OpenAI GPT-4.1 | 0.605 | 17 | 4.09 | 17 | No | Not shortlisted |
| Anthropic Claude Haiku 4.5 | 0.603 | 18 | 4.69 | 18 | No | Not shortlisted |
| MiniMax M2.5 | 0.529 | 20 | 6.73 | 21 | No | Not shortlisted |
| NiuTrans | 0.514 | 21 | 6.42 | 19 | No | Not shortlisted |
| Cohere Command A Translate | 0.479 | 22 | 7.43 | 22 | No | Not shortlisted |
| Alibaba Machine Translation API | 0.163 | 23 | 18.60 | 23 | No | Not shortlisted |
| Google Gemini 3.1 Pro Preview | 0.691 | 1 | 2.46 | 1 | Yes | Top 8 by both metrics |
| OpenAI GPT-5.6 Sol | 0.690 | 2 | 2.52 | 3 | Yes | Top 8 by both metrics |
| Anthropic Claude Fable 5 | 0.682 | 3 | 2.60 | 5 | Yes | Top 8 by both metrics |
| Google Gemini 3.5 Flash | 0.675 | 4 | 2.48 | 2 | Yes | Top 8 by both metrics |
| OpenAI GPT-5.5 | 0.672 | 5 | 2.64 | 6 | Yes | Top 8 by both metrics |
| Anthropic Claude Sonnet 4.6 | 0.670 | 6 | 2.77 | 8 | Yes | Top 8 by both metrics |
| Google Gemma 4 | 0.670 | 7 | 3.28 | 13 | Yes | Top 8 by COMET-Kiwi |
| xAI Grok 4.20 (non-reasoning) | 0.669 | 8 | 2.84 | 10 | Yes | Top 8 by COMET-Kiwi |
| Anthropic Claude Opus 4.8 | 0.665 | 12 | 2.72 | 7 | Yes | Top 8 by MetricX |
| Mistral Large 3 | 0.664 | 13 | 2.53 | 4 | Yes | Top 8 by MetricX |
| Google Cloud Advanced Translation API | 0.650 | 15 | 3.56 | 15 | Yes | Real-time place |
| OpenAI gpt-oss-120b | 0.532 | 19 | 6.53 | 20 | Yes | Real-time place |
| Model | Solution complexity | COMET-Kiwi rank | COMET-Kiwi | MetricX rank | MetricX | Shortlist |
|---|---|---|---|---|---|---|
| Google Gemini 3.1 Pro Preview | Adjusted LLM | 1 | 0.691 | 1 | 2.46 | Top 8 by both metrics |
| OpenAI GPT-5.6 Sol | Adjusted LLM | 2 | 0.690 | 3 | 2.52 | Top 8 by both metrics |
| Anthropic Claude Fable 5 | Adjusted LLM | 3 | 0.682 | 5 | 2.60 | Top 8 by both metrics |
| Google Gemini 3.5 Flash | Adjusted LLM | 4 | 0.675 | 2 | 2.48 | Top 8 by both metrics |
| OpenAI GPT-5.5 | Adjusted LLM | 5 | 0.672 | 6 | 2.64 | Top 8 by both metrics |
| Anthropic Claude Sonnet 4.6 | Adjusted LLM | 6 | 0.670 | 8 | 2.77 | Top 8 by both metrics |
| Google Gemma 4 | Adjusted LLM | 7 | 0.670 | 13 | 3.28 | Top 8 by COMET-Kiwi |
| xAI Grok 4.20 (non-reasoning) | Adjusted LLM | 8 | 0.669 | 10 | 2.84 | Top 8 by COMET-Kiwi |
| Anthropic Claude Sonnet 5 | Adjusted LLM | 9 | 0.669 | 9 | 2.82 | Not shortlisted |
| z.ai GLM-5 | Adjusted LLM | 10 | 0.668 | 14 | 3.29 | Not shortlisted |
| DeepSeek V3.2 | Adjusted LLM | 11 | 0.666 | 11 | 2.99 | Not shortlisted |
| Anthropic Claude Opus 4.8 | Adjusted LLM | 12 | 0.665 | 7 | 2.72 | Top 8 by MetricX |
| Mistral Large 3 | Adjusted LLM | 13 | 0.664 | 4 | 2.53 | Top 8 by MetricX |
| MoonshotAI Kimi K2.5 | Adjusted LLM | 14 | 0.658 | 12 | 3.16 | Not shortlisted |
| Google Cloud Advanced Translation API | Off-the-shelf NMT | 15 | 0.650 | 15 | 3.56 | Real-time place |
| Mistral Medium 3.1 | Adjusted LLM | 16 | 0.623 | 16 | 3.83 | Not shortlisted |
| OpenAI GPT-4.1 | Adjusted LLM | 17 | 0.605 | 17 | 4.09 | Not shortlisted |
| Anthropic Claude Haiku 4.5 | Adjusted LLM | 18 | 0.603 | 18 | 4.69 | Not shortlisted |
| OpenAI gpt-oss-120b | Off-the-shelf LLM | 19 | 0.532 | 20 | 6.53 | Real-time place |
| MiniMax M2.5 | Adjusted LLM | 20 | 0.529 | 21 | 6.73 | Not shortlisted |
| NiuTrans | Off-the-shelf NMT | 21 | 0.514 | 19 | 6.42 | Not shortlisted |
| Cohere Command A Translate | Adjusted LLM | 22 | 0.479 | 22 | 7.43 | Not shortlisted |
| Alibaba Machine Translation API | Off-the-shelf NMT | 23 | 0.163 | 23 | 18.60 | Not shortlisted |
| OpenAI GPT-5.6 Luna | Adjusted LLM | — | 0.684 | — | 2.56 | Same model family as OpenAI GPT-5.6 Sol |
| OpenAI GPT-5.6 Terra | Adjusted LLM | — | 0.674 | — | 2.91 | Same model family as OpenAI GPT-5.6 Sol |
| xAI Grok 4.20 (reasoning) | Adjusted LLM | — | 0.667 | — | 2.86 | Same model family as xAI Grok 4.20 (non-reasoning) |
Every model we got a translation from, as the shortlist compared it: off the shelf, without context in the request, or adjusted, with it. A model that did not make the shortlist is shown adjusted, unless off the shelf it scores higher by 0.03 or more on COMET-Kiwi. Ranks among 23 models, with each model family counted once, and the top 8 of each metric in bold. Higher is better on COMET-Kiwi, lower on MetricX. The scores rank models within this language and are not a requirement result.
COMET-Kiwi alone put Google Gemma 4 and xAI Grok 4.20 (non-reasoning) on the list, 7th and 8th, while MetricX ranks them 13th and 10th. MetricX alone put Mistral Large 3 and Anthropic Claude Opus 4.8 on it, 4th and 7th, while COMET-Kiwi ranks them 13th and 12th. Nearest the cut is Anthropic Claude Sonnet 5, 9th on both metrics, at 0.669 on COMET-Kiwi like xAI Grok 4.20 (non-reasoning) in 8th place.
| Model | US dollars per million characters | Class |
|---|---|---|
| Google Gemma 4 | $0.16 | Open-weight LLMs |
| OpenAI GPT-5.6 Luna | $0.51 | LLMs |
| MiniMax M2.5 | $0.59 | LLMs |
| Mistral Large 3 | $0.86 | LLMs |
| DeepSeek V3.2 | $1.04 | LLMs |
| Mistral Medium 3.1 | $1.04 | LLMs |
| xAI Grok 4.20 (reasoning) | $1.25 | LLMs |
| xAI Grok 4.20 (non-reasoning) | $1.26 | LLMs |
| MoonshotAI Kimi K2.5 | $1.35 | LLMs |
| z.ai GLM-5 | $2.14 | LLMs |
| Anthropic Claude Haiku 4.5 | $2.48 | LLMs |
| OpenAI GPT-4.1 | $3.39 | LLMs |
| Google Gemini 3.5 Flash | $3.93 | LLMs |
| OpenAI GPT-5.6 Terra | $5.12 | LLMs |
| Google Gemini 3.1 Pro Preview | $5.24 | LLMs |
| Anthropic Claude Sonnet 5 | $6.26 | LLMs |
| Alibaba Machine Translation API | $7 | MT engines |
| Anthropic Claude Sonnet 4.6 | $7.5 | LLMs |
| OpenAI GPT-5.6 Sol | $8.84 | LLMs |
| OpenAI GPT-5.5 | $12.81 | LLMs |
| Anthropic Claude Opus 4.8 | $15.59 | LLMs |
| Google Cloud Advanced Translation API | $20 | MT engines |
| Anthropic Claude Fable 5 | $31.31 | LLMs |
| Model | Tokens per character | Class |
|---|---|---|
| Anthropic Claude Sonnet 5 | 0.548 | LLMs |
| Anthropic Claude Opus 4.8 | 0.545 | LLMs |
| Anthropic Claude Fable 5 | 0.544 | LLMs |
| Mistral Medium 3.1 | 0.474 | LLMs |
| Mistral Large 3 | 0.471 | LLMs |
| Anthropic Claude Haiku 4.5 | 0.445 | LLMs |
| Anthropic Claude Sonnet 4.6 | 0.443 | LLMs |
| Cohere Command A Translate | 0.415 | LLMs |
| DeepSeek V3.2 | 0.413 | LLMs |
| z.ai GLM-5 | 0.406 | LLMs |
| MoonshotAI Kimi K2.5 | 0.397 | LLMs |
| Google Gemini 3.1 Pro Preview | 0.383 | LLMs |
| Google Gemini 3.5 Flash | 0.382 | LLMs |
| OpenAI GPT-5.6 Sol | 0.38 | LLMs |
| OpenAI GPT-4.1 | 0.379 | LLMs |
| xAI Grok 4.20 (non-reasoning) | 0.378 | LLMs |
| OpenAI GPT-5.5 | 0.377 | LLMs |
| OpenAI GPT-5.6 Luna | 0.377 | LLMs |
| OpenAI GPT-5.6 Terra | 0.377 | LLMs |
| Google Gemma 4 | 0.376 | Open-weight LLMs |
| xAI Grok 4.20 (reasoning) | 0.375 | LLMs |
| MiniMax M2.5 | 0.351 | LLMs |
Over the 22 models we counted tokens for, a translation into Kurdish (Kurmanji) has a median 0.39 tokens per character, 1.6 times as many as English text. Tokens alone do not set the cost. Translations from Google Gemini 3.5 Flash and OpenAI GPT-5.6 Sol have 0.382 and 0.38 tokens per character. Off the shelf, they cost $3.93 and $8.84 per million characters, and for OpenAI GPT-5.6 Sol that is a lower bound. OpenAI gpt-oss-120b has no price, so it is not on the chart.
| Model | Seconds per 1,000 words | Class |
|---|---|---|
| Google Cloud Advanced Translation API | 0.1 | MT engines |
| OpenAI gpt-oss-120b | 2.8 | Open-weight LLMs |
| xAI Grok 4.20 (non-reasoning) | 24 | LLMs |
| OpenAI GPT-5.6 Luna | 25 | LLMs |
| Alibaba Machine Translation API | 30 | MT engines |
| OpenAI GPT-4.1 | 35 | LLMs |
| OpenAI GPT-5.6 Terra | 50 | LLMs |
| Mistral Medium 3.1 | 51 | LLMs |
| Anthropic Claude Haiku 4.5 | 52 | LLMs |
| DeepSeek V3.2 | 54 | LLMs |
| xAI Grok 4.20 (reasoning) | 57 | LLMs |
| NiuTrans | 60 | MT engines |
| MiniMax M2.5 | 62 | LLMs |
| Google Gemini 3.1 Pro Preview | 79 | LLMs |
| Google Gemini 3.5 Flash | 80 | LLMs |
| Cohere Command A Translate | 85 | LLMs |
| Anthropic Claude Sonnet 5 | 86 | LLMs |
| Mistral Large 3 | 89 | LLMs |
| Anthropic Claude Fable 5 | 92 | LLMs |
| OpenAI GPT-5.5 | 100 | LLMs |
| Anthropic Claude Sonnet 4.6 | 100 | LLMs |
| OpenAI GPT-5.6 Sol | 113 | LLMs |
| Anthropic Claude Opus 4.8 | 113 | LLMs |
| MoonshotAI Kimi K2.5 | 182 | LLMs |
| Google Gemma 4 | 191 | Open-weight LLMs |
| z.ai GLM-5 | 250 | LLMs |
Real-time models are those the shortlist counts as real-time, under 2 seconds per text across the languages we timed. The chart shows the time of each model into Kurdish (Kurmanji).
Of the 26 models we timed, the two real-time models are the MT engine Google Cloud Advanced Translation API and the open-weight OpenAI gpt-oss-120b, both shortlisted as real-time solutions. OpenAI gpt-oss-120b takes 2.8 seconds per 1,000 words. Then come translations from xAI Grok 4.20 (non-reasoning), at 24 seconds per 1,000 words. Translations from the MT engines Alibaba Machine Translation API and NiuTrans take 30 and 60 seconds per 1,000 words.
For tone of voice in Kurdish (Kurmanji) we check the formal register (hûn), with a bar of 100% of the sentences it applies to. Of the 12 shortlisted solutions, 10 meet it. Translations from Google Cloud Advanced Translation API have 95.74% of those sentences in the formal register, and from OpenAI gpt-oss-120b 21.3%. The 12 shortlisted solutions have 92.6% to 96.4% of the sentences about people gender-neutral.
Leave feedback