The State of Translation Automation 2026 · Languages
What it takes to automate translation from English into Urdu against 10 requirements in three levels: the solutions that meet them, what they cost and how fast they are, from the 2026 benchmark of 57 AI models and MT engines.
A solution is a model plus its harness: prompts, glossaries and other context, pre- and post-processing. It meets a quality level when it meets every requirement of the level.
| Requirement | Description | Met when |
|---|---|---|
| Basic requirements for any company | ||
| General translation quality | Translation errors of every type, such as mistranslations, omissions and grammar errors, classified by type and severity following MQM, the industry standard. | No critical error, at most 3 major errors in the four test texts (3,034 words) and at most 20 errors per 1,000 words, weighted by severity (minor 1, major 5, critical 10). |
| Standard requirements adds the rules companies set for their content | ||
| Terminology | The approved translation of every glossary term, in any of its correct forms. | 100% of the glossary terms. We test it in the 16 languages with a glossary, so it does not apply in Urdu. |
| Tone of voice | The formal register (آپ) in every sentence in the second person or the imperative. | 100% of those sentences. |
| Tags | Every HTML tag of the source, with its attributes, in the translation. | 100% of the tags. |
| Unicode character integrity | Every character intact, with no corrupted characters and no letter from another script that only looks like the right one. | 100% of the characters. |
| Locale conventions | The punctuation marks of the target language in place of the English ones. | No pass-or-fail bar. We report it as a share, because whether to localize punctuation is a matter of house style. |
| HTML entities | No HTML character code, such as the code for a quotation mark, that the source does not carry. | No added HTML entity. |
| Valid JSON | The translated JSON document still parses. | The document parses. |
| Spacing at tag boundaries | The spaces next to HTML tags stay as in the source. | 100% of the tag boundaries. |
| Advanced requirements adds a requirement for some companies and use cases | ||
| Gender neutrality | A gender-neutral translation wherever the English source leaves the gender of a person open, as in “your manager” or “the expert”. | 100% of the sentences that refer to a person. |
Pick a quality level, or your own requirements, bars and constraints, to see the shortlisted solutions that meet them in Urdu, with a column for each requirement you select. Under Optimize for, choose which of them comes first: the one with the fewest total errors, the lowest cost or the shortest time. Click a column heading to sort by it.
The shortlisted solutions on 12 models, with their Intento LQA results. Total errors per 1,000 words add the weighted general translation quality errors (minor 1, major 5, critical 10) to the missed glossary terms, sentences in the wrong register, tag errors and sentences that fail gender neutrality, each counted once. Formatting counts as met or missed and does not add to the total errors. The models that did not make the shortlist have automatic scores only: the shortlist. Gender neutrality: the Advanced requirements count the two best solutions of the language, with the gender-neutrality agent where they need it. The results of the other solutions are shown for information.
Two solutions meet every Standard requirement in Urdu. Neither leads on errors, cost and time at once. The three-step solution (translation + LQA + repair) on Anthropic Claude Fable 5 has 22.6 total errors per 1,000 words, at $603 per million characters and 267 seconds per 1,000 words. The two-step solution (translation + post-edit) on OpenAI GPT-5.5 has 35.8 total errors per 1,000 words, at $11.28 per million characters and 114 seconds per 1,000 words. Optimizing for quality selects the first, for cost or latency the second.
None of the solutions that meet even the Basic requirements in Urdu cost less than $11.27 per million characters or take less than 96 seconds per 1,000 words, so a real-time limit rules them all out. For the Standard requirements, only the two-step solution (translation + post-edit) on OpenAI GPT-5.5 costs less than $603 per million characters, takes less than 267 seconds per 1,000 words or has fewer than three steps.
The gender-neutrality agent works with both solutions that meet every Standard requirement. On Anthropic Claude Fable 5, total errors fall from 22.6 to 7.6 per 1,000 words, the cost rises from $603 to $991 per million characters and the time from 267 to 303 seconds per 1,000 words. On OpenAI GPT-5.5, the version with the agent runs three steps, not two. Total errors fall from 35.8 to 9.9 per 1,000 words, the cost rises from $11.28 to $963 per million characters and the time from 114 to 370 seconds per 1,000 words.
Anthropic Claude Fable 5
one-step LLM adjusted for Standard requirements
OpenAI GPT-5.5
one-step LLM adjusted for Standard requirements
Anthropic Claude Fable 5
one-step LLM adjusted for Standard requirements
Anthropic Claude Fable 5
three-step
OpenAI GPT-5.5
two-step
OpenAI GPT-5.5
two-step
Anthropic Claude Fable 5
three-step with the gender-neutrality agent
Misses gender neutrality.
Optimize for selects one of the solutions that meet the level: the one with the fewest total errors, the lowest cost or the shortest time. Where no solution meets the level, the card shows the available solution with the fewest total errors and what it misses. Total errors count every requirement with countable errors.
| Quality level | Tier | Solutions | Simplest solution | Model | Total errors per 1,000 words | Cost per million characters | Seconds per 1,000 words |
|---|---|---|---|---|---|---|---|
| Basic | Tier 1: simple solution + backup | ★★ | one-step | OpenAI GPT-5.5 | 36.5 | $11.28 | 114 |
| Standard | Tier 2: two or more solutions, none simple | ☆☆ | two-step | OpenAI GPT-5.5 | 35.8 | $11.28 | 114 |
| Advanced | Tier 4: no solution | No solution | — | — | — |
One solution per level: the simplest that meets the level, the cheapest of them where several are equally simple. ★ a one-step solution, ☆ a solution with more steps or the gender-neutrality agent.
The two one-step solutions that meet every Basic requirement in Urdu, on Anthropic Claude Fable 5 and OpenAI GPT-5.5, have 99.2% and 99.6% of the tags right, so the Standard requirements remove both. With a post-edit step, translations from OpenAI GPT-5.5 have every tag right. On Anthropic Claude Fable 5 it takes three steps, as the two-step solution has 20.1 weighted errors per 1,000 words and 4 major errors. For the Basic requirements, optimizing for quality or latency selects Anthropic Claude Fable 5, and for cost OpenAI GPT-5.5.
| Model | US dollars per million characters | Weighted errors per 1,000 words | Group |
|---|---|---|---|
| Google Gemma 4 | 0.14 | 45.8 | Misses a Standard requirement |
| OpenAI GPT-4.1 | 3.17 | 38.2 | Misses a Standard requirement |
| Google Gemini 3.5 Flash | 3.41 | 27.4 | Misses a Standard requirement |
| Google Gemini 3.1 Pro Preview | 4.61 | 31.3 | Misses a Standard requirement |
| OpenAI GPT-5.6 Sol | 7.76 | 21.4 | Misses a Standard requirement |
| Anthropic Claude Sonnet 5 | 7.94 | 24.7 | Misses a Standard requirement |
| Microsoft Translator | 10 | 81.4 | Misses a Standard requirement |
| Anthropic Claude Sonnet 4.6 | 10.95 | 28 | Misses a Standard requirement |
| OpenAI GPT-5.5 | 11.28 | 19.8 | Misses a Standard requirement |
| Translated Lara 2 | 24.99 | 42.2 | Misses a Standard requirement |
| DeepL Next Gen | 27.5 | 46.5 | Misses a Standard requirement |
| Anthropic Claude Fable 5 | 39.78 | 18.5 | Misses a Standard requirement |
In Urdu, the one-step solutions on the MT engine and the two LLM-based MT engines cost $10 to $27.50 per million characters, and their translations have 42.2 to 81.4 weighted errors per 1,000 words. Six LLM solutions cost less than $10, from $0.14 to $7.94 per million characters, and every LLM solution but the one on Google Gemma 4 has fewer weighted errors per 1,000 words than any of the three.
| Requirement | Median errors per 1,000 words |
|---|---|
| General translation quality | 29.6 |
| Gender neutrality | 15.2 |
| Tag handling | 1.3 |
| Tone of voice | 0 |
The requirements with countable errors. Every solution with its result on each requirement: the explorer.
In Urdu, the three-step solutions (translation + LQA + repair) on Google Gemini 3.1 Pro Preview and Google Gemini 3.5 Flash meet every Standard requirement but general translation quality. Their translations have every tag right and 11.2 and 19.8 weighted errors per 1,000 words, with 4 major errors in the first and 1 critical error in the second. Of the one-step solutions that miss general translation quality, the one on OpenAI GPT-5.6 Sol comes nearest, with 21.4 weighted errors per 1,000 words.
| Model | Weighted errors per 1,000 words of the shortlisted one-step solutions in Urdu | Class |
|---|---|---|
| Microsoft Translator | 81.4 | MT engines |
| DeepL Next Gen | 46.5 | LLM-based MT engines |
| Google Gemma 4 | 45.8 | Open-weight LLMs |
| Translated Lara 2 | 42.2 | LLM-based MT engines |
| OpenAI GPT-4.1 | 38.2 | LLMs |
| Google Gemini 3.1 Pro Preview | 31.3 | LLMs |
| Anthropic Claude Sonnet 4.6 | 28 | LLMs |
| Google Gemini 3.5 Flash | 27.4 | LLMs |
| Anthropic Claude Sonnet 5 | 24.7 | LLMs |
| OpenAI GPT-5.6 Sol | 21.4 | LLMs |
| OpenAI GPT-5.5 | 19.8 | LLMs |
| Anthropic Claude Fable 5 | 18.5 | LLMs |
All nine shortlisted LLMs, the open-weight Google Gemma 4 among them, support Urdu by our measured criteria, and translations from all nine meet formatting. In one step, translations from the eight that are not open-weight have 18.5 to 38.2 weighted errors per 1,000 words and 2 to 4 major errors. Those from the MT engine and the two LLM-based MT engines have 42.2 to 81.4 weighted errors per 1,000 words and, from DeepL Next Gen, the only critical error among the shortlisted one-step solutions. None of the three meet formatting or tag handling.
| Model | COMET-Kiwi | Rank on COMET-Kiwi | MetricX | Rank on MetricX | Shortlisted | Shortlist place |
|---|---|---|---|---|---|---|
| Mistral Large 3 | 0.762 | 9 | 1.68 | 17 | No | Not shortlisted |
| Anthropic Claude Opus 4.8 | 0.762 | 10 | 1.68 | 16 | No | Not shortlisted |
| xAI Grok 4.20 (reasoning) | 0.758 | 13 | 1.58 | 11 | No | Not shortlisted |
| MoonshotAI Kimi K2.5 | 0.758 | 14 | 1.72 | 19 | No | Not shortlisted |
| z.ai GLM-5 | 0.757 | 15 | 1.69 | 18 | No | Not shortlisted |
| DeepSeek V3.2 | 0.757 | 16 | 1.61 | 13 | No | Not shortlisted |
| Mistral Medium 3.1 | 0.747 | 19 | 1.62 | 14 | No | Not shortlisted |
| Anthropic Claude Haiku 4.5 | 0.746 | 20 | 1.84 | 20 | No | Not shortlisted |
| Google Translation LLM | 0.741 | 21 | 2.00 | 24 | No | Not shortlisted |
| OpenAI gpt-oss-120b | 0.736 | 22 | 1.67 | 15 | No | Not shortlisted |
| ModernMT Enterprise Edition | 0.735 | 23 | 1.98 | 23 | No | Not shortlisted |
| Alibaba Qwen3-235B-A22B | 0.735 | 24 | 1.89 | 21 | No | Not shortlisted |
| Cohere Command A Translate | 0.731 | 25 | 2.11 | 25 | No | Not shortlisted |
| NiuTrans | 0.725 | 26 | 2.35 | 27 | No | Not shortlisted |
| MiniMax M2.5 | 0.720 | 27 | 2.57 | 29 | No | Not shortlisted |
| Yandex Cloud Translate v2 | 0.713 | 28 | 2.73 | 30 | No | Not shortlisted |
| Swiss AI Apertus-8B-2509 | 0.707 | 29 | 2.40 | 28 | No | Not shortlisted |
| Cohere Tiny Aya | 0.707 | 30 | 2.31 | 26 | No | Not shortlisted |
| Google Cloud Advanced Translation API | 0.702 | 31 | 3.02 | 31 | No | Not shortlisted |
| Amazon Translate | 0.668 | 32 | 3.16 | 32 | No | Not shortlisted |
| SYSTRAN PNMT API | 0.633 | 33 | 4.74 | 33 | No | Not shortlisted |
| AppTek NMT | 0.620 | 34 | 4.91 | 34 | No | Not shortlisted |
| AI21 Jamba Large 1.7 | 0.593 | 35 | 5.27 | 35 | No | Not shortlisted |
| Alibaba Machine Translation API | 0.338 | 36 | 11.82 | 36 | No | Not shortlisted |
| OpenAI GPT-5.6 Sol | 0.773 | 1 | 1.43 | 7 | Yes | Top 8 by both metrics |
| Google Gemini 3.5 Flash | 0.771 | 2 | 1.42 | 6 | Yes | Top 8 by both metrics |
| Anthropic Claude Fable 5 | 0.770 | 3 | 1.37 | 3 | Yes | Top 8 by both metrics |
| Google Gemini 3.1 Pro Preview | 0.770 | 4 | 1.35 | 2 | Yes | Top 8 by both metrics |
| Anthropic Claude Sonnet 5 | 0.765 | 5 | 1.42 | 5 | Yes | Top 8 by both metrics |
| Google Gemma 4 | 0.764 | 6 | 1.54 | 9 | Yes | Top 8 by COMET-Kiwi |
| OpenAI GPT-4.1 | 0.764 | 7 | 1.35 | 1 | Yes | Top 8 by both metrics |
| OpenAI GPT-5.5 | 0.764 | 8 | 1.55 | 10 | Yes | Top 8 by COMET-Kiwi |
| DeepL Next Gen | 0.761 | 11 | 1.51 | 8 | Yes | Top 8 by MetricX |
| Anthropic Claude Sonnet 4.6 | 0.760 | 12 | 1.39 | 4 | Yes | Top 8 by MetricX |
| Translated Lara 2 | 0.755 | 17 | 1.93 | 22 | Yes | Real-time place |
| Microsoft Translator | 0.752 | 18 | 1.60 | 12 | Yes | Real-time place |
| Model | Solution complexity | COMET-Kiwi rank | COMET-Kiwi | MetricX rank | MetricX | Shortlist |
|---|---|---|---|---|---|---|
| OpenAI GPT-5.6 Sol | Adjusted LLM | 1 | 0.773 | 7 | 1.43 | Top 8 by both metrics |
| Google Gemini 3.5 Flash | Adjusted LLM | 2 | 0.771 | 6 | 1.42 | Top 8 by both metrics |
| Anthropic Claude Fable 5 | Adjusted LLM | 3 | 0.770 | 3 | 1.37 | Top 8 by both metrics |
| Google Gemini 3.1 Pro Preview | Adjusted LLM | 4 | 0.770 | 2 | 1.35 | Top 8 by both metrics |
| Anthropic Claude Sonnet 5 | Adjusted LLM | 5 | 0.765 | 5 | 1.42 | Top 8 by both metrics |
| Google Gemma 4 | Adjusted LLM | 6 | 0.764 | 9 | 1.54 | Top 8 by COMET-Kiwi |
| OpenAI GPT-4.1 | Adjusted LLM | 7 | 0.764 | 1 | 1.35 | Top 8 by both metrics |
| OpenAI GPT-5.5 | Adjusted LLM | 8 | 0.764 | 10 | 1.55 | Top 8 by COMET-Kiwi |
| Mistral Large 3 | Adjusted LLM | 9 | 0.762 | 17 | 1.68 | Not shortlisted |
| Anthropic Claude Opus 4.8 | Adjusted LLM | 10 | 0.762 | 16 | 1.68 | Not shortlisted |
| DeepL Next Gen | Off-the-shelf NMT | 11 | 0.761 | 8 | 1.51 | Top 8 by MetricX |
| Anthropic Claude Sonnet 4.6 | Adjusted LLM | 12 | 0.760 | 4 | 1.39 | Top 8 by MetricX |
| xAI Grok 4.20 (reasoning) | Adjusted LLM | 13 | 0.758 | 11 | 1.58 | Not shortlisted |
| MoonshotAI Kimi K2.5 | Adjusted LLM | 14 | 0.758 | 19 | 1.72 | Not shortlisted |
| z.ai GLM-5 | Adjusted LLM | 15 | 0.757 | 18 | 1.69 | Not shortlisted |
| DeepSeek V3.2 | Adjusted LLM | 16 | 0.757 | 13 | 1.61 | Not shortlisted |
| Translated Lara 2 | Adjusted NMT | 17 | 0.755 | 22 | 1.93 | Real-time place |
| Microsoft Translator | Off-the-shelf NMT | 18 | 0.752 | 12 | 1.60 | Real-time place |
| Mistral Medium 3.1 | Adjusted LLM | 19 | 0.747 | 14 | 1.62 | Not shortlisted |
| Anthropic Claude Haiku 4.5 | Adjusted LLM | 20 | 0.746 | 20 | 1.84 | Not shortlisted |
| Google Translation LLM | Adjusted NMT | 21 | 0.741 | 24 | 2.00 | Not shortlisted |
| OpenAI gpt-oss-120b | Adjusted LLM | 22 | 0.736 | 15 | 1.67 | Not shortlisted |
| ModernMT Enterprise Edition | Off-the-shelf NMT | 23 | 0.735 | 23 | 1.98 | Not shortlisted |
| Alibaba Qwen3-235B-A22B | Adjusted LLM | 24 | 0.735 | 21 | 1.89 | Not shortlisted |
| Cohere Command A Translate | Adjusted LLM | 25 | 0.731 | 25 | 2.11 | Not shortlisted |
| NiuTrans | Off-the-shelf NMT | 26 | 0.725 | 27 | 2.35 | Not shortlisted |
| MiniMax M2.5 | Adjusted LLM | 27 | 0.720 | 29 | 2.57 | Not shortlisted |
| Yandex Cloud Translate v2 | Off-the-shelf NMT | 28 | 0.713 | 30 | 2.73 | Not shortlisted |
| Swiss AI Apertus-8B-2509 | Adjusted LLM | 29 | 0.707 | 28 | 2.40 | Not shortlisted |
| Cohere Tiny Aya | Off-the-shelf LLM | 30 | 0.707 | 26 | 2.31 | Not shortlisted |
| Google Cloud Advanced Translation API | Off-the-shelf NMT | 31 | 0.702 | 31 | 3.02 | Not shortlisted |
| Amazon Translate | Off-the-shelf NMT | 32 | 0.668 | 32 | 3.16 | Not shortlisted |
| SYSTRAN PNMT API | Off-the-shelf NMT | 33 | 0.633 | 33 | 4.74 | Not shortlisted |
| AppTek NMT | Off-the-shelf NMT | 34 | 0.620 | 34 | 4.91 | Not shortlisted |
| AI21 Jamba Large 1.7 | Adjusted LLM | 35 | 0.593 | 35 | 5.27 | Not shortlisted |
| Alibaba Machine Translation API | Off-the-shelf NMT | 36 | 0.338 | 36 | 11.82 | Not shortlisted |
| OpenAI GPT-5.6 Luna | Adjusted LLM | — | 0.768 | — | 1.39 | Same model family as OpenAI GPT-5.6 Sol |
| OpenAI GPT-5.6 Terra | Adjusted LLM | — | 0.770 | — | 1.48 | Same model family as OpenAI GPT-5.6 Sol |
| xAI Grok 4.20 (non-reasoning) | Adjusted LLM | — | 0.752 | — | 1.72 | Same model family as xAI Grok 4.20 (reasoning) |
Every model we got a translation from, as the shortlist compared it: off the shelf, without context in the request, or adjusted, with it. A model that did not make the shortlist is shown adjusted, unless off the shelf it scores higher by 0.03 or more on COMET-Kiwi. Ranks among 36 models, with each model family counted once, and the top 8 of each metric in bold. Higher is better on COMET-Kiwi, lower on MetricX. The scores rank models within this language and are not a requirement result.
OpenAI GPT-5.5 is on the shortlist through COMET-Kiwi alone, eighth there and 10th on MetricX. By Intento LQA, its one-step solution has 19.8 weighted errors per 1,000 words, one of the two that meet general translation quality. The one on OpenAI GPT-4.1, first on MetricX, has 38.2 weighted errors per 1,000 words, and the one on Google Gemma 4, sixth on COMET-Kiwi, 45.8. Nearest the cut in the tail are Mistral Large 3, ninth on COMET-Kiwi, and xAI Grok 4.20 (reasoning), 11th on MetricX.
| Quality level | Cheapest solution that meets every requirement of the level | Cost per million characters |
|---|---|---|
| Basic | OpenAI GPT-5.5, one-step LLM adjusted for Standard requirements | $11.28 |
| Standard | OpenAI GPT-5.5, two-step | $11.28 |
| Advanced | No solution | — |
| Model | US dollars per million characters | Class |
|---|---|---|
| Google Gemma 4 | $0.14 | Open-weight LLMs |
| OpenAI GPT-4.1 | $3.17 | LLMs |
| Google Gemini 3.5 Flash | $3.41 | LLMs |
| Google Gemini 3.1 Pro Preview | $4.61 | LLMs |
| OpenAI GPT-5.6 Sol | $7.76 | LLMs |
| Anthropic Claude Sonnet 5 | $7.94 | LLMs |
| Microsoft Translator | $10 | MT engines |
| Anthropic Claude Sonnet 4.6 | $10.95 | LLMs |
| OpenAI GPT-5.5 | $11.28 | LLMs |
| Translated Lara 2 | $24.99 | LLM-based MT engines |
| DeepL Next Gen | $27.5 | LLM-based MT engines |
| Anthropic Claude Fable 5 | $39.78 | LLMs |
| Model | Tokens per character | Class |
|---|---|---|
| Mistral Medium 3.1 | 0.849 | LLMs |
| Mistral Large 3 | 0.847 | LLMs |
| Anthropic Claude Opus 4.8 | 0.715 | LLMs |
| Anthropic Claude Fable 5 | 0.713 | LLMs |
| Anthropic Claude Sonnet 5 | 0.707 | LLMs |
| Anthropic Claude Sonnet 4.6 | 0.686 | LLMs |
| Anthropic Claude Haiku 4.5 | 0.671 | LLMs |
| AI21 Jamba Large 1.7 | 0.621 | LLMs |
| Cohere Command A Translate | 0.592 | LLMs |
| Alibaba Qwen3-235B-A22B | 0.59 | Open-weight LLMs |
| MiniMax M2.5 | 0.544 | LLMs |
| z.ai GLM-5 | 0.524 | LLMs |
| DeepSeek V3.2 | 0.464 | LLMs |
| MoonshotAI Kimi K2.5 | 0.43 | LLMs |
| xAI Grok 4.20 (reasoning) | 0.402 | LLMs |
| xAI Grok 4.20 (non-reasoning) | 0.398 | LLMs |
| Cohere Tiny Aya | 0.345 | LLMs |
| OpenAI GPT-4.1 | 0.342 | LLMs |
| OpenAI GPT-5.6 Sol | 0.333 | LLMs |
| OpenAI GPT-5.6 Terra | 0.33 | LLMs |
| OpenAI GPT-5.5 | 0.329 | LLMs |
| OpenAI GPT-5.6 Luna | 0.329 | LLMs |
| Google Gemini 3.1 Pro Preview | 0.325 | LLMs |
| Google Gemini 3.5 Flash | 0.323 | LLMs |
| Google Gemma 4 | 0.321 | Open-weight LLMs |
The tokenizers of the three Google models on the shortlist produce 0.321 to 0.325 tokens per character of the Urdu translation, and those of the three Anthropic Claude models 0.686 to 0.713. Tokens do not set the cost alone: at 0.707 and 0.713 tokens per character, the one-step solutions on Anthropic Claude Sonnet 5 and Anthropic Claude Fable 5 cost $7.94 and $39.78 per million characters. The LQA and repair steps raise it: the three-step solution (translation + LQA + repair) on Anthropic Claude Fable 5 costs $603 per million characters.
| Quality level | Fastest solution that meets every requirement of the level | Seconds per 1,000 words |
|---|---|---|
| Basic | Anthropic Claude Fable 5, one-step LLM adjusted for Standard requirements | 97 |
| Standard | OpenAI GPT-5.5, two-step | 114 |
| Advanced | No solution | — |
| Model | Seconds per 1,000 words | Class |
|---|---|---|
| Microsoft Translator | 0.2 | MT engines |
| Google Cloud Advanced Translation API | 0.3 | MT engines |
| ModernMT Enterprise Edition | 0.4 | MT engines |
| DeepL Next Gen | 0.7 | LLM-based MT engines |
| Amazon Translate | 1.2 | MT engines |
| Google Translation LLM | 1.6 | LLM-based MT engines |
| SYSTRAN PNMT API | 1.6 | MT engines |
| Translated Lara 2 | 1.7 | LLM-based MT engines |
| OpenAI gpt-oss-120b | 2.4 | Open-weight LLMs |
| Yandex Cloud Translate v2 | 3.1 | MT engines |
| OpenAI GPT-5.6 Luna | 15 | LLMs |
| Cohere Tiny Aya | 20 | LLMs |
| xAI Grok 4.20 (non-reasoning) | 22 | LLMs |
| Alibaba Machine Translation API | 26 | MT engines |
| OpenAI GPT-4.1 | 36 | LLMs |
| DeepSeek V3.2 | 43 | LLMs |
| OpenAI GPT-5.6 Terra | 45 | LLMs |
| Anthropic Claude Haiku 4.5 | 49 | LLMs |
| Mistral Medium 3.1 | 51 | LLMs |
| NiuTrans | 57 | MT engines |
| Google Gemini 3.5 Flash | 58 | LLMs |
| MiniMax M2.5 | 58 | LLMs |
| xAI Grok 4.20 (reasoning) | 64 | LLMs |
| Cohere Command A Translate | 66 | LLMs |
| Mistral Large 3 | 68 | LLMs |
| AI21 Jamba Large 1.7 | 72 | LLMs |
| Anthropic Claude Sonnet 5 | 79 | LLMs |
| Google Gemini 3.1 Pro Preview | 84 | LLMs |
| Anthropic Claude Fable 5 | 97 | LLMs |
| Anthropic Claude Sonnet 4.6 | 102 | LLMs |
| AppTek NMT | 102 | MT engines |
| Anthropic Claude Opus 4.8 | 106 | LLMs |
| OpenAI GPT-5.5 | 114 | LLMs |
| Swiss AI Apertus-8B-2509 | 120 | Open-weight LLMs |
| OpenAI GPT-5.6 Sol | 128 | LLMs |
| Google Gemma 4 | 169 | Open-weight LLMs |
| Alibaba Qwen3-235B-A22B | 183 | Open-weight LLMs |
| z.ai GLM-5 | 197 | LLMs |
| MoonshotAI Kimi K2.5 | 207 | LLMs |
Real-time models are those the shortlist counts as real-time, under 2 seconds per text across the languages we timed. The chart shows the time of each model into Urdu.
The nine real-time models in Urdu are five MT engines, three LLM-based MT engines and the open-weight OpenAI gpt-oss-120b, the one LLM among them. Translations from the three on the shortlist meet tone of voice. None of the three meet tag handling or formatting. The time grows with the requirements: the fastest solution that meets every Basic requirement takes 97 seconds per 1,000 words, the fastest that meets every Standard requirement 114, and the best available for the Advanced requirements 303.
| The two best solutions | Solution | Sentences about people that are gender-neutral | Meets every Advanced requirement | Cost per million characters | Seconds per 1,000 words |
|---|---|---|---|---|---|
| Anthropic Claude Fable 5 | three-step, with the gender-neutrality agent | 98.2% | ✗ | $991 | 303 |
| OpenAI GPT-5.5 | three-step, with the gender-neutrality agent | 98.1% | ✗ | $963 | 370 |
For Urdu we check the formal register, with آپ, and gender-neutral wording through neutral nouns and neutral rephrasing, with no default gender. The other options in Urdu are a masculine default, and explicit double forms with neutral nouns and neutral rephrasing. With the gender-neutrality agent, the two best solutions meet every Advanced requirement but gender neutrality, for which we require 100% of the sentences about people to be gender-neutral.
For the 12 shortlisted solutions in Urdu, the median share of sentences on which the three LQA judges agree is 95.8%. Both pairs of automatic metrics agree above the threshold, and neither result is borderline. So the error counts of the 12 shortlisted solutions and the automatic ranking behind the shortlist rest on measures that agree in Urdu.
Leave feedback