The State of Translation Automation 2026 · Languages
What it takes to automate translation from English into Chinese Traditional against 10 requirements in three levels: the solutions that meet them, what they cost and how fast they are, from the 2026 benchmark of 57 AI models and MT engines.
A solution is a model plus its harness: prompts, glossaries and other context, pre- and post-processing. It meets a quality level when it meets every requirement of the level.
| Requirement | Description | Met when |
|---|---|---|
| Basic requirements for any company | ||
| General translation quality | Translation errors of every type, such as mistranslations, omissions and grammar errors, classified by type and severity following MQM, the industry standard. | No critical error, at most 3 major errors in the four test texts (3,034 words) and at most 20 errors per 1,000 words, weighted by severity (minor 1, major 5, critical 10). |
| Standard requirements adds the rules companies set for their content | ||
| Terminology | The approved translation of every glossary term, in any of its correct forms. | 100% of the glossary terms. We test it in the 16 languages with a glossary, so it does not apply in Chinese Traditional. |
| Tone of voice | The formal register (您) in every sentence in the second person or the imperative. | 100% of those sentences. |
| Tags | Every HTML tag of the source, with its attributes, in the translation. | 100% of the tags. |
| Unicode character integrity | Every character intact, with no corrupted characters and no letter from another script that only looks like the right one. | 100% of the characters. |
| Locale conventions | The punctuation marks of the target language in place of the English ones, such as the punctuation of Chinese. | No pass-or-fail bar. We report it as a share, because whether to localize punctuation is a matter of house style. |
| HTML entities | No HTML character code, such as the code for a quotation mark, that the source does not carry. | No added HTML entity. |
| Valid JSON | The translated JSON document still parses. | The document parses. |
| Spacing at tag boundaries | The spaces next to HTML tags stay as in the source. | 100% of the tag boundaries. |
| Advanced requirements adds a requirement for some companies and use cases | ||
| Gender neutrality | A gender-neutral translation wherever the English source leaves the gender of a person open, as in “your manager” or “the expert”. | 100% of the sentences that refer to a person. |
Pick a quality level, or your own requirements, bars and constraints, to see the shortlisted solutions that meet them in Chinese Traditional, with a column for each requirement you select. Under Optimize for, choose which of them comes first: the one with the fewest total errors, the lowest cost or the shortest time. Click a column heading to sort by it.
The shortlisted solutions on 13 models, with their Intento LQA results. Total errors per 1,000 words add the weighted general translation quality errors (minor 1, major 5, critical 10) to the missed glossary terms, sentences in the wrong register, tag errors and sentences that fail gender neutrality, each counted once. Formatting counts as met or missed and does not add to the total errors. The models that did not make the shortlist have automatic scores only: the shortlist. Inconclusive: the three LQA judges agree on fewer than 90% of the sentences of the solution, so it has no general translation quality result (09-02). Gender neutrality: the Advanced requirements count the two best solutions of the language, with the gender-neutrality agent where they need it. The results of the other solutions are shown for information.
Among the six three-step solutions (translation + LQA + repair) that meet every Standard requirement in Chinese Traditional, the one on OpenAI GPT-5.5 has the fewest total errors, 11.2 per 1,000 words, at $490 per million characters. The one on OpenAI GPT-5.6 Sol costs the least, $490 per million characters, and takes the shortest time, 140 seconds per 1,000 words, with 14.8 total errors per 1,000 words. Optimizing for quality selects the first, and optimizing for cost or latency the second.
None of the solutions under $489 per million characters meet even the Basic requirements in Chinese Traditional. The one-step solutions cost $0.50 to $27.50 per million characters, and the two-step ones (translation + post-edit) $1.46 to $52.32. The six solutions that meet every Standard requirement cost $490 to $561 per million characters. Three of them, on OpenAI GPT-5.6 Sol, OpenAI GPT-5.5 and Anthropic Claude Fable 5, take 140 to 177 seconds per 1,000 words, and the one on MoonshotAI Kimi K2.5 takes 752.
The Advanced requirements count only the two best solutions, so in Chinese Traditional they leave out the three-step solution (translation + LQA + repair) on OpenAI GPT-5.6 Sol, the cheapest and fastest of those that meet every Standard requirement. The cheapest and fastest left is the one on OpenAI GPT-5.5, at $490 per million characters and 172 seconds per 1,000 words. Optimizing for quality, cost or latency selects it. Both work without the gender-neutrality agent, so their cost and time do not change.
OpenAI GPT-5.5
three-step
OpenAI GPT-5.6 Sol
three-step
Tied with OpenAI GPT-5.5 (three-step).
OpenAI GPT-5.6 Sol
three-step
OpenAI GPT-5.5
three-step
OpenAI GPT-5.6 Sol
three-step
Tied with OpenAI GPT-5.5 (three-step).
OpenAI GPT-5.6 Sol
three-step
OpenAI GPT-5.5
three-step
OpenAI GPT-5.5
three-step
OpenAI GPT-5.5
three-step
Optimize for selects one of the solutions that meet the level: the one with the fewest total errors, the lowest cost or the shortest time. Where no solution meets the level, the card shows the available solution with the fewest total errors and what it misses. Total errors count every requirement with countable errors.
| Quality level | Tier | Solutions | Simplest solution | Model | Total errors per 1,000 words | Cost per million characters | Seconds per 1,000 words |
|---|---|---|---|---|---|---|---|
| Basic | Tier 2: two or more solutions, none simple | ☆×7 | three-step | OpenAI GPT-5.6 Sol | 14.8 | $490 | 140 |
| Standard | Tier 2: two or more solutions, none simple | ☆×6 | three-step | OpenAI GPT-5.6 Sol | 14.8 | $490 | 140 |
| Advanced | Tier 2: two or more solutions, none simple | ☆☆ | three-step | OpenAI GPT-5.5 | 11.2 | $490 | 172 |
One solution per level: the simplest that meets the level, the cheapest of them where several are equally simple. ★ a one-step solution, ☆ a solution with more steps or the gender-neutrality agent.
In Chinese Traditional, seven solutions meet every Basic requirement, all of them three-step solutions (translation + LQA + repair). The Standard requirements drop the one on Anthropic Claude Opus 4.8, which gets 99.6% of the tags right, against the 100% we require. The Advanced requirements count only the two best solutions, and both, on OpenAI GPT-5.5 and Anthropic Claude Fable 5, meet them. Optimizing for quality, cost or latency selects the same solutions under the Basic requirements as under the Standard ones.
| Model | US dollars per million characters | Weighted errors per 1,000 words | Group |
|---|---|---|---|
| MiniMax M2.5 | 0.5 | 34.3 | Misses a Standard requirement |
| Mistral Large 3 | 0.7 | 58.3 | Misses a Standard requirement |
| MoonshotAI Kimi K2.5 | 1.06 | 36.6 | Misses a Standard requirement |
| z.ai GLM-5 | 1.65 | 31.3 | Misses a Standard requirement |
| Google Gemini 3.5 Flash | 2.75 | 50.4 | Misses a Standard requirement |
| Google Gemini 3.1 Pro Preview | 3.66 | 31.3 | Misses a Standard requirement |
| Anthropic Claude Sonnet 5 | 4.68 | 41.2 | Misses a Standard requirement |
| OpenAI GPT-5.6 Sol | 6.82 | 24.7 | Misses a Standard requirement |
| OpenAI GPT-5.5 | 9.84 | 25.4 | Misses a Standard requirement |
| Anthropic Claude Opus 4.8 | 11.81 | 23.7 | Misses a Standard requirement |
| Anthropic Claude Fable 5 | 23.33 | 33 | Misses a Standard requirement |
| DeepL Next Gen | 27.5 | 40.5 | Misses a Standard requirement |
Paying more for a one-step solution in Chinese Traditional brings fewer errors only in part. The one on Anthropic Claude Opus 4.8 has the fewest, 23.7 weighted errors per 1,000 words, at $11.81 per million characters. The two dearest with an Intento LQA result, on DeepL Next Gen and Anthropic Claude Fable 5, cost $27.50 and $23.33 per million characters and have 40.5 and 33.0 weighted errors per 1,000 words. The cheapest, on MiniMax M2.5, costs $0.50 per million characters and has 34.3 weighted errors per 1,000 words.
| Requirement | Median errors per 1,000 words |
|---|---|
| General translation quality | 33.6 |
| Tag handling | 1 |
| Tone of voice | 0.1 |
| Gender neutrality | 0 |
The requirements with countable errors. Every solution with its result on each requirement: the explorer.
In Chinese Traditional, the number of steps follows from general translation quality. The one-step solutions nearest its bar, on Anthropic Claude Opus 4.8, OpenAI GPT-5.6 Sol and OpenAI GPT-5.5, have 23.7 to 25.4 weighted errors per 1,000 words. The two-step solution (translation + post-edit) on OpenAI GPT-5.5 has 21.4 weighted errors per 1,000 words. Seven of the nine three-step solutions (translation + LQA + repair) meet the requirement, at 11.2 to 14.8, and the other two have 26.7 and 39.2 weighted errors per 1,000 words.
| Model | Weighted errors per 1,000 words of the shortlisted one-step solutions in Chinese Traditional | Class |
|---|---|---|
| Mistral Large 3 | 58.3 | LLMs |
| Google Gemini 3.5 Flash | 50.4 | LLMs |
| Anthropic Claude Sonnet 5 | 41.2 | LLMs |
| DeepL Next Gen | 40.5 | LLM-based MT engines |
| MoonshotAI Kimi K2.5 | 36.6 | LLMs |
| MiniMax M2.5 | 34.3 | LLMs |
| Anthropic Claude Fable 5 | 33 | LLMs |
| z.ai GLM-5 | 31.3 | LLMs |
| Google Gemini 3.1 Pro Preview | 31.3 | LLMs |
| OpenAI GPT-5.5 | 25.4 | LLMs |
| OpenAI GPT-5.6 Sol | 24.7 | LLMs |
| Anthropic Claude Opus 4.8 | 23.7 | LLMs |
The 13 shortlisted solutions in Chinese Traditional are on 11 LLMs and two LLM-based MT engines, DeepL Next Gen and Supertext LLM. None of the ten MT engines that are not LLM-based made the shortlist, and COMET-Kiwi and MetricX both rank the best placed of them, Microsoft Translator, 23rd. Translations from DeepL Next Gen have 1.71 times as many weighted errors as those from Anthropic Claude Opus 4.8, which have the fewest among the shortlisted LLMs. The solution on Supertext LLM has no conclusive Intento LQA result.
| Model | COMET-Kiwi | Rank on COMET-Kiwi | MetricX | Rank on MetricX | Shortlisted | Shortlist place |
|---|---|---|---|---|---|---|
| xAI Grok 4.20 (non-reasoning) | 0.783 | 10 | 1.28 | 14 | No | Not shortlisted |
| DeepSeek V3.2 | 0.782 | 11 | 1.29 | 16 | No | Not shortlisted |
| Cohere Command A Translate | 0.781 | 13 | 1.27 | 13 | No | Not shortlisted |
| OpenAI GPT-4.1 | 0.779 | 14 | 1.26 | 12 | No | Not shortlisted |
| Google Gemma 4 | 0.779 | 15 | 1.29 | 17 | No | Not shortlisted |
| Mistral Medium 3.1 | 0.778 | 17 | 1.26 | 11 | No | Not shortlisted |
| Widn AI Tower Sugarloaf | 0.776 | 18 | 1.34 | 21 | No | Not shortlisted |
| Google Translation LLM | 0.774 | 21 | 1.66 | 31 | No | Not shortlisted |
| Anthropic Claude Sonnet 4.6 | 0.772 | 22 | 1.33 | 20 | No | Not shortlisted |
| Microsoft Translator | 0.772 | 23 | 1.36 | 23 | No | Not shortlisted |
| Anthropic Claude Haiku 4.5 | 0.771 | 24 | 1.36 | 22 | No | Not shortlisted |
| Swiss AI Apertus-8B-2509 | 0.768 | 25 | 1.36 | 24 | No | Not shortlisted |
| OpenAI gpt-oss-120b | 0.766 | 26 | 1.49 | 27 | No | Not shortlisted |
| AI21 Jamba Large 1.7 | 0.765 | 27 | 1.54 | 28 | No | Not shortlisted |
| ModernMT Enterprise Edition | 0.764 | 28 | 1.48 | 26 | No | Not shortlisted |
| Alibaba Qwen3-235B-A22B | 0.761 | 29 | 1.47 | 25 | No | Not shortlisted |
| Google Cloud Advanced Translation API | 0.751 | 30 | 2.18 | 38 | No | Not shortlisted |
| Cohere Tiny Aya | 0.750 | 31 | 1.58 | 29 | No | Not shortlisted |
| SYSTRAN PNMT API | 0.746 | 32 | 1.61 | 30 | No | Not shortlisted |
| Tencent Cloud TMT | 0.739 | 33 | 1.80 | 34 | No | Not shortlisted |
| Alibaba Machine Translation API | 0.738 | 34 | 1.70 | 32 | No | Not shortlisted |
| Amazon Translate | 0.735 | 35 | 1.71 | 33 | No | Not shortlisted |
| Youdao Translation Service API | 0.727 | 36 | 2.07 | 36 | No | Not shortlisted |
| Naver Papago NMT Commercial | 0.716 | 37 | 1.99 | 35 | No | Not shortlisted |
| Oracle Cloud Language Service | 0.691 | 38 | 2.14 | 37 | No | Not shortlisted |
| Widn AI Tower Vesuvius | 0.469 | 39 | 10.50 | 39 | No | Not shortlisted |
| Widn AI Tower Anthill | 0.459 | 40 | 10.70 | 40 | No | Not shortlisted |
| DeepL Next Gen | 0.789 | 1 | 1.19 | 1 | Yes | Top 8 by both metrics |
| OpenAI GPT-5.6 Sol | 0.789 | 2 | 1.25 | 9 | Yes | Top 8 by COMET-Kiwi |
| MiniMax M2.5 | 0.788 | 3 | 1.23 | 7 | Yes | Top 8 by both metrics |
| Anthropic Claude Opus 4.8 | 0.787 | 4 | 1.30 | 18 | Yes | Top 8 by COMET-Kiwi |
| OpenAI GPT-5.5 | 0.786 | 5 | 1.29 | 15 | Yes | Top 8 by COMET-Kiwi |
| Anthropic Claude Fable 5 | 0.785 | 6 | 1.26 | 10 | Yes | Top 8 by COMET-Kiwi |
| z.ai GLM-5 | 0.785 | 7 | 1.31 | 19 | Yes | Top 8 by COMET-Kiwi |
| Google Gemini 3.1 Pro Preview | 0.785 | 8 | 1.21 | 3 | Yes | Top 8 by both metrics |
| MoonshotAI Kimi K2.5 | 0.783 | 9 | 1.24 | 8 | Yes | Top 8 by MetricX |
| Anthropic Claude Sonnet 5 | 0.781 | 12 | 1.22 | 5 | Yes | Top 8 by MetricX |
| Google Gemini 3.5 Flash | 0.779 | 16 | 1.23 | 6 | Yes | Top 8 by MetricX |
| Supertext LLM | 0.776 | 19 | 1.22 | 4 | Yes | Top 8 by MetricX |
| Mistral Large 3 | 0.775 | 20 | 1.20 | 2 | Yes | Top 8 by MetricX |
| Model | Solution complexity | COMET-Kiwi rank | COMET-Kiwi | MetricX rank | MetricX | Shortlist |
|---|---|---|---|---|---|---|
| DeepL Next Gen | Off-the-shelf NMT | 1 | 0.789 | 1 | 1.19 | Top 8 by both metrics |
| OpenAI GPT-5.6 Sol | Adjusted LLM | 2 | 0.789 | 9 | 1.25 | Top 8 by COMET-Kiwi |
| MiniMax M2.5 | Adjusted LLM | 3 | 0.788 | 7 | 1.23 | Top 8 by both metrics |
| Anthropic Claude Opus 4.8 | Adjusted LLM | 4 | 0.787 | 18 | 1.30 | Top 8 by COMET-Kiwi |
| OpenAI GPT-5.5 | Adjusted LLM | 5 | 0.786 | 15 | 1.29 | Top 8 by COMET-Kiwi |
| Anthropic Claude Fable 5 | Adjusted LLM | 6 | 0.785 | 10 | 1.26 | Top 8 by COMET-Kiwi |
| z.ai GLM-5 | Adjusted LLM | 7 | 0.785 | 19 | 1.31 | Top 8 by COMET-Kiwi |
| Google Gemini 3.1 Pro Preview | Adjusted LLM | 8 | 0.785 | 3 | 1.21 | Top 8 by both metrics |
| MoonshotAI Kimi K2.5 | Adjusted LLM | 9 | 0.783 | 8 | 1.24 | Top 8 by MetricX |
| xAI Grok 4.20 (non-reasoning) | Adjusted LLM | 10 | 0.783 | 14 | 1.28 | Not shortlisted |
| DeepSeek V3.2 | Adjusted LLM | 11 | 0.782 | 16 | 1.29 | Not shortlisted |
| Anthropic Claude Sonnet 5 | Adjusted LLM | 12 | 0.781 | 5 | 1.22 | Top 8 by MetricX |
| Cohere Command A Translate | Adjusted LLM | 13 | 0.781 | 13 | 1.27 | Not shortlisted |
| OpenAI GPT-4.1 | Adjusted LLM | 14 | 0.779 | 12 | 1.26 | Not shortlisted |
| Google Gemma 4 | Adjusted LLM | 15 | 0.779 | 17 | 1.29 | Not shortlisted |
| Google Gemini 3.5 Flash | Adjusted LLM | 16 | 0.779 | 6 | 1.23 | Top 8 by MetricX |
| Mistral Medium 3.1 | Off-the-shelf LLM | 17 | 0.778 | 11 | 1.26 | Not shortlisted |
| Widn AI Tower Sugarloaf | Off-the-shelf NMT | 18 | 0.776 | 21 | 1.34 | Not shortlisted |
| Supertext LLM | Off-the-shelf NMT | 19 | 0.776 | 4 | 1.22 | Top 8 by MetricX |
| Mistral Large 3 | Adjusted LLM | 20 | 0.775 | 2 | 1.20 | Top 8 by MetricX |
| Google Translation LLM | Adjusted NMT | 21 | 0.774 | 31 | 1.66 | Not shortlisted |
| Anthropic Claude Sonnet 4.6 | Adjusted LLM | 22 | 0.772 | 20 | 1.33 | Not shortlisted |
| Microsoft Translator | Off-the-shelf NMT | 23 | 0.772 | 23 | 1.36 | Not shortlisted |
| Anthropic Claude Haiku 4.5 | Adjusted LLM | 24 | 0.771 | 22 | 1.36 | Not shortlisted |
| Swiss AI Apertus-8B-2509 | Adjusted LLM | 25 | 0.768 | 24 | 1.36 | Not shortlisted |
| OpenAI gpt-oss-120b | Adjusted LLM | 26 | 0.766 | 27 | 1.49 | Not shortlisted |
| AI21 Jamba Large 1.7 | Adjusted LLM | 27 | 0.765 | 28 | 1.54 | Not shortlisted |
| ModernMT Enterprise Edition | Off-the-shelf NMT | 28 | 0.764 | 26 | 1.48 | Not shortlisted |
| Alibaba Qwen3-235B-A22B | Adjusted LLM | 29 | 0.761 | 25 | 1.47 | Not shortlisted |
| Google Cloud Advanced Translation API | Off-the-shelf NMT | 30 | 0.751 | 38 | 2.18 | Not shortlisted |
| Cohere Tiny Aya | Off-the-shelf LLM | 31 | 0.750 | 29 | 1.58 | Not shortlisted |
| SYSTRAN PNMT API | Off-the-shelf NMT | 32 | 0.746 | 30 | 1.61 | Not shortlisted |
| Tencent Cloud TMT | Off-the-shelf NMT | 33 | 0.739 | 34 | 1.80 | Not shortlisted |
| Alibaba Machine Translation API | Off-the-shelf NMT | 34 | 0.738 | 32 | 1.70 | Not shortlisted |
| Amazon Translate | Off-the-shelf NMT | 35 | 0.735 | 33 | 1.71 | Not shortlisted |
| Youdao Translation Service API | Off-the-shelf NMT | 36 | 0.727 | 36 | 2.07 | Not shortlisted |
| Naver Papago NMT Commercial | Off-the-shelf NMT | 37 | 0.716 | 35 | 1.99 | Not shortlisted |
| Oracle Cloud Language Service | Off-the-shelf NMT | 38 | 0.691 | 37 | 2.14 | Not shortlisted |
| Widn AI Tower Vesuvius | Adjusted NMT | 39 | 0.469 | 39 | 10.50 | Not shortlisted |
| Widn AI Tower Anthill | Adjusted NMT | 40 | 0.459 | 40 | 10.70 | Not shortlisted |
| OpenAI GPT-5.6 Luna | Adjusted LLM | — | 0.782 | — | 1.31 | Same model family as OpenAI GPT-5.6 Sol |
| OpenAI GPT-5.6 Terra | Adjusted LLM | — | 0.787 | — | 1.29 | Same model family as OpenAI GPT-5.6 Sol |
| xAI Grok 4.20 (reasoning) | Adjusted LLM | — | 0.781 | — | 1.35 | Same model family as xAI Grok 4.20 (non-reasoning) |
Every model we got a translation from, as the shortlist compared it: off the shelf, without context in the request, or adjusted, with it. A model that did not make the shortlist is shown adjusted, unless off the shelf it scores higher by 0.03 or more on COMET-Kiwi. Ranks among 40 models, with each model family counted once, and the top 8 of each metric in bold. Higher is better on COMET-Kiwi, lower on MetricX. The scores rank models within this language and are not a requirement result.
COMET-Kiwi ranks Anthropic Claude Opus 4.8 fourth in Chinese Traditional and MetricX ranks it 18th, so it is on the shortlist through COMET-Kiwi alone. Yet by Intento LQA, the one-step solution on this model has the fewest weighted errors of the shortlist, 23.7 per 1,000 words. The one on Mistral Large 3, second on MetricX and 20th on COMET-Kiwi, has the most, 58.3 weighted errors per 1,000 words. In the tail, xAI Grok 4.20 (non-reasoning) is nearest the cut on COMET-Kiwi, in 10th place, and Mistral Medium 3.1 on MetricX, in 11th.
| Quality level | Cheapest solution that meets every requirement of the level | Cost per million characters |
|---|---|---|
| Basic | OpenAI GPT-5.6 Sol, three-step | $490 |
| Standard | OpenAI GPT-5.6 Sol, three-step | $490 |
| Advanced | OpenAI GPT-5.5, three-step | $490 |
| Model | US dollars per million characters | Class |
|---|---|---|
| MiniMax M2.5 | $0.5 | LLMs |
| Mistral Large 3 | $0.7 | LLMs |
| MoonshotAI Kimi K2.5 | $1.06 | LLMs |
| z.ai GLM-5 | $1.65 | LLMs |
| Google Gemini 3.5 Flash | $2.75 | LLMs |
| Google Gemini 3.1 Pro Preview | $3.66 | LLMs |
| Anthropic Claude Sonnet 5 | $4.68 | LLMs |
| OpenAI GPT-5.6 Sol | $6.82 | LLMs |
| OpenAI GPT-5.5 | $9.84 | LLMs |
| Anthropic Claude Opus 4.8 | $11.81 | LLMs |
| Anthropic Claude Fable 5 | $23.33 | LLMs |
| Supertext LLM | $23.4 | LLM-based MT engines |
| DeepL Next Gen | $27.5 | LLM-based MT engines |
| Model | Tokens per character | Class |
|---|---|---|
| Mistral Medium 3.1 | 0.785 | LLMs |
| Mistral Large 3 | 0.781 | LLMs |
| AI21 Jamba Large 1.7 | 0.778 | LLMs |
| Anthropic Claude Fable 5 | 0.765 | LLMs |
| Anthropic Claude Opus 4.8 | 0.765 | LLMs |
| Anthropic Claude Sonnet 5 | 0.762 | LLMs |
| Anthropic Claude Sonnet 4.6 | 0.715 | LLMs |
| Anthropic Claude Haiku 4.5 | 0.71 | LLMs |
| MoonshotAI Kimi K2.5 | 0.606 | LLMs |
| OpenAI GPT-5.6 Sol | 0.581 | LLMs |
| z.ai GLM-5 | 0.578 | LLMs |
| OpenAI GPT-5.5 | 0.576 | LLMs |
| OpenAI GPT-4.1 | 0.572 | LLMs |
| OpenAI GPT-5.6 Terra | 0.572 | LLMs |
| OpenAI GPT-5.6 Luna | 0.57 | LLMs |
| Widn AI Tower Vesuvius | 0.517 | LLM-based MT engines |
| Cohere Command A Translate | 0.516 | LLMs |
| Alibaba Qwen3-235B-A22B | 0.509 | Open-weight LLMs |
| Cohere Tiny Aya | 0.509 | LLMs |
| Widn AI Tower Sugarloaf | 0.504 | LLM-based MT engines |
| Google Gemini 3.5 Flash | 0.503 | LLMs |
| Google Gemini 3.1 Pro Preview | 0.501 | LLMs |
| Google Gemma 4 | 0.498 | Open-weight LLMs |
| DeepSeek V3.2 | 0.493 | LLMs |
| xAI Grok 4.20 (non-reasoning) | 0.481 | LLMs |
| xAI Grok 4.20 (reasoning) | 0.477 | LLMs |
| MiniMax M2.5 | 0.444 | LLMs |
| Widn AI Tower Anthill | 0.415 | LLM-based MT engines |
Per character of the translation into Chinese Traditional, the tokenizer of Mistral Large 3 produces 0.781 tokens, and those of the three shortlisted Anthropic Claude models 0.762 to 0.765. Tokens do not set the price alone. The one-step solution on Mistral Large 3 costs $0.70 per million characters, against $4.68 to $23.33 on the three Anthropic Claude models. The steps weigh far more than the tokens. On OpenAI GPT-5.5 a translation costs $9.84 per million characters in one step, $39.15 in two and $490 in three.
| Quality level | Fastest solution that meets every requirement of the level | Seconds per 1,000 words |
|---|---|---|
| Basic | OpenAI GPT-5.6 Sol, three-step | 140 |
| Standard | OpenAI GPT-5.6 Sol, three-step | 140 |
| Advanced | OpenAI GPT-5.5, three-step | 172 |
| Model | Seconds per 1,000 words | Class |
|---|---|---|
| Microsoft Translator | 0.2 | MT engines |
| ModernMT Enterprise Edition | 0.5 | MT engines |
| Google Cloud Advanced Translation API | 0.6 | MT engines |
| Google Translation LLM | 1.1 | LLM-based MT engines |
| SYSTRAN PNMT API | 1.1 | MT engines |
| Amazon Translate | 1.4 | MT engines |
| DeepL Next Gen | 2 | LLM-based MT engines |
| OpenAI gpt-oss-120b | 2.1 | Open-weight LLMs |
| Oracle Cloud Language Service | 3.9 | MT engines |
| Widn AI Tower Anthill | 6.8 | LLM-based MT engines |
| xAI Grok 4.20 (non-reasoning) | 12 | LLMs |
| Google Gemini 3.5 Flash | 13 | LLMs |
| OpenAI GPT-5.6 Luna | 14 | LLMs |
| Cohere Tiny Aya | 15 | LLMs |
| OpenAI GPT-5.6 Terra | 17 | LLMs |
| Supertext LLM | 25 | LLM-based MT engines |
| Anthropic Claude Haiku 4.5 | 27 | LLMs |
| Cohere Command A Translate | 28 | LLMs |
| MiniMax M2.5 | 32 | LLMs |
| DeepSeek V3.2 | 33 | LLMs |
| Widn AI Tower Vesuvius | 33 | LLM-based MT engines |
| Mistral Medium 3.1 | 34 | LLMs |
| OpenAI GPT-4.1 | 35 | LLMs |
| Naver Papago NMT Commercial | 37 | MT engines |
| Alibaba Machine Translation API | 39 | MT engines |
| xAI Grok 4.20 (reasoning) | 39 | LLMs |
| OpenAI GPT-5.6 Sol | 40 | LLMs |
| OpenAI GPT-5.5 | 40 | LLMs |
| Anthropic Claude Sonnet 5 | 42 | LLMs |
| Anthropic Claude Fable 5 | 49 | LLMs |
| AI21 Jamba Large 1.7 | 51 | LLMs |
| Anthropic Claude Sonnet 4.6 | 54 | LLMs |
| Anthropic Claude Opus 4.8 | 55 | LLMs |
| Mistral Large 3 | 55 | LLMs |
| Swiss AI Apertus-8B-2509 | 57 | Open-weight LLMs |
| Youdao Translation Service API | 60 | MT engines |
| Google Gemini 3.1 Pro Preview | 65 | LLMs |
| Tencent Cloud TMT | 65 | MT engines |
| Alibaba Qwen3-235B-A22B | 77 | Open-weight LLMs |
| Google Gemma 4 | 125 | Open-weight LLMs |
| z.ai GLM-5 | 138 | LLMs |
| MoonshotAI Kimi K2.5 | 187 | LLMs |
Real-time models are those the shortlist counts as real-time, under 2 seconds per text across the languages we timed. The chart shows the time of each model into Chinese Traditional.
Of the eight real-time (<2 sec) models in Chinese Traditional, only DeepL Next Gen is on the shortlist. Its translations miss general translation quality, with 40.5 weighted errors per 1,000 words and one critical error, and also tone of voice, tag handling and formatting, with 89.36% of the sentences in the formal register and 94.1% of the tags right. The fastest shortlisted LLM solution, on Google Gemini 3.5 Flash, takes 13 seconds per 1,000 words and has 50.4 weighted errors per 1,000 words.
| The two best solutions | Solution | Sentences about people that are gender-neutral | Meets every Advanced requirement | Cost per million characters | Seconds per 1,000 words |
|---|---|---|---|---|---|
| OpenAI GPT-5.5 | three-step | 100% | ✓ | $490 | 172 |
| Anthropic Claude Fable 5 | three-step | 100% | ✓ | $561 | 177 |
In Chinese Traditional, tone of voice means the formal register, with 您, and gender neutrality means neutral nouns and rephrasing, with no forced 他 or 她 and no default gender. The other options in the language are a generic 他, the split 他·她 and TA. Translations from all 11 shortlisted LLMs keep the formal register in every sentence in the second person or the imperative. Those from the two LLM-based MT engines, DeepL Next Gen and Supertext LLM, keep it in 89.36% and 72.3% of those sentences.
In Chinese Traditional the three LQA judges agree on a median of 97.2% of the sentences across the 13 shortlisted solutions. The similarity pair of automatic metrics agrees at 0.63, against a threshold of 0.46, and the quality-estimation pair at 0.61, against 0.53, with neither near its threshold. So the shortlist rests on automatic metrics that agree, and the Intento LQA error counts on LQA judges that agree on most of the sentences.
Leave feedback