The State of Translation Automation 2026 · Languages
What it takes to automate translation from English into Croatian against 10 requirements in three levels: the solutions that meet them, what they cost and how fast they are, from the 2026 benchmark of 57 AI models and MT engines.
A solution is a model plus its harness: prompts, glossaries and other context, pre- and post-processing. It meets a quality level when it meets every requirement of the level.
| Requirement | Description | Met when |
|---|---|---|
| Basic requirements for any company | ||
| General translation quality | Translation errors of every type, such as mistranslations, omissions and grammar errors, classified by type and severity following MQM, the industry standard. | No critical error, at most 3 major errors in the four test texts (3,034 words) and at most 20 errors per 1,000 words, weighted by severity (minor 1, major 5, critical 10). |
| Standard requirements adds the rules companies set for their content | ||
| Terminology | The approved translation of every glossary term, in any of its correct forms. | 100% of the glossary terms. We test it in the 16 languages with a glossary, so it does not apply in Croatian. |
| Tone of voice | The formal register (Vi) in every sentence in the second person or the imperative. | 100% of those sentences. |
| Tags | Every HTML tag of the source, with its attributes, in the translation. | 100% of the tags. |
| Unicode character integrity | Every character intact, with no corrupted characters and no letter from another script that only looks like the right one. | 100% of the characters. |
| Locale conventions | The punctuation marks of the target language in place of the English ones. | No pass-or-fail bar. We report it as a share, because whether to localize punctuation is a matter of house style. |
| HTML entities | No HTML character code, such as the code for a quotation mark, that the source does not carry. | No added HTML entity. |
| Valid JSON | The translated JSON document still parses. | The document parses. |
| Spacing at tag boundaries | The spaces next to HTML tags stay as in the source. | 100% of the tag boundaries. |
| Advanced requirements adds a requirement for some companies and use cases | ||
| Gender neutrality | A gender-neutral translation wherever the English source leaves the gender of a person open, as in “your manager” or “the expert”. | 100% of the sentences that refer to a person. |
Pick a quality level, or your own requirements, bars and constraints, to see the shortlisted solutions that meet them in Croatian, with a column for each requirement you select. Under Optimize for, choose which of them comes first: the one with the fewest total errors, the lowest cost or the shortest time. Click a column heading to sort by it.
The shortlisted solutions on 13 models, with their Intento LQA results. Total errors per 1,000 words add the weighted general translation quality errors (minor 1, major 5, critical 10) to the missed glossary terms, sentences in the wrong register, tag errors and sentences that fail gender neutrality, each counted once. Formatting counts as met or missed and does not add to the total errors. The models that did not make the shortlist have automatic scores only: the shortlist. Gender neutrality: the Advanced requirements count the two best solutions of the language, with the gender-neutrality agent where they need it. The results of the other solutions are shown for information.
None of the four solutions that meet every Standard requirement in Croatian leads on errors, cost and time at once. For quality, Optimize for selects the one on OpenAI GPT-5.5, with 15.9 total errors per 1,000 words at $10.83 per million characters. For cost it selects Google Gemini 3.1 Pro Preview, at $4.46 per million characters but 84 seconds per 1,000 words, the longest. For latency it selects Anthropic Claude Fable 5, at 61 seconds per 1,000 words but $28.30 per million characters, the highest cost.
The cheapest of these four is also the slowest, so a time limit of 65 to 84 seconds per 1,000 words leaves the three dearer ones, from $7.49 to $28.30 per million characters. The one-step solution on Google Gemini 3.5 Flash is cheaper and faster, at $3.30 per million characters and 15 seconds per 1,000 words, but it misses general translation quality, with 27.7 weighted errors per 1,000 words. The solutions that meet every Advanced requirement take two or three steps, so a limit of one step rules them out.
Both solutions with the gender-neutrality agent take more than one step. On OpenAI GPT-5.5, the two-step solution (translation + post-edit) with the agent costs $45.38 per million characters against $10.83 for the one-step solution, takes 172 seconds per 1,000 words against 65, and has 11.9 total errors per 1,000 words against 15.9. On OpenAI GPT-5.6 Sol, the three-step solution (translation + LQA + repair) with the agent costs $984 per million characters against $7.49, and has 4.6 total errors per 1,000 words against 20.1.
OpenAI GPT-5.5
one-step LLM adjusted for Standard requirements
Google Gemini 3.1 Pro Preview
one-step LLM adjusted for Standard requirements
Anthropic Claude Fable 5
one-step LLM adjusted for Standard requirements
OpenAI GPT-5.5
one-step LLM adjusted for Standard requirements
Google Gemini 3.1 Pro Preview
one-step LLM adjusted for Standard requirements
Anthropic Claude Fable 5
one-step LLM adjusted for Standard requirements
OpenAI GPT-5.6 Sol
three-step with the gender-neutrality agent
OpenAI GPT-5.5
two-step with the gender-neutrality agent
OpenAI GPT-5.5
two-step with the gender-neutrality agent
Tied with OpenAI GPT-5.6 Sol (three-step with the gender-neutrality agent).
Optimize for selects one of the solutions that meet the level: the one with the fewest total errors, the lowest cost or the shortest time. Where no solution meets the level, the card shows the available solution with the fewest total errors and what it misses. Total errors count every requirement with countable errors.
| Quality level | Tier | Solutions | Simplest solution | Model | Total errors per 1,000 words | Cost per million characters | Seconds per 1,000 words |
|---|---|---|---|---|---|---|---|
| Basic | Tier 1: simple solution + backup | ★×4 | one-step | Google Gemini 3.1 Pro Preview | 18.6 | $4.46 | 84 |
| Standard | Tier 1: simple solution + backup | ★×4 | one-step | Google Gemini 3.1 Pro Preview | 18.6 | $4.46 | 84 |
| Advanced | Tier 2: two or more solutions, none simple | ☆☆ | with the gender-neutrality agent | OpenAI GPT-5.5 | 11.9 | $45.38 | 172 |
One solution per level: the simplest that meets the level, the cheapest of them where several are equally simple. ★ a one-step solution, ☆ a solution with more steps or the gender-neutrality agent.
The same four one-step solutions meet the Basic and Standard requirements, so Optimize for makes the same selections under both. The Advanced requirements add gender neutrality. None of the four meets it. The two solutions that meet every Advanced requirement work with the gender-neutrality agent. Optimize for selects the three-step one on OpenAI GPT-5.6 Sol for quality, with 4.6 total errors per 1,000 words, and the two-step one on OpenAI GPT-5.5 for cost, at $45.38 per million characters. For latency the two tie at 172 seconds per 1,000 words.
| Model | US dollars per million characters | Weighted errors per 1,000 words | Group |
|---|---|---|---|
| Google Gemma 4 | 0.13 | 47.8 | Misses a Standard requirement |
| MoonshotAI Kimi K2.5 | 1.27 | 52.7 | Misses a Standard requirement |
| z.ai GLM-5 | 2.02 | 35.6 | Misses a Standard requirement |
| OpenAI GPT-4.1 | 2.98 | 27.7 | Misses a Standard requirement |
| Google Gemini 3.5 Flash | 3.3 | 27.7 | Misses a Standard requirement |
| Google Gemini 3.1 Pro Preview | 4.46 | 13.8 | Meets every Standard requirement |
| Anthropic Claude Sonnet 4.6 | 6.38 | 24.7 | Misses a Standard requirement |
| OpenAI GPT-5.6 Sol | 7.49 | 12.9 | Meets every Standard requirement |
| Microsoft Translator | 10 | 48.1 | Misses a Standard requirement |
| OpenAI GPT-5.5 | 10.83 | 9.6 | Meets every Standard requirement |
| Translated Lara 2 | 24.99 | 24.4 | Misses a Standard requirement |
| DeepL Next Gen | 27.5 | 47.8 | Misses a Standard requirement |
| Anthropic Claude Fable 5 | 28.3 | 13.5 | Meets every Standard requirement |
Every shortlisted one-step solution that costs more than the one on OpenAI GPT-5.5, at $10.83 per million characters, has more weighted errors per 1,000 words than its 9.6: the one on Translated Lara 2 has 24.4 at $24.99, the one on DeepL Next Gen 47.8 at $27.50 and the one on Anthropic Claude Fable 5 13.5 at $28.30.
| Requirement | Median errors per 1,000 words |
|---|---|
| General translation quality | 27.7 |
| Gender neutrality | 6.5 |
| Tone of voice | 0 |
| Tag handling | 0 |
The requirements with countable errors. Every solution with its result on each requirement: the explorer.
The one-step solution on Anthropic Claude Sonnet 4.6 misses general translation quality on weighted errors alone, with 24.7 per 1,000 words and no major or critical error, and misses tag handling by a small margin, with 99.2% of the tags kept.
| Model | Weighted errors per 1,000 words of the shortlisted one-step solutions in Croatian | Class |
|---|---|---|
| MoonshotAI Kimi K2.5 | 52.7 | LLMs |
| Microsoft Translator | 48.1 | MT engines |
| Google Gemma 4 | 47.8 | Open-weight LLMs |
| DeepL Next Gen | 47.8 | LLM-based MT engines |
| z.ai GLM-5 | 35.6 | LLMs |
| OpenAI GPT-4.1 | 27.7 | LLMs |
| Google Gemini 3.5 Flash | 27.7 | LLMs |
| Anthropic Claude Sonnet 4.6 | 24.7 | LLMs |
| Translated Lara 2 | 24.4 | LLM-based MT engines |
| Google Gemini 3.1 Pro Preview | 13.8 | LLMs |
| Anthropic Claude Fable 5 | 13.5 | LLMs |
| OpenAI GPT-5.6 Sol | 12.9 | LLMs |
| OpenAI GPT-5.5 | 9.6 | LLMs |
No translation into Croatian: SpeakLeash Bielik-11B-v3.0-Instruct (wrong language), Widn AI Tower Anthill (far too short, wrong language), Widn AI Tower Vesuvius (far too short, wrong language).
The four one-step solutions that meet general translation quality in Croatian are all LLMs, with 9.6 to 13.8 weighted errors per 1,000 words. Translations from the other six LLMs on the shortlist, the open-weight Google Gemma 4 among them, have 24.7 to 52.7 weighted errors per 1,000 words. All ten shortlisted LLMs support Croatian by our criteria. The output of three more models was in the wrong language: SpeakLeash Bielik-11B-v3.0-Instruct, Widn AI Tower Anthill and Widn AI Tower Vesuvius.
| Model | COMET-Kiwi | Rank on COMET-Kiwi | MetricX | Rank on MetricX | Shortlisted | Shortlist place |
|---|---|---|---|---|---|---|
| xAI Grok 4.20 (reasoning) | 0.825 | 10 | 1.50 | 18 | No | Not shortlisted |
| Mistral Large 3 | 0.824 | 11 | 1.47 | 17 | No | Not shortlisted |
| Anthropic Claude Sonnet 5 | 0.824 | 13 | 1.44 | 10 | No | Not shortlisted |
| Anthropic Claude Opus 4.8 | 0.823 | 16 | 1.55 | 20 | No | Not shortlisted |
| DeepSeek V3.2 | 0.819 | 18 | 1.47 | 15 | No | Not shortlisted |
| Pangeanic PangeaMT | 0.819 | 19 | 1.67 | 27 | No | Not shortlisted |
| Tilde TildeOpen LLM engine | 0.819 | 20 | 1.51 | 19 | No | Not shortlisted |
| Google Translation LLM | 0.817 | 21 | 1.78 | 33 | No | Not shortlisted |
| Swiss AI Apertus-8B-2509 | 0.815 | 22 | 1.64 | 26 | No | Not shortlisted |
| OpenAI gpt-oss-120b | 0.814 | 23 | 1.67 | 29 | No | Not shortlisted |
| Mistral Medium 3.1 | 0.814 | 24 | 1.57 | 22 | No | Not shortlisted |
| ModernMT Enterprise Edition | 0.813 | 25 | 1.44 | 12 | No | Not shortlisted |
| MiniMax M2.5 | 0.810 | 26 | 1.62 | 25 | No | Not shortlisted |
| Cohere Command A Translate | 0.809 | 27 | 1.46 | 14 | No | Not shortlisted |
| Anthropic Claude Haiku 4.5 | 0.809 | 28 | 1.59 | 24 | No | Not shortlisted |
| Supertext LLM | 0.800 | 29 | 1.57 | 21 | No | Not shortlisted |
| Google Cloud Advanced Translation API | 0.797 | 30 | 2.29 | 36 | No | Not shortlisted |
| Alibaba Qwen3-235B-A22B | 0.795 | 31 | 1.70 | 30 | No | Not shortlisted |
| Amazon Translate | 0.792 | 32 | 1.67 | 28 | No | Not shortlisted |
| AI21 Jamba Large 1.7 | 0.778 | 33 | 1.73 | 32 | No | Not shortlisted |
| SYSTRAN PNMT API | 0.778 | 34 | 1.72 | 31 | No | Not shortlisted |
| Cohere Tiny Aya | 0.776 | 35 | 1.92 | 34 | No | Not shortlisted |
| Yandex Cloud Translate v2 | 0.754 | 36 | 2.06 | 35 | No | Not shortlisted |
| AppTek NMT | 0.635 | 37 | 5.71 | 37 | No | Not shortlisted |
| Google Gemini 3.5 Flash | 0.832 | 1 | 1.35 | 2 | Yes | Top 8 by both metrics |
| OpenAI GPT-4.1 | 0.831 | 2 | 1.59 | 23 | Yes | Top 8 by COMET-Kiwi |
| OpenAI GPT-5.6 Sol | 0.830 | 3 | 1.35 | 1 | Yes | Top 8 by both metrics |
| Google Gemini 3.1 Pro Preview | 0.828 | 4 | 1.42 | 7 | Yes | Top 8 by both metrics |
| OpenAI GPT-5.5 | 0.828 | 5 | 1.45 | 13 | Yes | Top 8 by COMET-Kiwi |
| Translated Lara 2 | 0.828 | 6 | 1.44 | 11 | Yes | Top 8 by COMET-Kiwi |
| Anthropic Claude Sonnet 4.6 | 0.825 | 7 | 1.43 | 9 | Yes | Top 8 by COMET-Kiwi |
| Anthropic Claude Fable 5 | 0.825 | 8 | 1.43 | 8 | Yes | Top 8 by both metrics |
| DeepL Next Gen | 0.825 | 9 | 1.40 | 6 | Yes | Top 8 by MetricX |
| Google Gemma 4 | 0.824 | 12 | 1.36 | 4 | Yes | Top 8 by MetricX |
| z.ai GLM-5 | 0.824 | 14 | 1.35 | 3 | Yes | Top 8 by MetricX |
| MoonshotAI Kimi K2.5 | 0.823 | 15 | 1.37 | 5 | Yes | Top 8 by MetricX |
| Microsoft Translator | 0.823 | 17 | 1.47 | 16 | Yes | Real-time place |
| Model | Solution complexity | COMET-Kiwi rank | COMET-Kiwi | MetricX rank | MetricX | Shortlist |
|---|---|---|---|---|---|---|
| Google Gemini 3.5 Flash | Adjusted LLM | 1 | 0.832 | 2 | 1.35 | Top 8 by both metrics |
| OpenAI GPT-4.1 | Adjusted LLM | 2 | 0.831 | 23 | 1.59 | Top 8 by COMET-Kiwi |
| OpenAI GPT-5.6 Sol | Adjusted LLM | 3 | 0.830 | 1 | 1.35 | Top 8 by both metrics |
| Google Gemini 3.1 Pro Preview | Adjusted LLM | 4 | 0.828 | 7 | 1.42 | Top 8 by both metrics |
| OpenAI GPT-5.5 | Adjusted LLM | 5 | 0.828 | 13 | 1.45 | Top 8 by COMET-Kiwi |
| Translated Lara 2 | Adjusted NMT | 6 | 0.828 | 11 | 1.44 | Top 8 by COMET-Kiwi |
| Anthropic Claude Sonnet 4.6 | Adjusted LLM | 7 | 0.825 | 9 | 1.43 | Top 8 by COMET-Kiwi |
| Anthropic Claude Fable 5 | Adjusted LLM | 8 | 0.825 | 8 | 1.43 | Top 8 by both metrics |
| DeepL Next Gen | Off-the-shelf NMT | 9 | 0.825 | 6 | 1.40 | Top 8 by MetricX |
| xAI Grok 4.20 (reasoning) | Adjusted LLM | 10 | 0.825 | 18 | 1.50 | Not shortlisted |
| Mistral Large 3 | Adjusted LLM | 11 | 0.824 | 17 | 1.47 | Not shortlisted |
| Google Gemma 4 | Adjusted LLM | 12 | 0.824 | 4 | 1.36 | Top 8 by MetricX |
| Anthropic Claude Sonnet 5 | Adjusted LLM | 13 | 0.824 | 10 | 1.44 | Not shortlisted |
| z.ai GLM-5 | Adjusted LLM | 14 | 0.824 | 3 | 1.35 | Top 8 by MetricX |
| MoonshotAI Kimi K2.5 | Adjusted LLM | 15 | 0.823 | 5 | 1.37 | Top 8 by MetricX |
| Anthropic Claude Opus 4.8 | Adjusted LLM | 16 | 0.823 | 20 | 1.55 | Not shortlisted |
| Microsoft Translator | Off-the-shelf NMT | 17 | 0.823 | 16 | 1.47 | Real-time place |
| DeepSeek V3.2 | Adjusted LLM | 18 | 0.819 | 15 | 1.47 | Not shortlisted |
| Pangeanic PangeaMT | Off-the-shelf NMT | 19 | 0.819 | 27 | 1.67 | Not shortlisted |
| Tilde TildeOpen LLM engine | Adjusted NMT | 20 | 0.819 | 19 | 1.51 | Not shortlisted |
| Google Translation LLM | Adjusted NMT | 21 | 0.817 | 33 | 1.78 | Not shortlisted |
| Swiss AI Apertus-8B-2509 | Off-the-shelf LLM | 22 | 0.815 | 26 | 1.64 | Not shortlisted |
| OpenAI gpt-oss-120b | Adjusted LLM | 23 | 0.814 | 29 | 1.67 | Not shortlisted |
| Mistral Medium 3.1 | Adjusted LLM | 24 | 0.814 | 22 | 1.57 | Not shortlisted |
| ModernMT Enterprise Edition | Off-the-shelf NMT | 25 | 0.813 | 12 | 1.44 | Not shortlisted |
| MiniMax M2.5 | Adjusted LLM | 26 | 0.810 | 25 | 1.62 | Not shortlisted |
| Cohere Command A Translate | Adjusted LLM | 27 | 0.809 | 14 | 1.46 | Not shortlisted |
| Anthropic Claude Haiku 4.5 | Adjusted LLM | 28 | 0.809 | 24 | 1.59 | Not shortlisted |
| Supertext LLM | Off-the-shelf NMT | 29 | 0.800 | 21 | 1.57 | Not shortlisted |
| Google Cloud Advanced Translation API | Off-the-shelf NMT | 30 | 0.797 | 36 | 2.29 | Not shortlisted |
| Alibaba Qwen3-235B-A22B | Adjusted LLM | 31 | 0.795 | 30 | 1.70 | Not shortlisted |
| Amazon Translate | Off-the-shelf NMT | 32 | 0.792 | 28 | 1.67 | Not shortlisted |
| AI21 Jamba Large 1.7 | Adjusted LLM | 33 | 0.778 | 32 | 1.73 | Not shortlisted |
| SYSTRAN PNMT API | Off-the-shelf NMT | 34 | 0.778 | 31 | 1.72 | Not shortlisted |
| Cohere Tiny Aya | Off-the-shelf LLM | 35 | 0.776 | 34 | 1.92 | Not shortlisted |
| Yandex Cloud Translate v2 | Off-the-shelf NMT | 36 | 0.754 | 35 | 2.06 | Not shortlisted |
| AppTek NMT | Off-the-shelf NMT | 37 | 0.635 | 37 | 5.71 | Not shortlisted |
| OpenAI GPT-5.6 Luna | Adjusted LLM | — | 0.829 | — | 1.33 | Same model family as OpenAI GPT-5.6 Sol |
| OpenAI GPT-5.6 Terra | Adjusted LLM | — | 0.827 | — | 1.34 | Same model family as OpenAI GPT-5.6 Sol |
| xAI Grok 4.20 (non-reasoning) | Adjusted LLM | — | 0.820 | — | 1.49 | Same model family as xAI Grok 4.20 (reasoning) |
Every model we got a translation from, as the shortlist compared it: off the shelf, without context in the request, or adjusted, with it. A model that did not make the shortlist is shown adjusted, unless off the shelf it scores higher by 0.03 or more on COMET-Kiwi. Ranks among 37 models, with each model family counted once, and the top 8 of each metric in bold. Higher is better on COMET-Kiwi, lower on MetricX. The scores rank models within this language and are not a requirement result.
The two models COMET-Kiwi ranks highest in Croatian are Google Gemini 3.5 Flash and OpenAI GPT-4.1, and by Intento LQA translations from both have 27.7 weighted errors per 1,000 words, too many for general translation quality. MetricX ranks OpenAI GPT-4.1 23rd, so COMET-Kiwi alone brought it onto the list. Nearest the cut in the tail are xAI Grok 4.20 (reasoning), rank 10 on COMET-Kiwi, and Anthropic Claude Sonnet 5, rank 10 on MetricX.
| Quality level | Cheapest solution that meets every requirement of the level | Cost per million characters |
|---|---|---|
| Basic | Google Gemini 3.1 Pro Preview, one-step LLM adjusted for Standard requirements | $4.46 |
| Standard | Google Gemini 3.1 Pro Preview, one-step LLM adjusted for Standard requirements | $4.46 |
| Advanced | OpenAI GPT-5.5, two-step with the gender-neutrality agent | $45.38 |
| Model | US dollars per million characters | Class |
|---|---|---|
| Google Gemma 4 | $0.13 | Open-weight LLMs |
| MoonshotAI Kimi K2.5 | $1.27 | LLMs |
| z.ai GLM-5 | $2.02 | LLMs |
| OpenAI GPT-4.1 | $2.98 | LLMs |
| Google Gemini 3.5 Flash | $3.3 | LLMs |
| Google Gemini 3.1 Pro Preview | $4.46 | LLMs |
| Anthropic Claude Sonnet 4.6 | $6.38 | LLMs |
| OpenAI GPT-5.6 Sol | $7.49 | LLMs |
| Microsoft Translator | $10 | MT engines |
| OpenAI GPT-5.5 | $10.83 | LLMs |
| Translated Lara 2 | $24.99 | LLM-based MT engines |
| DeepL Next Gen | $27.5 | LLM-based MT engines |
| Anthropic Claude Fable 5 | $28.3 | LLMs |
| Model | Tokens per character | Class |
|---|---|---|
| Anthropic Claude Fable 5 | 0.474 | LLMs |
| Anthropic Claude Opus 4.8 | 0.473 | LLMs |
| Anthropic Claude Sonnet 5 | 0.471 | LLMs |
| Mistral Medium 3.1 | 0.428 | LLMs |
| Mistral Large 3 | 0.417 | LLMs |
| AI21 Jamba Large 1.7 | 0.415 | LLMs |
| Alibaba Qwen3-235B-A22B | 0.375 | Open-weight LLMs |
| DeepSeek V3.2 | 0.368 | LLMs |
| MoonshotAI Kimi K2.5 | 0.367 | LLMs |
| z.ai GLM-5 | 0.364 | LLMs |
| Cohere Command A Translate | 0.362 | LLMs |
| Anthropic Claude Sonnet 4.6 | 0.356 | LLMs |
| Anthropic Claude Haiku 4.5 | 0.355 | LLMs |
| xAI Grok 4.20 (non-reasoning) | 0.337 | LLMs |
| xAI Grok 4.20 (reasoning) | 0.335 | LLMs |
| Google Gemini 3.1 Pro Preview | 0.312 | LLMs |
| MiniMax M2.5 | 0.311 | LLMs |
| Google Gemini 3.5 Flash | 0.309 | LLMs |
| Google Gemma 4 | 0.309 | Open-weight LLMs |
| OpenAI GPT-5.5 | 0.306 | LLMs |
| OpenAI GPT-5.6 Luna | 0.306 | LLMs |
| OpenAI GPT-5.6 Sol | 0.306 | LLMs |
| OpenAI GPT-5.6 Terra | 0.306 | LLMs |
| OpenAI GPT-4.1 | 0.304 | LLMs |
| Cohere Tiny Aya | 0.28 | LLMs |
Meeting the Standard requirements costs the same as meeting the Basic ones in Croatian: the cheapest solution for both is the one on Google Gemini 3.1 Pro Preview, at $4.46 per million characters. The tokenizers produce a median 0.35 tokens per character of Croatian, 1.3 times their count for English. The tokenizers of OpenAI GPT-5.5 and OpenAI GPT-5.6 Sol produce the same 0.306 tokens per character of Croatian, but the solutions on them cost $10.83 and $7.49 per million characters.
| Quality level | Fastest solution that meets every requirement of the level | Seconds per 1,000 words |
|---|---|---|
| Basic | Anthropic Claude Fable 5, one-step LLM adjusted for Standard requirements | 61 |
| Standard | Anthropic Claude Fable 5, one-step LLM adjusted for Standard requirements | 61 |
| Advanced | OpenAI GPT-5.5, two-step with the gender-neutrality agent | 172 |
| Model | Seconds per 1,000 words | Class |
|---|---|---|
| Google Cloud Advanced Translation API | 0.2 | MT engines |
| Microsoft Translator | 0.3 | MT engines |
| ModernMT Enterprise Edition | 0.4 | MT engines |
| Pangeanic PangeaMT | 0.5 | MT engines |
| Google Translation LLM | 0.5 | LLM-based MT engines |
| DeepL Next Gen | 0.6 | LLM-based MT engines |
| Amazon Translate | 1.1 | MT engines |
| Translated Lara 2 | 1.2 | LLM-based MT engines |
| OpenAI gpt-oss-120b | 2.1 | Open-weight LLMs |
| SYSTRAN PNMT API | 2.6 | MT engines |
| Yandex Cloud Translate v2 | 4.4 | MT engines |
| Tilde TildeOpen LLM engine | 7 | LLM-based MT engines |
| Google Gemini 3.5 Flash | 15 | LLMs |
| Supertext LLM | 16 | LLM-based MT engines |
| OpenAI GPT-5.6 Luna | 16 | LLMs |
| Cohere Tiny Aya | 17 | LLMs |
| xAI Grok 4.20 (non-reasoning) | 21 | LLMs |
| OpenAI GPT-5.6 Terra | 27 | LLMs |
| OpenAI GPT-4.1 | 28 | LLMs |
| AppTek NMT | 33 | MT engines |
| MiniMax M2.5 | 38 | LLMs |
| Anthropic Claude Haiku 4.5 | 38 | LLMs |
| Mistral Medium 3.1 | 41 | LLMs |
| Swiss AI Apertus-8B-2509 | 42 | Open-weight LLMs |
| DeepSeek V3.2 | 49 | LLMs |
| Cohere Command A Translate | 50 | LLMs |
| AI21 Jamba Large 1.7 | 56 | LLMs |
| Anthropic Claude Sonnet 5 | 58 | LLMs |
| Mistral Large 3 | 59 | LLMs |
| xAI Grok 4.20 (reasoning) | 60 | LLMs |
| Anthropic Claude Fable 5 | 61 | LLMs |
| OpenAI GPT-5.6 Sol | 62 | LLMs |
| OpenAI GPT-5.5 | 65 | LLMs |
| Anthropic Claude Sonnet 4.6 | 66 | LLMs |
| Anthropic Claude Opus 4.8 | 74 | LLMs |
| Google Gemini 3.1 Pro Preview | 84 | LLMs |
| Alibaba Qwen3-235B-A22B | 134 | Open-weight LLMs |
| Google Gemma 4 | 148 | Open-weight LLMs |
| z.ai GLM-5 | 236 | LLMs |
| MoonshotAI Kimi K2.5 | 258 | LLMs |
Real-time models are those the shortlist counts as real-time, under 2 seconds per text across the languages we timed. The chart shows the time of each model into Croatian.
Of the ten real-time models in Croatian, nine are MT engines or LLM-based MT engines, and the other is the open-weight OpenAI gpt-oss-120b, which is not on the shortlist. None of the three real-time solutions on the shortlist meet general translation quality or formatting. The solutions that meet every Standard requirement take 61 to 84 seconds per 1,000 words. With the gender-neutrality agent, the two that meet every Advanced requirement take 172 seconds per 1,000 words.
| The two best solutions | Solution | Sentences about people that are gender-neutral | Meets every Advanced requirement | Cost per million characters | Seconds per 1,000 words |
|---|---|---|---|---|---|
| OpenAI GPT-5.5 | two-step, with the gender-neutrality agent | 100% | ✓ | $45.38 | 172 |
| OpenAI GPT-5.6 Sol | three-step, with the gender-neutrality agent | 100% | ✓ | $984 | 172 |
For tone of voice in Croatian we check the formal register, with Vi as the form of address. For gender neutrality, the strategy chosen for Croatian is neutral nouns, impersonal forms and recasting, not plurals, which are masculine by rule. The one-step solutions have a median of 6.5 gender-neutrality errors per 1,000 words. With the gender-neutrality agent, 100% of the sentences about people are gender-neutral in the translations from both of the two best solutions, on OpenAI GPT-5.5 and OpenAI GPT-5.6 Sol.
The three LQA judges agree on a median of 96.6% of the sentences of the shortlisted solutions in Croatian, so the Intento LQA results on this page carry weight. The automatic scores carry less, since only one of the two pairs of metrics agrees here. The scores only rank the models for the shortlist, and the weighted errors on this page come from Intento LQA.
Leave feedback