Method

The 2026 State of Translation Automation measures the quality achievable in each target language. We evaluate solutions: a model plus its harness of prompts, glossaries and other context, pre- and post-processing.

The three quality levels define the requirements we check. Basic covers general translation quality. Standard adds terminology, tone of voice and six formatting requirements. Advanced adds gender neutrality.

The benchmark uses four English business texts totaling ~3,000 words and covers 100 target languages. We add complexity when a simpler solution does not meet the requirements.

Our Language engineers run the evaluation using Intento Language Hub. The same team of forward-deployed engineers implements production AI setups for 60+ enterprise customers across hundreds of language pairs.

A solution is a model plus its harness: prompts, glossaries and other context, pre- and post-processing. We use the harness to supply business context and requirements. We implement these harnesses in Intento Language Hub: one-step solutions, workflows with post-editing, LQA and repair, and multi-agent solutions for gender neutrality.

The framework defines eight levels of solution complexity. Fine-tuned models are out of scope in the 2026 edition. One model can form the basis of several solutions, with different harnesses for the requirements we check. We use the same approach when building solutions for customers.

We start with every one-step solution available in each language. Where a language does not have two solutions that meet all the requirements, we build two-step solutions (translation + post-edit) on the one-step solutions that miss general translation quality (no critical error, at most three major errors and at most 20 weighted errors per 1,000 words) or one of the Standard requirements, and repair tags and formatting by code.

In the languages that still lack two such solutions, we build three-step solutions (translation + LQA + repair), with a repair bounded to the issues the LQA found. Finally, we add a gender-neutrality agent to the best solutions with gender-neutrality issues.

See the complexity levels and model configurations →  ·  Explore interactively →

We build the three LLM judges on Claude Opus 5, GPT-5.6 Sol and Gemini 3.1 Pro, using models from three vendors. Each judge receives the source sentence and its translation, without a reference translation, and identifies errors by type and severity using the MQM framework for classifying translation errors. We anonymize the three error lists before combining the findings with Claude Fable 5.

For gender-neutral translations, the LQA agent consults the gender-neutrality agent. We reassess findings on sentences that refer to a person against the relevant gender rule, so required neutral rewording does not count as a translation error.

We assess general translation quality with three LLM judges built on Claude Opus 5, GPT-5.6 Sol and Gemini 3.1 Pro, and combine their findings with Claude Fable 5. For tone of voice, terminology and gender neutrality, we use GPT-5.6 Sol to check each sentence where the requirement applies and Claude Opus 5, from another vendor, to review the findings.

The formatting checks cover tags, Unicode character integrity, locale conventions, HTML entities, valid JSON and spacing at tag boundaries. These checks give the same result for the same translation every time.

The corpus covers four content types: an editorial article on thinking skills, a legal privacy notice for a loyalty program, an institutional mission statement and planning guide for an international education organization, and an operational travel-safety alert.

English often leaves gender unspecified where a language with grammatical gender requires a choice. "Your manager", "the expert" and "a prospective client" must each be given a gender in Spanish, German or Arabic. In these texts that happens constantly, and the source almost never carries the answer: one "she" is the only gendered third-person pronoun in the corpus.

Three kinds of words are marked for gender in a grammatical-gender language: generic role references, generic second-person address and generic references to people. The four texts carry them unevenly. The editorial article has 35 generic human references and 20 second-person references in 969 words. The privacy notice has 58 second-person references in ~1,000 words, plus a corporate "We" that is marked for gender in some Slavic languages.

The education text holds ten personal names, one of them the referent of that single "she". The travel alert contains mostly instructions and few references to people. Gender neutrality is evaluated only on segments with a human referent, ~115 of the 177 segments the texts are split into.

See details →

Before ranking solutions, we check whether the metrics provide a reliable assessment in that language. A plausible-looking score alone does not establish reliability.

We use two pairs of automatic metrics, two semantic-similarity metrics and two quality-estimation metrics, and the two metrics in a pair are built on different underlying models. A language qualifies when the two metrics of either pair agree with each other at least as well as they do on a set of reference languages.

For 96 of the 100 languages at least one pair agrees. For the other four, Haitian Creole, Maltese, Turkmen and Uyghur, the similarity pair falls short, or for Maltese is on the threshold, and the quality-estimation pair cannot be applied: one of its models does not support Haitian Creole, Maltese and Turkmen, and the other does not cover Turkmen and Uyghur.

For those four languages we publish no ranking and run no Intento LQA evaluation. The result for them is "not reliably measurable": no two independent metrics confirm each other there, so these metrics do not provide independent confirmation for a quality ranking.

See details →  ·  Explore interactively →

We measure agreement by how similarly two metrics rank translated sentences. For each metric pair, we set the threshold 0.05 below the lowest agreement among the reference languages, counting identical translations of a sentence once. A language qualifies when the lower end of the 90% confidence interval is above the threshold.

Both pairs of metrics agree in 73 languages. In 12, only one pair applies, and it agrees. In 11, one pair agrees and the other does not. COMET-Kiwi and MetricX agree less in Indic languages, and least of all in Gujarati. MetricX has no direct support for Croatian, Bosnian and Latin-script Serbian and scores them through Serbian. There, its agreement with COMET-Kiwi is among the lowest. Maltese is right at the threshold, and we leave it out of the ranking.

In 15 of these languages, the three LQA judges disagree more often: for 50% or more of the shortlisted solutions, they agree on fewer than 90% of the sentences, and in Igbo, Malagasy, Xhosa and Zulu for all of them.

For each language, we compare solutions using confidence intervals over the scored segments. Solutions share a tier where the intervals overlap, and we show an order only where the data supports a difference. These tiers group translation solutions within a language.

For each sentence, we take the most severe error each judge flags (none, minor, major or critical), and we count agreement when the three severity ratings are at most one level apart. Two judges can name the same error differently, so we compare severity only. The judges agree less where translations have more errors.

In 14 of the 96 languages, fewer than 50% of the shortlisted solutions pass this check, so we cannot assess quality there. These are Hausa, Igbo, Javanese, Kinyarwanda, Kurdish (Kurmanji), Malagasy, Scottish Gaelic, Sesotho, Shona, Somali, Wolof, Xhosa, Yoruba and Zulu, and in Igbo, Malagasy, Xhosa and Zulu no solution passes.

In these 14 languages, we can still choose the best solution, but cannot precisely assess its quality. Together with Haitian Creole, Maltese, Turkmen and Uyghur, where we cannot assess quality at all, that makes 18 languages.

Gender-neutral translation rewords some sentences, for example "the teaching staff" for "each teacher", and a standard LQA flags such rewording as a mistranslation or an omission. So for gender neutrality, the LQA agent consults the gender-neutrality agent. With this adjusted LQA, 17% of the errors found by LQA on the gender-neutral translations turn out to be the required rewording, and they make up ~60% of the loss in accuracy against the original translations.

To test the adjusted LQA, we also ran it on sentences that refer to no person, where the gender rule changes nothing. There, it kept 98% to 100% of the errors.

When we translate the source text again with the gender-neutrality agent, the sentences that refer to a person have ~11% more errors, weighted by severity, than in the original translation, while the other sentences have about as many as before. That ~11% is the remaining accuracy difference in this comparison.

A solution meets general translation quality with no critical error, at most three major errors and at most 20 weighted errors per 1,000 words. Where a language has fewer than two such candidates, we also take one-step solutions that miss general translation quality and one other requirement. We leave out solutions that miss general translation quality and two or more other requirements because they fall short on too many requirements, and that includes 99% of the MT engine solutions on our shortlists.

In two-step solutions, we use four different automatic post-editing (APE) steps, depending on the missing requirements: one for general translation quality, one for tone of voice, one for terminology and one for general translation quality together with tone of voice or terminology, and we repair tags and formatting by code. We choose the APE step in advance, based on the known flaws of the particular AI model in the language.

If a language still has fewer than two solutions that meet all the requirements, we build three-step solutions on the same one-step solutions and choose the APE dynamically, from the Intento LQA findings on each translation.