Method

How the benchmark was run: source texts, languages, quality levels, automatic scoring, LLM-as-a-judge review and the selection of solutions.

Takeaways for , , , and

Takeaway 1

We assess solutions built from 57 AI models and MT engines against three levels of translation quality.

The 2026 State of Translation Automation measures the quality achievable in each target language. We evaluate solutions: a model plus its harness of prompts, glossaries and other context, pre- and post-processing.

The three quality levels define the requirements we check. Basic covers general translation quality. Standard adds terminology, tone of voice and six formatting requirements. Advanced adds gender neutrality.

The benchmark uses four English business texts totaling ~3,000 words and covers 100 target languages. We add complexity when a simpler solution does not meet the requirements.

Our Language engineers run the evaluation using Intento Language Hub. The same team of forward-deployed engineers implements production AI setups for 60+ enterprise customers across hundreds of language pairs.

38
vendors
57
models
129
model-and-harness configurations
3
quality levels
3,000
source words
100
target languages
Takeaway 2

We measure which translation requirements solutions can meet in each language, and what it takes to meet them.

A metric score alone does not show which requirements a translation meets or what still needs improvement. We assess quality against the Basic, Standard and Advanced requirements, together with confidence in the assessment, solution complexity, cost and latency.

Takeaway 3

We add complexity to a translation solution only when a simpler solution cannot meet the requirements.

A solution is a model plus its harness: prompts, glossaries and other context, pre- and post-processing. We use the harness to supply business context and requirements. We implement these harnesses in Intento Language Hub: one-step solutions, workflows with post-editing, LQA and repair, and multi-agent solutions for gender neutrality.

The framework defines eight levels of solution complexity. Fine-tuned models are out of scope in the 2026 edition. One model can form the basis of several solutions, with different harnesses for the requirements we check. We use the same approach when building solutions for customers.

We start with every one-step solution available in each language. Where a language does not have two solutions that meet all the requirements, we build two-step solutions (translation + post-edit) on the one-step solutions that miss general translation quality (no critical error, at most three major errors and at most 20 weighted errors per 1,000 words) or one of the Standard requirements, and repair tags and formatting by code.

In the languages that still lack two such solutions, we build three-step solutions (translation + LQA + repair), with a repair bounded to the issues the LQA found. Finally, we add a gender-neutrality agent to the best solutions with gender-neutrality issues.

See the complexity levels and model configurations →  ·  Explore interactively →

Eight levels of solution complexityNMT solutions1Off-the-shelfNMTA neural machinetranslation engine withdefault settings and noglossary.2Adjusted NMTAn NMT solution withglossaries, tone of voice,tag handling and othersettings supported throughthe API.LLM solutions3Off-the-shelfLLMAn LLM with a simpletranslation prompt.4Adjusted LLMAn LLM with glossaryretrieval and promptstailored to tone of voice,tags and otherrequirements.Fine-tuned models5Fine-tuned modelsNMT domain adaptation or LLM fine-tuning, combined withother customization features. Out of scope in 2026.Solutions with multiple steps or agents6Two-stepsolutionA translation step usingNMT or an LLM, followed byautomatic post-editing withan LLM to address unmetrequirements.7Multi-stepMore than two steps in oneworkflow: translation,validation and repair.8Multi-agent systemsAgents responsible for different groups of requirements worktogether to meet all of them.We add complexity only when a simplersolution cannot meet the requirements.
Eight levels of solution complexity
Takeaway 4

We aim to find two solutions based on different models for each language: a main solution and a backup.

Both solutions must meet all the requirements we check. The backup keeps translation available when the main solution is unavailable.

We add complexity only when a simpler solution misses a requirement, and stop once we have two qualifying solutions for the language. A gender-neutral version uses the same model as the original solution, so we count it as a version of that solution rather than a separate backup.

Takeaway 5

We shortlist solutions with quality-estimation metrics, then assess them against translation requirements with Intento LQA.

Through the Intento Language Hub API, we obtain COMET-Kiwi and MetricX scores for language assessment and shortlisting. We use these two low-cost quality-estimation metrics to select 10 to 13 solutions per language for further assessment. They estimate accuracy and fluency, but do not check whether the translation meets terminology, tone-of-voice or gender-neutrality requirements.

A higher metric score does not always mean better compliance. In the glossary comparison, the COMET score fell in most cases where the translation used the approved terminology.

We then assess the shortlisted solutions against the 10 requirements grouped into Basic, Standard and Advanced, using Intento LQA. This automatic assessment combines LLM checks with checks in code.

See the quality checks →

Takeaway 6

We assess general translation quality with three LLM judges and combine their findings with a fourth model.

We build the three LLM judges on Claude Opus 5, GPT-5.6 Sol and Gemini 3.1 Pro, using models from three vendors. Each judge receives the source sentence and its translation, without a reference translation, and identifies errors by type and severity using the MQM framework for classifying translation errors. We anonymize the three error lists before combining the findings with Claude Fable 5.

For gender-neutral translations, the LQA agent consults the gender-neutrality agent. We reassess findings on sentences that refer to a person against the relevant gender rule, so required neutral rewording does not count as a translation error.

Intento LQA assessment of general translation qualitySourcesentencesandtranslationsJudge 1Claude Opus 5Judge 2GPT-5.6 SolJudge 3Gemini 3.1 ProJudgeagreementAt least 90% ofsegmentsCombinefindingsClaude Fable 5Reassess errors incontextConsult thegender-neutralityagentFor gender-neutraltranslationsWeighted errors per1,000 wordsThe judges assess each source sentence and its translation without a reference translation. Sentences that all three judges find error-freeskip the merge. The final score weights errors by severity. Dashed connection: Consultation for gender-neutral translations.Company, product and model names and logos are shown for informational purposes only. All trademarks and logosbelong to their respective owners. Their use does not imply affiliation or endorsement.
Intento LQA assessment of general translation quality
Takeaway 7

We use LLM checks for four linguistic requirements and code to check the six formatting requirements.

We assess general translation quality with three LLM judges built on Claude Opus 5, GPT-5.6 Sol and Gemini 3.1 Pro, and combine their findings with Claude Fable 5. For tone of voice, terminology and gender neutrality, we use GPT-5.6 Sol to check each sentence where the requirement applies and Claude Opus 5, from another vendor, to review the findings.

The formatting checks cover tags, Unicode character integrity, locale conventions, HTML entities, valid JSON and spacing at tag boundaries. These checks give the same result for the same translation every time.

Assessment methods for the 10 translation requirementsGeneral translation qualityTone of voiceTerminologyGender neutralityTagsUnicode character integrityLocale conventionsHTML entitiesValid JSONSpacing at tag boundariesLLM assessmentThree judges; a fourth model combines findingsLLM assessmentOne model checks; another reviewsChecks in codeRepeatable rulesTerminology also uses exact matching; LLM checks resolve inconclusive matches.
Assessment methods for the 10 translation requirements
Takeaway 8

Enterprise localization teams contributed the source texts with permission to use them for evaluation.

The teams supplied terminology, tone-of-voice, tagging and formatting requirements with the texts. We use the content only for evaluation and only with AI models and MT engines covered by a guarantee that the content will not be used for training.

Takeaway 9

We use the same English source texts across target languages so comparisons use the same source material.

The corpus covers four content types: an editorial article on thinking skills, a legal privacy notice for a loyalty program, an institutional mission statement and planning guide for an international education organization, and an operational travel-safety alert.

4
source texts
2,978
source words
100
target languages
Takeaway 10

The four content types contain different features to preserve in translation.

The editorial article is the only source text with idiomatic English. Expressions such as “at face value”, “off-the-shelf” and “on autopilot” have meanings beyond the literal meanings of the individual words; three idioms appear in the glossary.

The education text uses a formal institutional register, with direct questions to the reader and personal names to preserve. The travel-safety alert contains agency acronyms, place names, dates and counts, with direct instructions in 10 of its 17 sentences.

The privacy notice contains legal wording and a sentence of 56 words. It uses capitalized “We”, “Our” and “Us” for contract-defined parties, a convention absent from most target languages.

Takeaway 11

The English source texts almost never specify gender, so we can assess gender neutrality in translation.

English often leaves gender unspecified where a language with grammatical gender requires a choice. "Your manager", "the expert" and "a prospective client" must each be given a gender in Spanish, German or Arabic. In these texts that happens constantly, and the source almost never carries the answer: one "she" is the only gendered third-person pronoun in the corpus.

Three kinds of words are marked for gender in a grammatical-gender language: generic role references, generic second-person address and generic references to people. The four texts carry them unevenly. The editorial article has 35 generic human references and 20 second-person references in 969 words. The privacy notice has 58 second-person references in ~1,000 words, plus a corporate "We" that is marked for gender in some Slavic languages.

The education text holds ten personal names, one of them the referent of that single "she". The travel alert contains mostly instructions and few references to people. Gender neutrality is evaluated only on segments with a human referent, ~115 of the 177 segments the texts are split into.

See details →

Second-person references per text0204060Editorial article: 2020Editorial articleEducation guide: 1212Education guideTravel alert: 11Travel alertPrivacy notice: 5858Privacy noticeOccurrences of you, your, yours, yourself and yourselves in the plain text of each document.Each one takes a gender or a register choice in many target languages.
Second-person references per text
Takeaway 12

All 46 glossary terms occur in the source texts, covering a different kind of terminology in each content type.

The glossary draws on all four source texts: 12, 12, 9 and 13 per text, 121 occurrences in 2,970 words, one every 25 words.

The kinds of terms differ. The editorial article carries a four-part framework whose terms are defined inside the text, so the terms are coined and the test is consistency. The education text carries terms with official renderings in the termbase the organization maintains.

The travel alert carries operational terms, including a colour-coded severity scale whose levels must stay parallel. The privacy notice carries statutory terms, each with one official rendering per language in EU law, plus contract-defined names that must stay capitalized and paired. We assess terminology against these defined or approved forms.

See details →

Takeaway 13

Native-speaker review found two context errors in the glossary generated with Claude Opus.

We generated the 46 terms per language with Claude Opus from the four source texts, and native-speaker reviewers then checked the lists for the 16 languages. In both corrections the target term repeated a word that is already next to it in the sentence, an organization name in one case and a legal phrase in the other. Reviewers in several languages caught it independently. We shortened the targets to the term itself, in all 16 languages for the first term and in 15 for the second, where the Estonian target still carries the extra words.

Takeaway 14

We compare terminology within each language because reviewed glossaries cover only 16 of the 100 languages.

We built glossaries only for languages with an available reviewer inside Intento. Including terminology in a cross-language comparison would therefore reflect reviewer availability as well as translation performance.

See details →

Takeaway 15

We translate each source text separately, using sentence-level requests where API limits prevent full-document requests.

In a merged document, terminology and tone from one text can appear in the translation of the next, so a score in the report is always the score of one document. For five of the 57 models, a whole document exceeds the request size limit or the time-out. To those models, we send the texts as 177 sentence-level pieces, because otherwise we would score the request limit, not the translation.

If a request still fails after automatic retries and another run on a later date, we keep the error text in place of the translation, so we can tell an error apart from missing language support. Errors remain for four models. From ALL-Lab we got time-outs in eight of its 14 languages: Amharic, Igbo, Malagasy, Sesotho, Shona, Xhosa, Yoruba and Zulu.

From Widn AI Tower Anthill and Widn AI Tower Vesuvius we got errors in 16 languages: Arabic (MSA), Bulgarian, Catalan, Croatian, Estonian, Greek, Hebrew, Indonesian, Latvian, Lithuanian, Malay, Slovak, Slovenian, Thai, Turkish and Vietnamese. In these languages all our requests to the two models carried the tone of voice setting, and for Arabic (MSA) and Estonian also the glossary. The errors from Tilde are in three of its 31 languages: Lithuanian, Maltese and Swedish.

Takeaway 16

We check translation length before scoring to detect cut-off or nearly empty texts.

From 13 of the 57 models, we got texts cut off part-way, from two other models error messages in place of the translation, and from one more a refusal to translate. Normal length differs by language, so we flag a text shorter than 50% of the median from the other models. We check each of the four texts on its own, because with one text cut short, the average of the four often stays above the threshold.

If two or more texts are cut off or nearly empty, we exclude the translation from scoring. If only one is affected, we score the translation and assign the lowest automatic-metric values to the missing text. A language where we exclude every translation of a model counts as nominally supported for that model.

See details →

Takeaway 17

We combine sentence-level and full-text checks into one quality assessment for each solution in each language.

Translation quality depends on the whole text, while we evaluate general translation quality, tone of voice, gender neutrality and terminology on a segment level. In a score per document, a critical error in one sentence disappears into the average of the rest. We need a data point per sentence for a confidence interval, and requirements such as tone of voice and terminology are sentence-level by nature.

On the full text with its markup, we check tags, HTML entities and spacing at tag boundaries. We also check that the translation of the JSON text is valid JSON. Over all four texts, we count the translation errors per 1,000 source words, weighted by severity. Where we get two source sentences back merged into one, or one split into three, we still map the translation onto the source segments.

See details →

Takeaway 18

We rank solutions only in languages where at least one pair of independent metrics meets the agreement threshold.

Before ranking solutions, we check whether the metrics provide a reliable assessment in that language. A plausible-looking score alone does not establish reliability.

We use two pairs of automatic metrics, two semantic-similarity metrics and two quality-estimation metrics, and the two metrics in a pair are built on different underlying models. A language qualifies when the two metrics of either pair agree with each other at least as well as they do on a set of reference languages.

For 96 of the 100 languages at least one pair agrees. For the other four, Haitian Creole, Maltese, Turkmen and Uyghur, the similarity pair falls short, or for Maltese is on the threshold, and the quality-estimation pair cannot be applied: one of its models does not support Haitian Creole, Maltese and Turkmen, and the other does not cover Turkmen and Uyghur.

For those four languages we publish no ranking and run no Intento LQA evaluation. The result for them is "not reliably measurable": no two independent metrics confirm each other there, so these metrics do not provide independent confirmation for a quality ranking.

See details →  ·  Explore interactively →

Metric-pair agreement per languageAfrikaans (af): both pairs agreeafAfrikaansAmharic (am): both pairs agreeamAmharicArabic (MSA) (ar): both pairs agreearArabic (MS…Armenian (hy): both pairs agreehyArmenianAzerbaijani (az): both pairs agreeazAzerbaijaniBasque (eu): both pairs agreeeuBasqueBengali (bn): both pairs agreebnBengaliBosnian (bs): both pairs agreebsBosnianBulgarian (bg): both pairs agreebgBulgarianBurmese (my): both pairs agreemyBurmeseCatalan (ca): both pairs agreecaCatalanChinese Simplified (zh): both pairs agreezhChinese Si…Chinese Traditional (zh-hant): both pairs agreezh-hantChinese Tr…Czech (cs): both pairs agreecsCzechDanish (da): both pairs agreedaDanishDutch (nl): both pairs agreenlDutchEstonian (et): both pairs agreeetEstonianFarsi (fa): both pairs agreefaFarsiFinnish (fi): both pairs agreefiFinnishFrench (fr): both pairs agreefrFrenchGeorgian (ka): both pairs agreekaGeorgianGerman (de): both pairs agreedeGermanGreek (el): both pairs agreeelGreekHausa (ha): both pairs agreehaHausaHebrew (he): both pairs agreeheHebrewHindi (hi): both pairs agreehiHindiHungarian (hu): both pairs agreehuHungarianIcelandic (is): both pairs agreeisIcelandicIndonesian (id): both pairs agreeidIndonesianIrish (ga): both pairs agreegaIrishItalian (it): both pairs agreeitItalianJapanese (ja): both pairs agreejaJapaneseJavanese (jv): both pairs agreejvJavaneseKannada (kn): both pairs agreeknKannadaKazakh (kk): both pairs agreekkKazakhKhmer (km): both pairs agreekmKhmerKorean (ko): both pairs agreekoKoreanKurdish (Kurmanji) (ku): both pairs agreekuKurdish (K…Kyrgyz (ky): both pairs agreekyKyrgyzLatvian (lv): both pairs agreelvLatvianLithuanian (lt): both pairs agreeltLithuanianMacedonian (mk): both pairs agreemkMacedonianMalagasy (mg): both pairs agreemgMalagasyMalay (ms): both pairs agreemsMalayMalayalam (ml): both pairs agreemlMalayalamMarathi (mr): both pairs agreemrMarathiMongolian (mn): both pairs agreemnMongolianNepali (ne): both pairs agreeneNepaliNorwegian (no): both pairs agreenoNorwegianPashto (ps): both pairs agreepsPashtoPolish (pl): both pairs agreeplPolishPortuguese (European) (pt): both pairs agreeptPortuguese…Punjabi (pa): both pairs agreepaPunjabiRomanian (ro): both pairs agreeroRomanianRussian (ru): both pairs agreeruRussianSinhala (si): both pairs agreesiSinhalaSlovak (sk): both pairs agreeskSlovakSlovenian (sl): both pairs agreeslSlovenianSomali (so): both pairs agreesoSomaliSpanish (European) (es): both pairs agreeesSpanish (E…Swahili (sw): both pairs agreeswSwahiliSwedish (sv): both pairs agreesvSwedishTagalog (tl): both pairs agreetlTagalogTamil (ta): both pairs agreetaTamilTelugu (te): both pairs agreeteTeluguThai (th): both pairs agreethThaiTurkish (tr): both pairs agreetrTurkishUkrainian (uk): both pairs agreeukUkrainianUrdu (ur): both pairs agreeurUrduUzbek (uz): both pairs agreeuzUzbekVietnamese (vi): both pairs agreeviVietnameseWelsh (cy): both pairs agreecyWelshYiddish (yi): both pairs agreeyiYiddishAssamese (as): one pair agrees, the other cannot be appliedasAssameseCebuano (ceb): one pair agrees, the other cannot be appliedcebCebuanoIgbo (ig): one pair agrees, the other cannot be appliedigIgboKinyarwanda (rw): one pair agrees, the other cannot be appliedrwKinyarwandaOdia (Oriya) (or): one pair agrees, the other cannot be appliedorOdia (Oriy…Sesotho (st): one pair agrees, the other cannot be appliedstSesothoShona (sn): one pair agrees, the other cannot be appliedsnShonaTajik (tg): one pair agrees, the other cannot be appliedtgTajikTatar (tt): one pair agrees, the other cannot be appliedttTatarWolof (wo): one pair agrees, the other cannot be appliedwoWolofYoruba (yo): one pair agrees, the other cannot be appliedyoYorubaZulu (zu): one pair agrees, the other cannot be appliedzuZuluAlbanian (sq): one pair agrees, the other disagreessqAlbanianBelarusian (be): one pair agrees, the other disagreesbeBelarusianCroatian (hr): one pair agrees, the other disagreeshrCroatianFrisian (fy): one pair agrees, the other disagreesfyFrisianGalician (gl): one pair agrees, the other disagreesglGalicianGujarati (gu): one pair agrees, the other disagreesguGujaratiLao (lo): one pair agrees, the other disagreesloLaoScottish Gaelic (gd): one pair agrees, the other disagreesgdScottish G…Serbian (sr-latn): one pair agrees, the other disagreessr-latnSerbianSundanese (su): one pair agrees, the other disagreessuSundaneseXhosa (xh): one pair agrees, the other disagreesxhXhosaHaitian Creole (ht): not reliably measurablehtHaitian Cr…Maltese (mt): not reliably measurablemtMalteseTurkmen (tk): not reliably measurabletkTurkmenUyghur (ug): not reliably measurableugUyghurboth pairs agree (73)one pair agrees, the other cannot be applied (12)one pair agrees, the other disagrees (11)not reliably measurable (4)100 target languages. A language is measurable when the two metrics of either pair agree.
Metric-pair agreement per language
Takeaway 19

We set metric-agreement thresholds using 16 high-resource languages, then apply them to the other 84.

We measure agreement by how similarly two metrics rank translated sentences. For each metric pair, we set the threshold 0.05 below the lowest agreement among the reference languages, counting identical translations of a sentence once. A language qualifies when the lower end of the 90% confidence interval is above the threshold.

Metric agreement and thresholds in 16 reference languagesSemantic similarity: LaBSE and Cohere embed v3Quality estimation: COMET-Kiwi XL and MetricX-24Threshold for semantic similarityThreshold for quality estimation0.400.450.500.550.600.650.70Spanish (European)Spanish (European), Semantic similarity: LaBSE and Cohere embed v3: 0.620Spanish (European), Quality estimation: COMET-Kiwi XL and MetricX-24: 0.650GermanGerman, Semantic similarity: LaBSE and Cohere embed v3: 0.618German, Quality estimation: COMET-Kiwi XL and MetricX-24: 0.621FrenchFrench, Semantic similarity: LaBSE and Cohere embed v3: 0.648French, Quality estimation: COMET-Kiwi XL and MetricX-24: 0.607Chinese SimplifiedChinese Simplified, Semantic similarity: LaBSE and Cohere embed v3: 0.605Chinese Simplified, Quality estimation: COMET-Kiwi XL and MetricX-24: 0.627ItalianItalian, Semantic similarity: LaBSE and Cohere embed v3: 0.666Italian, Quality estimation: COMET-Kiwi XL and MetricX-24: 0.598Arabic (MSA)Arabic (MSA), Semantic similarity: LaBSE and Cohere embed v3: 0.592Arabic (MSA), Quality estimation: COMET-Kiwi XL and MetricX-24: 0.618RussianRussian, Semantic similarity: LaBSE and Cohere embed v3: 0.591Russian, Quality estimation: COMET-Kiwi XL and MetricX-24: 0.659DutchDutch, Semantic similarity: LaBSE and Cohere embed v3: 0.602Dutch, Quality estimation: COMET-Kiwi XL and MetricX-24: 0.591CzechCzech, Semantic similarity: LaBSE and Cohere embed v3: 0.590Czech, Quality estimation: COMET-Kiwi XL and MetricX-24: 0.599SwedishSwedish, Semantic similarity: LaBSE and Cohere embed v3: 0.575Swedish, Quality estimation: COMET-Kiwi XL and MetricX-24: 0.666HindiHindi, Semantic similarity: LaBSE and Cohere embed v3: 0.551Hindi, Quality estimation: COMET-Kiwi XL and MetricX-24: 0.584TurkishTurkish, Semantic similarity: LaBSE and Cohere embed v3: 0.545Turkish, Quality estimation: COMET-Kiwi XL and MetricX-24: 0.619JapaneseJapanese, Semantic similarity: LaBSE and Cohere embed v3: 0.534Japanese, Quality estimation: COMET-Kiwi XL and MetricX-24: 0.672Portuguese (European)Portuguese (European), Semantic similarity: LaBSE and Cohere embed v3: 0.529Portuguese (European), Quality estimation: COMET-Kiwi XL and MetricX-24: 0.630PolishPolish, Semantic similarity: LaBSE and Cohere embed v3: 0.529Polish, Quality estimation: COMET-Kiwi XL and MetricX-24: 0.589KoreanKorean, Semantic similarity: LaBSE and Cohere embed v3: 0.511Korean, Quality estimation: COMET-Kiwi XL and MetricX-24: 0.689Agreement is the Spearman rank correlation of the two metrics over the unique translations of each sentence, inthe translations of the four texts from the 57 models, without and with context in the request. Each threshold is0.05 below the lowest agreement of its pair.Company, product and model names and logos are shown for informational purposes only. Alltrademarks and logos belong to their respective owners. Their use does not imply affiliation orendorsement.
Metric agreement and thresholds in 16 reference languages
Takeaway 20

Metric agreement varies across the 96 qualifying languages, so confidence in the assessment varies too.

Both pairs of metrics agree in 73 languages. In 12, only one pair applies, and it agrees. In 11, one pair agrees and the other does not. COMET-Kiwi and MetricX agree less in Indic languages, and least of all in Gujarati. MetricX has no direct support for Croatian, Bosnian and Latin-script Serbian and scores them through Serbian. There, its agreement with COMET-Kiwi is among the lowest. Maltese is right at the threshold, and we leave it out of the ranking.

In 15 of these languages, the three LQA judges disagree more often: for 50% or more of the shortlisted solutions, they agree on fewer than 90% of the sentences, and in Igbo, Malagasy, Xhosa and Zulu for all of them.

Agreement between COMET-Kiwi and MetricX, by languageIndic languagesCroatian, Bosnian and Latin-script SerbianOther languagesThreshold for quality estimation00.20.40.60.8Farsi: 0.7242FarsiKhmer: 0.7154KhmerBurmese: 0.7104BurmeseSinhala: 0.7091SinhalaLao: 0.6925LaoKorean: 0.6893KoreanGeorgian: 0.6763GeorgianAmharic: 0.676AmharicUrdu: 0.6746UrduLithuanian: 0.6739LithuanianBelarusian: 0.6736BelarusianBasque: 0.6723BasqueJapanese: 0.6722JapaneseArmenian: 0.6693ArmenianAzerbaijani: 0.6683AzerbaijaniMalayalam: 0.6679MalayalamIrish: 0.6676IrishSwedish: 0.6658SwedishGreek: 0.6653GreekBulgarian: 0.6614BulgarianRussian: 0.6591RussianPashto: 0.658PashtoThai: 0.6571ThaiVietnamese: 0.6563VietnameseKurdish (Kurmanji): 0.655Kurdish (Kurmanji)Indonesian: 0.6511IndonesianSpanish (European): 0.6503Spanish (European)Hungarian: 0.6479HungarianKyrgyz: 0.6448KyrgyzCatalan: 0.644CatalanSlovak: 0.6396SlovakSlovenian: 0.6396SlovenianFinnish: 0.6392FinnishUkrainian: 0.6372UkrainianUzbek: 0.6352UzbekMalay: 0.6342MalayScottish Gaelic: 0.6341Scottish GaelicEstonian: 0.6306EstonianNorwegian: 0.6302NorwegianMongolian: 0.63MongolianPortuguese (European): 0.6298Portuguese (European)Afrikaans: 0.6291AfrikaansKazakh: 0.6272KazakhChinese Simplified: 0.6267Chinese SimplifiedHebrew: 0.6211HebrewRomanian: 0.621RomanianGerman: 0.6208GermanAlbanian: 0.6197AlbanianTamil: 0.6194TamilTurkish: 0.6187TurkishArabic (MSA): 0.6177Arabic (MSA)Somali: 0.6152SomaliMacedonian: 0.6132MacedonianIcelandic: 0.613IcelandicChinese Traditional: 0.6129Chinese TraditionalLatvian: 0.6087LatvianSwahili: 0.6081SwahiliFrench: 0.6068FrenchDanish: 0.6036DanishCzech: 0.5988CzechItalian: 0.598ItalianYiddish: 0.5941YiddishDutch: 0.5906DutchGalician: 0.5897GalicianPolish: 0.5888PolishNepali: 0.5855NepaliHindi: 0.5837HindiKannada: 0.5826KannadaSundanese: 0.5768SundaneseMalagasy: 0.5722MalagasyFrisian: 0.571FrisianBengali: 0.5692BengaliJavanese: 0.568JavaneseSerbian: 0.5678SerbianHausa: 0.5671HausaBosnian: 0.5571BosnianPunjabi: 0.5535PunjabiWelsh: 0.5503WelshMarathi: 0.5448MarathiTagalog: 0.54TagalogTelugu: 0.5344TeluguCroatian: 0.4928CroatianXhosa: 0.4034XhosaGujarati: 0.3629GujaratiAgreement is the Spearman rank correlation of COMET-Kiwi XL and MetricX-24 over the unique translations of each sentence, in the 84 languages with both metrics. Languages below the threshold qualify throughsemantic similarity.
Agreement between COMET-Kiwi and MetricX, by language
Takeaway 21

We group solutions into quality tiers when their scores are too close to distinguish statistically.

For each language, we compare solutions using confidence intervals over the scored segments. Solutions share a tier where the intervals overlap, and we show an order only where the data supports a difference. These tiers group translation solutions within a language.

Solutions grouped by statistically distinguishable qualitySolution A: 93 (90.5–95.5)Solution ASolution B: 91.5 (89–94)Solution BSolution C: 86 (84–88)Solution CSolution D: 79 (76.5–81.5)Solution Done tier: intervals overlapscoreIllustration. Four solutions with their scores and confidence intervals: the intervals ofthe first two overlap, so they share a tier; the other two stand apart.
Solutions grouped by statistically distinguishable quality
Takeaway 22

COMET-Kiwi and MetricX produce similar rankings, but their top-10 lists share only ~7 models.

In the 16 high-resource languages, their rank correlation is ~0.89, and the first model on each list is always on the other list. The lists differ at the bottom. So we put the top-8 solutions of each metric on the shortlist, and 17% of the shortlist comes from MetricX alone. In Intento LQA, these solutions score as high on general translation quality as those only in the COMET-Kiwi top-8, and solutions in both top-8 lists score higher.

Takeaway 23

We accept an Intento LQA result only when the three LLM judges agree on at least 90% of sentences.

For each sentence, we take the most severe error each judge flags (none, minor, major or critical), and we count agreement when the three severity ratings are at most one level apart. Two judges can name the same error differently, so we compare severity only. The judges agree less where translations have more errors.

In 14 of the 96 languages, fewer than 50% of the shortlisted solutions pass this check, so we cannot assess quality there. These are Hausa, Igbo, Javanese, Kinyarwanda, Kurdish (Kurmanji), Malagasy, Scottish Gaelic, Sesotho, Shona, Somali, Wolof, Xhosa, Yoruba and Zulu, and in Igbo, Malagasy, Xhosa and Zulu no solution passes.

Shortlisted solutions meeting the judge-agreement threshold, by languageLanguages we assessLanguages we cannot assessHalf of the shortlisted solutions020406080100Igbo: 0IgboMalagasy: 0MalagasyZulu: 0ZuluXhosa: 0XhosaSesotho: 8SesothoWolof: 8WolofYoruba: 8YorubaKinyarwanda: 9KinyarwandaShona: 9ShonaSomali: 9SomaliJavanese: 15JavaneseScottish Gaelic: 25Scottish GaelicHausa: 27HausaKurdish (Kurmanji): 33Kurdish (Kurmanji)Cebuano: 50CebuanoBurmese: 55BurmeseYiddish: 58YiddishIcelandic: 62IcelandicFrisian: 69FrisianSwahili: 73SwahiliTatar: 73TatarKyrgyz: 73KyrgyzUzbek: 73UzbekAssamese: 73AssameseSundanese: 75SundaneseKazakh: 75KazakhKhmer: 75KhmerMongolian: 80MongolianPashto: 80PashtoAzerbaijani: 82AzerbaijaniMalay: 82MalayBasque: 83BasqueIrish: 83IrishLithuanian: 83LithuanianSinhala: 83SinhalaJapanese: 85JapaneseGeorgian: 91GeorgianLao: 91LaoOdia (Oriya): 91Odia (Oriya)Armenian: 91ArmenianNepali: 91NepaliPortuguese (European): 91Portuguese (European)Tajik: 92TajikHebrew: 92HebrewGujarati: 92GujaratiThai: 92ThaiDanish: 92DanishChinese Traditional: 92Chinese TraditionalAmharic: 100AmharicKannada: 100KannadaMarathi: 100MarathiWelsh: 100WelshGerman: 100GermanMalayalam: 100MalayalamTelugu: 100TeluguArabic (MSA): 100Arabic (MSA)Belarusian: 100BelarusianSpanish (European): 100Spanish (European)Tagalog: 100TagalogTurkish: 100TurkishHungarian: 100HungarianVietnamese: 100VietnameseEstonian: 100EstonianIndonesian: 100IndonesianMacedonian: 100MacedonianPunjabi: 100PunjabiTamil: 100TamilUkrainian: 100UkrainianChinese Simplified: 100Chinese SimplifiedPolish: 100PolishRomanian: 100RomanianUrdu: 100UrduBengali: 100BengaliCatalan: 100CatalanFarsi: 100FarsiFinnish: 100FinnishGalician: 100GalicianHindi: 100HindiItalian: 100ItalianKorean: 100KoreanLatvian: 100LatvianDutch: 100DutchRussian: 100RussianAlbanian: 100AlbanianSerbian: 100SerbianSwedish: 100SwedishNorwegian: 100NorwegianBulgarian: 100BulgarianCroatian: 100CroatianSlovak: 100SlovakAfrikaans: 100AfrikaansBosnian: 100BosnianCzech: 100CzechSlovenian: 100SlovenianGreek: 100GreekFrench: 100FrenchShare of the shortlisted one-step solutions in each of the 96 measurable languages, in %. For each sentence, the judges agree when the most severe errors they flag (none, minor, major or critical) are atmost one level apart.
Shortlisted solutions meeting the judge-agreement threshold, by language
Takeaway 24

We assess gender-neutral rewording against the gender rules before treating it as a translation error.

Gender-neutral translation rewords some sentences, for example "the teaching staff" for "each teacher", and a standard LQA flags such rewording as a mistranslation or an omission. So for gender neutrality, the LQA agent consults the gender-neutrality agent. With this adjusted LQA, 17% of the errors found by LQA on the gender-neutral translations turn out to be the required rewording, and they make up ~60% of the loss in accuracy against the original translations.

To test the adjusted LQA, we also ran it on sentences that refer to no person, where the gender rule changes nothing. There, it kept 98% to 100% of the errors.

When we translate the source text again with the gender-neutrality agent, the sentences that refer to a person have ~11% more errors, weighted by severity, than in the original translation, while the other sentences have about as many as before. That ~11% is the remaining accuracy difference in this comparison.

Errors per 1,000 words, originals and gender-neutral versions05101520Original: 13.613.6OriginalNeutral, standard LQA: 1919.0Neutral, standard LQANeutral, adjusted LQA: 15.815.8Neutral, adjusted LQAMean over the 99 gender-neutral versions of the two best solutions and theiroriginals, weighted by severity: minor 1, major 5, critical 10. Standard LQA: withoutthe gender-neutrality agent. Adjusted LQA: the LQA agent consults thegender-neutrality agent.
Errors per 1,000 words, originals and gender-neutral versions
Takeaway 25

We add steps to solutions with limited requirement failures when a language lacks two solutions that meet the requirements.

A solution meets general translation quality with no critical error, at most three major errors and at most 20 weighted errors per 1,000 words. Where a language has fewer than two such candidates, we also take one-step solutions that miss general translation quality and one other requirement. We leave out solutions that miss general translation quality and two or more other requirements because they fall short on too many requirements, and that includes 99% of the MT engine solutions on our shortlists.

In two-step solutions, we use four different automatic post-editing (APE) steps, depending on the missing requirements: one for general translation quality, one for tone of voice, one for terminology and one for general translation quality together with tone of voice or terminology, and we repair tags and formatting by code. We choose the APE step in advance, based on the known flaws of the particular AI model in the language.

If a language still has fewer than two solutions that meet all the requirements, we build three-step solutions on the same one-step solutions and choose the APE dynamically, from the Intento LQA findings on each translation.

One-step solutions by the Standard requirements they missMeet every requirementMiss general quality onlyMiss other requirements onlyGeneral quality and one otherGeneral quality and two or more020406080100MT engines, off the shelfMT engines, off the shelf: Meet every requirement 0 (0.0%)MT engines, off the shelf: Miss general quality only 0 (0.0%)MT engines, off the shelf: Miss other requirements only 0 (0.0%)MT engines, off the shelf: General quality and one other 0 (0.0%)MT engines, off the shelf: General quality and two or more 100 (100.0%)MT engines, adjustedMT engines, adjusted: Meet every requirement 0 (0.0%)MT engines, adjusted: Miss general quality only 0 (0.0%)MT engines, adjusted: Miss other requirements only 4 (4.5%)MT engines, adjusted: General quality and one other 0 (0.0%)MT engines, adjusted: General quality and two or more 95 (95.5%)LLMs, adjustedLLMs, adjusted: Meet every requirement 10 (10.3%)LLMs, adjusted: Miss general quality only 27 (27.3%)LLMs, adjusted: Miss other requirements only 7 (7.9%)LLMs, adjusted: General quality and one other 35 (35.0%)LLMs, adjusted: General quality and two or more 19 (19.5%)% of the solutions of each typePercent of the one-step solutions with a conclusive LQA result in the 96 measurable languages. Off-the-shelf LLMs are left out: 4solutions.
One-step solutions by the Standard requirements they miss

More on Method

Takeaway 0

We translate the texts in their original file formats and assess formatting before linguistic quality.

Markup accounts for 17% to 36% of the source characters sent for translation, depending on the document.

None of the four documents is plain text. The editorial article is web HTML with the CMS class names still on it. The education text is an XML export from a content system. The privacy notice is page HTML. The travel alert is a JSON object whose data fields carry HTML inside them. Across the four documents, markup is 24.5% of the translated characters, one tag every ~6 words. Heavily tagged documents are in the corpus on purpose: a solution that translates the words well but breaks the file has not done the job.

We translate the documents with tags in place and assess tag handling on the tagged translation. We then remove the tags and assess linguistic requirements on the plain text, so a broken tag counts once, as a formatting error, and not again as a language error.

When we remove the tags, we restore a space lost at a tag boundary, so two words glued together do not count against the language. Spacing at tag boundaries is a formatting requirement of its own. The JSON document has its own check, because a corrupted data field is a different failure from a mistranslated sentence.

See details →

Source characters sent for translation, by documentTextWhitespaceMarkup02,0004,0006,0008,00010,000Editorial articleEditorial article: Text 4,899 (65.9%)Editorial article: Whitespace 964 (13.0%)Editorial article: Markup 1,569 (21.1%)21% markupEducation guideEducation guide: Text 4,449 (55.6%)Education guide: Whitespace 677 (8.5%)Education guide: Markup 2,877 (35.9%)36% markupTravel alertTravel alert: Text 1,590 (67.3%)Travel alert: Whitespace 279 (11.8%)Travel alert: Markup 494 (20.9%)21% markupPrivacy noticePrivacy notice: Text 5,250 (70.5%)Privacy notice: Whitespace 960 (12.9%)Privacy notice: Markup 1,241 (16.7%)17% markupcharactersCharacters of the tagged text as sent for translation. Markup: tags, and for the JSON document its fieldnames, quote marks, braces and indentation. Whitespace: spaces and line breaks inside the content. Text: everyother character.
Source characters sent for translation, by document
Takeaway 0

We score the API response as received, including extra text beyond the requested translation.

Sixteen models returned extra text inside otherwise usable output: two leaked their reasoning into the translation, twelve wrapped the translation in code-block markers, one MT engine inserted an HTML header into every text, and two models of one vendor returned their glossary matches as visible markup. We retain that extra text for scoring.

The prompt asked for the translation only, so anything else in the response is a critical error, and we leave it in to assess the response as received. We make no exception for terminology: when a model returns its glossary matches as visible markup, we score the output as returned, and the markup counts as an error even though it comes from the glossary feature we asked the model to use.

See details →

Takeaway 0

We check the language and script of each translation before scoring.

We got such text from 16 of the 57 models, in 58 languages: English, a copy of the English source, another language or the wrong script. We flag a translated text when it is a copy of the source, when less than 50% of its letters are in the script of the target language, or when the language detectors label it as another language.

We use two open-source detectors: GlotLID, which covers more than 1,600 languages, and Lingua, which covers 75. Neither separates Traditional from Simplified Chinese, and GlotLID labels 254 of the 292 Serbian translations as Bosnian, so we accept a close relative or another variant of the target language. For text in other languages, we often get English from Lingua with full confidence, so we use its label only for its 75.

See details →

Takeaway 0

We align ~84% of translated sentences without an LLM and use an LLM for doubtful passages.

We split each translation into sentences using three sources together: a neural splitter, the markup and the punctuation of each script, because each source is unreliable in some languages. We then match the sentences by meaning with LaBSE, a public multilingual sentence-embedding model from Google for 109 languages, and use GPT-5.5 from OpenAI as the LLM.

We keep the match of a single sentence with low similarity, because in the calibration sample, all 64 such matches were correct. To the LLM, we send a few doubtful or missing sentences in a row with a confidently matched sentence on each side, or the whole text when its similarity is low overall or many sentences are missing.

From the LLM alignment, we use only the split positions and keep the original text of the translation.

Sentence alignment with and without an LLM, by source textWithout an LLMPassage sent to an LLMWhole text sent to an LLM020406080100All four textsAll four texts: Without an LLM 83 (83.9%)All four texts: Passage sent to an LLM 3 (3.2%)All four texts: Whole text sent to an LLM 12 (12.9%)84% without an LLMEditorial articleEditorial article: Without an LLM 75 (75.3%)Editorial article: Passage sent to an LLM 2 (2.6%)Editorial article: Whole text sent to an LLM 22 (22.1%)75% without an LLMEducation guideEducation guide: Without an LLM 86 (86.9%)Education guide: Passage sent to an LLM 4 (4.0%)Education guide: Whole text sent to an LLM 9 (9.1%)87% without an LLMTravel alertTravel alert: Without an LLM 78 (78.4%)Travel alert: Passage sent to an LLM 0 (0.8%)Travel alert: Whole text sent to an LLM 20 (20.8%)78% without an LLMPrivacy noticePrivacy notice: Without an LLM 90 (90.8%)Privacy notice: Passage sent to an LLM 3 (3.8%)Privacy notice: Whole text sent to an LLM 5 (5.4%)91% without an LLM% of the segments of the textSegments of the translations of the four texts from the 57 models, without the sentence-by-sentencetranslations, already aligned to the source. A passage is a few doubtful segments in a row, with the alignedsegments on each side.
Sentence alignment with and without an LLM, by source text
Takeaway 0

We preserve missing translations as empty segments during alignment.

During alignment, text from a neighboring sentence can fill the position of an omitted sentence. That would hide the omission and raise the quality score. We reward the aligner for leaving missing segments empty and assign them the lowest automatic-metric scores. Empty segments account for 1.2% of the scored segments.

Takeaway 0

We test sentence alignment in each language and use an LLM for every Wolof text.

In Wolof, a West African language spoken mainly in Senegal and one of the 25 lowest-resource languages among the 100, the LaBSE similarity threshold below which we treat an alignment as doubtful is 0.31. The threshold is 0.66 or higher in every other measured language. With similarity that low, we cannot tell a good match from a bad one. We sent at least one text whole to the LLM in each of the 100 languages.

Full-text LLM alignment, by target languageTexts sent whole to GPT-5.5 for alignment, % of the aligned texts in the language020406080100Wolof: 100WolofArmenian: 56ArmenianThai: 43ThaiBurmese: 27BurmeseKannada: 26KannadaPunjabi: 23PunjabiOdia (Oriya): 21Odia (Oriya)Assamese: 19AssameseMalay: 19MalayMacedonian: 18MacedonianKhmer: 18KhmerLao: 18LaoBengali: 17BengaliAlbanian: 17AlbanianBosnian: 17BosnianVietnamese: 16VietnameseMalayalam: 16MalayalamSinhala: 16SinhalaUyghur: 16UyghurBasque: 16BasqueTatar: 16TatarAmharic: 16AmharicItalian: 16ItalianIrish: 16IrishMaltese: 16MalteseMarathi: 16MarathiTamil: 16TamilBulgarian: 15BulgarianKinyarwanda: 15KinyarwandaTelugu: 15TeluguMongolian: 15MongolianKyrgyz: 15KyrgyzGujarati: 14GujaratiKurdish (Kurmanji): 14Kurdish (Kurmanji)Igbo: 14IgboSwedish: 14SwedishUrdu: 14UrduScottish Gaelic: 14Scottish GaelicShona: 14ShonaCzech: 13CzechKazakh: 13KazakhTagalog: 13TagalogNepali: 13NepaliDanish: 13DanishBelarusian: 13BelarusianCebuano: 13CebuanoAfrikaans: 13AfrikaansTurkish: 13TurkishMalagasy: 12MalagasyTajik: 12TajikZulu: 12ZuluXhosa: 12XhosaLatvian: 12LatvianWelsh: 12WelshYiddish: 12YiddishDutch: 12DutchHungarian: 12HungarianLithuanian: 12LithuanianSlovenian: 12SlovenianSundanese: 12SundaneseYoruba: 12YorubaFinnish: 12FinnishHausa: 11HausaCroatian: 11CroatianSomali: 11SomaliPortuguese (European): 11Portuguese (European)Slovak: 11SlovakRomanian: 11RomanianJavanese: 11JavaneseGerman: 11GermanUkrainian: 11UkrainianGeorgian: 11GeorgianPashto: 11PashtoSesotho: 11SesothoHaitian Creole: 10Haitian CreoleChinese Traditional: 10Chinese TraditionalAzerbaijani: 10AzerbaijaniPolish: 10PolishNorwegian: 10NorwegianUzbek: 10UzbekSpanish (European): 10Spanish (European)Hindi: 9HindiSerbian: 9SerbianEstonian: 9EstonianChinese Simplified: 9Chinese SimplifiedFarsi: 9FarsiSwahili: 9SwahiliTurkmen: 8TurkmenJapanese: 8JapaneseFrench: 8FrenchGalician: 8GalicianIcelandic: 8IcelandicGreek: 7GreekRussian: 7RussianFrisian: 7FrisianCatalan: 6CatalanArabic (MSA): 6Arabic (MSA)Indonesian: 6IndonesianKorean: 3KoreanHebrew: 3HebrewTexts of the translations of the four texts from the 57 models, without the sentence-by-sentence translations, already aligned to the source. We send a text whole to the LLM when its LaBSE similarity is low overall or many of itssegments are empty, and in Wolof we send every text whole after the language check.Company, product and model names and logos are shown for informational purposes only. All trademarks and logos belong to their respective owners. Their use does not imply affiliation or endorsement.
Full-text LLM alignment, by target language
Takeaway 0

We check that MetricX distinguishes between translated sentences in each language.

If its scores are almost the same for every sentence, MetricX cannot distinguish the translations in that language, and we cannot read its low agreement with COMET-Kiwi as evidence of poor translation quality. MetricX passed this check in all 93 languages it scores.

Its training data has human quality ratings for some languages only, yet among the 16 high-resource languages, MetricX agrees with COMET-Kiwi just as well where it had no such ratings. Another version of MetricX would give different scores, so we score every translation with the same version we used to set the quality threshold.

Takeaway 0

We use MetricX to rank solutions in seven languages without COMET-Kiwi support.

COMET-Kiwi does not support 13 of the 100 languages. MetricX provides the ranking in seven of those languages.

In the 84 languages with both metrics, the average rank correlation between them is 0.89. For the combined ranking of the semantic similarity metrics LaBSE and Cohere, the correlation with COMET-Kiwi is only 0.32, so it would be a poor replacement. Kinyarwanda, Tatar and Wolof have no quality metric, so we rank their solutions by semantic similarity alone, which does not measure fluency. Haitian Creole, Maltese and Turkmen are not measurable.

Takeaway 0

We give missing translations the worst metric scores when selecting the shortlist.

MetricX counts errors, so a score of zero means a perfect translation, and we give a missing sentence the worst MetricX score, 25. On COMET-Kiwi, LaBSE and Cohere, where higher is better, we score a missing sentence at 0. Without a fixed score, the penalty for a missing sentence would differ by metric and by language.

We leave missing sentences out when we measure how well two metrics agree. Both quality metrics assign their worst scores to missing sentences. Including these fixed scores would increase measured agreement automatically and could affect whether a language qualifies.

Takeaway 0

We shortlist 10 to 13 solutions per language, including at least three real-time solutions.

We start by building the available one-step solutions and scoring them using fast quality-estimation scores (COMET-Kiwi and MetricX). For each language we then take the top-8 solutions by COMET-Kiwi and the top-8 by MetricX, so a strong solution stays in even if one of the two metrics underrates it. Several harness variants of the same model, and sibling models whose scores are statistically indistinguishable, count once. The two top-8 lists together give 8 to 13 candidates per language, depending on how much the metrics agree.

Finally, we add real-time solutions until at least three are on the list, up to 13 solutions in total. Real-time means a measured median of two seconds or less per document, not a label on the model: translation takes 67 seconds per document with one MT engine and under two seconds with one LLM.

See details →  ·  Explore interactively →

COMET-Kiwi rankMetricX rankShortlistboth metrics agree1234567891012349101112561234real-time5678910real-time111214additionalreal-time
Shortlisting solutions
Takeaway 0

We check tone of voice, terminology and gender neutrality only where each requirement applies.

Of the 177 sentences, 47 address the reader or give instructions, 73 contain a glossary term, and 115 refer to a person. We assess tone of voice, terminology and gender neutrality on these respective subsets. Terminology applies only in the 16 languages with a glossary.

For tone of voice, we use an LLM to identify the register and a second LLM from another vendor to review the finding, then compare it with the requested register in code.

Takeaway 0

We use an LLM to assess terminology in its correct inflected forms, rather than requiring an exact glossary match.

A glossary has one form per term, and in a sentence the form often changes. Estonian nouns decline, and a Korean rendering changes with the particle after it, so the approved form is right only in some positions. A simple check that looks for the exact term misses a declined one. So in Intento LQA we use an LLM to check the terms, and it counts a term as applied in any of its correct forms.

Takeaway 0

We reassess major and critical errors in context, and ~one in three becomes a minor error or no error.

We assess translations with LLM judges one sentence at a time, so the judges count as errors a pronoun or a tense that depends on the previous sentence, content moved across a sentence boundary, or a heading translated as a noun phrase. We send each major and critical error to Claude Fable 5 with the two sentences before and the one after, in the source and in the translation, and ask whether the error still stands. We downgrade most of the changed errors to minor and drop the rest, and we keep nearly every critical error as critical.

Nearly all solutions that reach the limit of three major errors in the four texts only after reassessment use LLMs. Most MT engine translations remain above that limit. Claude Fable 5 is also one of the models we evaluate, and the re-check changes errors in its own translations as often as in those of the other leading LLMs.

Major and critical errors after reassessment in context, by source textKeptDowngraded to minorDropped020406080100Editorial articleEditorial article: Kept 60 (60.0%)60Editorial article: Downgraded to minor 29 (29.0%)29Editorial article: Dropped 11 (11.0%)11Education guideEducation guide: Kept 71 (71.0%)71Education guide: Downgraded to minor 21 (21.0%)21Education guide: Dropped 8 (8.0%)8Travel alert (JSON)Travel alert (JSON): Kept 78 (78.0%)78Travel alert (JSON): Downgraded to minor 13 (13.0%)13Travel alert (JSON): Dropped 9 (9.0%)9Privacy noticePrivacy notice: Kept 74 (74.0%)74Privacy notice: Downgraded to minor 22 (22.0%)22Privacy notice: Dropped 4 (4.0%)4% of the major and critical errors in the text13,768 major and critical errors found by the LQA judges in the translations of the shortlisted solutionsin 91 languages, re-checked with the two sentences before and the one after, in the source and thetranslation.
Major and critical errors after reassessment in context, by source text
Takeaway 0

In our tests, examples in the prompts reduced LLM-judge agreement, so we use prompts without examples.

We assess each translation with three LLM judges, built on Claude Opus 5, GPT-5.6 Sol and Gemini 3.1 Pro. We tested one set of examples for all languages and one set per language on translations into Spanish and Ukrainian from the 2025 edition of the benchmark, which was assessed by human reviewers.

With the set for all languages, the Gemini 3.1 Pro judge flagged an error in 7% of the sentences, against 36% without examples, while the two other judges barely changed, and we saw the same drop when we repeated the test twice on translations from 2026. Of the Ukrainian sentences where human reviewers marked a major or critical error, the three judges agreed on such an error in 6% with the set for all languages, against 46% without examples.

In one of the examples, an English term left untranslated in a Chinese translation was a minor error. With this example in the prompt, the judges also rated an English term left untranslated in a Ukrainian translation as minor, while human reviewers rated it as major. We rely on the agreement of the three judges rather than on human ratings, because in the 2025 human review, reviewers rated severity inconsistently across languages, and sometimes within one language.

Takeaway 0

We found no consistent vendor preference in the LQA judges based on Gemini 3.1 Pro, GPT-5.6 Sol and Claude Opus 5.

Intento LQA scores translations with a panel of three LLM judges: Gemini 3.1 Pro, GPT-5.6 Sol and Claude Opus 5. Each judge is itself a solution, a model in a harness that scores translations.

Two of the three judge models also appear as translation solutions in the shortlist, so we tested whether a judge scores translations from its own vendor higher. For every scored solution we compared each judge with the average of the three, separately for solutions from the same vendor as the judge and for all others. The same-vendor difference is below 0.2 points on the 0–100 general translation quality score and goes in different directions for different judges, so we found no consistent vendor preference in this test.

The same test showed that the judges differ in strictness. Claude Opus 5, the strictest judge, scores 2.7 points below Gemini 3.1 Pro, the most lenient one, while the best and worst shortlisted solutions in a language differ by about 12 points. A judge that is strict or lenient across the board does not change the order of solutions within a language, because every judge scores every solution. Absolute scores do depend on which judges sit on the panel, so we keep the same panel for the whole 2026 edition.

Score difference for each judge relative to the other two-2-1012Gemini 3.1 Pro: 1.3+1.30Gemini 3.1 ProGPT-5.6 Sol: 0.13+0.13GPT-5.6 SolClaude Opus 5: -1.44-1.44Claude Opus 5Points on the 0–100 general translation quality score, 293 solutions scored by allthree judges. The best and worst shortlisted solutions in a language differ by about12 points.Company, product and model names and logos are shown for informationalpurposes only. All trademarks and logos belong to their respective owners.Their use does not imply affiliation or endorsement.
Score difference for each judge relative to the other two

Automate translation to meet your requirements

See Intento Language Hub in action.

Book a demo
Leave feedback