We rank solutions only in languages where at least one pair of independent metrics meets the agreement threshold.

We measure agreement separately for each metric pair and language. We calculate rank correlation across unique translations of each source sentence, counting identical translations from several solutions once.

• Similarity pair: each metric turns the source segment and its translation into a vector and scores how close the two meanings are. The two are built on different encoders.
  • LaBSE (Language-agnostic BERT Sentence Embedding, Google Research, 2020): an open model that embeds sentences from 109 languages into one shared space, trained so that a sentence and its translation land close together.
  • Cohere embed v3: a commercial multilingual embedding model served through the Cohere API, covering more than 100 languages, built for semantic similarity and search.
• Quality-estimation pair: each metric estimates quality using the source text and its translation, without a reference translation. The two are built on different base models and trained on different human annotations.
  • COMET-Kiwi XL (Unbabel, WMT 2023): a reference-free quality-estimation model built on XLM-R XL, trained to predict the quality score a human annotator would give a translation of the source.
  • MetricX-24 (Google, WMT 2024): a learned metric built on mT5 that predicts an error score for a translation, lower being better; we use the hybrid version in its reference-free mode.

Each metric pair has a minimum agreement threshold. The minimums are 0.461 for the similarity pair and 0.534 for the quality-estimation pair. Each is the weakest agreement the pair reaches on 16 reference languages (German, French, Spanish, Italian, Portuguese, Dutch, Russian, Polish, Czech, Swedish, Turkish, Arabic, Chinese, Japanese, Korean, Hindi), minus a margin of 0.05: for the similarity pair the weakest reference language is Korean at 0.511, for the quality-estimation pair Hindi at 0.584.

The idea is that a language qualifies when its metrics agree at least about as well as they do on a well-resourced language where the metrics are known to work. The agreement of a language is an estimate, so a language qualifies when the lower end of its 90% confidence interval is at the minimum or above. Either pair is enough, because requiring both would exclude languages on evidence no weaker than the evidence that admits others.

The quality-estimation pair only counts where MetricX separates good translations from bad ones (a discrimination score of at least 1.35); in this benchmark that condition never removed a language. Both minimums were derived once on the final corpus and then frozen, so they do not move with the translations being scored.

Fixed thresholds keep the admission rule consistent. A minimum recomputed every time new translations are scored would depend on the reference languages rather than on the language being assessed: any change in how well the two metrics agree on Korean, the weakest reference language, would move the similarity minimum and could admit or exclude an unrelated language whose own agreement had not changed. So we set each minimum once and keep it as a fixed constant.

Both metric pairs meet the threshold in most languages. Both pairs meet the threshold in 73 languages, one pair only in 23, and neither in 4. Resource level tracks the outcome most closely: both pairs agree in 38 of the 40 high-resource languages, in 26 of the 35 medium-resource languages and in 9 of the 25 low-resource languages, and all 4 not reliably measurable languages are low-resource. The languages where only one pair agrees also have fewer solutions scored (a median of 54 to 61 against 69 where both pairs agree), so their agreement figures rest on fewer translations.

Among the 11 languages where one pair agrees and the other disagrees, it is the similarity pair that falls short in 8 (Albanian, Belarusian, Frisian, Galician, Lao, Scottish Gaelic, Serbian and Sundanese, with quality-estimation agreement between 0.57 and 0.69) and the quality-estimation pair in 3 (Gujarati, Croatian, Xhosa).

Agreement varies within language families. No family falls entirely below the threshold: both pairs agree in 6 of 7 Romance, 8 of 9 Germanic, 9 of 12 Slavic and all 3 Uralic languages. Two families stand out on the low side, Bantu, where both pairs agree in 1 of 6, and Turkic, with 5 of 8 and two of the four languages that are not measurable (Turkmen and Uyghur). Both families are largely low-resource with few solutions scored.

The Indic languages show the opposite split: across the 14 Indo-Aryan and Dravidian languages the similarity pair agrees better than elsewhere (mean 0.586 against 0.532 for the other languages) while the quality-estimation pair agrees worse (median 0.58 against 0.63 overall), and one of the three languages where quality estimation disagrees is Indic (Gujarati).

The writing system shows no pattern: the similarity pair agrees at 0.531 on average in Latin-script languages and 0.552 in the others, and both pairs agree in 7 of the 10 Cyrillic-script languages. One more observation: of the three closely related languages Bosnian, Croatian and Serbian, both pairs agree only for Bosnian, while the quality-estimation pair falls short for Croatian and the similarity pair for Serbian.