We verify language support by checking whether each model returns a usable translation in that language.

A model supports a language when its output in that language is a translation. We send every model the same source texts, language by language, and check each output as a whole before any quality scoring. An output counts as a translation when it passes five checks: it is in the target language, it is in the script of the target language, it is not garbage or near-empty, it is not cut off part-way, and it is not far shorter than the source. One translation from any solution of the model is enough. When we get output in a language from a model, but no output passes all five checks, the language counts as nominally supported. When the language is not available for the model at all, it counts as not supported.

Language lists built from documentation are wrong in both directions. For one MT engine, the list built from its documentation held 45 of our 100 languages without a translation from the engine, and for another, 74. For one LLM, we got translations in more languages than there are in its documentation.

We got text in the wrong language from 19 models, in 69 languages, more than for any other check.

One MT engine is available to us under a license limited to a few languages. We did not evaluate its outputs in the other languages, and those languages count as outside the license, not as unsupported.