We exclude unusable translations before scoring quality.
Excluded translations by model: every translation, or one of the two
| Model | Every translation excluded | Only the translation without context excluded | Only the translation with context excluded |
|---|---|---|---|
| Apertus-8B-2509 | Amharic, Armenian, Assamese, Hausa, Igbo, Khmer, Kurdish (Kurmanji), Lao, Odia (Oriya), Russian, Scottish Gaelic, Sesotho, Shona, Tatar, Turkmen, Uyghur, Wolof | Belarusian, Bengali, Cebuano, Chinese Simplified, Czech, Danish, Dutch, Gujarati, Italian, Kazakh, Latvian, Pashto, Serbian, Slovenian, Spanish (European), Ukrainian, Welsh, Yiddish, Zulu | Azerbaijani, Basque, Burmese, Farsi, French, Frisian, Georgian, German, Hebrew, Indonesian, Irish, Japanese, Kyrgyz, Lithuanian, Macedonian, Malagasy, Malayalam, Maltese, Marathi, Polish, Punjabi, Sinhala, Slovak, Somali, Sundanese, Swahili, Tajik, Telugu, Turkish, Uzbek, Xhosa, Yoruba |
| AI21 Jamba Large 1.7 | Amharic, Armenian, Assamese, Burmese, Georgian, Gujarati, Hausa, Igbo, Kannada, Khmer, Kinyarwanda, Kurdish (Kurmanji), Kyrgyz, Lao, Malagasy, Malayalam, Maltese, Mongolian, Odia (Oriya), Pashto, Punjabi, Sesotho, Shona, Sinhala, Somali, Sundanese, Tajik, Tamil, Tatar, Telugu, Turkmen, Uyghur, Wolof, Xhosa, Yoruba, Zulu | Belarusian, Irish, Kazakh, Nepali, Scottish Gaelic, Swahili, Uzbek | Bengali, Yiddish |
| SpeakLeash Bielik-11B-v3.0-Instruct | Belarusian, Bosnian, Bulgarian, Croatian, Danish, Dutch, Finnish, Greek, Hungarian, Italian, Latvian, Lithuanian, Romanian, Russian, Serbian, Slovak, Slovenian, Swedish, Ukrainian | Albanian, Czech, Estonian, German | Portuguese (European) |
| Widn AI Tower Vesuvius | Czech, Danish, Dutch, Finnish, French, German, Hindi, Hungarian, Icelandic, Italian, Japanese, Korean, Norwegian, Polish, Portuguese (European), Romanian, Russian, Spanish (European), Swedish, Ukrainian | ||
| Widn AI Tower Anthill | Czech, Danish, Finnish, Hindi, Hungarian, Icelandic, Korean, Norwegian, Polish, Russian, Swedish, Ukrainian | French, German, Romanian | |
| Cohere Command A Translate | Amharic, Shona, Tajik, Wolof, Zulu | Igbo, Kinyarwanda, Sinhala, Somali, Yoruba | Kannada, Scottish Gaelic, Xhosa |
| OpenAI gpt-oss-120b | Lao, Wolof | Amharic, Danish, Odia (Oriya), Scottish Gaelic, Sinhala | Javanese, Kurdish (Kurmanji), Uyghur |
| Cohere Tiny Aya | Wolof | Javanese, Serbian, Zulu | Catalan, Khmer, Shona, Xhosa, Yoruba |
| ALL-Lab | Amharic, Igbo, Malagasy, Sesotho, Shona, Xhosa, Yoruba, Zulu | ||
| M2.5 | Lao, Telugu | Burmese, Khmer | Wolof, Yiddish |
| Tilde | Lithuanian, Maltese, Serbian | Swedish | |
| Anthropic Claude Haiku 4.5 | Assamese | Malagasy | Hausa |
| Google Translation LLM | Serbian | Korean | |
| Youdao | Kurdish (Kurmanji), Serbian | ||
| Alibaba | Burmese | ||
| Anthropic Claude Sonnet 5 | Uyghur | ||
| DeepL Next Gen | Serbian | ||
| Google Cloud Advanced Translation API | Serbian | ||
| Microsoft Translator | Kurdish (Kurmanji) | ||
| Mistral Medium 3.1 | Sesotho |
We have one or two translations from a model in each language: one without context, and one with context in the request (the glossary and the instructions for tone of voice and formatting, in the prompt or in API parameters). Every translation excluded includes the languages where we have only one.
Excluded translations by model and reason
| Why we excluded the translation | Languages |
|---|---|
| Apertus-8B-2509: 68 languages, every translation excluded in 17 | |
| cut off part-way | Azerbaijani, Bengali, Burmese, Cebuano, Chinese Simplified, Danish, Farsi, French, Frisian, Georgian, German, Hebrew, Indonesian, Irish, Italian, Japanese, Kazakh, Khmer, Lao, Lithuanian, Malayalam, Odia (Oriya), Punjabi, Russian, Serbian, Slovak, Slovenian, Spanish (European), Sundanese, Tatar, Turkish, Ukrainian, Yiddish |
| garbage or near-empty | Amharic, Armenian, Basque, Gujarati, Hausa, Igbo, Kurdish (Kurmanji), Kyrgyz, Macedonian, Malagasy, Maltese, Marathi, Pashto, Scottish Gaelic, Sesotho, Shona, Somali, Swahili, Tajik, Telugu, Turkmen, Uyghur, Welsh, Wolof, Xhosa, Yoruba, Zulu |
| wrong language | Armenian, Assamese, Belarusian, Czech, Dutch, Lao, Latvian, Odia (Oriya), Russian, Sesotho, Tatar, Turkmen, Uyghur, Wolof, Yiddish |
| wrong script | Armenian, Lao, Odia (Oriya), Russian, Tatar, Uyghur, Yiddish |
| far too short | Lao, Odia (Oriya), Polish, Sinhala, Uzbek |
| AI21 Jamba Large 1.7: 45 languages, every translation excluded in 36 | |
| garbage or near-empty | Amharic, Assamese, Burmese, Hausa, Igbo, Kazakh, Khmer, Kinyarwanda, Kurdish (Kurmanji), Kyrgyz, Lao, Malagasy, Malayalam, Maltese, Mongolian, Nepali, Odia (Oriya), Pashto, Punjabi, Scottish Gaelic, Shona, Sinhala, Somali, Sundanese, Tajik, Tamil, Tatar, Telugu, Turkmen, Uyghur, Uzbek, Wolof, Xhosa, Yiddish, Yoruba, Zulu |
| wrong language | Assamese, Belarusian, Burmese, Gujarati, Hausa, Irish, Khmer, Kurdish (Kurmanji), Lao, Malagasy, Maltese, Nepali, Odia (Oriya), Pashto, Punjabi, Scottish Gaelic, Sesotho, Sinhala, Somali, Swahili, Tajik, Tatar, Turkmen, Uyghur, Yoruba |
| wrong script | Assamese, Belarusian, Burmese, Gujarati, Khmer, Lao, Malayalam, Nepali, Odia (Oriya), Pashto, Punjabi, Sinhala, Tajik, Tatar, Uyghur |
| cut off part-way | Armenian, Assamese, Bengali, Georgian, Gujarati, Kannada, Khmer, Malayalam, Odia (Oriya), Punjabi, Tamil, Telugu |
| far too short | Irish |
| SpeakLeash Bielik-11B-v3.0-Instruct: 24 languages, every translation excluded in 19 | |
| wrong language | Albanian, Belarusian, Bosnian, Bulgarian, Croatian, Czech, Danish, Dutch, Estonian, Finnish, German, Greek, Hungarian, Italian, Latvian, Lithuanian, Portuguese (European), Romanian, Russian, Serbian, Slovak, Slovenian, Swedish, Ukrainian |
| wrong script | Belarusian, Bulgarian, Greek, Russian, Ukrainian |
| Widn AI Tower Vesuvius: 20 languages, every translation excluded in 20 | |
| cut off part-way | Czech, Danish, Dutch, Finnish, French, German, Hindi, Hungarian, Icelandic, Italian, Japanese, Korean, Norwegian, Polish, Portuguese (European), Romanian, Russian, Spanish (European), Swedish, Ukrainian |
| Widn AI Tower Anthill: 15 languages, every translation excluded in 12 | |
| cut off part-way | Czech, Danish, Finnish, French, German, Hindi, Hungarian, Icelandic, Korean, Norwegian, Polish, Romanian, Russian, Swedish, Ukrainian |
| wrong language | Czech, Danish, Finnish, German, Icelandic, Korean, Norwegian, Polish, Swedish, Ukrainian |
| wrong script | Korean, Ukrainian |
| Cohere Command A Translate: 13 languages, every translation excluded in 5 | |
| garbage or near-empty | Amharic, Igbo, Kannada, Kinyarwanda, Scottish Gaelic, Shona, Sinhala, Somali, Wolof, Xhosa, Yoruba, Zulu |
| wrong language | Tajik |
| wrong script | Tajik |
| OpenAI gpt-oss-120b: 10 languages, every translation excluded in 2 | |
| garbage or near-empty | Amharic, Javanese, Kurdish (Kurmanji), Lao, Odia (Oriya), Sinhala, Uyghur, Wolof |
| wrong language | Danish, Scottish Gaelic |
| far too short | Scottish Gaelic |
| Cohere Tiny Aya: 9 languages, every translation excluded in 1 | |
| garbage or near-empty | Javanese, Khmer, Wolof, Yoruba, Zulu |
| wrong language | Javanese, Serbian, Shona, Wolof |
| cut off part-way | Catalan, Xhosa |
| wrong script | Serbian |
| ALL-Lab: 8 languages, every translation excluded in 8 | |
| an error message instead of the translation | Amharic, Igbo, Malagasy, Sesotho, Shona, Xhosa, Yoruba, Zulu |
| M2.5: 6 languages, every translation excluded in 2 | |
| wrong language | Lao, Telugu, Wolof, Yiddish |
| garbage or near-empty | Burmese, Khmer, Lao, Telugu |
| wrong script | Lao, Telugu |
| cut off part-way | Telugu |
| far too short | Wolof |
| Tilde: 4 languages, every translation excluded in 3 | |
| an error message instead of the translation | Lithuanian, Maltese, Swedish |
| wrong language | Serbian |
| wrong script | Serbian |
| Anthropic Claude Haiku 4.5: 3 languages, every translation excluded in 1 | |
| wrong language | Assamese, Malagasy |
| garbage or near-empty | Hausa |
| Google Translation LLM: 2 languages, every translation excluded in 1 | |
| wrong language | Serbian |
| garbage or near-empty | Korean |
| wrong script | Serbian |
| Youdao: 2 languages, every translation excluded in 2 | |
| wrong language | Kurdish (Kurmanji), Serbian |
| cut off part-way | Serbian |
| wrong script | Serbian |
| Alibaba: 1 language, every translation excluded in 1 | |
| cut off part-way | Burmese |
| Anthropic Claude Sonnet 5: 1 language, every translation excluded in 0 | |
| wrong language | Uyghur |
| wrong script | Uyghur |
| DeepL Next Gen: 1 language, every translation excluded in 1 | |
| wrong language | Serbian |
| wrong script | Serbian |
| Google Cloud Advanced Translation API: 1 language, every translation excluded in 1 | |
| wrong language | Serbian |
| wrong script | Serbian |
| Microsoft Translator: 1 language, every translation excluded in 1 | |
| wrong language | Kurdish (Kurmanji) |
| Mistral Medium 3.1: 1 language, every translation excluded in 1 | |
| garbage or near-empty | Sesotho |
| cut off part-way | Sesotho |
A language appears under every check one of its translations failed.