We check translation length before scoring to detect cut-off or nearly empty texts.
Four models account for ~85% of the texts cut off part-way: Widn AI Tower Vesuvius, Apertus-8B-2509, Widn AI Tower Anthill and AI21 Jamba Large 1.7.
From TartuNLP, we got one text cut off part-way in each of its seven languages, so we scored all seven translations.
Cut-off and nearly empty translations by model
| Model | Vendor | What we got | Excluded from scoring | Scored, with one text cut short | Note | Languages |
|---|---|---|---|---|---|---|
| Apertus-8B-2509 | Swiss AI |
|
| Apertus-8B-2509: available through one hosting service, and part of the cut-off translations trace to dropped connections on that service. | 51 | |
| Widn AI Tower Vesuvius | TransPerfect |
|
|
| 22 | |
| Cohere Tiny Aya | Cohere |
|
|
| 21 | |
| Widn AI Tower Anthill | TransPerfect |
|
|
| 21 | |
| AI21 Jamba Large 1.7 | AI21 |
|
|
| 16 | |
| ALL-Lab | ALL-Lab |
|
| ALL-Lab: time-out errors in place of the translation, under the load of parallel requests. | 8 | |
| TartuNLP | TartuNLP |
|
| 7 | ||
| M2.5 | MiniMax |
|
|
| 4 | |
| Tilde | Tilde |
|
| Tilde: a server error for every segment, so the translation holds an error message in place of each sentence. | 3 | |
| Alibaba | Alibaba |
|
|
| 2 | |
| Grok 4.20 (non-reasoning) | xAI |
|
| 2 | ||
| Mistral Medium 3.1 | Mistral |
|
|
| 2 | |
| Youdao | Youdao |
|
|
| 2 | |
| Cohere Command A Translate | Cohere |
|
| 1 | ||
| Mistral Large 3 | Mistral |
|
| 1 | ||
| OpenAI gpt-oss-120b | OpenAI |
|
| OpenAI gpt-oss-120b: a refusal to translate one of the texts into Scottish Gaelic. | 1 |
A language can appear in both columns when we excluded the translation at one quality level and kept it at the other.