We benchmark 57 models from 38 vendors: 28 translation APIs and 29 LLMs prompted for translation.
The 57 models in the benchmark and their vendors
| Model | Type | Model version | Translated | Note |
|---|---|---|---|---|
| AI21 | ||||
| Jamba Large 1.7 | LLM | jamba-large-1.7-2025-07 | 24 June to 28 July 2026 | |
| Alibaba | ||||
| Machine Translation API | MT engine | 24 June to 29 June 2026 | ||
| Qwen3-235B-A22B | open-weight LLM | qwen3-235b-a22b | 24 June to 27 July 2026 | |
| ALL-Lab | ||||
| ALL-Lab | MT engine | 24 June to 28 July 2026 | ||
| Amazon | ||||
| Translate | MT engine | 24 June to 29 June 2026 | ||
| Anthropic | ||||
| Claude Fable 5 | LLM | claude-fable-5 | 24 July to 28 July 2026 | |
| Claude Haiku 4.5 | LLM | claude-haiku-4-5-20251001 | 24 June to 28 July 2026 | |
| Claude Opus 4.8 | LLM | claude-opus-4-8 | 24 June to 28 July 2026 | |
| Claude Sonnet 4.6 | LLM | claude-sonnet-4-6 | 24 June to 28 July 2026 | |
| Claude Sonnet 5 | LLM | claude-sonnet-5 | 14 July to 28 July 2026 | |
| AppTek | ||||
| NMT | MT engine | 1 July to 3 July 2026 | ||
| Baidu | ||||
| Translate API | MT engine | 24 June 2026 | ||
| Cohere | ||||
| Command A Translate | LLM | command-a-translate-08-2025 | 24 June to 28 July 2026 | |
| Tiny Aya | LLM | tiny-aya-global | 24 June to 16 July 2026 | |
| DeepL | ||||
| Next Gen | LLM-based MT engine | 24 June to 29 June 2026 | ||
| DeepSeek | ||||
| V3.2 | LLM | deepseek-reasoner | 24 June to 28 July 2026 | |
| Elia | ||||
| Machine Translation System | MT engine | 8 July 2026 | ||
| Cloud Advanced Translation API | MT engine | 24 June to 29 June 2026 | ||
| Gemini 3.1 Pro Preview | LLM | gemini-3.1-pro-preview | 24 June to 28 July 2026 | |
| Gemini 3.5 Flash | LLM | gemini-3.5-flash | 24 June to 28 July 2026 | |
| Gemma 4 | open-weight LLM | gemma-4-31b-it | 24 June to 28 July 2026 | |
| Translation LLM | LLM-based MT engine | 24 June to 13 August 2026 | ||
| iFlyTek | ||||
| MT | MT engine | 7 July 2026 | ||
| Meta | ||||
| Llama 4 Maverick | open-weight LLM | meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8 | 2 September 2026 | Runs on the FP8 build its inference provider serves. The one model added after the 16 July freeze, on 1 September. |
| Microsoft | ||||
| Translator | MT engine | 24 June to 29 June 2026 | ||
| MiniMax | ||||
| M2.5 | LLM | MiniMax-M2.5 | 1 July to 28 July 2026 | |
| Mistral | ||||
| Large 3 | LLM | mistral-large-2512 | 24 June to 28 July 2026 | |
| Medium 3.1 | LLM | mistral-medium-2508 | 24 June to 28 July 2026 | |
| MoonshotAI | ||||
| Kimi K2.5 | LLM | kimi-k2.5 | 2 July to 28 July 2026 | |
| Naver | ||||
| Papago NMT Commercial | MT engine | 24 June to 25 June 2026 | ||
| NiuTrans | ||||
| NiuTrans | MT engine | 14 July to 15 July 2026 | ||
| OpenAI | ||||
| GPT-4.1 | LLM | gpt-4.1 | 24 June to 28 July 2026 | |
| GPT-5.5 | LLM | gpt-5.5 | 24 June to 28 July 2026 | |
| GPT-5.6 Luna | LLM | gpt-5.6-luna | 14 July to 28 July 2026 | |
| GPT-5.6 Sol | LLM | gpt-5.6-sol | 14 July to 28 July 2026 | |
| GPT-5.6 Terra | LLM | gpt-5.6-terra | 14 July to 28 July 2026 | |
| gpt-oss-120b | open-weight LLM | openai/gpt-oss-120b | 2 September 2026 | |
| Oracle | ||||
| Cloud Language Service | MT engine | 24 June 2026 | ||
| Pangeanic | ||||
| PangeaMT | MT engine | 24 June to 26 June 2026 | ||
| SpeakLeash | ||||
| Bielik-11B-v3.0-Instruct | open-weight LLM | speakleash/Bielik-11B-v3.0-Instruct | 2 September 2026 | |
| Supertext | ||||
| LLM | LLM-based MT engine | 1 July to 3 July 2026 | ||
| Swiss AI | ||||
| Apertus-8B-2509 | open-weight LLM | swiss-ai/Apertus-8B-Instruct-2509 | 2 September 2026 | |
| SYSTRAN | ||||
| PNMT API | MT engine | 30 June to 22 September 2026 | ||
| Tarjama | ||||
| NMT | MT engine | 24 June 2026 | ||
| TartuNLP | ||||
| TartuNLP | MT engine | 1 July 2026 | ||
| Tencent | ||||
| Cloud TMT | MT engine | 24 June 2026 | ||
| Tilde | ||||
| TildeOpen LLM engine | LLM-based MT engine | 24 July to 28 July 2026 | ||
| Translated | ||||
| Lara 2 | LLM-based MT engine | 14 July to 29 July 2026 | Translated launched a new version of Lara after the model freeze, so the results describe Lara 2. | |
| ModernMT Enterprise Edition | MT engine | 24 June to 13 August 2026 | ||
| TransPerfect | ||||
| Widn AI Tower Anthill | LLM-based MT engine | anthill | 24 June to 28 July 2026 | Widn.AI was discontinued on 27 August 2026, after the model freeze. |
| Widn AI Tower Sugarloaf | LLM-based MT engine | sugarloaf | 7 July to 29 July 2026 | Widn.AI was discontinued on 27 August 2026, after the model freeze. |
| Widn AI Tower Vesuvius | LLM-based MT engine | vesuvius | 24 June to 28 July 2026 | Widn.AI was discontinued on 27 August 2026, after the model freeze. |
| xAI | ||||
| Grok 4.20 (non-reasoning) | LLM | grok-4.20-0309-non-reasoning | 24 June to 28 July 2026 | |
| Grok 4.20 (reasoning) | LLM | grok-4.20-0309-reasoning | 24 June to 28 July 2026 | |
| Yandex | ||||
| Cloud Translate v2 | MT engine | 8 July to 13 August 2026 | ||
| Youdao | ||||
| Translation Service API | MT engine | 8 July to 13 July 2026 | ||
| z.ai | ||||
| GLM-5 | LLM | glm-5.1 | 1 July to 28 July 2026 | |
Translated: the dates of the first and the last translation of the model, across the three quality levels. Model version: the model ID we called, where the API takes one.