The State of Machine Translation 2024

An independent multi-domain evaluation of
Machine Translation engines and Large Language Models

MT Engines and LLMs

Language pairs

Content Domains

MT Engines and LLMs

Language pairs

Content Domains

First name*

Last name*

Work email*

Company name*

Phone number

Consent to news and insights emails

I want to receive industry news and expert insights from Intento

consent_to_e2f_communications

I agree to share my data with e2f, an expert in data annotation and our 2024 report partner

Trusted globally by

Trusted by the global enterprise

What‘s changed in the MT market since 2023

LLMs reshaped the MT landscape and now account for 55% of top-performing models
Quality varies by language and domain, with LLMs excelling in Colloquial, Education, and Entertainment
Traditional Machine Translation (MT) still outperforms in specific language pairs and domains
LLMs are less expensive but slower than MT engines
Open-source LLMs lag behind commercial offerings
Customization through translation memories, glossaries, and prompt engineering is key to eliminating errors

52 Evaluated MT Engines and LLMs

MT Engines

LLMs

MT Engines

LLMs

18 providers show the best results and 9 MT Engines excel among real-time providers

18 providers show the best results

Our datasets cover:

22 language pairs (English-to-Spanish, French, Italian, Portuguese, German, Dutch, Ukrainian, Korean, Japanese, Chinese, and Arabic)

9 content domains (General, Colloquial, IT, Entertainment, Hospitality, Education, Healthcare, Legal, and Financial)

9 MT Engines excel among real-time providers

“We launched our inaugural MT report in 2017, at a time when there were only eleven stock models available. Fast forward six years, and the MT landscape has exploded, with six-fold growth in MT platforms and 15x more language pairs. In addition, we see new automated customization possibilities such as domain adaptation and custom glossaries, and the emergence of Large Language Models, which can handle translation among other tasks.”

“We’re pleased to extend our expertise to drive exceptional MT evaluation results in the latest Intento report. The caliber of our lingosets (golden datasets) is backed by a talented force of translation and data scientists that have enabled e2f to partner with some of the top players in the localization and AI landscape. As a result, enterprises are attaining a new standard of excellence for conversational AI.”

The State of Machine Translation 2024

What‘s changed in the MT market since 2023

52 Evaluated MT Engines and LLMs

Alibaba Cloud

Alibaba

Amazon

Baidu

DeepL

Elia

HiThink

Globalese

Google Cloud

Kawamura by NICT

Lingua Custodia

Microsoft

Mirai

ModernMT

Naver

NiuTrans

Oracle

Pangeanic

PROMT

SYSTRAN

Tarjama

TartuNLP

Tencent Cloud

Tilde

TREBE

Ubiqus

Yandex

Youdao

Aya-101

Claude 3 Haiku

Claude 3 Opus

Claude 3 Sonnet

Command-R

Command-R+

Gemini Pro

Gemini Pro 1.5

GPT-3.5 Turbo

GPT-4

GPT-4о

GPT-4о

GPT-4 Turbo

AI21

LLaMA-2

LLaMA-3

Mistral Large

Mixtral 8x7B

GPaLM2 Chat Bison

PaLM2 Text Bison

PaLM2 Unicorn

RakutenAI 7B Instruct

Titan Text Express

TowerInstruct 13B v0.1

TowerInstruct 7B v0.2