We were super busy last half year with product development, but finally, it’s time to catch up on what changed in the Machine Translation market. We run our regular benchmark for pre-trained Machine Translation engines and eager to share our findings:
- 📊 Overall MT Quality significantly improved for Finnish<>English, German<>French, Romanian>English, Russian>English, Chinese>English. We have updated some of the datasets to WMT-2019, which led to quality fluctuations, but the language pairs above improved beyond that.
- 🏆 The best MT provider has changed for 19 language pairs since January 2019. To get the best quality across 48 language pairs, one needs 8 different engines.
- 🌏 Many engines increased their language coverage: Google, Amazon, Kakao, Systran PNMT, SDL, PROMT, ModernMT, IBM. Google actually added Portuguese Brazilian (although this is not reflected in docs).
- 📦 New pre-trained engines: Alibaba, Tencent, Tilde (EN-PL) and Cloud Translation.
Read the report on Slideshare.
If Slideshare is blocked by your government or company, we have another copy here: https://docsend.com/view/z3f2pes
Overall MT Quality
The number of language pairs with an average score over 0.7 continues to increase. Based on the projects with LQA we had recently, below 0.7 clearly indicate linguistic errors, hence MT still has a lot of room to grow.
Sanity check
Although human evaluation is beyond the scope and budget of our public reports, we have showcased how we validate our results for the cases we have questions about.
For example, we had a concern related to a strong leader in Finnish > English, so we checked a number of median sentences with high score deviation to make sure those are attributed to different intelligibility.
Please note that the border between “perfect” and flawed translations is very different across different language pairs. Perhaps this is correlated with the language pair “complexity”.
Also, as you may notice in the 0.7–0.75 range the mistakes are about wrong words in the right places, which in practice can be cured with domain adaptation and glossaries.
MT Quality changes
The fragmentation among 4–5 leaders increases. Also, we start to see cases that significantly more expensive MT (e.g. DeepL or ModernMT) is significantly better for some language pairs. It’s still far from the case when all leading MT engines provide a comparable level of quality for all language pairs.
A quick summary of changes among the best-performing engines:
New MT Engines
- Alibaba launched its Machine Translation on the international site of Alibaba Cloud, with a list price of $15–33 per 1M characters translated (we haven’t evaluated its quality yet).
- Yandex Cloud also launched its Machine Translation for international clients, further reducing the price to $5.74.
- Tencent launched Machine Translation in production for its users in Mainland China, for 58 CNY (~$8) per 1M character. We found it’s the best for English to Chinese — please let us know what you think!
- Another Chinese MT provider, Cloud Translation, offers custom-built NMT engines. Recently they started to provide vertical-specific pre-trained NMT engines. We have their EN-ZH Medicine available via Intento.
- Tilde also started to offer its baseline engines, with the first one being general-purpose English-to-Polish engine, available via Intento.
Conclusions
- Machine Translation landscape continues to evolve, both in terms of quality and price.
- Language coverage is increasing faster than ever.
- Even for the general domain, having the best quality across 48 language pairs requires 8 engines used simultaneously (and those are different from half a year ago).
- Re-evaluate your MT choice often to stay competitive.
Our MT API Gateway helps to evaluate and use multiple MT vendors — you’re welcome to try it as well, just register at https://console.inten.to. Also, we have plugins for MemoQ, SDL Trados, Matecat, and may integrate with your favorite TMS via XLIFF and webhooks — let us know!
