Glossary support in 57 MT engines and LLMs

Which MT engines and LLMs support glossaries?

Of the 57 models we evaluated, 42 have a way to take a glossary and 15 have none. For the LLMs we use retrieval-augmented generation (RAG): we insert the terms found in each segment into the prompt with the instruction to translate them exactly as specified. DeepL, Google Cloud Advanced, Google Translation LLM, ModernMT, SYSTRAN and Tilde take a glossary uploaded in advance, and Widn AI, ALL-Lab, Supertext and Yandex a dynamic glossary sent with each request, each with limits by language. For each target language, we use Intento Language Hub to deliver the same glossary through these different interfaces.

Glossary support has limits missing from the vendor documentation

Some models reject a glossary for some languages. Yandex rejects it for Belarusian, Ukrainian and Serbian in Latin script, Google Translation LLM for Serbian in Latin script, and DeepL for Belarusian and Serbian. ModernMT accepts a glossary for Arabic but does not apply any of its terms.

Both Google models return their internal glossary markup as visible text in one segment of every translation with a glossary. We score the translation as returned, because that is what a customer calling the API receives.

Terms set up inside a vendor account are a different mechanism from a glossary sent with the request. Tilde takes its terms that way, and the same request sent from another account comes back without them.

The glossary holds nouns, our matcher is case-sensitive, and we count declined forms as applied

The glossary holds nouns and noun phrases. When we chose the 46 terms we left out verbs, proper nouns that stay as they are, and generic vocabulary, so no model has to inflect a verb from a glossary entry.

SYSTRAN assigns a part of speech to each glossary entry itself and treats entries as hints. It never applies some exact matches, inflects the terms it inserts, and with a dictionary attached returns different output from run to run.

Our matcher is case-sensitive and matches whole words. For RAG and the dynamic glossary we send a term to the model only when the segment contains it in the casing used in the glossary, so a term in a different form in the source is not sent. Yandex also applies each term pair only as an exact match.

In scoring we accept a glossary term in any inflected or compound form, as long as its words stay. A synonym or a paraphrase is a miss. So is an abbreviation where the glossary has the full term, or the full term where the glossary has an abbreviation. LLMs often leave out an abbreviation in brackets that the glossary adds and the English source lacks, such as (IB) or (TSM).