MT engines miss general translation quality, tag handling and formatting in nearly every language, off the shelf or adjusted with a glossary and formality.
For LLMs, we assess the adjusted version. 2% of the one-step LLM solutions on our shortlists are off the shelf, because we bring forward the adjusted version unless the off-the-shelf one scores measurably higher on the automatic metrics.
Share of one-step solutions meeting each requirement
| Requirement | MT engines, off the shelf | MT engines, adjusted | LLMs, adjusted |
|---|---|---|---|
| General translation quality | 0% | 5% | 18% |
| Tone of voice | 56% | 66% | 89% |
| Terminology | 0% | 27% | 53% |
| Tag handling | 1% | 0% | 57% |
| Formatting | 1% | 5% | 78% |
Share of the one-step solutions with a conclusive LQA result that meet the requirement. Adjusted MT engines: a glossary and a formality setting through the API. Adjusted LLMs: tone of voice, terminology and other context in the request. Terminology only in the 16 languages with a glossary.