Valid JSON output from LLM and MT engine translations

How do LLMs and MT engines compare on valid JSON?

Translations from LLMs have an invalid JSON document in 3% of the cases, from open-weight LLMs never, from LLM-based MT engines in 74% and from other MT engines in 92%. Translations from Translated Lara 2, Microsoft Translator, Google Translation LLM and Widn AI Tower Sugarloaf have one in every language where we shortlisted them, while translations from Google Gemini 3.1 Pro Preview and OpenAI GPT-5.5 are valid in every language.

Translations from every shortlisted model return a valid JSON document in Frisian, Kazakh and Lithuanian

In Marathi, Georgian and Mongolian, most of the invalid documents come from Translated Lara 2, DeepL Next Gen and Google Translation LLM.

Translations from Anthropic Claude Sonnet 4.6 return a valid JSON document in 21 of 31 languages

Translations from the three OpenAI GPT-5.6 models, Anthropic Claude Opus 4.8 and Anthropic Claude Sonnet 5 return a valid one in every language where we shortlisted them, and those from DeepL Next Gen in 40 of 69.

Every two-step and three-step solution returns a valid JSON document

This includes the two-step solutions (translation + post-edit) and three-step solutions (translation + LQA + repair) built on a one-step solution with an invalid JSON document. In two-step solutions, we also repair tags and formatting by code. We measured these fixes on LLM solutions, since we leave out the MT engine solutions as too far off.

Disclaimers

• The JSON document is valid when the translation still parses as JSON. We check it on the one JSON text of the four.
• We assess only the shortlisted one-step solutions (10 to 13 per language by COMET-Kiwi or MetricX, with at least three real-time solutions among them), so a model has no result in a language where it is not on the shortlist.
• The results come from four business texts of ~3,000 words. Other content can show other results.
• We build two-step and three-step solutions only in the languages with fewer than two solutions that meet every Standard requirement, so their results describe those languages.