Localized punctuation in LLM and MT engine translations

How do LLMs and MT engines compare on localized punctuation?

At the median, 25% of the punctuation in translations from LLMs follows the conventions of the target language, against 50% for LLM-based MT engines, 23% for open-weight LLMs and 0% for other MT engines. By model, the median goes from 0% for Microsoft Translator and Anthropic Claude Sonnet 4.6 to 98% for Google Translation LLM.

Languages written in Japanese, Chinese, Arabic, Greek, Armenian or Cyrillic script have the most localized punctuation

At the median, 95% of the punctuation in Japanese translations follows the conventions of the target language, 88% in languages written in Chinese script, 80% in Arabic script, 74% in Greek and Armenian, and 73% in Cyrillic script. Languages written in Latin script have a median of 24%, from 0% in Basque, Norwegian and Slovenian to 90% in Lithuanian, and languages written in the scripts of South and Southeast Asia, Korean and Hebrew 10% to 22%.

Translations from most LLMs follow the target-language conventions for less than 50% of the punctuation

In translations from OpenAI GPT-5.6 Luna and the two xAI Grok 4.20 models, 72% to 74% of the punctuation follows them, in those from Google Gemini 3.1 Pro Preview 15% and in those from z.ai GLM-5 10%.

Disclaimers

• Locale conventions cover punctuation such as guillemets, the punctuation marks of Chinese and Japanese, and the Spanish ¿ and ¡. The share counts the punctuation in the translation that has a localized form in the target language.
• We assess only the shortlisted one-step solutions (10 to 13 per language by COMET-Kiwi or MetricX, with at least three real-time solutions among them), so a model has no result in a language where it is not on the shortlist.
• The results come from four business texts of ~3,000 words. Other content can show other results.