Gender neutrality in LLM and MT engine translations

How do LLMs and MT engines compare on gender neutrality?

The language matters more than the type of model. Translations into genderless languages, which have no gendered nouns or pronouns, have a median of 0 gender-neutrality errors per 1,000 words, against 3.1 in gendered and natural gender languages. By type, translations from LLMs have gender-neutrality errors in 57% of the cases and translations from MT engines in 78%, with a median of 1.0 and 3.1 gender-neutrality errors per 1,000 words.

Two or more one-step solutions are fully gender-neutral in 40 of the 82 languages

25 of the 40 are genderless languages, such as Armenian, Estonian, Hungarian, Kazakh and Malay. Translations into Arabic, Hebrew and Urdu have the highest medians, 26.7, 22.6 and 15.2 gender-neutrality errors per 1,000 words, and translations into 35 languages a median of 0.

Translations from Anthropic Claude Opus 4.8, Google Translation LLM and OpenAI GPT-5.6 Luna have a median of 0 to 0.2 gender-neutrality errors per 1,000 words, among the fewest

Translations from Google Gemini 3.1 Pro Preview have a median of 1.4 gender-neutrality errors per 1,000 words, from DeepL Next Gen 2.4, from Microsoft Translator 3.0 and from Cohere Command A Translate 3.2.

With the gender-neutrality agent, two-step and three-step solutions have a median of 0 gender-neutrality errors per 1,000 words

With the agent, 47% of the one-step versions are fully gender-neutral, 54% of the two-step versions and 68% of the three-step versions. In a one-step solution, the agent works while we translate the source sentences, and the median number of gender-neutrality errors per 1,000 words goes from 4.1 to 0.3. In two-step solutions (translation + post-edit) it goes from 3.1 to 0, and in three-step solutions (translation + LQA + repair), where the agent works in the repair with the gender findings, from 1.9 to 0.

Disclaimers

• A gender-neutrality error is a sentence that refers to a person and is not gender-neutral. Each language has one gender-neutrality strategy, chosen from the options that language offers: in the 27 genderless languages it is not to add a gender the source does not specify, and in the gendered and natural gender languages it is neutral rephrasing. We group the languages as Prewitt-Freilino, Caswell and Laakso (Sex Roles, 2012) do: gendered languages have masculine and feminine words, natural gender languages gender only their pronouns (他 and 她 in Chinese, han and hun in Danish), and genderless languages have neither.
• We use the rebuilt gender judge for every solution. It does not count non-person terms, epicene forms (one form for every gender) and organizations named by a role as references to a person.
• We assess only the shortlisted one-step solutions (10 to 13 per language by COMET-Kiwi or MetricX, with at least three real-time solutions among them), so a model has no result in a language where it is not on the shortlist.
• The results come from four business texts of ~3,000 words. Other content can show other results.