Gender neutrality in LLM and MT engine translations

How do LLMs and MT engines compare on gender neutrality?

The language matters more than the type of model. Translations into genderless languages, which have no gendered nouns or pronouns, have a median of 0 gender-neutrality errors per 1,000 words, against 5.1 in gendered and natural gender languages. By type, translations from LLMs have gender-neutrality errors in 67% of the cases and translations from MT engines in 86%, with a median of 1.1 and 5.5 gender-neutrality errors per 1,000 words.

Two or more one-step solutions are fully gender-neutral in 34 of the 82 languages

23 of the 34 are genderless languages, such as Armenian, Estonian, Hungarian, Kazakh and Malay. Translations into Hebrew, Arabic and Lithuanian have the highest medians, 27.0, 22.4 and 19.7 gender-neutrality errors per 1,000 words, and translations into 27 languages a median of 0.

Translations from Google Translation LLM, OpenAI GPT-5.5 and Anthropic Claude Opus 4.8 have a median of 0.3 to 0.4 gender-neutrality errors per 1,000 words, among the fewest

Translations from Google Gemini 3.1 Pro Preview have a median of 1.3 gender-neutrality errors per 1,000 words, from DeepL Next Gen 2.3, from Microsoft Translator 4.6 and from Cohere Command A Translate 11.8.

With the gender-neutrality agent, two-step and three-step solutions have a median of 0 gender-neutrality errors per 1,000 words

With the agent, 47% of the one-step versions are fully gender-neutral, 54% of the two-step versions and 68% of the three-step versions. In a one-step solution, the agent works while we translate the source sentences, and the median number of gender-neutrality errors per 1,000 words goes from 6.6 to 0.3. In two-step solutions (translation + post-edit) it goes from 3.6 to 0, and in three-step solutions (translation + LQA + repair), where the agent works in the repair with the gender findings, from 2.5 to 0.

Disclaimers

• A gender-neutrality error is a sentence that refers to a person and is not gender-neutral. Each language has one gender-neutrality strategy, chosen from the options that language offers: in the 27 genderless languages it is not to add a gender the source does not specify, and in the gendered and natural gender languages it is neutral rephrasing. We group the languages as Prewitt-Freilino, Caswell and Laakso (Sex Roles, 2012) do: gendered languages have masculine and feminine words, natural gender languages gender only their pronouns (他 and 她 in Chinese, han and hun in Danish), and genderless languages have neither.
• For the two best solutions of each language we use the rebuilt gender judge, which does not count non-person terms, epicene forms (one form for every gender) and organizations named by a role as references to a person. For the other one-step solutions we use the earlier judge, so we mix the two judges in the medians by model and by language.
• We assess only the shortlisted one-step solutions (10 to 13 per language by COMET-Kiwi or MetricX, with at least three real-time solutions among them), so a model has no result in a language where it is not on the shortlist.
• The results come from four business texts of ~3,000 words. Other content can show other results.