We shortlist solutions with quality-estimation metrics, then assess them against translation requirements with Intento LQA.
Two metrics estimate quality without a reference translation. We assess each translation together with its source text. COMET-Kiwi and MetricX use different base models and training annotations.
- COMET-Kiwi XL (Unbabel, WMT 2023) uses XLM-R XL to predict the quality score a human annotator would give a translation.
- MetricX-24 (Google, WMT 2024) uses mT5 to predict an error score; lower scores indicate better quality. We use the hybrid version in reference-free mode.
Intento LQA assesses the 10 translation requirements. We assess every shortlisted solution using the same 177 segments per language, without a reference translation. General translation quality uses three LLM judges; tone of voice, terminology and gender neutrality have dedicated checks. Code checks formatting.
- Basic requirements
- General translation quality. We use three LLM judges built on Claude Opus 5, GPT-5.6 Sol and Gemini 3.1 Pro to identify errors by type and severity using the MQM framework. We assign a final score only when the judges agree on at least 90% of the segments, and combine their anonymized findings with Claude Fable 5.
- Additional Standard requirements
- Terminology. We check the translation against the 46 approved glossary terms using exact matching. Where matching alone is inconclusive, we use GPT-5.6 Sol for the check and Claude Opus 5 to review the findings.
- Tone of voice. We check sentences that address the reader or give instructions. We use GPT-5.6 Sol to identify the register and Claude Opus 5 to review the findings; the requirement is met when the register matches the requested formal tone.
- Tags. Each source tag must appear the same number of times in the translation, with the same attributes.
- Unicode character integrity. We check for replacement characters, control characters, text corrupted by incorrect encoding, and stray byte-order marks.
- Locale conventions. We check punctuation against target-locale conventions, including quotation marks, CJK punctuation and Spanish inverted marks. We report the rate as advisory.
- HTML entities. We check that the translation introduces no HTML character references beyond those in the source.
- Valid JSON. The translated document must remain valid JSON, with its keys unchanged.
- Spacing at tag boundaries. We check that tags do not join or split words, introduce double spaces, or add spaces before punctuation or after opening brackets.
- Additional Advanced requirement
- Gender neutrality. We check the segments that refer to people, ~115 of the 177 per language, using language-specific rules written by linguists. We use GPT-5.6 Sol to check whether each segment follows the rules and Claude Opus 5 to review the findings. Compliance is the percentage of segments that follow the rules.