About half the time, a translation that breaks a client requirement gets a higher COMET score than one that follows it. We scored 84,000 machine translations twice, once with reference-based COMET and once with requirements-based Intento LQA, and on locale formatting the violation scores higher 36 to 47% of the time in all four languages. On gender neutrality in German and Spanish it does so more often than not. Of the 9,217 requirement violations we identified, 46% are above 0.90 on the COMET scale of 0 to 1.
That gap changes which solution goes to production: a buyer who ranks six solutions by COMET shortlists one that meets 13% of the German gender-neutrality rule over one that meets 75%.
What does a COMET score actually measure?
Machine translation evaluation started in research, where the question is which system is better on average, but a client comes to us with a different question. They have a style guide, a termbase, placeholder conventions, a locale spec, sometimes a gender-neutrality rule, and they want to know whether a candidate setup satisfies what they wrote down.
A COMET score of 0.89 says only that the translation is close to one reference sentence somebody wrote at some point, possibly years ago and possibly under a different style guide, while the buyer asked four separate things: is our termbase applied, does the translation follow our style guide, what happened to the placeholders, does it comply with our gender-neutrality rule. One number can’t answer any of them on its own, and in earlier evaluation work we kept finding requirement failures only after go-live, once general-quality scoring had passed them, which is the most expensive moment to find them.
How do you test MT output against client requirements?
We composed the test set rather than sampling real content, because real content tests requirements unevenly and a thousand production segments might have three glossary terms and no placeholders at all. So we built about 450 sentence frames of enterprise IT and SaaS content (documentation, UI strings, release notes) and filled them from controlled lists (terms from a real termbase, placeholder families, register markers), generating source and references for all five languages together. The result is 3,500 source segments in four sub-sets: 2,000 GENERAL, 500 GLOSSARY covering all 213 termbase entries, 500 TOV with formal register throughout (Sie, usted, です・ます, 您), and 500 TAGS covering seven placeholder and markup families. Each segment went through six solutions into German, Spanish, Japanese and Chinese: 84,000 translations.
The sentences and the references are synthetic. The termbase isn’t: it started from real client glossaries, which we merged, reworked and normalized, so it derives from client data without reproducing any of it. The brief is one translation prompt per language standing in for a client style guide, covering terminology, register, tag handling and locale formatting, and in German and Spanish a gender-neutrality rule: no defaulting to the generic masculine, so a neutral noun, a rephrase or, in German, the colon form (Nutzer:innen).
Two instruments scored every translation. Reference-based COMET reads the source, the translation and one reference translation and returns a similarity score between 0 and 1. Intento LQA, the requirement framework, runs six checks: general quality (an MQM review in which a second LLM re-examines the severity ratings the first one assigned), tone of voice, glossary, formatting and locale, tags and placeholders, and the check we added for this study from the client questionnaire, an LLM judge applying the German and Spanish gender-neutrality rules. Two of the six are fixed rule-based checks, and none of them reads the reference, since each compares the translation only with the source and with the stated rule. Most of what follows comes from that one property.
How often does COMET rank a requirement violation above a correct translation?
For each requirement we label every segment pass or fail with its check, then ask how well the COMET score separates the two groups. The number on the chart is the answer to one question: pick a violation and a correct translation at random, and count how often COMET scores the violation higher. Zero would be a perfect metric, fifty is random, and above fifty the violation scores higher. We computed it over every segment in the set, so nothing here is a picked example.
The gender-neutrality row depends on one fact about the test set: the references don’t comply with that policy at all, using the generic masculine throughout, in both languages, in all 3,500 segments.
Gender neutrality is to the right of the 50% line: the violation scores higher than the correct translation 54% of the time in German and 56% in Spanish, above random on that requirement. Locale formatting is between 36% and 47% in all four languages, close to random in every language. Tone of voice and glossary are better than random and nowhere near well enough to gate on, and two sections down there are tone and terminology failures at 0.95 and above. On tags the gap is largest, not because of any markup evaluation but because a mangled placeholder produces a broken sentence, and broken sentences score lower on COMET.
We started with the share of failures above 0.90 and stopped using it for comparisons, because COMET averages differ across languages (Japanese averages 0.930, German 0.890). The fair comparison is against the average for compliant output in the same language, and by that measure about two thirds of gender-neutrality failures score above the compliant average: 61% in German, 65% in Spanish.
The ordering is the same in four unrelated languages: gender neutrality, then formatting, then tone, then glossary, then tags, with one tie between tone and glossary in Chinese. Two cells are empty on purpose: we scored gender neutrality on German and Spanish only, since both mark gender on the noun itself and we haven’t tested whether an equivalent requirement is meaningful in Japanese or Chinese. Japanese tone of voice has three violations in 3,000 segments, so neither instrument has enough data to compare.
Can you pick a translation solution by COMET score?
Scored both ways in German, the six solutions come out in different orders: on requirements the solution labeled LLM 2 ranks first and LLM 3 second, while on COMET, NMT 2 is second and LLM 3 is fifth. A buyer following the COMET ranking would shortlist a solution that meets 13% of the gender-neutrality rule over one that meets 75%.
The other columns show the same problem: NMT 3 fails the register requirement on 35% of German segments, which for B2B documentation is a rejected delivery rather than a matter of degree, while all six solutions are within about two points on COMET.
In Spanish the gap is wider still, with five of six positions differing between the two rankings on a 35-point requirement spread against 27 in German. NMT 2 is third on COMET, above LLM 3, while meeting 18% of the gender-neutrality rule and failing register on 40% of segments, and LLM 3, which meets 96% of the rule, is fourth. NMT 1 is last on requirements, with 25% register compliance and 20% on the gender-neutrality rule, and fifth on COMET, two points off the leader.
Japanese and Chinese behave differently, because without a gender-neutrality requirement in the brief five of the six solutions land within about three points of each other on the spec, and at that spread a buyer learns little from either ranking. Japanese technically reorders, Chinese mostly doesn’t, and we wouldn’t read anything into either. The two rankings differ where the requirement set separates the candidates, and in German and Spanish that separation comes almost entirely from gender neutrality, the one requirement where the COMET score is below chance.
Why NMT engines can’t follow a style guide
Part of the gap is a delivery gap, and it depends on what a solution is: a model plus the harness around it, meaning the prompt, the glossary, the pre- and post-processing. The three solutions built on LLMs received the translation brief alongside their glossaries, while the three built on NMT APIs received the glossaries and nothing else, because a bare NMT API has nowhere to put a style guide. On the German gender-neutrality rule the prompted solutions score 75 to 95% and the unprompted ones 9 to 17%, and Spanish has the same shape, 72 to 98% against 16 to 20%. The gap applies only to requirements the brief contains, so on terminology, which is provided through the API, the NMT-based solutions stay between 78% and 97%.
A solution that never received the brief can’t follow it, and a general-quality score does not make that visible. NMT 2 is still second on COMET in German and third in Spanish. Before any ranking of outputs comes the question of whether a solution can take the requirements in at all.
Why optimizing for the reference translation can be the failure
The brief required the German gender neutrality, while the reference corpus, generated before that rule existed, uses the generic masculine throughout. The two were out of sync, and we counted zero compliant forms in the 3,500 German references and zero in the 3,500 Spanish ones.
Three solutions rendered one source sentence, “The incident responders receive a monthly summary”, three different ways.
LLM 3 produced the generic masculine, identical to the reference and failing the requirement, at 0.975. LLM 2 used the compliant colon form at 0.943, and NMT 1 produced a neutral collective noun, the form the policy prefers most, at 0.856. On COMET the three scored in the reverse of what the client asked for.
The pattern isn’t limited to one segment: 333 of the violations match the reference word for word, 111 in German and 222 in Spanish, and they average 0.964. Inside the German output, the most common compliant form (the colon) averages 0.867, while the average violation is 0.880.
Zero of the 3,500 references in either language comply with the gender neutrality the brief required.
Whether or not a test set falls out of sync like this on purpose, it is the ordinary case in production, where companies rewrite style guides after a rebrand or a compliance review and nobody goes back and updates ten years of translation memory. A metric based on the reference reflects whichever rule was in force when somebody wrote that reference, while the requirement checks are independent of the reference, so none of this affects them.
Does COMET penalize correct translations that differ from the reference?
Scored against a reference, a different answer and a wrong answer look alike. We included 187 segments per language in which the reference uses one valid construction while a competent solution would pick another, equally valid one. Some of those segments contain role nouns, so the gender-neutrality effect from the previous section could affect them, and everything in the table below excludes role-bearing segments entirely.
Three of the eleven comparison structures show a clear effect: when a solution turns a verb phrase into a noun phrase (−0.036 in German, −0.041 in Spanish), uses a general verb plus noun instead of a specific verb (Spanish, −0.028) or drops the subject pronoun (German, −0.027), the output scores lower on COMET, and in every case those segments score above the corpus average on the requirement checks, because they are clean. Three of eleven is a weaker result than we’d have liked, and we would rather say so than show the eleven-row version.
In the example below the English reads “Once you submit the form, you cannot edit the access review any further.” The reference rewrites it as a noun phrase and drops the “you”, while LLM 1 kept the second person, which mirrors the English and uses exactly the formal address the brief asked for. No issues came up in the checks, the segment scored 100 of 100, and the COMET score was 0.764 against a German average of 0.890.
Distance from a reference cannot separate a wrong answer from a different one.
A team that tunes output to raise a COMET score ends up matching the reference, whether or not the client asked for it.
High COMET scores on translations that break the spec
The references above are out of date, so the next eight failures, two per language and among the highest-scoring in each, all come from segments where the reference complies with the rule the example tests.
German: informal address where the spec says formal, 0.951. Spanish: a missing space before the percent sign, a hard rule of Spanish typography, 0.970. Chinese: the informal possessive where the spec requires the polite form, 0.950. Japanese: a missing space between a number and its unit, 0.970. The Spanish and Japanese references have the space, and the Chinese reference uses the formal pronoun. On the specific rule each example tests, the reference gets it right, the solution broke it anyway, and all four are still at the top of the COMET range.
The terminology cards work the same way. In Spanish the termbase says garantía de calidad, quality assurance, and the solution wrote control de calidad, quality control, a different discipline. The COMET score is 0.958 against a compliant average of 0.913, German left “Preparation” untranslated at 0.944, and Japanese and Chinese picked near-synonyms the termbase forbids, 要約 for 概要 and 通知 for 注意, at 0.971 and 0.954. A termbase is the first requirement a buyer hands us, and no metric without a termbase can test for this.
The references here comply with the rule each example tests, and they are still not gender-neutral, which is the whole of the previous section. “Among the highest-scoring” is deliberate too: a German quote-rule failure at 0.953, two Japanese full-width-parenthesis failures at 0.976 and 0.972, and several Chinese date-spacing failures up to 0.970 score higher than the cards shown, all with compliant references. None of the eight is a matter of taste: each is binary and checkable, and a client rejects a delivery over any one of them.
Why one reference can never cover every valid translation
The German policy allows three valid forms, and we counted three across the six solutions, while the Spanish policy bans every glyph and allows six: collective nouns, neutral head nouns, relative clauses, the impersonal se, and so on. A reference contains exactly one of them, so you could regenerate every reference tomorrow under the new rule and any solution that picks a different valid form would still lose points for being correct.
A reference contains one of several valid forms, and a reference-based metric penalizes every other one.
Why an LLM judge does not catch the same requirements as COMET
The obvious next step is to drop the reference and score with a language model, so we tested that with our own general-quality judge over the same failing segments, where the violation scores higher between 41% and 55% of the time. Locale formatting is close to random in all four languages, and gender neutrality comes out at 48% in German and 55% in Spanish. COMET on the same subset lands between 37 and 59%. The judge scored the GENERAL set only, so we can’t test tone, glossary and tags against it, and there is no human review anywhere in this study: the judge is an LLM-as-a-judge metric like the rest of the requirement checks.
Replacing the instrument doesn’t change the question, and the question is still general quality. A judge configured without the client rule cannot test for it, and once you tell it the rule, you have written a requirement check.
How a new client requirement becomes an automated check
The gender neutrality also shows how the framework extends to new rules, because the German and Spanish rules are completely different: German permits the colon glyph, so compliance is often a single substitution, while Spanish bans every glyph and requires rephrasing. We encoded both as checks and got a per-solution result in one pass.
The spread between the best and the worst solution in Spanish is 82 points, on a requirement where the violation is on top in the COMET ranking 56% of the time. A requirement changes on a business timeline, after a rebrand or a compliance rewrite, and reference data isn’t updated that fast, since nobody regenerates a reference corpus because the style guide changed on Tuesday. A check built from the rule compares the translation with the source and with the rule, so it works from the day the requirement changes, while a reference-based score still reflects the old one.
When COMET is the right tool, and when it can’t gate a release
None of this makes COMET a bad metric. It does what its authors designed it to do, and two of its properties are ones our own checks lack.
It is consistent and cheap: our judge changed its score on 4.3% of repeated identical inputs, with an average difference of 14.2 points, and COMET never does. One class of error went the other way: “read only” rendered as lesegeschützt, read-protected, which inverts the meaning in a permissions context. Of the six cases in the study data, all from one solution, three scored a clean 100 on the judge, while all six scored 0.62 on COMET against a 0.89 German average. On damaged markup it ranks well, with a broken tag falling below a clean segment 94% of the time in Chinese, though that is the effect from Figure 1, where broken tags produce broken sentences and COMET scores broken sentences lower, rather than a markup check: in German, correctly repositioning a tag because word order moved it costs about 42% of what a real break costs.
It has no pass mark, because even a translation identical to the reference word for word averages 0.967, so there is no COMET value above which a translation passes, and no way to test register, termbase or locale. And at the error rates you see in production it can’t act as a gate even on tags, where it ranks best: tag defects make up one to two percent of segments, so if you use COMET as a filter to catch 80% of them, 98.5% of what it flags in German is actually clean, and 86.5% in Chinese.
To catch 80% of German tag defects by COMET score, a reviewer reads 2,057 clean segments for every 32 broken ones.
Ranking well and being usable as a gate are different things, and COMET is useful between releases: a score change means something changed.
Why “can I ship this” is the question enterprise buyers ask
These two instruments answer different questions: “how does this compare” and “can I ship this”. Both are legitimate, and enterprise buyers ask the second one.
From a score you learn how translations compare. From a requirement check you learn whether you can release.
I work at Intento, and the requirement checks in this study are the ones we built and run. What we do about the result above is the same instrument at three points of a translation program: we score candidate solutions against the client spec before delivery, we configure each run from the questionnaire that client filled in (their tone, their termbase, their tag conventions, their gender-neutrality rule), and we re-run the same checks on live content so that what breaks updates the prompts, glossaries and routing. That cycle is what the requirement checks exist for.
In short
- 84,000 machine translations (3,500 source segments, four languages, six solutions), each scored with reference-based COMET and with a check for every requirement in the brief.
- 9,217 requirement violations found, 46% of them above 0.90 on COMET.
- COMET ranks a violation above a correct translation 36 to 47% of the time on locale formatting in all four languages, and 54% (German) and 56% (Spanish) on the gender neutrality.
- Zero of the 3,500 references in either language comply with the gender-neutrality rule. 333 violations match the reference word for word and average 0.964.
- Eight failures with compliant references score 0.944 to 0.971, above the average for correct output in the same language.
- A buyer following the COMET ranking in German shortlists a solution at 13% gender-neutrality compliance over one at 75%. In Spanish five of six positions differ.
- A general-quality LLM judge over the same segments ranks the violation on top 41 to 55% of the time.
- COMET is consistent and useful for meaning inversions that the judge did not catch, but it cannot gate: to catch 80% of German tag defects, 98.5% of COMET flags are false alarms.
Method note
Corpus. We composed 3,500 source segments from about 450 sentence frames (roughly eight segments per frame, so patterns recur), enterprise IT and SaaS content, in four sub-sets: GENERAL 2,000 (187 segments where a correct translation must differ from the reference), GLOSSARY 500 (213 termbase entries, up to 42 do-not-translate terms per language), TOV 500 (formal register), TAGS 500 (seven placeholder and markup families). Target languages German, Spanish, Japanese, Chinese. Six solutions: three built on LLMs with the translation brief and a glossary lookup, three built on NMT APIs with glossaries only. We anonymize them as LLM 1–3 and NMT 1–3 throughout, because the study is about the instruments rather than the vendors.
Data. Sentences and references are synthetic. The termbase derives from real client glossaries, which we merged into one canonical English list. We regenerated or overrode a large share of targets, dropped 60 out-of-domain entries and normalized the rest. No client content appears in the set and we claim no real client: we specified the requirement set ourselves, modeled on the questionnaire we use in a requirements-based setup.
The gender-neutrality mismatch. We generated the references before the gender-neutrality rule entered the brief, and none comply with it. We counted the mismatch before scoring (0 compliant forms in either reference set) and did not fix it. It turned out to isolate the largest result in the study, and we don’t claim we engineered it.
Instruments. Reference-based COMET, source plus translation plus one reference, 0 to 1. Requirement checks: general quality (MQM categories, a second LLM re-examines the severity ratings), tone of voice (LLM, must quote the phrase behind its call), glossary (exact term matching, LLM only for inflection conflicts), formatting and locale (fixed rules per locale), tags and placeholders (rule-based source-target comparison), gender neutrality (LLM judge, German and Spanish only). Every LLM-based check is an LLM-as-a-judge metric. There is no human MQM anywhere in this study, and we checked every example by hand.
Statistics. “How often COMET ranks the violation above a correct translation” is the share of random violation-vs-correct pairs where COMET scores the violation higher, computed over every segment in the sub-set, and covers the six study solutions only. The judge comparison covers the GENERAL set only. The comparison of valid alternatives (Figure 5) excludes every segment with a role noun, and 3 of 11 comparison structures remain, with 48 to 102 role-free rows per remaining cell, so the combined result and the German and Spanish subsets are solid while single cells are indicative. Formatting compliance is 95 to 100% across solutions, so formatting failures concentrate in two of them. Chinese confirms the ordering and is the weakest language for the argument: the combined comparison result is not significant there and the rank flip does not replicate. Japanese tone of voice has three violations, so we make no register claim for Japanese in either direction. We recomputed every figure from the raw scoring and check files before publication. Daria Sinitsyna presented this work at AMTA 2026 in Québec City on September 2.
References. Rei, R., Stewart, C., Farinha, A. C., Lavie, A. (2020). COMET: A Neural Framework for MT Evaluation. EMNLP 2020. https://aclanthology.org/2020.emnlp-main.213/ · Freitag, M., Mathur, N., Lo, C. et al. (2023). Results of WMT23 Metrics Shared Task: Metrics Might Be Guilty but References Are Not Innocent. WMT 2023. https://aclanthology.org/2023.wmt-1.51/ · Lommel, A., Uszkoreit, H., Burchardt, A. (2014). Multidimensional Quality Metrics (MQM): A Framework for Declaring and Describing Translation Quality Metrics. Tradumàtica 12. https://ddd.uab.cat/record/130144
