{"id":11,"date":"2022-11-23T15:25:18","date_gmt":"2022-11-23T15:25:18","guid":{"rendered":"https:\/\/mtuniversity.inten.to\/?p=11"},"modified":"2023-05-02T10:20:48","modified_gmt":"2023-05-02T10:20:48","slug":"automated-scoring-and-evaluation-of-mt-engines","status":"publish","type":"post","link":"https:\/\/inten.to\/machine-translation-university\/automated-scoring-and-evaluation-of-mt-engines\/","title":{"rendered":"Automated scoring and evaluation of MT engines"},"content":{"rendered":"

MT Evaluation Goals<\/span><\/h2>\n

Evaluating machine translation may appear to be solely focused on determining translation quality, but there is more to it than that. The chosen models must be approved by various departments, including security, legal, and procurement, ensuring they meet all requirements.<\/span><\/p>\n

To effectively evaluate MT engines, know why you are using MT and what you hope to achieve. Here are some common goals for MT evaluation:<\/span><\/p>\n

    \n
  1. Selecting the best MT model<\/b>: Pick the most suitable machine translation model that aligns with your domain, language pair, and content type for optimal results.<\/span><\/li>\n
  2. Gathering data to enhance MT and identifying bottlenecks<\/b>: Collect relevant data to improve MT performance and identify areas of improvement, ensuring a smoother translation process<\/span><\/li>\n
  3. Assessing risk factors<\/b>: Identify potential risk factors associated with implementing MT, such as data security, to mitigate any negative impact on your project.<\/span><\/li>\n
  4. Evaluating the source content’s compatibility with MT<\/b>: Assess how well your source content aligns with MT capabilities, making any necessary adjustments to ensure a seamless translation<\/span><\/li>\n
  5. Gaining end-user trust and confidence<\/b>: Foster confidence in end-users by collecting their feedback and demonstrating the effectiveness of your chosen MT solution.<\/span><\/li>\n
  6. Establishing fair machine translation post-editing (MTPE) rates<\/b>: Measure machine translation post-editing (MTPE) efforts to set reasonable rates to compensate editors for their work<\/span><\/li>\n
  7. Implementing translation triage<\/b>: Apply a translation prioritization system to allocate resources effectively, ensuring high-quality translations for the most critical content<\/span><\/li>\n
  8. Estimating return on investment (ROI)<\/b>: Calculate ROI for your MT project, considering cost savings, improved efficiency, and overall translation quality.<\/span><\/li>\n<\/ol>\n

    MT ROI Framework for Localization Use-Case<\/span><\/h3>\n

    In this section, we will focus on the post-editing case of the MT ROI Framework. There are two approaches to evaluation: automatic and human evaluation.<\/span><\/p>\n

    \"\"
    Figure 32. MT ROI framework for Localization use-case<\/figcaption><\/figure>\n

    MT Evaluation Types<\/span><\/h2>\n
      \n
    1. Automatic Evaluation:<\/span>\n
        \n
      1. Reference-Based Scoring: Compare MT output to a human-generated reference translation, gauging translation quality through quality metrics.<\/span><\/li>\n
      2. MTQE Scoring: Use Machine Translation Quality Estimation (MTQE) metrics to predict translation quality without a reference.<\/span><\/li>\n<\/ol>\n<\/li>\n
      3. Human Evaluation:<\/span>\n
          \n
        1. Linguistic Quality Assessment (LQA): Conduct LQA to measure post-editing effort and translation quality by translators and editors, considering factors such as accuracy, style, and consistency.<\/span><\/li>\n
        2. Holistic Evaluation: Gather feedback from end-users who assess the overall translation quality.<\/span><\/li>\n<\/ol>\n<\/li>\n<\/ol>\n

          Human evaluation is often considered the benchmark for determining translation quality. Human evaluators possess linguistic and cultural expertise, enabling them to comprehend nuances, idiomatic expressions, and context-specific meanings in both the source and target languages. Human evaluators can identify subtle errors or inconsistencies that automated evaluation methods might overlook.<\/span><\/p>\n

          However, human evaluation is labor-intensive and time-consuming, particularly when examining thousands of content segments. Human evaluators can only review some small parts when the vast content requires their attention. Consequently, evaluators often use sampling, selecting a subset of content for assessment. While this method can save time and resources, random sampling represents only some of the content adding the risk of missing business-critical errors.<\/span><\/p>\n

          We combine automatic and human evaluation, using smart sampling to make the process reliable, directing reviewers’ attention to the most relevant segments. Smart sampling guarantees critical information and context-specific nuances are captured, leading to more accurate and reliable translation assessments. This ensures that the effort spent evaluating 1,000 words represents the effort required for half a million words in production.<\/span><\/p>\n

          The primary objective is to reduce the content review volume by approximately 200 times, which correlates with reducing review time.\u00a0<\/span><\/p>\n

          Translation quality scores (metrics)<\/span><\/h2>\n

          Translation quality scores are numerical values assigned to machine-translated content to measure how close the translation is to a \u201cgolden\u201d reference. There are several ways to calculate them:<\/span><\/p>\n