如何评估机器翻译系统质量？主流系统公开指标结果是否可查？

Great question! Let’s break this down into two clear parts: how to evaluate machine translation (MT) quality, and where you can find benchmark results for the popular systems you mentioned.

There are two primary categories of evaluation methods, each with their own strengths:

自动评估指标 #

These are fast, scalable ways to score translations programmatically, using reference human translations as a baseline:

• BLEU (Bilingual Evaluation Understudy): The most widely used metric. It calculates the overlap of n-grams (word sequences) between the MT output and reference translations. It’s great for batch testing but can miss nuances like natural word order or synonym usage.
• METEOR (Metric for Evaluation of Translation with Explicit ORdering): Fixes some of BLEU’s gaps by accounting for synonyms, stem matching, and minor word order adjustments. It correlates better with human judgments but is slightly more computationally expensive.
• LEPOR: Combines lexical precision, recall, and a "position difference" score to account for word order—making it particularly useful for language pairs with big structural differences (like Chinese ↔ English).
• Bonus metrics: CHRF (Character-level n-gram matching, ideal for morphologically rich languages) and TER (Translation Edit Rate, which counts how many edits are needed to turn the MT output into a reference translation).

人工评估 #

The gold standard for quality, since no automated metric can fully capture human-like fluency and context understanding. Human evaluators typically score translations on two key dimensions:

• Fluency: Does the translation read naturally in the target language?
• Fidelity: Does it accurately convey the exact meaning of the source text?
  Some evaluations also add a "appropriateness" score for context-specific translations (like formal business writing vs. casual chat). The downside? It’s time-consuming and costly, so it’s usually reserved for small, high-stakes test sets.

Most commercial MT platforms don’t publish full, up-to-date official metrics across all language pairs, but you can find reliable data from academic benchmarks, third-party studies, and industry evaluations:

• Google Translate: Consistently tops leaderboards at events like WMT (Workshop on Machine Translation). For common language pairs (e.g., English ↔ French), its BLEU scores often land in the 35–45 range; for more challenging pairs (e.g., English ↔ Chinese), scores typically sit between 28–38.
• DeepL: Independent studies frequently show it outperforming Google on European language pairs, with slightly higher BLEU/METEOR scores and stronger human ratings for fluency. It’s particularly strong in Germanic and Romance languages.
• Microsoft Translate: A close competitor to Google in WMT benchmarks, with strong performance across high-resource and many low-resource languages. Its public BLEU scores are often on par with Google’s for most major language pairs.
• Yandex Translate: Shines in Russian and Eastern European language pairs. In WMT’s Russian-focused tasks, its BLEU scores are among the highest, and it’s known for accurate translations of Slavic languages.
• Papago (Naver): Specializes in East Asian language pairs (Korean ↔ Chinese, Korean ↔ Japanese). Third-party tests show its BLEU scores are comparable to Baidu’s, with a slight edge in Korean ↔ Chinese translations.
• 百度翻译: Delivers consistent results for Chinese-centric pairs (Chinese ↔ English, Chinese ↔ Japanese). Public benchmarks place its BLEU scores between 25–35 depending on the language pair, with steady improvements in low-resource language support.
• Promt: Focused heavily on Russian translation (Russian ↔ English, Russian ↔ Chinese). It has solid fidelity scores in these pairs, and you can find its BLEU results in Russian-language MT evaluations and academic papers.
• Apertium: An open-source rule-based MT system, ideal for low-resource and regional language pairs (e.g., Spanish ↔ Portuguese, Catalan ↔ Spanish). Its BLEU scores are generally lower than neural MT systems, but it’s valued for its transparency and customizability—you’ll find its metrics in academic studies comparing rule-based vs. neural MT.
• Openlogs: This is a more niche system focused on domain-specific translations. Public benchmark data is limited, but you can check its official documentation or related academic research for any published metrics.

If you’re hunting for exact, up-to-date scores, prioritize recent WMT reports, ACL/EMNLP conference papers, or independent MT comparison studies—just remember that MT systems are updated regularly, so older data might not reflect current performance.

内容的提问来源于stack exchange，提问作者Valeriu