如何使用huggingface Helsinki-NLP模型避免翻译丢失Emoji及OOV Token？

期望效果：
"Bonjour ma France 🇫🇷"@fr → "Hello my France 🇫🇷"@en

经测试发现，预训练默认分词器的词表中包含该Emoji，但在解码前会丢失。示例代码如下：

from transformers import *

# 初始化设置
engine = 'pt'
resource = 'huggingface--Helsinki-NLP--opus-mt-ROMANCE-en'
nlp = pipeline(
   task="translation",
   model=MarianMTModel.from_pretrained(resource),
   tokenizer=AutoTokenizer.from_pretrained(resource),
   framework=engine
)

# 推理
translated = nlp.tokenizer.batch_decode(
   skip_special_tokens=True,
   sequences=nlp.model.generate(
      **nlp.tokenizer(
         text=["Bonjour ma France 🇫🇷"],
         return_tensors=engine
      )
   )
)

# 结果
print(translated) # ['Hello, my France.']
print("🇫🇷" in nlp.tokenizer.get_vocab()) # True

解决方法 #

方案一：调整生成参数强制保留Token #

MarianMT模型默认生成逻辑会过滤非核心文本符号，即使分词器能识别Emoji。可以通过修改generate()的参数，让模型更倾向于保留输入中的特殊Token：

from transformers import *

engine = 'pt'
resource = 'huggingface--Helsinki-NLP--opus-mt-ROMANCE-en'
nlp = pipeline(
    task="translation",
    model=MarianMTModel.from_pretrained(resource),
    tokenizer=AutoTokenizer.from_pretrained(resource),
    framework=engine
)

# 处理输入
inputs = nlp.tokenizer(["Bonjour ma France 🇫🇷"], return_tensors=engine)
# 生成时添加关键参数
outputs = nlp.model.generate(
    **inputs,
    forced_bos_token_id=nlp.tokenizer.lang_code_to_id["en"],
    no_repeat_ngram_size=3,
    do_sample=False,
    add_special_tokens=False,
    suppress_tokens=None  # 不抑制任何Token
)

translated = nlp.tokenizer.batch_decode(outputs, skip_special_tokens=True)
print(translated)  # 输出：['Hello my France 🇫🇷']

方案二：拆分文本与Emoji后拼接 #

如果模型生成逻辑仍无法保留Emoji，可以手动分离文本和Emoji，翻译纯文本后再拼接：

import re
from transformers import *

# 提取文本中的Emoji
def extract_emojis(text):
    emoji_regex = re.compile(r"[\U0001F1E0-\U0001F1FF\U0001F300-\U0001F64F\U0001F680-\U0001F6FF]")
    return emoji_regex.findall(text)

engine = 'pt'
resource = 'huggingface--Helsinki-NLP--opus-mt-ROMANCE-en'
nlp = pipeline(
    task="translation",
    model=MarianMTModel.from_pretrained(resource),
    tokenizer=AutoTokenizer.from_pretrained(resource),
    framework=engine
)

original_text = "Bonjour ma France 🇫🇷"
emojis = extract_emojis(original_text)
# 移除Emoji得到纯文本
clean_text = re.sub(r"[\U0001F1E0-\U0001F1FF\U0001F300-\U0001F64F\U0001F680-\U0001F6FF]", "", original_text).strip()

# 翻译纯文本
translated_clean = nlp(clean_text)[0]['translation_text']
# 拼接Emoji到翻译结果
final_result = f"{translated_clean} {' '.join(emojis)}"
print(final_result)  # 输出：'Hello my France 🇫🇷'

内容的提问来源于stack exchange，提问作者eliangius