如何用HuggingFace的M2M模型单Pipeline实现多语言翻译及资源疑问

M2M模型基于约100种语言训练，可实现跨语言翻译，基础使用示例如下：

from transformers import pipeline

m2m100 = pipeline('translation', 'facebook/m2m100_418M', src_lang='en', tgt_lang="de")
m2m100(["hello world", "foo bar"])

输出：

[{'translation_text': 'Hallo Welt'}, {'translation_text': 'Die Fu Bar'}]

多目标语言翻译的常规实现方式 #

要实现多目标语言翻译，通常需要初始化多个pipeline，示例代码如下：

from transformers import pipeline

m2m100_en_de = pipeline('translation', 'facebook/m2m100_418M', src_lang='en', tgt_lang="de")
m2m100_en_fr = pipeline('translation', 'facebook/m2m100_418M', src_lang='en', tgt_lang="fr")

print(m2m100_en_de(["hello world", "foo bar"]))
print(m2m100_en_fr(["hello world", "foo bar"]))

输出：

[{'translation_text': 'Hallo Welt'}, {'translation_text': 'Die Fu Bar'}]
[{'translation_text': 'Bonjour Monde'}, {'translation_text': 'Le bar Fou'}]

能否通过单个pipeline实现M2M模型的多目标语言和/或多源语言翻译？ #

尝试传入语言列表给tgt_lang参数时会触发错误：

from transformers import pipeline

m2m100_en_defr = pipeline('translation', 'facebook/m2m100_418M', src_lang='en', tgt_lang=["de", "fr"])
print(m2m100_en_defr(["hello world", "foo bar"]))

报错信息：

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_28/3374873260.py in <module>
      3 m2m100_en_defr = pipeline('translation', 'facebook/m2m100_418M', src_lang='en', tgt_lang=["de", "fr"])
      4 
----> 5 print(m2m100_en_defr(["hello world", "foo bar"]))

/opt/conda/lib/python3.7/site-packages/transformers/pipelines/text2text_generation.py in __call__(self, *args, **kwargs)
    364               token ids of the translation.
    365         """
--> 366         return super().__call__(*args, **kwargs)

/opt/conda/lib/python3.7/site-packages/transformers/pipelines/text2text_generation.py in __call__(self, *args, **kwargs)
    163         """
    164 
--> 165         result = super().__call__(*args, **kwargs)
    166         if (
    167             isinstance(args[0], list)

/opt/conda/lib/python3.7/site-packages/transformers/pipelines/base.py in __call__(self, inputs, num_workers, batch_size, *args, **kwargs)
   1088                     inputs, num_workers, batch_size, preprocess_params, forward_params, postprocess_params
   1089                 )
-> 1090                 outputs = list(final_iterator)
   1091                 return outputs
   1092             else:

/opt/conda/lib/python3.7/site-packages/transformers/pipelines/pt_utils.py in __next__(self)
    122 
    123         # We're out of items within a batch
-> 124         item = next(self.iterator)
    125         processed = self.infer(item, **self.params)
    126         # We now have a batch of "inferred things".

/opt/conda/lib/python3.7/site-packages/transformers/pipelines/pt_utils.py in __next__(self)
    122 
    123         # We're out of items within a batch
-> 124         item = next(self.iterator)
    125         processed = self.infer(item, **self.params)
    126         # We now have a batch of "inferred things".

/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py in __next__(self)
    626                 # TODO(https://github.com/pytorch/pytorch/issues/76750)
    627                 self._reset()  # type: ignore[call-arg]
-> 628             data = self._next_data()
    629             self._num_yielded += 1
    630             if self._dataset_kind == _DatasetKind.Iterable and \

/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py in _next_data(self)
    669     def _next_data(self):
    670         index = self._next_index()  # may raise StopIteration
-> 671         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    672         if self._pin_memory:
    673             data = _utils.pin_memory.pin_memory(data, self._pin_memory_device)

/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
     56                 data = self.dataset.__getitems__(possibly_batched_index)
     57             else:
---> 58                 data = [self.dataset[idx] for idx in possibly_batched_index]
     59         else:
     60             data = self.dataset[possibly_batched_index]

/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py in <listcomp>(.0)
     56                 data = self.dataset.__getitems__(possibly_batched_index)
     57             else:
---> 58                 data = [self.dataset[idx] for idx in possibly_batched_index]
     59         else:
     60             data = self.dataset[possibly_batched_index]

/opt/conda/lib/python3.7/site-packages/transformers/pipelines/pt_utils.py in __getitem__(self, i)
     17     def __getitem__(self, i):
     18         item = self.dataset[i]
---> 19         processed = self.process(item, **self.params)
     20         return processed
     21 

/opt/conda/lib/python3.7/site-packages/transformers/pipelines/text2text_generation.py in preprocess(self, truncation, src_lang, tgt_lang, *args)
    313         if getattr(self.tokenizer, "_build_translation_inputs", None):
    314             return self.tokenizer._build_translation_inputs(
--> 315                 *args, return_tensors=self.framework, truncation=truncation, src_lang=src_lang, tgt_lang=tgt_lang
    316             )
    317         else:

/opt/conda/lib/python3.7/site-packages/transformers/models/m2m_100/tokenization_m2m_100.py in _build_translation_inputs(self, raw_inputs, src_lang, tgt_lang, **extra_kwargs)
    351         self.src_lang = src_lang
    352         inputs = self(raw_inputs, add_special_tokens=True, **extra_kwargs)
--> 353         tgt_lang_id = self.get_lang_id(tgt_lang)
    354         inputs["forced_bos_token_id"] = tgt_lang_id
    355         return inputs

/opt/conda/lib/python3.7/site-packages/transformers/models/m2m_100/tokenization_m2m_100.py in get_lang_id(self, lang)
    379 
    380     def get_lang_id(self, lang: str) -> int:
--> 381         lang_token = self.get_lang_token(lang)
    382         return self.lang_token_to_id[lang_token]
    383 

/opt/conda/lib/python3.7/site-packages/transformers/models/m2m_100/tokenization_m2m_100.py in get_lang_token(self, lang)
    376 
    377     def get_lang_token(self, lang: str) -> str:
--> 378         return self.lang_code_to_token[lang]
    379 
    380     def get_lang_id(self, lang: str) -> int:

TypeError: unhashable type: 'list'

预期输出格式：

{"de": [{'translation_text': 'Hallo Welt'}, {'translation_text': 'Die Fu Bar'}],
 "fr": [{'translation_text': 'Bonjour Monde'}, {'translation_text': 'Le Foo Bar'}]}

解决方案

当前transformers的M2M100 pipeline不支持直接传入多语言列表，但可以复用单个pipeline实例，动态指定目标语言实现需求：

from transformers import pipeline

# 初始化单个pipeline，不固定目标语言
m2m100 = pipeline('translation', 'facebook/m2m100_418M', src_lang='en')

texts = ["hello world", "foo bar"]
target_langs = ["de", "fr"]
results = {}

for lang in target_langs:
    # 调用时动态传入目标语言
    results[lang] = m2m100(texts, tgt_lang=lang)

print(results)

运行后即可得到预期的输出格式。

若使用多个pipeline，模型内存映射是否共享？会初始化多模型多tokenizer对，还是单模型多tokenizer？ #

当基于同一个模型路径初始化多个pipeline时：

1. 模型权重会共享内存映射：transformers默认会复用已加载的模型权重张量，不会重复加载多个模型实例，避免额外内存占用。
2. tokenizer默认是独立实例：每个pipeline会自动创建新的tokenizer对象，但可以手动传入同一个tokenizer实例来共享。

验证示例

情况1：默认初始化多个pipeline

from transformers import pipeline

pipeline1 = pipeline('translation', 'facebook/m2m100_418M', src_lang='en', tgt_lang='de')
pipeline2 = pipeline('translation', 'facebook/m2m100_418M', src_lang='en', tgt_lang='fr')

print(pipeline1.model is pipeline2.model)  # 输出: True（模型实例共享）
print(pipeline1.tokenizer is pipeline2.tokenizer)  # 输出: False（tokenizer是独立实例）

情况2：手动共享tokenizer

from transformers import pipeline, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('facebook/m2m100_418M')
pipeline1 = pipeline('translation', 'facebook/m2m100_418M', src_lang='en', tgt_lang='de', tokenizer=tokenizer)
pipeline2 = pipeline('translation', 'facebook/m2m100_418M', src_lang='en', tgt_lang='fr', tokenizer=tokenizer)

print(pipeline1.model is pipeline2.model)  # 输出: True
print(pipeline1.tokenizer is pipeline2.tokenizer)  # 输出: True（tokenizer实例共享）

内容的提问来源于stack exchange，提问作者alvas