AllenNLP Python接口加载dygiepp模型获取词汇表遇配置错误

问题场景 #

尝试通过AllenNLP的Python接口加载公开预训练的DyGIEpp SciERC模型以获取词汇表，执行代码如下：

from allennlp.models.model import Model

scierc_model = Model.from_archive('https://s3-us-west-2.amazonaws.com/ai2-s2-research/dygiepp/master/scierc.tar.gz')

执行后触发ConfigurationError，错误信息显示系统无法识别dygie类型的dataset_reader，错误栈如下：

---------------------------------------------------------------------------
ConfigurationError                        Traceback (most recent call last)
/tmp/local/63381207/ipykernel_7616/3549263982.py in <module>
----> 1 scierc_model = Model.from_archive('https://s3-us-west-2.amazonaws.com/ai2-s2-research/dygiepp/master/scierc.tar.gz')

~/anaconda3/envs/dygiepp/lib/python3.7/site-packages/allennlp/models/model.py in from_archive(cls, archive_file, vocab)
    480         from allennlp.models.archival import load_archive  # here to avoid circular imports
    481 
--> 482         model = load_archive(archive_file).model
    483         if vocab:
    484             model.vocab.extend_from_vocab(vocab)

~/anaconda3/envs/dygiepp/lib/python3.7/site-packages/allennlp/models/archival.py in load_archive(archive_file, cuda_device, overrides, weights_file)
    231         # Instantiate model and dataset readers. Use a duplicate of the config, as it will get consumed.
    232         dataset_reader, validation_dataset_reader = _load_dataset_readers(
--> 233             config.duplicate(), serialization_dir
    234         )
    235         model = _load_model(config.duplicate(), weights_path, serialization_dir, cuda_device)

~/anaconda3/envs/dygiepp/lib/python3.7/site-packages/allennlp/models/archival.py in _load_dataset_readers(config, serialization_dir)
    267 
    268     dataset_reader = DatasetReader.from_params(
--> 269         dataset_reader_params, serialization_dir=serialization_dir
    270     )
    271     validation_dataset_reader = DatasetReader.from_params(

~/anaconda3/envs/dygiepp/lib/python3.7/site-packages/allennlp/common/from_params.py in from_params(cls, params, constructor_to_call, constructor_to_inspect, **extras)
    586                 "type",
    587                 choices=as_registrable.list_available(),
--> 588                 default_to_first_choice=default_to_first_choice,
    589             )
    590             subclass, constructor_name = as_registrable.resolve_class_name(choice)

~/anaconda3/envs/dygiepp/lib/python3.7/site-packages/allennlp/common/params.py in pop_choice(self, key, choices, default_to_first_choice, allow_class_names)
    322                 """{"model": "my_module.models.MyModel"} to have it imported automatically."""
    323             )
--> 324             raise ConfigurationError(message)
    325         return value
    326 

ConfigurationError: dygie not in acceptable choices for dataset_reader.type: ['babi', 'conll2003', 'interleaving', 'multitask', 'multitask_shim', 'sequence_tagging', 'sharded', 'text_classification_json']. You should either use the --include-package flag to make sure the correct module is loaded, or use a fully qualified class name in your config file like {"model": "my_module.models.MyModel"} to have it imported automatically.

仅尝试import dygie无法解决问题，以下是可行的Python端解决方法：

可行解决方法 #

方法1：手动注册DyGIE DatasetReader #

利用AllenNLP的注册机制，在加载模型前手动注册DyGIE的DatasetReader：

from allennlp.models.model import Model
from dygie.data.dataset_readers.dygie import DyGIEDatasetReader
from allennlp.data.dataset_readers import DatasetReader

# 注册dygie类型的DatasetReader
DatasetReader.register("dygie", constructor="from_params")(DyGIEDatasetReader)

# 加载模型并获取词汇表
scierc_model = Model.from_archive('https://s3-us-west-2.amazonaws.com/ai2-s2-research/dygiepp/master/scierc.tar.gz')
vocab = scierc_model.vocab

方法2：使用include_package加载DyGIE模块 #

AllenNLP提供include_package函数可直接加载指定包，让系统识别DyGIE的组件：

from allennlp.models.model import Model
from allennlp.common.util import include_package

# 加载dygie包，注册其中的组件
include_package("dygie")

# 加载模型并获取词汇表
scierc_model = Model.from_archive('https://s3-us-west-2.amazonaws.com/ai2-s2-research/dygiepp/master/scierc.tar.gz')
vocab = scierc_model.vocab

方法3：修改模型配置文件（本地存档场景） #

若已下载模型存档到本地，可修改存档中的config.json文件，将dataset_reader.type替换为DyGIE DatasetReader的完整类路径：

"dataset_reader": {
  "type": "dygie.data.dataset_readers.dygie.DyGIEDatasetReader",
  // 保留原有其他配置项
}

修改完成后，加载本地修改后的存档即可。

内容的提问来源于stack exchange，提问作者SLotreck