使用spaCy自定义NER和RE时GPU训练报ValueError: Out shape is mismatched

使用spaCy进行自定义命名实体识别（NER）和关系抽取（RE），尝试在GPU上运行训练代码时遭遇ValueError: Out shape is mismatched错误。已尝试更新cupy，但问题未解决，推测是版本不兼容导致，但无法确定具体原因。

训练命令：

!python -m spacy train -g 0 config.cfg --output output --paths.train /content/spacyNER_data/train.spacy --paths.dev /content/spacyNER_data/valid.spacy

运行后错误日志：

ℹ Saving to output directory: output
ℹ Using GPU: 0

=========================== Initializing pipeline ===========================
[2022-07-26 09:01:41,051] [INFO] Set up nlp object from config
[2022-07-26 09:01:41,061] [INFO] Pipeline: ['transformer', 'ner']
[2022-07-26 09:01:41,066] [INFO] Created vocabulary
[2022-07-26 09:01:41,067] [INFO] Finished initializing nlp object
Some weights of the model checkpoint at /content/Indic-law-bert were not used when initializing BertModel: ['cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.predictions.bias']
- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertModel were not initialized from the model checkpoint at /content/Indic-law-bert and are newly initialized: ['bert.pooler.dense.weight', 'bert.pooler.dense.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.7/dist-packages/spacy/__main__.py", line 4, in <module>
    setup_cli()
  File "/usr/local/lib/python3.7/dist-packages/spacy/cli/_util.py", line 71, in setup_cli
    command(prog_name=COMMAND)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/typer/main.py", line 532, in wrapper
    return callback(**use_params)  # type: ignore
  File "/usr/local/lib/python3.7/dist-packages/spacy/cli/train.py", line 45, in train_cli
    train(config_path, output_path, use_gpu=use_gpu, overrides=overrides)
  File "/usr/local/lib/python3.7/dist-packages/spacy/cli/train.py", line 72, in train
    nlp = init_nlp(config, use_gpu=use_gpu)
  File "/usr/local/lib/python3.7/dist-packages/spacy/training/initialize.py", line 84, in init_nlp
    nlp.initialize(lambda: train_corpus(nlp), sgd=optimizer)
  File "/usr/local/lib/python3.7/dist-packages/spacy/language.py", line 1317, in initialize
    proc.initialize(get_examples, nlp=self, **p_settings)
  File "spacy/pipeline/transition_parser.pyx", line 575, in spacy.pipeline.transition_parser.Parser.initialize
  File "/usr/local/lib/python3.7/dist-packages/thinc/model.py", line 299, in initialize
    self.init(self, X=X, Y=Y)
  File "/usr/local/lib/python3.7/dist-packages/spacy/ml/tb_framework.py", line 47, in init
    lower.initialize()
  File "/usr/local/lib/python3.7/dist-packages/thinc/model.py", line 299, in initialize
    self.init(self, X=X, Y=Y)
  File "/usr/local/lib/python3.7/dist-packages/spacy/ml/_precomputable_affine.py", line 150, in init
    acts1 = predict(ids, tokvecs)
  File "/usr/local/lib/python3.7/dist-packages/spacy/ml/_precomputable_affine.py", line 131, in predict
    hiddens = model.predict(tokvecs[:-1])  # (nW, f, o, p)
  File "/usr/local/lib/python3.7/dist-packages/thinc/model.py", line 315, in predict
    return self._func(self, X, is_train=False)[0]
  File "/usr/local/lib/python3.7/dist-packages/spacy/ml/_precomputable_affine.py", line 29, in forward
    Yf[0] = model.get_param("pad")
  File "cupy/_core/core.pyx", line 1409, in cupy._core.core.ndarray.__setitem__
  File "cupy/_core/_routines_indexing.pyx", line 54, in cupy._core._routines_indexing._ndarray_setitem
  File "cupy/_core/_routines_indexing.pyx", line 959, in cupy._core._routines_indexing._scatter_op
  File "cupy/_core/_kernel.pyx", line 1161, in cupy._core._kernel.ufunc.__call__
  File "cupy/_core/_kernel.pyx", line 594, in cupy._core._kernel._get_out_args
ValueError: Out shape is mismatched

• 匹配spaCy、Thinc和CuPy版本：该错误多由版本不兼容引发，需对照spaCy官方文档的版本兼容矩阵，确保安装的Thinc、CuPy版本与当前spaCy版本完全匹配。不要单独升级CuPy，应整体调整到兼容的版本组合。
• 检查Transformer模型适配性：你使用的Indic-law-bert是自定义预训练模型，可能与spaCy的Transformer组件存在维度不匹配问题。可先在CPU上运行训练，确认模型本身无问题后再切换GPU；或在配置文件中调整Transformer的输出维度设置。
• 禁用预计算affine层：在config.cfg中找到[components.ner.model]相关配置，尝试关闭precomputable_affine选项，或调整其参数，避免维度计算错误导致形状不匹配。
• 清理缓存并重新初始化：删除之前生成的output目录和缓存文件，重新运行训练命令，避免旧模型参数干扰初始化过程。
• 降级CuPy版本：若升级CuPy无效，可尝试降级到与当前spaCy/Thinc版本兼容的旧版CuPy，比如CuPy v10.x或v11.x（具体版本需根据spaCy版本确定）。

内容的提问来源于stack exchange，提问作者Saran Pandian