在NVIDIA-Docker2容器运行PyTorch项目时遇weakcallableproxy弱引用错误

环境差异 #

• 本地环境：Python 3.10，PyTorch、CUDA版本与容器基本一致，项目运行正常
• 容器环境：Python 3.8，基于NVIDIA PyTorch 22.11镜像（PyTorch版本1.13.0a0+936e930）

容器运行命令与详情 #

➜ Face-Recognition-From-Crowd ⚡ 3 hours ago ( master)▶ sudo docker run --gpus all --device /dev/video0 --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 -it face-crowd bash

=============
== PyTorch ==
=============

NVIDIA Release 22.11 (build 48503342)
PyTorch Version 1.13.0a0+936e930

Container image Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

Copyright (c) 2014-2022 Facebook Inc.
Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)
Copyright (c) 2012-2014 Deepmind Technologies    (Koray Kavukcuoglu)
Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)
Copyright (c) 2011-2013 NYU                      (Clement Farabet)
Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston)
Copyright (c) 2006      Idiap Research Institute (Samy Bengio)
Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz)
Copyright (c) 2015      Google Inc.
Copyright (c) 2015      Yangqing Jia
Copyright (c) 2013-2016 The Caffe contributors
All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES.  All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

root@a08973389041:/app# python3 run.py --source live

报错栈信息 #

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2548, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2528, in wsgi_app
    response = self.handle_exception(e)
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2525, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1822, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/app/app.py", line 57, in video_feed
    video = Process(os.path.abspath('./temp'),
  File "/app/process.py", line 45, in __init__
    self.recognizer = Predictor(file=False, label=True)
  File "/app/scripts/FaceRecognition.py", line 31, in __init__
    self.model = SingleShotLearningFR(pretrained=True)
  File "/app/scripts/FRMethods/SingleShotLearningFR.py", line 29, in __init__
    super(SingleShotLearningFR, self).__init__()
  File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/core/module.py", line 124, in __init__
    self._register_sharded_tensor_state_dict_hooks_if_available()
  File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/core/module.py", line 2022, in _register_sharded_tensor_state_dict_hooks_if_available
    self.__class__._register_load_state_dict_pre_hook(
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1467, in _register_load_state_dict_pre_hook
    self._load_state_dict_pre_hooks[handle.id] = _WrappedHook(hook, self if with_module else None)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 54, in __init__
    self.module: weakref.ReferenceType["Module"] = weakref.ref(module)
TypeError: cannot create weak reference to 'weakcallableproxy' object

可能的原因分析 #

• Python版本兼容性问题：Python 3.8与3.10在weakref模块的对象处理逻辑上存在差异，容器内的Python 3.8无法为weakcallableproxy类型对象创建弱引用，而本地Python 3.10支持该操作。
• PyTorch Lightning与PyTorch版本适配问题：容器使用的是PyTorch nightly版本（1.13.0a0），搭配的PyTorch Lightning版本可能未针对该PyTorch版本或Python 3.8做充分适配，导致在注册状态字典钩子时触发错误。
• 依赖包版本不一致：虽然PyTorch和CUDA版本对齐，但项目依赖的其他包（如Flask、PyTorch Lightning）在本地与容器中的版本可能存在差异，间接引发对象代理处理的异常。

内容的提问来源于stack exchange，提问作者NavinKumarmMNK