Jetson Nano（JetPack 4.4）Docker镜像中PyTorch导入失败求助

问题描述 #

使用4GB版NVIDIA Jetson Nano，JetPack版本4.4，通过.whl文件成功安装PyTorch，Docker镜像可正常构建，但运行镜像时导入PyTorch出现如下报错：

==> Launching app...
Traceback (most recent call last):
  File "/code/catkin_ws/src/duckietown-safe-ml-demo/packages/safe_ml/src/ood_node.py", line 13, in <module>
    import torch
  File "/usr/local/lib/python3.8/dist-packages/torch/__init__.py", line 290, in <module>
    from torch._C import *  # noqa: F403
ImportError: /usr/local/lib/python3.8/dist-packages/torch/lib/../../torch.libs/libgomp-804f19d4.so.1.0.0: cannot allocate memory in static TLS block

我的Dockerfile配置 #

# parameters
ARG REPO_NAME="duckietown-safe-ml-demo"
ARG DESCRIPTION="Safe ML on the Duckie Town Platform"
ARG MAINTAINER="michaelj004@e.ntu.edu.sg"
# pick an icon from: https://fontawesome.com/v4.7.0/icons/
ARG ICON="cube"

# ==================================================>
# ==> Do not change the code below this line
ARG ARCH=arm32v7
ARG DISTRO=daffy
ARG BASE_TAG=${DISTRO}-${ARCH}
ARG BASE_IMAGE=dt-ros-commons
ARG LAUNCHER=default

# define base image
FROM duckietown/${BASE_IMAGE}:${BASE_TAG} as BASE

# Changed to have the full suite of DT-packages.
#ARG AIDO_REGISTRY=docker.io
#FROM ${AIDO_REGISTRY}/duckietown/dt-car-interface:${BASE_TAG} AS dt-car-interface
#FROM ${AIDO_REGISTRY}/duckietown/dt-core:${BASE_TAG} AS base

# recall all arguments
ARG ARCH
ARG DISTRO
ARG REPO_NAME
ARG DESCRIPTION
ARG MAINTAINER
ARG ICON
ARG BASE_TAG
ARG BASE_IMAGE
ARG LAUNCHER

# check build arguments
RUN dt-build-env-check "${REPO_NAME}" "${MAINTAINER}" "${DESCRIPTION}"

# define/create repository path
ARG REPO_PATH="${CATKIN_WS_DIR}/src/${REPO_NAME}"
ARG LAUNCH_PATH="${LAUNCH_DIR}/${REPO_NAME}"
RUN mkdir -p "${REPO_PATH}"
RUN mkdir -p "${LAUNCH_PATH}"
#WORKDIR "${REPO_PATH}"
WORKDIR /code
#COPY --from=dt-car-interface ${CATKIN_WS_DIR}/src/dt-car-interface ${CATKIN_WS_DIR}/src/dt-car-interface

# keep some arguments as environment variables
ENV DT_MODULE_TYPE "${REPO_NAME}"
ENV DT_MODULE_DESCRIPTION "${DESCRIPTION}"
ENV DT_MODULE_ICON "${ICON}"
ENV DT_MAINTAINER "${MAINTAINER}"
ENV DT_REPO_PATH "${REPO_PATH}"
ENV DT_LAUNCH_PATH "${LAUNCH_PATH}"
ENV DT_LAUNCHER "${LAUNCHER}"

# Daniel - Add some dependencies
RUN apt-get update -y && apt-get install -y --no-install-recommends \
         gcc \
         libc-dev\
         git \
         bzip2 \
         python3-tk \
         python3-wheel \
         python3-pip  \
         python3-pillow  \
         software-properties-common \
         #Pytorch dependencies
         libopenblas-dev \
         libblas-dev \
         libcairo2-dev \
         pkg-config \
         m4 \
         cmake \
         cython \
         python3-dev \
         python3-yaml \
         curl \
         python3-setuptools && \
     rm -rf /var/lib/apt/lists/*

RUN echo PYTHONPATH=$PYTHONPATH
RUN pip3 install -U "pip>=20.2" pipdeptree setuptools wheel
RUN pip3 install pycairo
COPY ./libraries "${REPO_PATH}/libraries"
RUN pip3 install "${REPO_PATH}/libraries/numpy-1.20.0-cp38-cp38-manylinux2014_aarch64.whl"
RUN pip3 install "${REPO_PATH}/libraries/torch-2.1.0a0+41361538.nv23.06-cp38-cp38-linux_aarch64.whl"
RUN pip3 install "${REPO_PATH}/libraries/torchvision-0.19.0-cp38-cp38-manylinux2014_aarch64.whl"

RUN pipdeptree
RUN pip list

RUN echo "deb https://packages.cloud.google.com/apt coral-edgetpu-stable main" | sudo tee /etc/apt/sources.list.d/coral-edgetpu.list && \
    curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && \
    sudo apt-get update && \
    sudo apt-get install libedgetpu1-std


# install apt dependencies
COPY ./dependencies-apt.txt "${REPO_PATH}/"
RUN dt-apt-install ${REPO_PATH}/dependencies-apt.txt

# install python3 dependencies
COPY ./dependencies-py3.txt "${REPO_PATH}/"
RUN pip3 install -r ${REPO_PATH}/dependencies-py3.txt

# copy the source code
COPY ./packages "${REPO_PATH}/packages"

# build packages
RUN . /opt/ros/${ROS_DISTRO}/setup.sh && \
  catkin build \
    --workspace ${CATKIN_WS_DIR}/

# install launcher scripts
COPY ./launchers/. "${LAUNCH_PATH}/"
COPY ./launchers/default.sh "${LAUNCH_PATH}/"
RUN dt-install-launchers "${LAUNCH_PATH}"

# define default command
CMD ["bash", "-c", "dt-launcher-${DT_LAUNCHER}"]

# store module metadata
LABEL org.duckietown.label.module.type="${REPO_NAME}" \
    org.duckietown.label.module.description="${DESCRIPTION}" \
    org.duckietown.label.module.icon="${ICON}" \
    org.duckietown.label.architecture="${ARCH}" \
    org.duckietown.label.code.location="${REPO_PATH}" \
    org.duckietown.label.code.version.distro="${DISTRO}" \
    org.duckietown.label.base.image="${BASE_IMAGE}" \
    org.duckietown.label.base.tag="${BASE_TAG}" \
    org.duckietown.label.maintainer="${MAINTAINER}"
# <== Do not change the code above this line
# <==================================================

解决方案 #

1. 修正架构不匹配问题（核心） #

Dockerfile指定了ARG ARCH=arm32v7（ARM32位架构），但安装的numpy、PyTorch、torchvision都是manylinux2014_aarch64（ARM64位）的包，架构不兼容是报错的主要原因。

修改步骤：

• 下载对应armv7l/arm32v7架构的whl包：
  • numpy：选择cp38版本的arm32包，如numpy-1.20.0-cp38-cp38-linux_armv7l.whl
  • PyTorch：选用JetPack4.4兼容的ARM32版本，如PyTorch 1.9.0的armv7l包
  • torchvision：匹配PyTorch版本的armv7l包，如0.10.0版本
• 替换Dockerfile中的安装命令：

RUN pip3 install "${REPO_PATH}/libraries/numpy-1.20.0-cp38-cp38-linux_armv7l.whl"
RUN pip3 install "${REPO_PATH}/libraries/torch-1.9.0-cp38-cp38-linux_armv7l.whl"
RUN pip3 install "${REPO_PATH}/libraries/torchvision-0.10.0-cp38-cp38-linux_armv7l.whl"

2. 预加载libgomp库解决TLS内存问题 #

架构修正后若仍报错，可通过预加载libgomp库解决：
在Dockerfile中PyTorch安装完成后添加环境变量：

ENV LD_PRELOAD=/usr/local/lib/python3.8/dist-packages/torch.libs/libgomp-804f19d4.so.1.0.0

（若文件名有变动，替换为实际路径即可）

3. 安装系统版libgomp #

安装系统自带的libgomp1，替换PyTorch打包的版本避免冲突：

• 在Dockerfile的apt依赖安装部分添加libgomp1：

RUN apt-get update -y && apt-get install -y --no-install-recommends \
    # 原有依赖...
    libgomp1 \
    # 其他依赖...

• 设置预加载系统库：

ENV LD_PRELOAD=/usr/lib/arm-linux-gnueabihf/libgomp.so.1

4. 调整容器内存限制（可选） #

Jetson Nano内存有限，运行容器时可增加内存限制：

docker run --memory=3g --memory-swap=4g [你的镜像名]

内容的提问来源于stack exchange，提问作者Vivek Velachhawala