树莓派4 rq worker运行Tensorflow时报libhdfs.so缺失与numpy不兼容如何修复

问题背景 #

部署在树莓派4上的歌曲流派预测项目，结合rq worker与Tensorflow实现麦克风录音后的流派识别功能。调用预测函数时rq worker抛出异常，且无需引入TensorFlow的其他rq任务可正常运行。

错误日志 #

2021-09-13 10:49:30.246680: E tensorflow/core/platform/hadoop/hadoop_file_system.cc:132] HadoopFileSystem load error: libhdfs.so: cannot open shared object file: No such file or directory
10:49:33 Traceback (most recent call last):
  File "/home/pi/.local/lib/python3.7/site-packages/rq/worker.py", line 1031, in perform_job
    rv = job.perform()
  File "/home/pi/.local/lib/python3.7/site-packages/rq/job.py", line 749, in perform
    self._result = self._execute()
  File "/home/pi/.local/lib/python3.7/site-packages/rq/job.py", line 772, in _execute
    result = self.func(*self.args, **self.kwargs)
  File "/home/pi/.local/lib/python3.7/site-packages/rq/job.py", line 225, in func
    return import_attribute(self.func_name)
  File "/home/pi/.local/lib/python3.7/site-packages/rq/utils.py", line 142, in import_attribute
    module = importlib.import_module(module_name)
  File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "./recognition.py", line 1, in <module>
    from tensorflow.keras import layers
  File "/home/pi/.local/lib/python3.7/site-packages/tensorflow/__init__.py", line 98, in <module>
    from tensorflow_core import *
  File "/home/pi/.local/lib/python3.7/site-packages/tensorflow_core/__init__.py", line 28, in <module>
    from tensorflow.python import pywrap_tensorflow  # pylint: disable=unused-import
  File "<frozen importlib._bootstrap>", line 1019, in _handle_fromlist
  File "/home/pi/.local/lib/python3.7/site-packages/tensorflow/__init__.py", line 50, in __getattr__
    module = self._load()
  File "/home/pi/.local/lib/python3.7/site-packages/tensorflow/__init__.py", line 44, in _load
    module = _importlib.import_module(self.__name__)
  File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/pi/.local/lib/python3.7/site-packages/tensorflow_core/python/__init__.py", line 88, in <module>
    from tensorflow.python import keras
  File "/home/pi/.local/lib/python3.7/site-packages/tensorflow_core/python/keras/__init__.py", line 26, in <module>
    from tensorflow.python.keras import activations
  File "/home/pi/.local/lib/python3.7/site-packages/tensorflow_core/python/keras/__init__.py", line 26, in <module>
    from tensorflow.python.keras import activations
  File "/home/pi/.local/lib/python3.7/site-packages/tensorflow_core/python/keras/activations.py", line 23, in <module>
    from tensorflow.python.keras.utils.generic_utils import deserialize_keras_object
  File "/home/pi/.local/lib/python3.7/site-packages/tensorflow_core/python/keras/utils/__init__.py", line 34, in <module>
    from tensorflow.python.keras.utils.io_utils import HDF5Matrix
  File "/home/pi/.local/lib/python3.7/site-packages/tensorflow_core/python/keras/utils/io_utils.py", line 30, in <module>
    import h5py
  File "/home/pi/.local/lib/python3.7/site-packages/h5py/__init__.py", line 25, in <module>
    from . import _errors
  File "h5py/_errors.pyx", line 1, in init h5py._errors
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 44 from C header, got 40 from PyObject

修复步骤 #

1. 核心错误修复 #

崩溃的直接原因是h5py与numpy的二进制版本不兼容：已安装的h5py是基于更高版本numpy编译的，当前环境numpy版本更低，导致结构体大小不匹配。执行以下命令重装适配版本：

# 先卸载现有不兼容版本
pip3 uninstall -y numpy h5py
# 优先安装树莓派官方源预编译的arm架构适配版本，避免编译问题
sudo apt update && sudo apt install -y python3-numpy python3-h5py
# 如果需要用pip安装，指定匹配Python3.7的兼容版本
pip3 install numpy==1.19.5 h5py==2.10.0

2. 非致命警告处理 #

日志中的Hadoop加载错误是TensorFlow默认编译带的HDFS依赖提示，不需要用到HDFS存储功能可以直接忽略，不会影响正常运行。如果要隐藏该警告，可以在导入TensorFlow的代码前添加环境变量配置：

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'

3. 可选优化 #

树莓派arm架构上建议安装官方适配的TensorFlow版本，避免通用pip版本的兼容性问题：

sudo apt install -y python3-tensorflow

所有操作完成后重启rq worker即可正常加载TensorFlow模块执行预测任务。

内容的提问来源于stack exchange，提问作者Alessio Rocchi