OpenShift下RHEL8非根容器中Pandarallel遇FIPS模式多进程错误

运行环境 #

• 平台：OpenShift，非root用户运行基于Red Hat Enterprise Linux 8.8的Docker容器
• 系统信息：

NAME="Red Hat Enterprise Linux"
VERSION="8.8 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.8 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
 
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.8
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.8"

• Python版本：3.10.11
• OpenSSL版本：OpenSSL 1.1.1k FIPS 25 Mar 2021
• 核心依赖：pandarallel==1.6.5、pandas==2.0.0，完整依赖清单：

python 3.10.11
asteroid==2.15.5
async-timeout==4.0.3
attrs==23.1.0
certify==2023.7.22
charset-normalizer==3.2.0
contourpy==1.1.0
coverage==7.2.7
cycler==0.11.0
debugpy==1.6.7
dill==0.3.6
exceptiongroup==1.1.1
execnet==1.9.0
fonttools==4.42.1
idna==3.4
iniconfig==2.0.0
isort==5.12.0
Jinja2==3.1.2
joblib==1.3.2
jsonschema==4.17.3
kiwisolver==1.4.5
lazy-object-proxy==1.9.0
MarkupSafe==2.1.3
matplotlib==3.7.2
mccabe==0.7.0
mlxtend==0.22.0
numpy==1.25.2
packaging==23.1
pandarallel==1.6.5
pandas==2.0.0
pika==1.3.1
Pillow==10.0.0
platformdirs==3.5.3
pluggy==1.0.0
psutil==5.9.5
py==1.11.0
py-cpuinfo==9.0.0
pylint==2.17.2
pyparsing==3.0.9
pyrsistent==0.19.3
pytest==7.3.1
pytest-benchmark==4.0.0
pytest-cov==4.0.0
pytest-html==3.2.0
pytest-metadata==3.0.0
pytest-mock==3.10.0
pytest-order==1.1.0
pytest-ordering==0.6
pytest-timeout==2.1.0
pytest-xdist==3.2.1
python-dateutil==2.8.2
pytz==2023.3
redis==4.5.4
requests==2.31.0
scikit-learn==1.2.2
scipy==1.10.1
seaborn==0.12.2
six==1.16.0
threadpoolctl==3.2.0
tomli==2.0.1
tomlkit==0.11.8
typing_extensions==4.6.3
tzdata==2023.3
urllib3==2.0.4
wrapt==1.15.0

问题复现 #

执行以下代码时触发错误：

import pandas as pd
from pandarallel import pandarallel

# Initialize pandarallel
pandarallel.initialize(use_memory_fs=False)

# Create a sample DataFrame
data = {'A': range(1, 11), 'B': range(11, 21)}
df = pd.DataFrame(data)

# Define a function that will be applied to each row in the DataFrame
def custom_function(row):
    return row['A'] + row['B']

# Use pandarallel to apply the function in parallel
result = df.parallel_apply(custom_function, axis=1)

print(result)

报错信息 #

INFO: Pandarallel will run on 2 workers.
INFO: Pandarallel will use standard multiprocessing data transfer (pipe) to transfer data between the main process and workers.
Traceback (most recent call last):
  File "/usr/app/x.py", line 17, in <module>
result = df.parallel_apply(custom_function, axis=1)
  File "/usr/local/lib/python3.10/site-packages/pandarallel/core.py", line 368, in closure
master_workers_queue = manager.Queue()
  File "/usr/local/lib/python3.10/multiprocessing/managers.py", line 723, in temp
token, exp = self._create(typeid, *args, **kwds)
  File "/usr/local/lib/python3.10/multiprocessing/managers.py", line 606, in _create
conn = self._Client(self._address, authkey=self._authkey)
  File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 508, in Client
answer_challenge(c, authkey)
  File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 755, in answer_challenge
digest = hmac.new(authkey, message, 'md5').digest()
  File "/usr/local/lib/python3.10/hmac.py", line 184, in new
return HMAC(key, msg, digestmod)
  File "/usr/local/lib/python3.10/hmac.py", line 60, in __init__
self._init_hmac(key, msg, digestmod)
  File "/usr/local/lib/python3.10/hmac.py", line 67, in _init_hmac
self._hmac = _hashopenssl.hmac_new(key, msg, digestmod=digestmod)
ValueError: no reason supplied

问题原因 #

报错根源是Python的multiprocessing模块在创建跨进程连接时，默认使用MD5哈希算法生成HMAC摘要，但当前系统处于FIPS模式，MD5已被FIPS标准禁用，导致OpenSSL拒绝执行该算法，抛出ValueError。

解决方案 #

方案1：替换multiprocessing的默认哈希算法 #

在代码开头添加以下配置，改用FIPS允许的SHA-256算法替代MD5：

import multiprocessing
from multiprocessing.connection import answer_challenge
import hmac

# 重写认证函数，使用sha256
def patched_answer_challenge(connection, authkey):
    message = connection.recv_bytes(256)
    digest = hmac.new(authkey, message, 'sha256').digest()
    connection.send_bytes(digest)

# 替换原函数
multiprocessing.connection.answer_challenge = patched_answer_challenge

完成配置后再导入pandarallel执行后续代码。

方案2：使用fork启动进程（仅类Unix系统适用） #

修改pandarallel初始化参数，指定用fork方式创建子进程，避免使用需要跨进程认证的Manager：

pandarallel.initialize(use_memory_fs=False, fork=True)

该方式直接通过fork复制父进程，无需建立跨进程认证连接，从而绕过MD5限制。

方案3：临时禁用FIPS模式（仅测试用，不推荐） #

若环境允许，可临时设置环境变量关闭FIPS模式：

export OPENSSL_FIPS=0

此方法会降低系统安全性，仅建议用于问题验证。

内容的提问来源于stack exchange，提问作者Omry Atia