创建Dataproc集群时初始化脚本执行失败求助

问题 #

创建Dataproc集群时运行初始化脚本，将纯Python wheel包从GCS复制到集群并安装，几周前脚本正常运行，但使用新版本wheel包（仅修改Python代码）时失败，Dataproc镜像版本为1.5.53-debian10，报错日志如下：

pip is already installed.
Traceback (most recent call last):
    sys.exit(main())
  File "/opt/conda/miniconda3/lib/python3.7/site-packages/pip/_internal/main.py", line 45, in main
    command = create_command(cmd_name, isolated=("--isolated" in cmd_args))
  File "/opt/conda/miniconda3/lib/python3.7/site-packages/pip/_internal/commands/__init__.py", line 96, in create_command
    module = importlib.import_module(module_path)
  File "/opt/conda/miniconda3/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/conda/miniconda3/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 23, in <module>
    from pip._internal.cli.req_command import RequirementCommand
  File "/opt/conda/miniconda3/lib/python3.7/site-packages/pip/_internal/cli/req_command.py", line 20, in <module>
    from pip._internal.network.session import PipSession
  File "/opt/conda/miniconda3/lib/python3.7/site-packages/pip/_internal/network/session.py", line 17, in <module>
    from pip._vendor import requests, six, urllib3
  File "/opt/conda/miniconda3/lib/python3.7/site-packages/pip/_vendor/requests/__init__.py", line 97, in <module>
    from pip._vendor.urllib3.contrib import pyopenssl
  File "/opt/conda/miniconda3/lib/python3.7/site-packages/pip/_vendor/urllib3/contrib/pyopenssl.py", line 46, in <module>
    import OpenSSL.SSL
  File "/opt/conda/miniconda3/lib/python3.7/site-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import crypto, SSL
  File "/opt/conda/miniconda3/lib/python3.7/site-packages/OpenSSL/crypto.py", line 1553, in <module>
    class X509StoreFlags(object):
  File "/opt/conda/miniconda3/lib/python3.7/site-packages/OpenSSL/crypto.py", line 1573, in X509StoreFlags
    CB_ISSUER_CHECK = _lib.X509_V_FLAG_CB_ISSUER_CHECK
AttributeError: module 'lib' has no attribute 'X509_V_FLAG_CB_ISSUER_CHECK'

使用的初始化脚本如下：

#!/bin/bash 
function install_pip() {
  if command -v pip >/dev/null; then
      echo "pip is already installed."
      return 0
  fi   
  if command -v easy_install >/dev/null; then
    echo "Installing pip with easy_install..."
    easy_install pip
    return 0
  fi   
  echo "Installing python-pip..."
  apt update
  apt install python-pip -y
} 
# install pip in the cluster
install_pip 

# Get the GCS location for SDK from cluster metadata attributes
SDK_GCS_LOCATION="$(/usr/share/google/get_metadata_value attributes/sdk-gcs-location)"
SDK_FILE_NAME="$(/usr/share/google/get_metadata_value attributes/sdk-file-name)"
readonly SDK_GCS_LOCATION
readonly SDK_FILE_NAME
SDK_WHEEL=$SDK_GCS_LOCATION/$SDK_FILE_NAME

# Copy wheel file from GCS to cluster
gsutil cp $SDK_WHEEL .

# Install wheel in cluster
pip install $SDK_FILE_NAME

解决方案 #

该错误由pyopenssl库与系统底层OpenSSL版本不兼容导致，以下是几种可行解决办法：

• 重新安装兼容版本的pyopenssl
  在初始化脚本中，install_pip步骤之后、安装wheel之前添加修复命令：

  pip install --force-reinstall pyopenssl==22.0.0
  

  此版本适配Dataproc 1.5环境中的OpenSSL版本，可解决属性缺失问题。

• 明确使用Dataproc自带的conda pip
  Dataproc 1.5默认使用conda环境，脚本可能误调用系统pip，修改安装命令为：

  /opt/conda/miniconda3/bin/pip install $SDK_FILE_NAME
  

  同时可删除自定义的install_pip函数，集群默认已预装pip，无需重复安装。

• 升级pip至稳定版本
  旧版pip可能存在依赖兼容缺陷，先升级pip再安装wheel：

  pip install --upgrade pip
  

  将该命令放在安装wheel步骤之前，提升pip的依赖处理能力。

• 升级Dataproc镜像版本
  若上述方法无效，可尝试切换至Dataproc 2.x系列镜像（如2.1-debian11），新版本环境的依赖库版本更协调，能从根源避免此类兼容性问题。

内容的提问来源于stack exchange，提问作者Rajnil Guha