为何OpenSSL版本问题导致Foundry Build Service出现无法解决的构建依赖Bug？

问题描述 #

在Foundry Build Service环境中使用PySpark的@Functions.pandas_udf装饰器时，会触发无法解决的依赖冲突Bug：PyArrow依赖的OPENSSL_3.4.0版本在构建系统环境中不存在，即使通过meta.yml在用户/项目环境中安装该版本也无法解决。

报错信息 #

ImportError: PyArrow >= 4.0.0 must be installed; however, it was not found.

Traceback (most recent call last):
  File "/app/work-dir/__python_runtime_environment__/__SYMLINKS__/site-packages/pyspark/sql/pandas/utils.py", line 53, in require_minimum_pyarrow_version
    import pyarrow
  File "/app/work-dir/__environment__/__SYMLINKS__/site-packages/pyarrow/__init__.py", line 65, in <module>
    import pyarrow.lib as _lib
ImportError: /app/work-dir/__python_runtime_environment__/__SYMLINKS__/lib-dynload/../../libcrypto.so.3: version OPENSSL_3.4.0' not found (required by /app/work-dir/__environment__/__SYMLINKS__/site-packages/pyarrow/../../../././libssl.so.3)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/work-dir/__python_runtime_environment__/__SYMLINKS__/site-packages/transforms_spark_module/delegate.py", line 100, in _execute_job
    result = job.run(
  File "/app/work-dir/__python_runtime_environment__/__SYMLINKS__/site-packages/transforms/_build.py", line 329, in run
    self._transform.compute(**kwargs, **parameters)
  File "/app/work-dir/__python_runtime_environment__/__SYMLINKS__/site-packages/transforms/api/_transform.py", line 334, in compute
    output_df: Union[DataFrame, Any] = self(**kwargs)
  File "/app/work-dir/__python_runtime_environment__/__SYMLINKS__/site-packages/transforms/api/_transform.py", line 183, in __call__
    return self._compute_func(*args, **kwargs)
  File "/app/work-dir/__user_code_environment__/__SYMLINKS__/site-packages/myproject/datasets/data_anon.py", line 15, in compute
    "case_weight": redist(df, "case_weight"),
  File "/app/work-dir/__user_code_environment__/__SYMLINKS__/site-packages/myproject/datasets/utils.py", line 32, in redist
    df = df.withColumn(column_name, add_noise(column_name, dist))
  File "/app/work-dir/__user_code_environment__/__SYMLINKS__/site-packages/myproject/datasets/utils.py", line 14, in add_noise
    @F.pandas_udf(DoubleType())
  File "/app/work-dir/__python_runtime_environment__/__SYMLINKS__/site-packages/pyspark/sql/pandas/functions.py", line 338, in pandas_udf
    require_minimum_pyarrow_version()
  File "/app/work-dir/__python_runtime_environment__/__SYMLINKS__/site-packages/pyspark/sql/pandas/utils.py", line 60, in require_minimum_pyarrow_version
    raise ImportError(
ImportError: PyArrow >= 4.0.0 must be installed; however, it was not found.

解决思路与方案 #

1. 降级PyArrow到兼容版本 #

选择对OpenSSL版本要求更低且与PySpark兼容的PyArrow版本（如10.x或11.x系列），这类版本通常依赖OpenSSL 3.0.x，适配Foundry默认环境。在meta.yml中明确指定：

dependencies:
  - pyarrow=10.0.1
  - pyspark=3.3.0 # 确保PySpark与PyArrow版本匹配

2. 强制用户环境库路径优先级 #

在代码执行前，手动将用户环境的库路径插入到系统路径最前端，让Python优先加载用户安装的依赖：

import sys
import os

# 指向用户环境的lib目录
user_lib_dir = "/app/work-dir/__environment__/__SYMLINKS__/lib"
if user_lib_dir not in sys.path:
    sys.path.insert(0, user_lib_dir)

# 重新导入依赖库
import pyarrow
from pyspark.sql import functions as F

3. 替换为普通Spark UDF #

如果PyArrow依赖问题无法绕过，将pandas_udf逻辑改写为普通Spark UDF，完全避免PyArrow依赖：

from pyspark.sql.types import DoubleType
from pyspark.sql.functions import udf
import numpy as np

# 替代原pandas_udf的单值处理UDF
@udf(DoubleType())
def add_noise_udf(value):
    if value is None:
        return None
    # 复制原pandas_udf的噪声逻辑
    return value + np.random.normal(loc=0, scale=0.1)

4. 寻求Foundry官方支持 #

若以上方案均无效，说明是Build Service环境的底层依赖缺失，可提交工单给Foundry支持团队，请求升级或添加所需的OpenSSL版本。

内容的提问来源于stack exchange，提问作者G-Unit