如何在无权限的Spark集群远程节点添加Python依赖库？

Let's work through your problem step by step—you're dealing with two main challenges: getting the module tool to recognize your local libraries, and making sure Spark properly loads your custom dependencies when running on the cluster. Here's how to fix both:

1. Force module to Look for Your Local Module Files #

Since you can't modify system-wide module directories, you can tell the module tool to check your home directory first by adjusting the MODULEPATH environment variable. Here's what to do:

Step 1: Create a Local Module Directory #

Make a folder in your home directory to store your custom module files:

mkdir -p ~/modulefiles/networkx/2.4
mkdir -p ~/modulefiles/shapely/1.7.1  # Use your actual Shapely version

Step 2: Write Custom Lua Module Files #

Create a .lua file for each library, mirroring the structure of the system's Python module. For example, ~/modulefiles/networkx/2.4.lua:

help([[Local installation of NetworkX 2.4 for restricted cluster nodes]])
whatis("Name: networkx")
whatis("Version: 2.4")
whatis("Description: Custom NetworkX module using local home directory libraries")

-- Prepend your local dist-packages to PYTHONPATH
prepend_path("PYTHONPATH", "/home/your_username/local/lib/python3.6/dist-packages")
-- Add any required library paths for compiled extensions (like Shapely's .libs folder)
prepend_path("LD_LIBRARY_PATH", "/home/your_username/local/lib/python3.6/dist-packages/shapely/.libs")

Replace /home/your_username with your actual home directory path—don't use ~/ here, as the module tool may not resolve it correctly.

Step 3: Update MODULEPATH #

Add your local module directory to the MODULEPATH so the module tool finds it:

export MODULEPATH=$HOME/modulefiles:$MODULEPATH

To make this permanent, add the line above to your ~/.bashrc or ~/.profile file.

Step 4: Load Your Modules #

Now you can load your custom modules alongside the system Python module:

module load python/3.6.5
module load networkx/2.4
module load shapely/1.7.1

2. Fix Spark's PYTHONPATH Configuration #

Your original Spark config had two key issues: using ~/ (which Spark doesn't resolve) and pointing to incorrect paths. Here's the corrected approach:

Key Fixes for Spark Conf #

• Use absolute paths instead of ~/
• Point PYTHONPATH directly to your dist-packages folder (where NetworkX/Shapely are stored)
• Set both executor and driver environment variables (since the driver node also needs the libraries)

Updated code snippet:

from pyspark import SparkConf, SparkContext

def calculate(sc):
    text_file = sc.textFile("nevergonnagive.txt")
    counts = text_file.flatMap(lambda line: line.split(" ")) \
        .map(lambda word: (word, 1)) \
        .reduceByKey(lambda a, b: a + b)
    counts.saveAsTextFile("word_count_OUT")
    return sc

# Initialize SparkConf with correct paths
conf_spark = SparkConf()
# Replace with your actual home directory path
local_lib_path = "/home/your_username/local/lib/python3.6/dist-packages"
system_python_lib = "/some/path/Python/3.6.5/lib/python3.6/site-packages"

# Set PYTHONPATH for both executor and driver
conf_spark.set('spark.executorEnv.PYTHONPATH', f"{local_lib_path}:{system_python_lib}")
conf_spark.set('spark.driverEnv.PYTHONPATH', f"{local_lib_path}:{system_python_lib}")

# Set LD_LIBRARY_PATH for compiled extensions (like Shapely)
conf_spark.set('spark.executorEnv.LD_LIBRARY_PATH', f"{local_lib_path}/shapely/.libs:/some/path/Python/3.6.5/lib")
conf_spark.set('spark.driverEnv.LD_LIBRARY_PATH', f"{local_lib_path}/shapely/.libs:/some/path/Python/3.6.5/lib")

# Create SparkContext with the updated config
sc = SparkContext.getOrCreate(conf_spark)

# Add library imports AFTER setting up SparkContext (or add sys.path fix here)
import sys
sys.path.insert(0, local_lib_path)  # Fallback to ensure driver loads local libs
import itertools
import networkx as nx
from networkx.algorithms.connectivity import local_edge_connectivity
import random
from shapely.geometry import Polygon
from shapely.ops import cascaded_union
import xml.etree.ElementTree as ET

sc = calculate(sc)
sc.close()

3. Quick Test to Validate Setup #

Before running your full Spark script, test if the libraries load correctly in a local Python session:

module load python/3.6.5 networkx/2.4
python3 -c "import networkx; import shapely; print('Libraries loaded successfully!')"

If this works, your module setup is correct. If not, double-check the paths in your .lua files and MODULEPATH.

Important Notes #

• Avoid copying entire system Python folders: Instead of copying /usr/lib/python3.6 from your Mint machine, consider installing NetworkX and Shapely directly into your home directory using pip install --user networkx==2.4 shapely==1.7.1 (if pip is available on the cluster). This ensures compatibility with Red Hat 7.
• Check extension compatibility: Shapely uses compiled C extensions—if you copied libraries from Mint to RHEL, make sure they're compatible (both are x86_64, so this should work, but if you get import errors, try re-installing via pip --user).

内容的提问来源于stack exchange，提问作者Jessica Chambers