Docker部署Spark3.3.2环境触发TypeError: 'JavaPackage'对象不可调用

云上老周

2026-7-21

PySpark容器重建后出现TypeError: 'JavaPackage' object is not callable错误

我构建了包含Ubuntu 22.04、Python 3.10、Spark 3.3.2、Hadoop 3、Scala 2.13及OpenJDK 19的Docker容器，作为AWS部署前的测试环境。该容器此前运行正常，但近期重建后，执行生成1000万随机点的PySpark测试脚本时，出现TypeError: 'JavaPackage' object is not callable错误。Dockerfile未做修改，且在不同机器/环境重建均复现该问题。

Dockerfile

# Container for Apache Spark, PySpark-Pandas, RasterFrames, GeoParquet
# FROM ubuntu:23:04 note Spark 3.4.0 will support Python 3.11
FROM ubuntu:22.04

# Setup Spark Version Requirements
ENV SPARK_HOME=/usr/local/spark
ARG SPARK_V="3.3.2"
ARG HADOOP_V="3"
ARG SCALA_V="13"
ARG OPENJDK_V="19"

# Apache Spark 3.3.2 with Scale2.13 Checksum from https://spark.apache.org/downloads.html
ARG spark_checksum="3ce800ca3e0569ccb8736e4fcdb8146ec6d3070da7622dcc9d0edbeb2dc9524224f3a082a70a0faff91306369a837caa13291a09f3ad0d2b0b51548365f90ead"

# Handle user-prompt for Ubuntu installation time zone selection
ENV TZ=America
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && \
    echo $TZ > /etc/timezone

# Update Ubuntu
RUN apt-get update && apt-get upgrade -y && \
    apt-get install -y apt-utils

# Install python
RUN apt-get update --fix-missing && apt-get upgrade -y && \
    apt-get install -y sudo dialog git openssh-server wget \
    curl cmake nano python3 python3-pip python3-setuptools \
    build-essential libpq-dev gdal-bin libgdal-dev

RUN apt-get update --yes && \
    apt-get install --yes --no-install-recommends \
    "openjdk-${OPENJDK_V}-jre-headless" \
    ca-certificates-java && \
    apt-get clean && rm -rf /var/lib/apt/lists/*
  
RUN apt-get autoclean && apt-get autoremove && \
    apt-get update --fix-missing && apt-get upgrade -y && \
    dpkg --configure -a && apt-get install -f
  
# Install Non-Spark Python Packages
RUN pip install numpy GDAL pandas \
    geopandas geoparquet rtree \
    plotly rasterio folium descartes 

# Spark installation
WORKDIR /tmp
RUN wget -qO "spark.tgz" "https://archive.apache.org/dist/spark/spark-${SPARK_V}/spark-${SPARK_V}-bin-hadoop${HADOOP_V}-scala2.${SCALA_V}.tgz"
RUN echo "${spark_checksum} *spark.tgz" | sha512sum -c -
RUN tar xzf "spark.tgz" -C /usr/local --owner root --group root --no-same-owner
RUN rm "spark.tgz"

# Handle spark home error for /usr/local/spark
#RUN rm /usr/local/spark
RUN mv /usr/local/spark-3.3.2-bin-hadoop3-scala2.13 /usr/local/spark

# Configure Spark
ENV SPARK_OPTS="--driver-java-options=-Xms1024M --driver-java-options=-Xmx4096M --driver-java-options=-Dlog4j.logLevel=info" PATH="${PATH}:${SPARK_HOME}/bin"
RUN mkdir -p /usr/local/bin/before-notebook.d
RUN ln -s "/usr/local/spark/sbin/spark-config.sh" /usr/local/bin/before-notebook.d/spark-config.sh

# Install Spark Python Packages
RUN pip install pyarrow py4j scipy \
    pyspark pyspark[sql] pyspark[pandas_on_spark]

WORKDIR /home

测试脚本 test-pyspark.py

import os, sys, json, io, rtree, fiona, math, findspark

import plotly.express as px
import plotly.graph_objects as go
from random import random
from scipy.special import expit as logistic

import numpy as np
import pandas as pd
import geopandas as gpd
import pyspark.pandas as ps

import pyarrow as pa
import pyarrow.parquet as pq

import shapely
from shapely import wkb, wkt
from shapely.geometry.base import BaseGeometry
from shapely.geometry.collection import GeometryCollection
from shapely.geometry import \
    LineString, MultiLineString, MultiPoint, MultiPolygon, Point, Polygon

import pyspark
from pyspark import SparkContext
from pyspark.sql import SparkSession
from pyspark.sql.functions import \
    col, lag, first, last, count, udf, lit, pandas_udf, split, array_contains
from pyspark.sql.types import \
    UserDefinedType, StructField, BinaryType, StructType, \
    StringType, IntegerType, FloatType, DoubleType, DecimalType, LongType

from ast import literal_eval as make_tuple

from IPython.display import display

os.environ["PYARROW_IGNORE_TIMEZONE"] = "1"
os.environ['PYSPARK_PYTHON'] = sys.executable
os.environ['PYSPARK_DRIVER_PYTHON'] = sys.executable

findspark.init() 

job_name = "can300-june-pothole"
spark = SparkSession.builder.master("local[*]") \
    .appName(job_name) \
    .config("spark.config.option", "true") \
    .config("spark.executor.memory", "32g") \
    .config("spark.driver.memory", "32g") \
    .config("spark.sql.debug.maxToStringFields",1000000) \
    .getOrCreate()

spark_context_few_slices = 64
spark_context_some_slices = 256
spark_context_many_slices = 16384
plot_size = 800

def test_para():
    # Generate Number of Random Samples for Spatial Join: 10 million
    num_datapoints_to_match = int(10e6) # 10m datapoints

    data = spark.sparkContext.parallelize(
        [[random()*3, random()*3 ] for x in range(num_datapoints_to_match) ],
        numSlices=spark_context_some_slices)
    sdf0 = data \
        .map(lambda x: str(x)) \
        .map(lambda w: w[1:-1].split(',')) \
        .toDF()
    sdf0 = sdf0 \
        .withColumnRenamed('_1', 'x') \
        .withColumnRenamed('_2', 'y') \

if __name__ == "__main__":
    print("Start!")
    test_para()

错误信息

root@b2924c37ab5a:/home/bl300# python3 test-pyspark.py 
/usr/local/lib/python3.10/dist-packages/pyspark/pandas/__init__.py:50: UserWarning: 'PYARROW_IGNORE_TIMEZONE' environment variable was not set. It is required to set this environment variable to '1' in both driver and executor sides if you use pyarrow>=2.0.0. pandas-on-Spark will set it for you but it does not work if there is a Spark context already launched.
  warnings.warn(
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/05/18 20:14:02 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Start!
Traceback (most recent call last):
  File "/home/bl300/test-pyspark.py", line 107, in <module>
    test_join()
  File "/home/bl300/test-pyspark.py", line 100, in test_join
    .toDF()
  File "/usr/local/lib/python3.10/dist-packages/pyspark/sql/session.py", line 115, in toDF
    return sparkSession.createDataFrame(self, schema, sampleRatio)
  File "/usr/local/lib/python3.10/dist-packages/pyspark/sql/session.py", line 1276, in createDataFrame
    return self._create_dataframe(
  File "/usr/local/lib/python3.10/dist-packages/pyspark/sql/session.py", line 1316, in _create_dataframe
    rdd, struct = self._createFromRDD(data.map(prepare), schema, samplingRatio)
  File "/usr/local/lib/python3.10/dist-packages/pyspark/sql/session.py", line 931, in _createFromRDD
    struct = self._inferSchema(rdd, samplingRatio, names=schema)
  File "/usr/local/lib/python3.10/dist-packages/pyspark/sql/session.py", line 874, in _inferSchema
    first = rdd.first()
  File "/usr/local/lib/python3.10/dist-packages/pyspark/rdd.py", line 2869, in first
    rs = self.take(1)
  File "/usr/local/lib/python3.10/dist-packages/pyspark/rdd.py", line 2836, in take
    res = self.context.runJob(self, takeUpToNumLeft, p)
  File "/usr/local/lib/python3.10/dist-packages/pyspark/context.py", line 2319, in runJob
    sock_info = self._jvm.PythonRDD.runJob(self._jsc.sc(), mappedRDD._jrdd, partitions)
  File "/usr/local/lib/python3.10/dist-packages/pyspark/rdd.py", line 5441, in _jrdd
    wrapped_func = _wrap_function(
  File "/usr/local/lib/python3.10/dist-packages/pyspark/rdd.py", line 5243, in _wrap_function
    return sc._jvm.SimplePythonFunction(
TypeError: 'JavaPackage' object is not callable

问题原因与解决方法

这个错误的核心是pip安装的pyspark版本与容器中手动部署的Spark二进制包版本不兼容。你指定了Spark 3.3.2，但pip install pyspark会默认拉取最新版pyspark，新版本的Java类路径和旧版Spark二进制包不匹配，导致找不到SimplePythonFunction类，抛出JavaPackage不可调用的错误。

具体修复步骤：

锁定pyspark版本与Spark二进制包一致
修改Dockerfile中安装pyspark的命令，指定版本为3.3.2：
```
RUN pip install pyarrow py4j scipy \
    pyspark==3.3.2 pyspark[sql]==3.3.2 pyspark[pandas_on_spark]==3.3.2
```

调整环境变量顺序，确保容器内Spark优先
将Dockerfile中PATH的配置移到Spark安装完成后，让SPARK_HOME/bin的优先级高于pip安装的pyspark路径：

# 移到Spark安装完成后配置
ENV SPARK_OPTS="--driver-java-options=-Xms1024M --driver-java-options=-Xmx4096M --driver-java-options=-Dlog4j.logLevel=info"
ENV PATH="${SPARK_HOME}/bin:${PATH}"

简化测试脚本初始化逻辑
移除findspark.init()，因为已通过环境变量指定SPARK_HOME，无需额外初始化。同时明确指定Python路径，避免路径问题：
```
# 移除 findspark.init()
os.environ["PYARROW_IGNORE_TIMEZONE"] = "1"
os.environ['PYSPARK_PYTHON'] = '/usr/bin/python3'
os.environ['PYSPARK_DRIVER_PYTHON'] = '/usr/bin/python3'
```

优化RDD转DataFrame的逻辑
原脚本中把列表转字符串再拆分的操作完全多余，直接用toDF指定列名即可，减少不必要的RDD操作：

def test_para():
    num_datapoints_to_match = int(10e6)
    data = spark.sparkContext.parallelize(
        [[random()*3, random()*3 ] for x in range(num_datapoints_to_match) ],
        numSlices=spark_context_some_slices)
    # 直接指定列名创建DataFrame
    sdf0 = data.toDF(["x", "y"])

验证修复

重新构建Docker容器后运行测试脚本，错误应该会消失。如果仍有问题，进入容器执行以下命令检查版本一致性：

pyspark --version
pip show pyspark

确保两者版本均为3.3.2。

内容的提问来源于stack exchange，提问作者Bamu

Docker部署Spark3.3.2环境触发TypeError: 'JavaPackage'对象不可调用

问题原因与解决方法 #

具体修复步骤： #

验证修复 #

问题原因与解决方法

具体修复步骤：

验证修复