VS Code Docker容器中PySpark连接PostgreSQL时Py4JJavaError及连接拒绝问题排查求助

大家好，我最近碰到一个很费解的问题，想请教下各位开发者：

我在VS Code中运行着一个Docker容器，里面用PySpark连接本地机器上的PostgreSQL数据库，代码如下：

from pyspark.sql import SparkSession

spark = SparkSession \
.builder \
.appName("Python Spark SQL basic example") \
.config("spark.jars", "/opt/spark/jars/postgresql-42.2.5.jar") \
.getOrCreate()

df = spark.read \
.format("jdbc") \
.option("url", "jdbc:postgresql://host.docker.internal:5432/postgres") \
.option("dbtable", "chicago_crime") \
.option("user", "postgres") \
.option("password", "postgres") \
.option("driver", "org.postgresql.Driver") \
.load()

type(df)

执行后返回的类型是pyspark.sql.dataframe.DataFrame，看起来DataFrame是创建成功了。

目前有些操作能正常运行，比如：

• df.printSchema() 可以正常输出表结构
• df.select('ogc_fid').show() 有时候能成功执行，但偶尔也会抛出Py4JJavaError

但像df.show(1)这类操作完全无法运行，会直接抛出Py4JJavaError，底层还附带ConnectionRefusedError，完整错误日志如下：

Output exceeds the size limit. Open the full output data in a text editor

---

Py4JJavaError Traceback (most recent call last)
[... skipping hidden 1 frame]

Cell In[2], line 1
----> 1 df.show(1)

File /usr/local/lib/python3.9/site-packages/pyspark/sql/dataframe.py:606, in DataFrame.show(self, n, truncate, vertical)
605 if isinstance(truncate, bool) and truncate:
--> 606 print(self._jdf.showString(n, 20, vertical))
607 else:

File /usr/local/lib/python3.9/site-packages/py4j/java_gateway.py:1321, in JavaMember.call(self, *args)
1320 answer = self.gateway_client.send_command(command)
-> 1321 return_value = get_return_value(
1322 answer, self.gateway_client, self.target_id, self.name)
1324 for temp_arg in temp_args:

File /usr/local/lib/python3.9/site-packages/pyspark/sql/utils.py:190, in capture_sql_exception..deco(*a, **kw)
189 try:
--> 190 return f(*a, **kw)
191 except Py4JJavaError as e:

File /usr/local/lib/python3.9/site-packages/py4j/protocol.py:326, in get_return_value(answer, gateway_client, target_id, name)
325 if answer[1] == REFERENCE_TYPE:
...
--> 438 self.socket.connect((self.java_address, self.java_port))
439 self.stream = self.socket.makefile("rb")
440 self.is_connected = True

ConnectionRefusedError: [Errno 111] Connection refused

有没有大佬能解释下这个Py4JJavaError的根源？特别是为什么部分操作能正常执行，部分操作却会触发连接拒绝的错误？该怎么解决这个问题呢？

备注：内容来源于stack exchange，提问作者Herwini