PySpark中DataFrame无法控制台显示，报Py4JJavaError大小超限错误

问题场景 #

创建包含6列的Spark DataFrame后，调用pysparkDF.show(truncate=False)试图完整展示数据，却触发Py4JJavaError，提示输出超过大小限制。

用户代码 #

创建SparkSession代码 #

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

DataFrame创建代码 #

data = [("James","","Smith",30,"M",60000),
        ("Michael","Rose","",50,"M",70000),
        ("Robert","","Williams",42,"",400000),
        ("Maria","Anne","Jones",38,"F",500000),
        ("Jen","Mary","Brown",45,"F",0)]

columns = ["first_name","middle_name","last_name","Age","gender","salary"]
pysparkDF = spark.createDataFrame(data = data, schema = columns)
pysparkDF.printSchema()
pysparkDF.show(truncate=False)

报错信息 #

Output exceeds the size limit. Open the full output data in a text editor
---------------------------------------------------------------------------
Py4JJavaError                             Traceback (most recent call last)
Cell In[28], line 10
      8 pysparkDF = spark.createDataFrame(data = data, schema = columns)
      9 pysparkDF.printSchema()
---> 10 pysparkDF.show(truncate=False)

File c:\Users\baps\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyspark\sql\dataframe.py:615, in DataFrame.show(self, n, truncate, vertical)
    610 except ValueError:
    611     raise TypeError(
    612         "Parameter 'truncate={}' should be either bool or int.".format(truncate)
    613     )
---> 615 print(self._jdf.showString(n, int_truncate, vertical))

File c:\Users\baps\AppData\Local\Programs\Python\Python38-32\lib\site-packages\py4j\java_gateway.py:1321, in JavaMember.__call__(self, *args)
   1315 command = proto.CALL_COMMAND_NAME +\
   1316     self.command_header +\
   1317     args_command +\
   1318     proto.END_COMMAND_PART
   1320 answer = self.gateway_client.send_command(command)
---> 1321 return_value = get_return_value(
   1322     answer, self.gateway_client, self.target_id, self.name)
   1324 for temp_arg in temp_args:
   1325     temp_arg._detach()
...
    at java.lang.ProcessImpl.<init>(ProcessImpl.java:453)
    at java.lang.ProcessImpl.start(ProcessImpl.java:140)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
    ... 30 more

解决方案 #

1. 使用垂直展示模式 #

将展示模式改为垂直排列，每行数据按字段拆分显示，大幅减少单行输出长度，避免超限：

pysparkDF.show(truncate=False, vertical=True)

2. 截断过长字段 #

将truncate参数设为整数，指定字段的最大显示长度，控制整体输出大小：

# 示例：限制每个字段最多显示20个字符
pysparkDF.show(truncate=20)

3. 手动遍历打印数据 #

通过collect()获取所有行数据后，逐行打印，灵活控制输出格式：

for row in pysparkDF.collect():
    print(row)

4. 调整环境输出限制（针对Jupyter等交互式环境） #

如果在Jupyter Notebook/Lab中运行，可修改IPython的输出配置，增大允许的输出宽度：

from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.instance().display_formatter.max_width = 1000

内容的提问来源于stack exchange，提问作者GALAXY A50