Jupyter中PySpark DataFrame用df.show()无法正常显示为表格的原因

当你在Jupyter Notebook中用PySpark读取CSV后调用df.show()，因列数过多、内容过长，会出现表格排版散乱、内容截断错位的情况，就像你提供的示例那样：

+---------+-------------+--------------------+--------------------+--------------+-----------------+------------------+-------+----+-----+--------------------+-----------+--------+----------+--------------+-------------+---------+------+--------------------------+------------------------+-------------------------------------------------------------------------------------------+--------------------+-------+--------+----------+-----------+------------+
|SALE TYPE|    SOLD DATE|       PROPERTY TYPE|             ADDRESS|          CITY|STATE OR PROVINCE|ZIP OR POSTAL CODE|  PRICE|BEDS|BATHS|            LOCATION|SQUARE FEET|LOT SIZE|YEAR BUILT|DAYS ON MARKET|$/SQUARE FEET|HOA/MONTH|STATUS|NEXT OPEN HOUSE START TIME|NEXT OPEN HOUSE END TIME|URL (SEE https://www.redfin.com/buy-a-home/comparative-market-analysis FOR INFO ON PRICING)|              SOURCE|   MLS#|FAVORITE|INTERESTED|   LATITUDE|   LONGITUDE|
+---------+-------------+--------------------+--------------------+--------------+-----------------+------------------+-------+----+-----+--------------------+-----------+--------+----------+--------------+-------------+---------+------+--------------------------+------------------------+-------------------------------------------------------------------------------------------+--------------------+-------+--------+----------+-----------+------------+
|PAST SALE|April-10-2024|Single Family Res...|1016 Wyndham Hill Ln|      Franklin|               TN|             37069| 950000|   5|  3.0|    Fieldstone Farms|       3500|   21780|      1993|          NULL|          271|       75|  Sold|                      NULL|                    NULL|                                                                       https://www.redfi...|REALTRACS as Dist...|2641189|       N|         Y| 35.9697949| -86.8849545|
+---------+-------------+--------------------+--------------------+--------------+-----------------+------------------+-------+----+-----+--------------------+-----------+--------+----------+--------------+-------------+---------+------+--------------------------+------------------------+-------------------------------------------------------------------------------------------+--------------------+-------+--------+----------+-----------+------------+

以下是几种实用的解决方法：

1. 调整show()的参数 #

• 关闭内容截断：使用truncate=False完整显示所有字符串内容，避免省略号，同时可指定显示行数（比如n=5）：

  df.show(n=5, truncate=False)
  

• 垂直显示单条数据：如果列数太多，横向排版始终混乱，用vertical=True将每一行数据按列垂直展开，可读性更强：

  df.show(n=5, truncate=False, vertical=True)
  

2. 转换为Pandas DataFrame显示 #

PySpark DataFrame转成Pandas后，Jupyter会自动渲染为美观的HTML表格，支持自适应宽度：

# 限制行数避免内存溢出，大数据集务必加limit
df.limit(5).toPandas()

如果还是有列截断或宽度不足，可先设置Pandas的显示参数：

import pandas as pd
# 显示所有列
pd.set_option('display.max_columns', None)
# 显示完整列内容
pd.set_option('display.max_colwidth', None)
# 设置整体显示宽度
pd.set_option('display.width', 1000)

df.limit(5).toPandas()

3. 使用交互式显示函数（部分环境支持） #

如果你的Jupyter环境集成了Databricks或相关扩展，直接用display()函数可生成交互式表格，支持排序、过滤和列宽调整：

display(df.limit(5))

内容的提问来源于stack exchange，提问作者imaginationkid