Spark DataFrame转Pandas时出现'MultiIndex'无'labels'属性错误

问题场景 #

执行Spark DataFrame转Pandas DataFrame操作时触发错误：

df_temp_p = df_temp.toPandas()

错误详情 #

----> 2 df_temp_p = df_temp.toPandas()

~/.conda/envs/conda_py36_env/lib/python3.6/site-packages/pyspark/sql/dataframe.py in toPandas(self)
   2122                     if len(batches) > 0:
   2123                         table = pyarrow.Table.from_batches(batches)
-> 2124                         pdf = table.to_pandas()
   2125                         pdf = _check_dataframe_convert_date(pdf, self.schema)
   2126                         return _check_dataframe_localize_timestamps(pdf, timezone)

~/.conda/envs/conda_py36_env/lib/python3.6/site-packages/pyarrow/table.pxi in pyarrow.lib.Table.to_pandas (/arrow/python/build/temp.linux-x86_64-3.6/lib.cxx:46331)()

~/.conda/envs/conda_py36_env/lib/python3.6/site-packages/pyarrow/pandas_compat.py in table_to_blockmanager(options, table, memory_pool, nthreads, categoricals)
    568 
    569     # ARROW-1751: flatten a single level column MultiIndex for pandas 0.21.0
-> 570     columns = _flatten_single_level_multiindex(columns)
    571 
    572     axes = [columns, index]

~/.conda/envs/conda_py36_env/lib/python3.6/site-packages/pyarrow/pandas_compat.py in _flatten_single_level_multiindex(index)
    626     if isinstance(index, pd.MultiIndex) and index.nlevels == 1:
    627         levels, = index.levels
-> 628         labels, = index.labels
    629 
    630         # Cheaply check that we do not somehow have duplicate column names

AttributeError: 'MultiIndex' object has no attribute 'labels'

已尝试的无效方法 #

• 关闭Arrow优化：

  spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "false")
  

• 全量选择列后转换：

  df_temp_p = df_temp.select("*").toPandas()
  

Spark DataFrame信息 #

结构 #

root
 |-- decile_rank: integer (nullable = false)
 |-- oot1_count: long (nullable = false)

数据内容 #

+-----------+----------+
|decile_rank|oot1_count|
+-----------+----------+
|          1|    169060|
|          2|    237598|
|          3|     91971|
|          4|    214923|
|          5|     38040|
|          6|    282598|
|          7|     61426|
|          8|    473339|
|          9|     36807|
|         10|    249824|
+-----------+----------+

解决方案 #

该错误源于pyarrow与pandas版本不兼容：pandas 1.0+版本将MultiIndex.labels属性替换为MultiIndex.codes，但旧版pyarrow仍在调用已废弃的labels属性。可通过以下方式解决：

方法1：升级pyarrow到兼容版本（推荐） #

升级pyarrow至0.17.0及以上版本，该版本开始适配pandas 1.0的API变更：

pip install --upgrade pyarrow

方法2：降级pandas到1.0以下版本（仅在升级pyarrow有冲突时使用） #

pip install pandas==0.25.3

方法3：临时绕过pyarrow的转换逻辑 #

将Spark DataFrame先转为RDD再构建Pandas DataFrame，此方法会将全量数据拉取到Driver节点，仅适用于小数据集：

import pandas as pd
df_temp_p = pd.DataFrame(df_temp.collect(), columns=df_temp.columns)

内容的提问来源于stack exchange，提问作者karan