解决使用concat()拼接空DataFrame时的FutureWarning问题

场景：拼接多个可能为空的DataFrame，需要保留所有列名且最终结果无空行，目前已修复一条警告，但仍存在FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated.警告。

原代码如下：

import io
import pandas as pd

df_list = ['RevisionTime,Data,2019/Q2,2019/Q3,2019/Q4\r\n',
           'RevisionTime,Data,2019/Q3\r\n2019-08-17,10.5,10.5\r\n',
           'RevisionTime,Data,2019/Q3\r\n2019-09-18 08:10:00,51.0,51.0\r\n',
           'RevisionTime,Data,2019/Q3\r\n2019-10-18 08:10:00,111.5,111.5\r\n',
           'RevisionTime,Data,2019/Q3,2019/Q4\r\n2019-11-15 22:31:00,182.0,111.5,70.5\r\n']

# list with dataframes
df_list = [pd.read_csv(io.StringIO(df)) for df in df_list]

# to avoid 'The behaviour of array concatenation with empty entries is deprecated.'
# and to retain all column names
for i, df in enumerate(df_list):
    col_length = len(df.columns)
    template = pd.DataFrame(data=[[pd.NA] * col_length], columns=df.columns)
    df_list[i] = df if not df.empty else template

res_df = pd.concat(df_list) # warning here
res_df = res_df.dropna(how='all') # remove empty rows
print(res_df)

问题原因 #

原代码将空DataFrame替换为全NA的单行DataFrame，concat时这些全NA条目仍会触发警告。

修改方案 #

先收集所有列名，对齐每个DataFrame的列，再过滤掉空或全NA的DataFrame，最后拼接即可消除警告，同时保留所有列名和有效数据：

import io
import pandas as pd

df_list = ['RevisionTime,Data,2019/Q2,2019/Q3,2019/Q4\r\n',
           'RevisionTime,Data,2019/Q3\r\n2019-08-17,10.5,10.5\r\n',
           'RevisionTime,Data,2019/Q3\r\n2019-09-18 08:10:00,51.0,51.0\r\n',
           'RevisionTime,Data,2019/Q3\r\n2019-10-18 08:10:00,111.5,111.5\r\n',
           'RevisionTime,Data,2019/Q3,2019/Q4\r\n2019-11-15 22:31:00,182.0,111.5,70.5\r\n']

# 读取所有DataFrame
df_list = [pd.read_csv(io.StringIO(df)) for df in df_list]

# 收集所有列名，确保最终结果包含所有列
all_columns = set()
for df in df_list:
    all_columns.update(df.columns)
all_columns = list(all_columns)

# 处理每个DataFrame：对齐列，过滤空/全NA的DataFrame
processed_dfs = []
for df in df_list:
    # 对齐到完整列集合，缺失列填充NA
    aligned_df = df.reindex(columns=all_columns)
    # 过滤掉空或所有行都是全NA的DataFrame
    if not aligned_df.empty and not aligned_df.dropna(how='all').empty:
        processed_dfs.append(aligned_df)

# 拼接并移除全NA行（可选，因为已过滤）
res_df = pd.concat(processed_dfs)
res_df = res_df.dropna(how='all')
# 调整列顺序和原数据一致（可选）
res_df = res_df[['RevisionTime', 'Data', '2019/Q2', '2019/Q3', '2019/Q4']]
print(res_df)

输出结果 #

RevisionTime   Data 2019/Q2  2019/Q3  2019/Q4
0           2019-08-17   10.5     NaN     10.5      NaN
0  2019-09-18 08:10:00   51.0     NaN     51.0      NaN
0  2019-10-18 08:10:00  111.5     NaN    111.5      NaN
0  2019-11-15 22:31:00  182.0     NaN    111.5     70.5

内容的提问来源于stack exchange，提问作者Vitamin C