Pandas 2.0.3向DataFrame空列表列赋值列表报错的解决方法

问题描述 #

原有代码在Pandas 1.x版本可正常运行，升级至2.0.3版本后，无法将列表元素赋值给DataFrame的空列表列。执行赋值操作时抛出错误：ValueError: Must have equal len keys and value when setting with an iterable，DataFrame的目标列仍为空列表。

原问题代码：

import pandas as pd

data = {"ID":{"0":"2","1":"20","2":"201","3":"2011","4":"206","5":"2061","6":"2062","7":"2063"},"Name":{"0":"Page","1":"Detailspage","2":"User Page","3":"User detailed page","4":"Result","5":"Res","6":"Res","7":"Res3"},"PATH":{"0":"/","1":"/Detailspage","2":"/UserPage","3":"/User detailed page","4":"Result","5":"Result / Res1","6":"Result / Res2","7":"Result / Res3"},"TYPE":{"0":"Page","1":"Page","2":"P/T","3":"Tile","4":"P/T","5":"T","6":"T","7":"T","8":"T"}}

df = pd.DataFrame(data)

df['details'] = [[] for _ in range(df.shape[0])]

df['ID'] = df['ID'].astype(str)

for i in range(0, len(df)):
        i_id = df.iloc[i]['ID']

        temp_df = (df.loc[df.ID.str.contains(f'^{i_id}\\d$'), :])

        if len(temp_df) > 0:
                print("@@",temp_df['PATH'].values.tolist())
                df.at[i,'details'] = temp_df['PATH'].values.tolist()
print(df["details"])

原因分析 #

Pandas 2.x版本对at/iat这类单元素定位赋值的校验逻辑进行了严格升级：直接赋值可迭代对象（如列表）时，会被误认为是要给多个单元格赋值，而非给单个单元格赋值一个列表对象，因此抛出长度不匹配的错误。

解决方案 #

方案1：改用loc/iloc完成赋值 #

将循环中的df.at[i,'details'] = ...替换为以下任意一种写法即可：

# 使用loc定位赋值
df.loc[i, 'details'] = temp_df['PATH'].tolist()

# 或者使用iloc定位赋值
df['details'].iloc[i] = temp_df['PATH'].tolist()

loc/iloc在该场景下可以正确识别赋值目标为单个单元格，接受列表作为单元格的值。

方案2：矢量化处理（推荐，避免循环） #

循环处理效率较低，尤其当数据量较大时，可使用apply结合字符串匹配实现矢量化操作，无需提前初始化空列表列：

import pandas as pd

data = {"ID":{"0":"2","1":"20","2":"201","3":"2011","4":"206","5":"2061","6":"2062","7":"2063"},"Name":{"0":"Page","1":"Detailspage","2":"User Page","3":"User detailed page","4":"Result","5":"Res","6":"Res","7":"Res3"},"PATH":{"0":"/","1":"/Detailspage","2":"/UserPage","3":"/User detailed page","4":"Result","5":"Result / Res1","6":"Result / Res2","7":"Result / Res3"},"TYPE":{"0":"Page","1":"Page","2":"P/T","3":"Tile","4":"P/T","5":"T","6":"T","7":"T","8":"T"}}

df = pd.DataFrame(data)
df['ID'] = df['ID'].astype(str)

# 直接生成details列
df['details'] = df['ID'].apply(
    lambda x: df.loc[df['ID'].str.match(f'^{x}\\d$'), 'PATH'].tolist()
)

print(df["details"])

运行结果 #

执行后可得到预期输出：

0    ['/UserPage', 'Result']
1    ['/User detailed page']
2    ['Result / Res1', 'Result / Res2', 'Result / Res3']
3                          []
4                          []
5                          []
6                          []
7                          []
Name: details, dtype: object

内容的提问来源于stack exchange，提问作者Madhu