Pandas最新更新后，pd.CategoricalDtype分组过滤数据致px.bar报错的解决方法

我有一个DataFrame，打算用pd.CategoricalDtype做数据过滤，再用px.bar绘制柱状图。Pandas未更新前代码运行完全正常，但更新后绘制图表时崩溃，抛出如下错误：

Traceback (most recent call last): File "", line 1, in
File
"/home/marco/python-wsl/project_folder/venv/lib/python3.8/site-packages/plotly/express/_chart_types.py",
line 373, in bar
return make_figure( File "/home/marco/python-wsl/project_folder/venv/lib/python3.8/site-packages/plotly/express/_core.py",
line 2003, in make_figure
groups, orders = get_groups_and_orders(args, grouper) File "/home/marco/python-wsl/project_folder/venv/lib/python3.8/site-packages/plotly/express/_core.py",
line 1978, in get_groups_and_orders
groups = { File "/home/marco/python-wsl/project_folder/venv/lib/python3.8/site-packages/plotly/express/_core.py",
line 1979, in
sf: grouped.get_group(s if len(s) > 1 else s[0]) File "/home/marco/python-wsl/project_folder/venv/lib/python3.8/site-packages/pandas/core/groupby/groupby.py",
line 811, in get_group
raise KeyError(name) KeyError: 'C'

相关代码 #

# Code outside px.bar
old_df2 = pd.DataFrame({"name": ["A", "A", "A", "A", "B", "B", "B", "C"],
                       "id1": [18, 22, 19, 14, 14, 11, 20, 28],
                       "id2": [5, 7, 7, 9, 12, 9, 9, 4],
                       "id3": [11, 8, 10, 6, 6, 7, 9, 12]})


new_df = old_df2.groupby([pd.CategoricalDtype(old_df2.name),'id2'])['id3'].count().fillna(0)
    
# Transforms count from series to data frame
new_df = new_df.to_frame()

# rowname to index 
new_df.reset_index(inplace=True)

new_df = new_df[new_df["level_0"].isin(["A","B"])]

new_df .rename(columns={'level_0': 'name'}, inplace=True)

# Not working here the error 
fig_bar = px.bar(new_df.loc[::-1], x="id2", y="id3", color = "name", barmode="group")

# Working version identical data
new_df_list = new_df.to_dict("records")

unlinked_df = pd.DataFrame(new_df_list )

直接使用过滤后的new_df调用px.bar会触发错误，但将其转为字典再重新生成DataFrame后则可正常运行，请问该如何修复这个问题？

问题根源 #

问题出在分类变量的类别未同步更新：你通过pd.CategoricalDtype创建的分类包含原始数据的A、B、C三类，过滤后new_df里只剩A和B，但分类的元数据仍保留了C这个无效类别。Plotly Express在分组处理时会尝试遍历所有原始分类的分组，找不到C就抛出KeyError。

而转字典再重建DataFrame的操作，会把分类类型转为普通字符串，丢失了分类元数据，所以Plotly不会再去查找不存在的C分组。

修复方法 #

方法1：过滤后清理无效分类 #

在过滤并重命名列之后，调用cat.remove_unused_categories()删除不再存在的分类类别：

new_df.rename(columns={'level_0': 'name'}, inplace=True)
# 添加这行代码清理无效分类
new_df['name'] = new_df['name'].cat.remove_unused_categories()

# 现在可以正常绘图
fig_bar = px.bar(new_df.loc[::-1], x="id2", y="id3", color = "name", barmode="group")
fig_bar.show()

方法2：提前转换分类列 #

先把name列转换为分类类型，再进行分组操作，后续过滤后同样清理无效分类：

old_df2['name'] = pd.Categorical(old_df2['name'])
new_df = old_df2.groupby(['name','id2'])['id3'].count().fillna(0)

new_df = new_df.to_frame().reset_index()
new_df = new_df[new_df["name"].isin(["A","B"])]
# 清理无效分类
new_df['name'] = new_df['name'].cat.remove_unused_categories()

# 正常绘图
fig_bar = px.bar(new_df.loc[::-1], x="id2", y="id3", color = "name", barmode="group")
fig_bar.show()

内容的提问来源于stack exchange，提问作者DevLeb2022