使用列表推导式遍历双列表处理DataFrame为何仅保留最后一次输出

• 列表推导式逻辑错误：你写的[(g,p) for g in geo_list for p in prod_list]仅生成了所有(地理、产品)组合的元组列表，没有将后续的数据处理逻辑包含在循环内，遍历结束后变量g、p只会保留最后一组组合值，后续处理仅针对最后一组数据生效，之前的所有遍历结果都没有被处理。
• 没有累积每次处理的结果：你没有为每次循环的计算结果设置临时存储容器，每次计算的new_df都会直接覆盖上一次的内容，最终只能保留最后一次循环的输出。
• 不匹配数据的筛选逻辑错误：当前的unmatch筛选是基于最后一组的new_df生成的，筛选范围完全不符合预期。

import time
import numpy as np
import pandas as pd

t1 = time.time()

geo_list = list(data5['Geography'].unique())
prod_list = list(data5['Product'].unique())

# 初始化临时列表存储每个分组的处理结果
processed_dfs = []

# 遍历所有组合处理数据
for g in geo_list:
    for p in prod_list:
        tmp_df = data5[(data5['Geography']==g) & (data5['Product']==p)].copy()
        if len(tmp_df) == 0:
            continue
        # 计算衍生字段
        tmp_df['pn'] = tmp_df['Price']-tmp_df['Price'].shift(-1)
        tmp_df['price_d'] = np.divide((tmp_df['Price']+tmp_df['Price'].shift(1)),2)
        tmp_df['dn'] = tmp_df['Unit_Sales']-tmp_df['Unit_Sales'].shift(-1)
        tmp_df['dd'] = np.divide((tmp_df['Unit_Sales']+tmp_df['Unit_Sales'].shift(1)),2)
        tmp_df['Delta_Demand'] = np.divide(tmp_df['dn'].values,tmp_df['dd'].values)*100
        tmp_df['Delta_Price'] = np.divide(tmp_df['pn'].values,tmp_df['price_d'].values)*100
        tmp_df['ped'] = np.divide(tmp_df['Delta_Demand'].values,tmp_df['Delta_Price'].values)
        # 保留需要的字段
        tmp_df = tmp_df[['Geography', 'Product', 'Price', 'Unit_Sales', 'ped']]
        processed_dfs.append(tmp_df)

# 合并所有处理后的分组数据
new_df_all = pd.concat(processed_dfs, ignore_index=True)

# 筛选未匹配的原始数据
unmatch = data5[~data5.apply(lambda x: (x['Geography'], x['Product'], x['Price'], x['Unit_Sales']) in zip(new_df_all['Geography'], new_df_all['Product'], new_df_all['Price'], new_df_all['Unit_Sales']), axis=1)]

# 生成最终结果
df_final = pd.concat([new_df_all, unmatch], ignore_index=True)

t2 = time.time()
print("Time taken by Loop: %.6f" %(t2 - t1))
# 验证地理字段唯一值
print(df_final['Geography'].unique())

你当前的嵌套循环效率较低，可以直接用groupby(['Geography', 'Product'])实现分组计算，代码更简洁运行速度也更快，示例逻辑如下：

def calc_ped(group):
    group['pn'] = group['Price']-group['Price'].shift(-1)
    group['price_d'] = np.divide((group['Price']+group['Price'].shift(1)),2)
    group['dn'] = group['Unit_Sales']-group['Unit_Sales'].shift(-1)
    group['dd'] = np.divide((group['Unit_Sales']+group['Unit_Sales'].shift(1)),2)
    group['Delta_Demand'] = np.divide(group['dn'].values,group['dd'].values)*100
    group['Delta_Price'] = np.divide(group['pn'].values,group['price_d'].values)*100
    group['ped'] = np.divide(group['Delta_Demand'].values,group['Delta_Price'].values)
    return group[['Geography', 'Product', 'Price', 'Unit_Sales', 'ped']]

new_df_all = data5.groupby(['Geography', 'Product'], group_keys=False).apply(calc_ped)

内容的提问来源于stack exchange，提问作者Alex Navarro