如何用Pandas/Numpy无循环实现多维数据集行累计值重计算?
高效实现基于前序累计和的DataFrame计算
需求说明
给定n×n的DataFrame(df_a),需生成新的n×n DataFrame(df_b),计算逻辑为:df_b[i,j] = (df_a对应单元格的值 - 该列当前行之前所有df_b单元格的累计和) × 1×n Series(df_series)对应行的数值
示例数据与结果
n×n数据集df_a
| i | 0 | 1 | 2 | 3 | 4 | 5 | 6 |
|---|---|---|---|---|---|---|---|
| 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 2 | 200 | 0 | 0 | 0 | 0 | 0 | 0 |
| 3 | 200 | 150 | 0 | 0 | 0 | 0 | 0 |
| 4 | 200 | 150 | 125 | 0 | 0 | 0 | 0 |
| 5 | 200 | 150 | 125 | 100 | 0 | 0 | 0 |
| 6 | 200 | 150 | 125 | 100 | 75 | 0 | 0 |
1×n Series
| i | 0 |
|---|---|
| 0 | .000 |
| 1 | .001 |
| 2 | .002 |
| 3 | .003 |
| 4 | .004 |
| 5 | .005 |
| 6 | .006 |
计算结果df_b
| i | 0 | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|---|
| 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 1 | 0 | 0 | 0 | 0 | 0 | 0 |
| 2 | .4 [(200 - sum 0) * .002] | 0 | 0 | 0 | 0 | 0 |
| 3 | .5988 [(200 - sum .4) * .003] | .45 [(150 - sum 0) * .003] | 0 | 0 | 0 | 0 |
| 4 | .7960 [(200 - sum (.4, .5988)) * .004] | .5982 [(150 - sum .45) * .004] | .50 | 0 | 0 | 0 |
| 5 | .9910 [(200 - sum (.4, .5988, .7960)) * .005] | .7447 [(150 - sum .45, .5982) * .005] | .6225 | .50 | 0 | 0 |
| 6 | 1.1832 [(200 - sum (.4, .5988, .7960, .9910)) * .006] | .8892 [(150 - sum .45, .5982, .7447) * .006] | .74326 | .5970 | .450 | 0 |
当前实现(低效循环)
我熟练使用Pandas和Numpy,但目前通过双重循环实现该计算,耗时极长。现有循环代码如下:
import numpy as np import pandas as pd df_a =pd.DataFrame([[0,0,0,0,0,0,0],[0,0,0,0,0,0,0],[200,0,0,0,0,0,0],[200,150,0,0,0,0,0],[200,150,125,0,0,0,0],[200,150,125,100,0,0,0],[200,150,125,100,75,0,0]]).astype(float) df_series = pd.Series([0.0,.001, .002, .003, .004, .005, .006]) df_b = pd.DataFrame(np.arange(49).reshape(7,7)).astype(float) for i in range(df_a.shape[0]): # 遍历行 for j in range(df_a.shape[1]): # 遍历列 df_a_val = df_a.iloc[i, j] # 获取df_a对应单元格值 ser_val = df_series.iloc[i] # 获取series对应行的值 df_b_prev = df_b.iloc[:i,j] # 获取当前列之前所有df_b的值 df_b_sum = df_b_prev.sum() # 计算前序值的累计和 df_b.at[i, j] = (df_a_val-df_b_sum)*(ser_val) # 计算并赋值 print("(df_a_val: {} - df_b_sum: {}) * ser_val : {} = {}".format(df_a_val, df_b_sum, ser_val, df_b.at[i, j])) print(df_series) print(df_b)
求助目标
寻求无需循环的高效实现方法,利用Pandas或Numpy的向量化操作提升计算速度。
内容的提问来源于stack exchange,提问作者String
相关产品推荐
相关产品推荐

