Python For循环生成衰减权重的高性能优化实现方案
批量生成衰减权重的性能优化
我需要批量生成符合指定规则的衰减权重,现有Python实现逻辑符合预期,但其中的For循环执行效率极低,本地测试单循环运行耗时达到1分05秒。
原始实现代码
# Imports from sympy.solvers.solveset import solveset_real from sympy import Symbol, Eq # Define variables initial_value = 1 rate = Symbol('r') decay_obs_window = 1480346 target_decay = .15 # Solver to calculate decay rate decay_rate = solveset_real(Eq((initial_value - rate * decay_obs_window), target_decay), rate).args[0] # Generate weights weights = [] for i in range(5723673): # How to handle data BEYOND decay_obs_window if i > decay_obs_window and target_decay == 0: # Record a weight of zero weights.append(0) elif i > decay_obs_window and target_decay > 0: # Record the final target weight weights.append(decayed_weight) # How to handle data WITHIN decay_obs_window else: # Calculate the new slightly decayed weight decayed_weight = 1 - (decay_rate * i) weights.append(decayed_weight) weights[0:10]
我尝试改用列表推导式优化执行效率,代码可正常运行,但性能无明显提升,优化后代码如下:
weights = [0 if i > decay_obs_window and target_decay == 0 else decayed_weight if i > decay_obs_window and target_decay > 0 else (decayed_weight := 1 - (decay_rate * i)) for i in range(len(weights_df))]
请问有什么更高效的实现方案可以大幅降低运行耗时?
最终优化方案
最终采用了基于numpy向量化运算的优化方案,本地测试总运行耗时仅425ms,代码如下:
import numpy as np from sympy.solvers.solveset import solveset_real from sympy import Symbol, Eq # Define variables initial_value = 1 rate = Symbol('r') decay_obs_window = 1480346 target_decay = .15 # Instantiate weights array weights = np.zeros(5723673) # Solver to calculate decay rate decay_rate = solveset_real(Eq((initial_value - rate * decay_obs_window), target_decay), rate).args[0] # Fix a bug where numpy doesn't like sympy floats :( decay_rate = float(decay_rate) # How to weight observations WITHIN decay_obs_window weights[:decay_obs_window + 1] = 1 - np.arange(decay_obs_window + 1) * decay_rate # How to weight observations BEYOND decay_obs_window weights[decay_obs_window + 1 : 5723673] = target_decay weights
优化核心逻辑
- 抛弃逐元素遍历的Python原生循环,使用Numpy的向量化运算,底层由C实现,大幅提升大规模数值计算效率
- 直接按区间批量赋值,不需要对每个元素做分支判断,进一步减少运算开销
内容的提问来源于stack exchange,提问作者matteblack
相关产品推荐
相关产品推荐

