为何向量化未提升代码运行速度？附循环与向量化代码对比

问题描述 #

我写了两段功能一致的代码：遍历数值组合列表，找出每个组合对应的生效过滤器，最终筛选出唯一过滤器（比如过滤器(5,6)(5,6)(5,6)和(5,7)(5,7)(5,7)等效，不算唯一过滤器）。听说向量化能提升速度，我在基础for循环后做了向量化实现，但向量化版本运行速度反而慢很多（结果是对的），而且去重步骤耗时远超预期，求帮忙排查原因。

完整代码 #

#########################################
### Compare for loop to vectorization ###
#########################################
import pandas as pd
import itertools
import numpy as np
import time

size_value = 8
# df = pd.DataFrame({'var_1_lower_limit':[1,2,1,3,2],'var_1_upper_limit':[5,5,4,7,6],'var_2_lower_limit':[5,5,3,6,4],'var_2_upper_limit':[9,9,8,9,7],'var_3_lower_limit':[6,7,4,2,4],'var_3_upper_limit':[8,8,6,8,7]})
df = pd.DataFrame({'var_1_lower_limit':np.random.randint(0,5,size=size_value),'var_1_upper_limit':np.random.randint(5,10,size=size_value),'var_2_lower_limit':np.random.randint(0,5,size=size_value),'var_2_upper_limit':np.random.randint(5,10,size=size_value),'var_3_lower_limit':np.random.randint(0,5,size=size_value),'var_3_upper_limit':np.random.randint(5,10,size=size_value)})
df.sort_values(['var_1_lower_limit','var_1_upper_limit','var_2_lower_limit','var_2_upper_limit','var_3_lower_limit','var_3_upper_limit'],ascending=True, inplace=True) # I think important to sort here
display(df)


########## for loop
checkpoint_01 = time.perf_counter()
print('Beginning for loop run...')
default_possibilities = [0,1,2,3,4,5,6,7,8,9]
possibilities = list(itertools.product(default_possibilities,default_possibilities,default_possibilities,default_possibilities,default_possibilities,default_possibilities)) #Creates all combinations of possibilities
print('length of possibilities',len(possibilities))
possibilities = [item for item in possibilities if item[1]>=item[0] and item[3]>=item[2] and item[5]>=item[4]] # removes combinations that don't make sense such as first value is 5 and second value is 3
print('length of revised possibilities',len(possibilities))

filtering_output = []
checkpoint_02 = time.perf_counter()
for i in possibilities:
  a = i[0]
  b = i[1]
  c = i[2]
  d = i[3]
  e = i[4]
  f = i[5]
  for z in df.itertuples():
    if z.var_1_lower_limit <=a and z.var_1_upper_limit >=b and z.var_2_lower_limit <=c and z.var_2_upper_limit >=d and z.var_3_lower_limit <=e and z.var_3_upper_limit >=f:
      filtering_output.append({'combination':i,'var_1_lower_limit':z.var_1_lower_limit,'var_1_upper_limit':z.var_1_upper_limit, 'var_2_lower_limit':z.var_2_lower_limit,'var_2_upper_limit':z.var_2_upper_limit,'var_3_lower_limit':z.var_3_lower_limit,'var_3_upper_limit':z.var_3_upper_limit,'pass_or_fail':'YES'})
    else:
      continue

filtering_output_df = pd.DataFrame(filtering_output)
checkpoint_03 = time.perf_counter()
print('Iteration loop time was',(checkpoint_03-checkpoint_02)/60,'minutes')
unique_combinations = filtering_output_df.drop_duplicates(subset=['combination'], keep='first')
final_filters = unique_combinations.drop_duplicates(subset=['var_1_lower_limit','var_1_upper_limit','var_2_lower_limit','var_2_upper_limit','var_3_lower_limit','var_3_upper_limit'], keep='first') # since df was sorted, keeping only filters that had a passing combination
checkpoint_04 = time.perf_counter()
print('Done finding unique',(checkpoint_04-checkpoint_03)/60,'minutes')
display(final_filters)
checkpoint_05 = time.perf_counter()
print('Total time for loop',(checkpoint_05-checkpoint_01)/60,'minutes')

######### vectorization
checkpoint_06 = time.perf_counter()
print('Starting vectoriation...')
default_possibilities_new = np.array(list(itertools.product(range(10), repeat=6)))
print('Length of default possibilities',len(default_possibilities_new))

mask_new = (default_possibilities_new[:, 1] >= default_possibilities_new[:, 0]) & \
       (default_possibilities_new[:, 3] >= default_possibilities_new[:, 2]) & \
       (default_possibilities_new[:, 5] >= default_possibilities_new[:, 4])

possibilities_new = default_possibilities_new[mask_new]
print('length of revised possibilties',len(possibilities_new))

filtering_output_new = []
checkpoint_07 = time.perf_counter()
for i in possibilities_new:
    mask_new = (
            (df['var_1_lower_limit'] <= i[0]) & (df['var_1_upper_limit'] >= i[1]) &
            (df['var_2_lower_limit'] <= i[2]) & (df['var_2_upper_limit'] >= i[3]) &
            (df['var_3_lower_limit'] <= i[4]) & (df['var_3_upper_limit'] >= i[5])
    )

    if mask_new.any():
        row = df[mask_new].iloc[0]
        filtering_output_new.append({
            'combination': i,
            'var_1_lower_limit': row['var_1_lower_limit'],
            'var_1_upper_limit': row['var_1_upper_limit'],
            'var_2_lower_limit': row['var_2_lower_limit'],
            'var_2_upper_limit': row['var_2_upper_limit'],
            'var_3_lower_limit': row['var_3_lower_limit'],
            'var_3_upper_limit': row['var_3_upper_limit'],
            'pass_or_fail': 'YES'
        })

filtering_output_df_new = pd.DataFrame(filtering_output_new)
checkpoint_08 = time.perf_counter()
print('Done with vectorization',(checkpoint_08-checkpoint_06)/60,'minutes')
unique_combinations_new = filtering_output_df_new.drop_duplicates(subset=['combination'], keep='first')
final_filters_new = unique_combinations_new.drop_duplicates(subset=['var_1_lower_limit', 'var_1_upper_limit', 'var_2_lower_limit', 'var_2_upper_limit', 'var_3_lower_limit','var_3_upper_limit'], keep='first')
checkpoint_09 = time.perf_counter()
print('Done finding unique',(checkpoint_09-checkpoint_08)/60,'minutes')
display(final_filters_new)

checkpoint_10 = time.perf_counter()
print("Vectorization done in", (checkpoint_10 - checkpoint_06)/60,'minutes')

慢速原因分析 #

• 伪向量化，本质还是循环：你的向量化版本外层依然在遍历possibilities_new的每个元素，每次循环对整个df做布尔索引——这根本没用到numpy/pandas的向量化核心优势，向量化的关键是把整个数组作为整体运算，而非逐个元素循环处理。
• 数据类型拖慢去重：possibilities_new的元素是numpy数组，存入DataFrame后combination列会变成object类型；而for循环版本的combination是元组。object类型列去重时需要逐个比较元素，远慢于元组的哈希比较，这直接导致去重步骤耗时暴增。
• 循环内冗余操作：向量化版本每次循环都执行df[mask_new].iloc[0]，会生成新的DataFrame切片再取第一行，相比for循环用itertuples()直接访问属性，多了额外的对象创建和索引开销。

优化建议 #

• 真正的全向量化匹配：把possibilities_new作为二维数组，和df的上下限列做广播运算，一次性完成所有组合与过滤器的匹配，彻底避免逐元素循环。示例代码：

  # 提取df的上下限为numpy数组，形状(8,6)
  df_limits = df[['var_1_lower_limit','var_1_upper_limit','var_2_lower_limit','var_2_upper_limit','var_3_lower_limit','var_3_upper_limit']].values
  # 广播运算，一次性判断所有组合与过滤器的匹配关系
  mask = (df_limits[:,0::2] <= possibilities_new[:,0::2][:,None]) & (df_limits[:,1::2] >= possibilities_new[:,1::2][:,None])
  # 筛选出至少匹配一个过滤器的组合
  valid_mask = mask.all(axis=2).any(axis=1)
  # 获取每个有效组合匹配的第一个过滤器索引
  first_match_idx = mask.all(axis=2).argmax(axis=1)[valid_mask]
  # 构造结果DataFrame
  valid_combs = possibilities_new[valid_mask]
  result_df = pd.DataFrame(df_limits[first_match_idx], columns=df.columns[:6])
  result_df['combination'] = [tuple(comb) for comb in valid_combs]  # 转成元组方便后续去重
  result_df['pass_or_fail'] = 'YES'
  

• 优化去重步骤：提前把combination列转换成元组（可哈希类型），或者在构造结果时就避免重复组合，减少后续去重的压力。
• 利用排序特性提前过滤：df已经排序，可提前计算每个过滤器能覆盖的组合范围，减少需要处理的组合数量——比如最小的过滤器覆盖组合最多，先处理它，后续组合如果已被覆盖直接跳过。

内容的提问来源于stack exchange，提问作者Jim