Numpy meshgrid函数间歇性性能骤降问题排查
关于Numpy meshgrid特定样本量下性能骤降的问题
反复调用Numpy的meshgrid函数时,发现该函数在特定样本量下会毫无征兆地出现性能骤降。测试代码如下:
from time import time import numpy as np np.random.seed(1729) maxn = 760 times = np.empty(maxn+1) times[0] = time() for n in range(1, maxn+1): x = np.random.uniform(size=(n,1)) y = np.random.uniform(size=(n,1)) x_grid, y_grid = np.meshgrid(x, y) times[n] = time() last = 40 [f'{i+maxn-last+1}: {x:.3f}' for i, x in enumerate(np.diff(times)[-last:])]
运行后得到结果:
['721: 0.003', '722: 0.003', '723: 0.003', '724: 0.004', '725: 0.010', '726: 0.007', '727: 0.007', '728: 0.005', '729: 0.005', '730: 0.006', '731: 37.243', # Suddenly slow '732: 0.009', '733: 0.007', '734: 0.007', '735: 0.009', '736: 0.007', '737: 0.009', '738: 0.009', '739: 0.009', '740: 11.602', # Suddenly slow '741: 0.008', '742: 0.010', '743: 0.012', '744: 0.012', '745: 0.016', '746: 0.009', '747: 0.015', '748: 0.015', '749: 61.460', # Suddenly slow '750: 0.008', '751: 0.007', '752: 0.007', '753: 0.007', '754: 0.007', '755: 0.007', '756: 0.007', '757: 0.007', '758: 1.167', # Suddenly slow '759: 0.007', '760: 0.010']
可以看到,多数调用耗时极短,但部分特定调用(如示例中从731开始每隔9次调用)耗时骤增。不确定在其他机器上需要设置多大的maxn才能复现该现象,希望了解可能的原因。
环境信息:Arch Linux系统,Python 3.10.10,Numpy 1.24.2。
补充:perf stat测试结果
使用perf stat运行该代码片段后,性能骤降出现的位置有所不同:
721: 0.003 722: 0.003 723: 0.003 724: 0.004 725: 0.008 726: 0.009 727: 0.008 728: 0.009 729: 0.014 730: 0.015 731: 0.013 732: 0.009 733: 0.009 734: 16.755 735: 0.012 736: 0.011 737: 0.011 738: 0.011 739: 0.013 740: 0.010 741: 0.010 742: 0.013 743: 0.011 744: 22.704 745: 0.016 746: 0.012 747: 0.017 748: 0.016 749: 0.016 750: 0.012 751: 0.011 752: 0.016 753: 4.244 754: 0.018 755: 0.014 756: 0.018 757: 0.015 758: 0.017 759: 0.021 760: 0.018 Performance counter stats for 'python test.py': 45,125.79 msec task-clock:u # 0.998 CPUs utilized 0 context-switches:u # 0.000 /sec 0 cpu-migrations:u # 0.000 /sec 568,187 page-faults:u # 12.591 K/sec 1,150,132,500 cycles:u # 0.025 GHz 1,124,532,475 instructions:u # 0.98 insn per cycle 188,859,225 branches:u # 4.185 M/sec 3,959,138 branch-misses:u # 2.10% of all branches 45.194501699 seconds time elapsed 0.393526000 seconds user 44.537625000 seconds sys
补充2:meshgrid替代方案
已找到适用于该场景的meshgrid替代方案:
x_grid, y_grid = np.broadcast_arrays(x[:, np.newaxis], y[np.newaxis, :]) x_grid = x_grid.reshape((x.shape[0], y.shape[0])).T y_grid = y_grid.reshape((x.shape[0], y.shape[0])).T
内容的提问来源于stack exchange,提问作者Mauricio
相关产品推荐
相关产品推荐

