40万次高频调用下字符串解析为NumPy数组的最优实现
高并发字符串解析性能优化实践
业务场景
业务流程需要执行约40万次字符串解析操作,核心目标是匹配最高执行效率的实现方案。目前已经尝试多套实现思路并完成初步性能对比测试,持续探索比现有版本更优的落地路径。
测试数据集
可通过如下代码生成标准化测试集,所有技术术语保持原样:
random.seed(10) np.random.seed(10) def test_str(): n = 10000000 arr = np.random.randint(10000, size=n) sign = np.random.choice(['+','-'], size=n) return 'ID1' + '\t' + ' '.join(["{}{}".format(a,b) for a,b in zip(arr, sign)])
生成的字符串格式形如ID1\t7688+ 737+ 677+ 1508- 9251-......
各版本实现说明
完整代码可在本地环境正常运行(此前在在线协作环境运行时曾触发TypingError,本地无兼容问题),核心实现分为通用工具函数与4套可对比的解析版本:
通用工具函数
实现逻辑参考Numba社区过往讨论,根据社区实测反馈,这类手写实现可能给Numba引入大量额外开销,部分场景下原生NumPy的运行效率反而优于这类实现:
@nb.jit(nopython=True) def str_to_int(s): final_index, result = len(s) - 1, 0 for i,v in enumerate(s): result += (ord(v) - 48) * (10 ** (final_index - i)) return result
方案1:独立拆分处理函数版
@nb.jit(nopython=True) def process_number(numb, identifier, i): sign = 1 if numb[-1] == '+' else -1 return str_to_int(numb[:-1]), sign, i, identifier @nb.jit(nopython=True) def expand1(data): identifier, l = data.split('\t') identifier = str_to_int(identifier[-1]) numbers = l.split() # 初始化空NumPy数组 arr = np.empty(shape = (len(numbers), 4), dtype = np.int64) # 填充数组 for i, numb in enumerate(numbers): arr[i,:] = process_number(numb, identifier, i) return arr
方案2:列表预组装赋值版
@nb.jit(nopython=True) def expand2(data): identifier, l = data.split('\t') identifier = str_to_int(identifier[-1]) numbers = l.split() size = len(numbers) numbs = [ str_to_int(numb[:-1]) for numb in numbers ] signs = [ 1 if numb[:-1] =='+' else -1 for numb in numbers ] arr = np.empty(shape = (size, 4), dtype = np.int64) arr[:,0] = numbs arr[:,1] = signs arr[:,2] = np.arange(0, size) arr[:,3] = np.repeat(identifier, size) return arr
方案3:循环内直接内联赋值版
@nb.jit(nopython=True) def expand3(data): identifier, l = data.split('\t') identifier = str_to_int(identifier[-1]) numbers = l.split() arr = np.empty(shape = (len(numbers), 4), dtype = np.int64) for i, numb in enumerate(numbers): arr[i,:] = str_to_int(numb[:-1]), 1 if numb[:-1] =='+' else -1, i, identifier return arr
参考对照实现
def expand4(t): identifier, l = t.split('\t') identifier = np.int(identifier[-1]) numbers = np.array([np.int(k[:-1]) for k in l.split(' ')]) signs = np.array([(k[-1] == '+') for k in l.split(' ')]) * 2 - 1 N = len(numbers) arr = np.empty(shape = (N, 4), dtype = np.int64) arr[:, 0] = numbers arr[:, 1] = signs arr[:, 2] = identifier arr[:, 3] = np.arange(N) return arr
性能测试结果
测试统一采用7轮重复执行、每轮5次循环的统计方式,记录单次循环的平均耗时与波动标准差:
Expand 1 72.7 ms ± 177 ms per loop (mean ± std. dev. of 7 runs, 5 loops each) Expand 2 27.9 ms ± 67.1 ms per loop (mean ± std. dev. of 7 runs, 5 loops each) Expand 3 8.81 ms ± 20.3 ms per loop (mean ± std. dev. of 7 runs, 5 loops each) Expand 4 ANSWER 1 429 µs ± 63.4 µs per loop (mean ± std. dev. of 7 runs, 5 loops each)
内容的提问来源于stack exchange,提问作者CodeNoob
相关产品推荐
相关产品推荐

