更快删除雷达DataFrame中符合过滤条件的军用航班数据的方法

原有代码的耗时瓶颈来源于循环逻辑：你需要对60余个军用呼号逐个匹配，每次匹配都要全量遍历50万行的CALLSIGN列，同时每次过滤都会生成新的DataFrame产生重复内存拷贝，整体计算量是单轮匹配的60倍，处理大时间跨度数据时耗时会线性成倍增长。

将所有呼号合并为单个正则表达式，仅需一次全量遍历即可完成所有匹配，性能可以提升几十倍，原有30秒的单天处理任务优化后普遍可以降到2秒以内。
优化后代码如下：

import re
import time
import pandas as pd

tic = time.perf_counter()
before = len(df1)
print('Removing military flights...', end = ' ')
mil_callsigns = ['ALPHA', 'ANGEL', 'ANGRY', 'ANVIL', 'ARCHER', 'ARROW', 'AVENGER', 'AZTEC', 'BAF', 'BALLS', 'BASIC', 
                 'BEAST', 'BEAVER', 'BLACK', 'BLADE', 'BOLT', 'BONZO', 'BULDG', 'BULLET', 'BURST', 'COBRA', 'COWBOY', 
                 'CRACK', 'DAF', 'DAWG', 'DEVIL', 'EAGLE', 'EPIC', 'EVIL', 'FIST', 'FLAME', 'FLNKR', 'GHOST', 'GRZLY', 
                 'HAF', 'IGOR', 'IVAN', 'JEDI', 'KILLER', 'KILLR', 'KING', 'LION', 'MACE', 'METAL', 'MIG', 
                 'NAF', 'NOW', 'PAIN', 'POLLY', 'REBEL', 'REDSKIN', 'SAW', 'SHARK', 'SLAMMR', 'SLMMR', 'SNAKE', 'STING', 
                 'TIGER', 'VIPER', 'VOODO', 'WARP', 'WILDCAT']

# 合并所有呼号为正则规则，re.escape避免呼号含正则特殊字符时报错
mil_pattern = re.compile('|'.join(re.escape(s) for s in mil_callsigns))
# 单次过滤完成所有匹配，na=False 处理呼号为空的行避免报错
df1 = df1[~df1['CALLSIGN'].str.contains(mil_pattern, na=False)]

toc = time.perf_counter()
print('In', round(toc-tic, 1), 'seconds', before - len(df1), 'points from assumed military callsigns were removed')

• 如果你要匹配的是呼号完全等于列表中的值，而非包含关键词，使用isin方法性能会比正则匹配再高3-5倍：

df1 = df1[~df1['CALLSIGN'].isin(mil_callsigns)]

• 如果处理的是亿行级的全年汇总数据，可以替换Pandas为Polars库，基于Rust的向量化执行引擎比Pandas快5-10倍，内存占用也更低，参考代码如下：

import polars as pl
import re

# 可直接用polars读取原始数据，无需转Pandas
# df1 = pl.read_csv("你的数据文件路径")
mil_pattern = '|'.join(re.escape(s) for s in mil_callsigns)
df1 = df1.filter(~pl.col('CALLSIGN').str.contains(mil_pattern))

如果你的全年数据是按天拆分的独立文件，可以用多进程并行处理，速度和CPU核心数成正比，8核CPU可以再提速7倍左右：

import re
import pandas as pd
from concurrent.futures import ProcessPoolExecutor

mil_callsigns = ['ALPHA', 'ANGEL', 'ANGRY', 'ANVIL', 'ARCHER', 'ARROW', 'AVENGER', 'AZTEC', 'BAF', 'BALLS', 'BASIC', 
                 'BEAST', 'BEAVER', 'BLACK', 'BLADE', 'BOLT', 'BONZO', 'BULDG', 'BULLET', 'BURST', 'COBRA', 'COWBOY', 
                 'CRACK', 'DAF', 'DAWG', 'DEVIL', 'EAGLE', 'EPIC', 'EVIL', 'FIST', 'FLAME', 'FLNKR', 'GHOST', 'GRZLY', 
                 'HAF', 'IGOR', 'IVAN', 'JEDI', 'KILLER', 'KILLR', 'KING', 'LION', 'MACE', 'METAL', 'MIG', 
                 'NAF', 'NOW', 'PAIN', 'POLLY', 'REBEL', 'REDSKIN', 'SAW', 'SHARK', 'SLAMMR', 'SLMMR', 'SNAKE', 'STING', 
                 'TIGER', 'VIPER', 'VOODO', 'WARP', 'WILDCAT']
mil_pattern = re.compile('|'.join(re.escape(s) for s in mil_callsigns))

def process_single_day(file_path):
    df = pd.read_csv(file_path)
    df = df[~df['CALLSIGN'].str.contains(mil_pattern, na=False)]
    df.to_csv(f"processed_{file_path}", index=False)
    return len(df)

if __name__ == "__main__":
    # 替换为你全年所有数据文件的路径列表
    all_day_files = [f"day_{i}.csv" for i in range(1, 366)]
    with ProcessPoolExecutor() as executor:
        process_results = list(executor.map(process_single_day, all_day_files))

内容的提问来源于stack exchange，提问作者Job Brüggen