如何用Python实现PySpark中filter、map、reduce等效的列表筛选操作？

原始数据与需求 #

给定两个列表：

list1 = ["num", "categ"]
all_names = ["col_num1", "col_num2", "col_num3", "col_categ1", "col_categ2", "col_bol1", "col_bol2", "num_extra_1", "num_extra_2", "categ_extra_1", "categ_extra_2"]

需要生成满足以下两个条件的新列表：

• 元素不包含字符串"extra"
• 元素包含list1中的任一关键词

期望结果：

["col_num1", "col_num2", "col_num3", "col_categ1", "col_categ2"]

优化实现方法 #

方法1：一步列表推导 #

将两个筛选条件合并到单个列表推导中，利用any()函数判断是否包含list1中的关键词，一步完成筛选：

l = [name for name in all_names if "extra" not in name and any(kw in name for kw in list1)]
print(l)
# 输出: ['col_num1', 'col_num2', 'col_num3', 'col_categ1', 'col_categ2']

方法2：使用filter函数（贴近PySpark写法） #

如果习惯PySpark中filter的用法，可以用Python内置的filter()函数实现，逻辑和PySpark一致：

# 定义筛选函数
def filter_condition(name):
    return "extra" not in name and any(kw in name for kw in list1)

l = list(filter(filter_condition, all_names))
print(l)
# 输出: ['col_num1', 'col_num2', 'col_num3', 'col_categ1', 'col_categ2']

也可以用lambda简化为一行：

l = list(filter(lambda x: "extra" not in x and any(kw in x for kw in list1), all_names))

内容的提问来源于stack exchange，提问作者armin