如何用Pandas关联两个DataFrame按文本规则过滤生成标志列
实现方案
推荐用Pandas向量化操作实现,比遍历效率高得多,代码如下:
import pandas as pd # 1. 先构造id到text的映射字典,关联查询更高效 id_text_map = meta_df.set_index('id')['text'].to_dict() # 2. 给df匹配id1和id2对应的text值 df['text1'] = df['id1'].map(id_text_map) df['text2'] = df['id2'].map(id_text_map) # 3. 按规则生成flag字段 def check_flag(row): has_lower1 = 'lower' in row['text1'] has_upper1 = 'upper' in row['text1'] has_lower2 = 'lower' in row['text2'] has_upper2 = 'upper' in row['text2'] # 满足恰好一个含lower,一个含upper if (has_lower1 and has_upper2) or (has_upper1 and has_lower2): return 'yes' return 'no' df['flag'] = df.apply(check_flag, axis=1) # 4. 提取需要的三列作为结果 result = df[['id1', 'id2', 'flag']]
如果要修正你原来的遍历写法,错误点在于之前的过滤条件写错了,修正后代码如下(仅适合小数据量场景,大数据量不推荐):
flags = [] for row in df.itertuples(): # 取id1对应的text text1 = meta_df[meta_df['id'] == row.id1].iloc[0]['text'] # 取id2对应的text text2 = meta_df[meta_df['id'] == row.id2].iloc[0]['text'] if ('lower' in text1 and 'upper' in text2) or ('upper' in text1 and 'lower' in text2): flags.append('yes') else: flags.append('no') df['flag'] = flags result = df[['id1', 'id2', 'flag']]
内容的提问来源于stack exchange,提问作者Hadoop Developer
相关产品推荐
相关产品推荐

