如何修改MRJob的mapper代码使其输出年份与单个词汇的配对结果

你只需要把原代码中返回整个清洗后词汇列表的逻辑，改为遍历列表逐个输出词汇即可，修改后输出的每一条记录都是年份与单个词汇的配对格式，可直接用于后续reducer统计每年词频。

修改后的完整代码 #

def mapper(self, _, line):
    stop_words = set(["to", "a", "an", "the", "for", "in", "on", "of", "at", "over", "with", "after", "and", "from", "new", "us", "by", "as", "man", "up", "says", "in", "out", "is", "be", "are", "not", "pm", "am", "off", "more", "less", "no", "how"])
    (date,words) = line.strip().split(",")

    word_list = words.split()
    clean_words = [word for word in word_list if word not in stop_words]
    clean_words.sort()

    # 遍历每个清洗后的词汇，逐个输出年份+词汇的配对
    year = date[0:4]
    for word in clean_words:
        yield (year, word)

改动说明 #

• 原代码最后直接返回(年份, 清洗后词汇列表)，所以输出是年份对应整个列表的格式
• 修改后先提取年份变量，再循环遍历每一个清洗完成的词汇，逐个与年份配对返回，每一次yield对应一组"年份" "单个词汇"的记录，完全匹配你的需求，后续reducer可直接按年份分组统计每个词汇的出现次数，再筛选Top3高频词即可。

内容的提问来源于stack exchange，提问作者CKZ