如何在pandas DataFrame上迭代计算Jaccard相似度函数？

现存问题的根因 #

• 类型不匹配导致的属性报错：将df.sentence提取为单列Series后直接调用iterrows()，Series没有这个方法，触发'Series' object has no attribute 'iterrows'；补充的预处理代码直接对列表类型的base调用.lower()，列表不存在该方法
• 方法调用错误：方案B中错误给iterrows()传入df参数（该方法本身无入参），加上示例数据初始化语法完全不符合pandas规范，触发too many values to unpack
• 逻辑错位：循环内没有使用遍历到的单行数据，反而直接引用整个DataFrame/不存在的df.sentence()方法计算；且函数在循环第一次执行时就触发return，既没有实现逐行500+行的遍历，也把基准文本和整个数据集混为一谈做比较

你提交的存在语法/逻辑错误的实现代码 #

方案A #

sentences = pd.Series(df.sentence)
sentences = sentences.str.replace('[^A-z ]','').str.replace(' +',' ').str.strip()
splitwords = [ nltk.word_tokenize( str(sentence) ) for sentence in sentences ]
print(splitwords)
sentence = df.sentence
def Jaccard_Similarity(base, sentence):
    for i, row in sentence.iterrows():
        a = set(word for word in base)
        b = set(word for word in df.sentence())
        c = a.intersection(b)
        return(float(len(c)) / (len(a) + len(b) - len(c)), a, b)
Jaccard_Similarity(base, sentence)

方案B #

df = df.apply(lambda row: nltk.word_tokenize(row['sentence']), axis=1)
print(df)

def Jaccard_Similarity(bas, df):
    for row in df.iterrows(df):
        a = set(word for word in base)
        b = set(word for word in df)
        c = a.intersection(b)
        return(float(len(c)) / (len(a) + len(b) - len(c)), a, b)
Jaccard_Similarity(base, df)

你补充的预处理代码 #

# 错误写法：base、df是列表/DataFrame，不能直接调用.lower()
base = set(base.lower().split()) 
df = set(df.lower().split())

def Jaccard_Similarity(base, df): 
    intersection = base.intersection(df)
    union = base.union(df)
    return float(len(intersection)) / len(union)

错误的示例数据初始化写法 #

# 语法不符合pandas规范，会直接触发解包报错
base = ['Tom', 'eats', 'apple']    
df = (["Tom eats an apple"],
          ["Tom eats a pineapple"],
          ["Eva eats an apple"],
          ["Eva eats a pineapple"],
         columns = 'sentence')  

可直接运行的修正实现 #

保留原始DataFrame结构，逐行完成文本清洗、分词、相似度计算，适配500+行的批量处理场景：

import re
import pandas as pd
import nltk

# 基准文本提前处理为小写集合，避免循环内重复计算
base = ['Tom', 'eats', 'apple']
base_set = set([word.lower() for word in base])

# 正确初始化示例DataFrame
df = pd.DataFrame(
    {"sentence": [
        "Tom eats an apple",
        "Tom eats a pineapple",
        "Eva eats an apple",
        "Eva eats a pineapple"
    ]}
)

def calc_jaccard(row_sentence_text, base_set):
    # 对当前单行文本做正则清洗、大小写归一
    cleaned = re.sub(r'[^A-z ]', '', str(row_sentence_text))
    cleaned = re.sub(r' +', ' ', cleaned).strip().lower()
    # 分词后转集合
    row_tokens = nltk.word_tokenize(cleaned)
    row_set = set(row_tokens)
    # 计算交集、并集得到Jaccard值
    inter = base_set.intersection(row_set)
    union = base_set.union(row_set)
    return len(inter) / len(union)

# 逐行应用计算逻辑，结果存入新列
df['jaccard_score'] = df.apply(
    lambda row: calc_jaccard(row['sentence'], base_set),
    axis=1
)

实现特点 #

• 全程不覆盖原始DataFrame结构，不会丢失行索引
• 所有计算仅取当前遍历到的单行文本，不会出现基准和整个数据集混算的问题
• 规避了Series调用iterrows、错误给iterrows传参的语法问题，不会触发解包报错
• 加入大小写归一逻辑，避免示例中Tom/Eva这类大小写差异导致的计算偏差

内容的提问来源于stack exchange，提问作者SAPNONEXPERT