Naive Bayes训练中loglikelihood字典键数量不符问题排查

问题概述 #

实现朴素贝叶斯分类器时，loglikelihood字典的键数量与预期不符：预期9165个键，实际仅得到9161个。

相关代码 #

# Build the freqs dictionary for later uses
freqs = count_tweets({}, train_x, train_y)

# UNQ_C2 GRADED FUNCTION: train_naive_bayes

def train_naive_bayes(freqs, train_x, train_y):
    '''
    Input:
        freqs: dictionary from (word, label) to how often the word appears
        train_x: a list of tweets
        train_y: a list of labels correponding to the tweets (0,1)
    Output:
        logprior: the log prior. (equation 3 above)
        loglikelihood: the log likelihood of you Naive bayes equation. (equation 6 above)
    '''
    loglikelihood = {}
    logprior = 0

    ### START CODE HERE ###

    # calculate V, the number of unique words in the vocabulary
    vocab = set([pair[0] for pair in freqs.keys()])
    V = len(vocab)  

    # calculate N_pos, N_neg, V_pos, V_neg
    N_pos = N_neg =  0
    for pair in freqs.keys():
        # if the label is positive (greater than zero)
        if pair[1] > 0:

            # Increment the number of positive words by the count for this (word, label) pair
            N_pos += freqs[pair]

        # else, the label is negative
        else:

            # increment the number of negative words by the count for this (word,label) pair
            N_neg += freqs[pair]

    # Calculate D, the number of documents
    D = len(train_y)

    # Calculate D_pos, the number of positive documents
    D_pos = sum(train_y)

    # Calculate D_neg, the number of negative documents
    D_neg = D - D_pos

    # Calculate logprior
    logprior = np.log(D_pos) - np.log(D_neg)
    
    # For each word in the vocabulary...
    for word in vocab:
        # get the positive and negative frequency of the word
        freq_pos = lookup(freqs,word,1)
        freq_neg = lookup(freqs,word,0)

        # calculate the probability that each word is positive, and negative
        p_w_pos = (freq_pos + 1) / (N_pos + V)
        p_w_neg = (freq_neg + 1) / (N_neg + V)

        # calculate the log likelihood of the word
        loglikelihood[word] = np.log(p_w_pos / p_w_neg)

    ### END CODE HERE ###

    return logprior, loglikelihood


# UNQ_C3 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
logprior, loglikelihood = train_naive_bayes(freqs, train_x, train_y)
print(logprior)
print(len(loglikelihood))

错误输出 #

0.0
9161
Wrong number of keys in loglikelihood dictionary. 
    Expected: 9165.
    Got: 9161.

问题分析与解决方案 #

核心原因 #

当前代码依赖freqs字典生成词汇表（vocab），但freqs的键是(word, label)的组合，若训练集中存在部分单词未被count_tweets函数统计到（比如分词遗漏、大小写未统一、或count_tweets过滤了特定词汇），就会导致词汇表缺失部分单词，最终loglikelihood的键数量不足。

修复方案 #

1. 从训练集直接生成完整词汇表
   替换原词汇表生成逻辑，遍历所有训练推文，提取所有唯一单词，确保覆盖训练集中的全部词汇：

   # 替换原vocab生成代码段
   vocab = set()
   # 假设process_tweet是你用于分词/预处理推文的函数，需和count_tweets中使用的一致
   for tweet in train_x:
       processed_words = process_tweet(tweet)
       vocab.update(processed_words)
   V = len(vocab)
   

2. 验证count_tweets函数的正确性
   检查count_tweets函数是否正确统计了所有单词的正负频率，确保没有遗漏训练集中的词汇。同时确认lookup函数能正确处理不存在的(word, label)对，返回0值：

   def lookup(freqs, word, label):
       return freqs.get((word, label), 0)
   

3. 核对词汇表长度
   生成新的vocab后，打印len(vocab)确认是否为预期的9165，确保词汇表完整。

内容的提问来源于stack exchange，提问作者Electro01