Spark/Scala相似词分组统计：处理词尾s/ly的合并计数需求

算力阿峰

2026-5-14

Got it, let's fix this up for you! The key here is to add a word normalization step before counting—we need to strip off those s or ly suffixes to group related words correctly.

First, we'll create a helper function to handle the suffix replacement. We check for ly first (since it's a longer suffix) to avoid edge cases where a word ending with ly might mistakenly get treated as just ending with s. Then we check for the s suffix.

Here's the updated code:

# Define a function to normalize words by stripping target suffixes
def normalize_word(word):
    # Prioritize longer suffix 'ly' first
    if word.endswith('ly'):
        return word[:-2]
    # Then handle words ending with 's'
    elif word.endswith('s'):
        return word[:-1]
    # Keep the original word if no matching suffix
    else:
        return word

# Assume your input data is already an RDD (adjust if your data source is different)
data = sc.parallelize(["hi", "yes", "love", "know", "hi", "knows", "loves", "lovely"])

# Apply normalization, then perform counting
result = data.map(lambda word: (normalize_word(word), 1)) \
             .reduceByKey(lambda a, b: a + b) \
             .collect()

print(result)

Let's walk through what this does:

The normalize_word function adjusts each input word:
- Turns lovely into love by removing the last two characters
- Turns knows into know and loves into love by removing the final s
- Leaves words like hi and yes unchanged
We map each normalized word to a tuple with a count of 1 (using the normalized word as the key instead of the original)
reduceByKey then sums up the counts for each normalized key