Python新手咨询：能否用NLTK构建自定义词典清洗企业名称？

Absolutely, you can use NLTK for this merchant name standardization task—while it’s not the only tool in the box, it’s totally capable of handling your needs, especially when paired with a custom dictionary tailored to your messy merchant data. Let’s walk through how to make this work for your 1M+ rows of transactions:

1. Start with a Custom Mapping Dictionary #

First, you’ll want to build a dictionary that maps all messy variants to their standardized names. This is the foundation—since you already have examples like American Eagle#12455112 → American Eagle, you can structure it like this:

merchant_mapping = {
    "american eagle": "American Eagle",
    "american eag": "American Eagle",
    "american eagle#12455112": "American Eagle",
    "//##7555banana rep": "Banana Republic",
    "new york h&m": "H&M",
    "h&m chigago": "H&M"
    # Add as many variants as you can identify
}

Pro tip: Normalize all text to lowercase first when matching—this avoids missing hits due to capitalization differences.

2. Use NLTK for Preprocessing & Fuzzy Matching #

NLTK shines here for cleaning up messy strings and handling partial matches or typos:

Preprocessing Cleanup #

First, strip out garbage characters and tokenize the names to make matching easier. NLTK’s word_tokenize helps split strings into meaningful chunks, and you can pair it with regex to remove unwanted symbols:

import re
from nltk.tokenize import word_tokenize

def clean_merchant_name(raw_name):
    # Remove special characters (keep letters, numbers, &)
    cleaned = re.sub(r'[^a-zA-Z0-9&]', ' ', raw_name)
    # Convert to lowercase and tokenize
    tokens = word_tokenize(cleaned.lower())
    return ' '.join(tokens)

Fuzzy Matching for Uncovered Variants #

For variants you haven’t explicitly mapped (like typos or partial names), NLTK’s edit_distance can calculate how similar two strings are, helping you find the closest standard name:

from nltk.metrics.distance import edit_distance

# List of your official standardized merchant names
standard_names = ["American Eagle", "Banana Republic", "H&M"]

def find_best_match(cleaned_name):
    min_distance = float('inf')
    best_match = None
    for standard in standard_names:
        # Calculate similarity between cleaned name and lowercase standard
        distance = edit_distance(cleaned_name, standard.lower())
        # Set a threshold (adjust based on your data—here, <3 means close enough)
        if distance < min_distance and distance <= 3:
            min_distance = distance
            best_match = standard
    return best_match

3. Batch Process Your 1M+ Rows #

With 1 million+ rows, efficiency matters. Use pandas to apply your cleaning and matching logic at scale:

import pandas as pd

# Load your transaction data (adjust file path as needed)
df = pd.read_csv('transaction_data.csv')

# Clean all merchant names first
df['cleaned_name'] = df['merchant_name'].apply(clean_merchant_name)

# First, do exact matches using your custom dictionary
df['standardized_name'] = df['cleaned_name'].map(merchant_mapping)

# For rows that didn't get an exact match, use fuzzy matching as a fallback
def get_standard_name(row):
    if pd.notna(row['standardized_name']):
        return row['standardized_name']
    return find_best_match(row['cleaned_name'])

df['standardized_name'] = df.apply(get_standard_name, axis=1)

Quick Note: NLTK vs. Other Tools #

While NLTK works great here, you might also want to check out libraries like fuzzywuzzy (it’s built around Levenshtein distance and has more user-friendly fuzzy matching functions) if you find yourself needing more advanced matching. But NLTK is totally sufficient for your use case—especially since you’re already comfortable starting with it as a Python newbie.

内容的提问来源于stack exchange，提问作者user8785697