Great question! Using corpus-based semantic similarity models is absolutely a solid approach for your clustering task—way more effective than Levenshtein distance when you care about meaning rather than just character-level matches. Let’s break this down for you:

Absolutely yes. Here’s why:

• Levenshtein distance only measures character-level overlap, which fails for cases like "Hotel A" vs "Hotel B" (same semantic category but different characters) or typos that don’t change the core meaning (e.g., "Coffe Shop" vs "Coffee Shop").
• Semantic models capture the intent behind Vendor names and the standardized meaning of MCC codes, which aligns perfectly with your goal of grouping functionally similar transactions.

You have a few options depending on whether you want general-purpose or domain-specific tools:

• General-Purpose Pre-Trained Models: These work out of the box for most Vendor name scenarios:
  • Sentence-BERT: Ideal for generating meaningful embeddings for full Vendor names (e.g., "Hilton Downtown" vs "Marriott Midtown" will have close embeddings).
  • Word2Vec/GloVe: Trained on large corpora like Wikipedia, these can capture word-level similarities (e.g., "hotel" and "motel" will have high similarity scores).
• Finance/Domain-Specific Corpora:
  • Official MCC Code Tables: Every MCC maps to a standardized merchant category (e.g., MCC 7011 = "Hotel/Motel", MCC 5812 = "Eating Places"). You can use these category descriptions as a small, targeted corpus to fine-tune a model, or even just map MCCs to their category names first (this step alone will make clustering MCCs far more logical than using the raw code strings).
  • Public Financial Transaction Datasets: Datasets like credit card transaction logs can be used to fine-tune models to better understand merchant naming conventions in banking contexts.

1. Clean & Standardize MCC Data First:
   • Map each raw MCC value to its official category name using a public MCC code table. For example, convert "7011" to "Hotel/Motel"—this turns a cryptic code into semantically meaningful text.
2. Preprocess Vendor Names:
   • Clean the text: Remove special characters, standardize to lowercase, strip redundant suffixes like "LLC", "Inc.", or "Corp.".
   • For inconsistent spellings, you can run a quick fuzzy match (but only to fix obvious typos, since semantic models handle most of the heavy lifting).
3. Generate Semantic Embeddings:
   • Use Sentence-BERT to create fixed-length embeddings for each cleaned Vendor name and MCC category name.
4. Cluster Using Cosine Similarity:
   • Calculate cosine similarity between embeddings to measure semantic closeness.
   • Use clustering algorithms like DBSCAN (great for finding arbitrary-shaped clusters) or hierarchical clustering (if you want a tree-like structure of categories) to group similar entries.
5. Validate Clusters:
   • Use metrics like the silhouette score to assess clustering quality, and spot-check clusters against business logic (e.g., make sure all entries in a "Hotel" cluster are actually lodging merchants).

If you’re looking for a lighter-weight solution:

• Rule-Based Clustering: Extract key industry keywords from Vendor names (e.g., using TF-IDF to identify high-frequency terms like "restaurant", "gas station") and group entries by these keywords. Pair this with standardized MCC categories for better accuracy.
• Hybrid Feature Clustering: Combine MCC category embeddings with Vendor name embeddings into a single feature vector. This leverages the standardization of MCCs and the semantic richness of Vendor names for more robust clusters.

内容的提问来源于stack exchange，提问作者boozy