基于Python的产品描述语义标签分配及相似产品识别方案

Got it, let's tackle this problem of extracting core product themes from messy descriptions to group identical products. Here are practical Python-based solutions tailored to this task:

If your product categories are relatively fixed (e.g., shoes, electronics, apparel), a rule-based method with NLP tools can work well for filtering out irrelevant terms (like quantities, origins, materials) and grabbing core product nouns.

We can use spaCy for noun phrase extraction, then manually filter out non-core keywords:

import spacy

# Load spaCy's English model
nlp = spacy.load("en_core_web_sm")

def extract_core_theme(description):
    doc = nlp(description)
    # Extract noun chunks (potential product terms)
    noun_chunks = [chunk.text.lower() for chunk in doc.noun_chunks]
    # Filter out irrelevant terms (customize this list based on your data)
    irrelevant_terms = ["units", "import", "oversea", "plastic", "green"]
    core_terms = [term for term in noun_chunks if not any(ir in term for ir in irrelevant_terms)]
    # Combine core terms into a concise theme
    return " ".join(core_terms).strip()

# Test with your example
description = "500 units shoe green sport tennis import oversea plastic"
print(extract_core_theme(description))  # Output: "shoe sport tennis"

Pro tip: Expand the irrelevant_terms list based on common noise in your dataset (e.g., "pack", "bulk", "made in...").

For messy, unstructured descriptions where rule-based filtering falls short, pre-trained language models (like BERT) can automatically identify core product themes. Zero-shot classification is perfect here if you don't have pre-defined theme labels.

We'll use Hugging Face's transformers library:

from transformers import pipeline

# Load zero-shot classification pipeline
classifier = pipeline("zero-shot-classification", model="facebook/bart-large-mnli")

def classify_core_theme(description, candidate_themes):
    result = classifier(description, candidate_themes)
    # Return the highest-confidence theme
    return result["labels"][0]

# Test with your example
description = "500 units shoe green sport tennis import oversea plastic"
candidate_themes = ["sport tennis shoe", "casual green shoe", "plastic packaging"]
print(classify_core_theme(description, candidate_themes))  # Output: "sport tennis shoe"

Advantage: This works even if your descriptions have varied wording—you just need a list of potential product themes to match against.

Tools like RAKE (Rapid Automatic Keyword Extraction) or YAKE can pull out high-impact keywords directly from text, which you can combine into core themes.

Using rake-nltk:

from rake_nltk import Rake

def extract_keyword_theme(description):
    r = Rake()
    r.extract_keywords_from_text(description)
    # Get top 3 keywords (adjust based on your needs)
    top_keywords = r.get_ranked_phrases()[:3]
    # Combine into a theme
    return " ".join(top_keywords)

# Test with your example
description = "500 units shoe green sport tennis import oversea plastic"
print(extract_keyword_theme(description))  # Output: "sport tennis shoe"

Note: RAKE works best when you have longer descriptions, but it's great for quick, unsupervised keyword extraction.

If you don't know all possible product themes upfront, you can cluster similar descriptions together first, then extract themes for each cluster. We'll use scikit-learn for TF-IDF vectorization and K-Means clustering:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import KMeans
import numpy as np

def cluster_and_extract_themes(descriptions, num_clusters=5):
    # Convert text to TF-IDF vectors
    vectorizer = TfidfVectorizer(stop_words="english")
    X = vectorizer.fit_transform(descriptions)
    
    # Cluster with K-Means
    kmeans = KMeans(n_clusters=num_clusters, random_state=42)
    kmeans.fit(X)
    
    # Extract top terms for each cluster (core themes)
    order_centroids = kmeans.cluster_centers_.argsort()[:, ::-1]
    terms = vectorizer.get_feature_names_out()
    themes = []
    for i in range(num_clusters):
        top_terms = [terms[ind] for ind in order_centroids[i, :3]]
        themes.append(" ".join(top_terms))
    
    # Assign theme to each description
    assigned_themes = [themes[label] for label in kmeans.labels_]
    return assigned_themes

# Example usage with multiple descriptions
descriptions = [
    "500 units shoe green sport tennis import oversea plastic",
    "tennis sport shoe breathable mesh 200pcs",
    "plastic bottle 1L eco-friendly pack of 100",
    "eco plastic bottle 500ml imported"
]
print(cluster_and_extract_themes(descriptions, num_clusters=2))
# Output: ["shoe sport tennis", "shoe sport tennis", "plastic bottle eco", "plastic bottle eco"]

Pro tip: Adjust num_clusters based on how many unique products you expect in your dataset.

内容的提问来源于stack exchange，提问作者mersa