Python正则处理医疗推特数据集：提取推文适配多文件KMeans聚类

Let's fix your two issues step by step, starting with the tweet extraction since that's the root cause of your unexpected results.

1. Correctly Extract Tweet Content from Each Line #

Your original code reads the entire file as a single string and splits it, which combines all lines into one giant list—so you only end up with the very last field of the entire file, not individual tweets. The fix is to process each line in the file separately:

• Read each line one by one
• Strip whitespace/newlines to clean up the line
• Split the line by | and extract the third field (index 2, since split results are zero-indexed)
• Add safeguards for empty lines or malformed entries to avoid crashes

2. Map Tweets to Your Preset Labels #

The glob function doesn't guarantee file order matches your labels list. Instead, extract the channel name directly from each filename (e.g., bbchealth.txt → bbchealth) to ensure tweets are paired with the correct label.

Full Revised Code #

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import KMeans
from sklearn.metrics import adjusted_rand_score
import numpy as np
import glob
import os

# Preset channel labels
labels = ["bbchealth","cbchealth","cnnhealth","everydayhealth","foxnewshealth","gdnhealthcare","goodhealth","KaiserHealthNews","latimeshealth" ,"msnhealthnews","NBChealth","nprhealth","nytimeshealth","reuters_health","usnewshealth","wsjhealth"]
# Create a mapping from channel name to numerical index for evaluation
label_mapping = {label: idx for idx, label in enumerate(labels)}

# Get all text files in the target directory
file_list = glob.glob(os.path.join("E:/Health-News-Tweets/Health-Tweets", "*.txt"))
corpus = []
true_labels = []

for file_path in file_list:
    # Extract channel name from the filename (e.g., "bbchealth.txt" → "bbchealth")
    channel_name = os.path.basename(file_path).replace('.txt', '')
    
    # Skip files that don't match your preset labels (just in case)
    if channel_name not in label_mapping:
        print(f"Skipping unknown channel file: {file_path}")
        continue
    
    with open(file_path, 'r', encoding='utf-8') as f_input:
        for line in f_input:
            line = line.strip()
            # Skip empty lines
            if not line:
                continue
            # Split the line into fields
            line_parts = line.split('|')
            # Only process lines with at least 3 fields
            if len(line_parts) >= 3:
                tweet_content = line_parts[2]
                corpus.append(tweet_content)
                true_labels.append(channel_name)

# Convert string labels to numerical values for clustering evaluation
true_labels_num = [label_mapping[label] for label in true_labels]

# Vectorize the tweet text using TF-IDF
vectorizer = TfidfVectorizer(stop_words='english')
X = vectorizer.fit_transform(corpus)

# Run KMeans clustering
true_k = 16
model = KMeans(n_clusters=true_k, init='k-means++', max_iter=300, n_init=10, random_state=3425)
model.fit(X)

# Evaluate clustering performance (Adjusted Rand Index measures similarity to true labels)
ari_score = adjusted_rand_score(true_labels_num, model.labels_)
print(f"Adjusted Rand Index (clustering vs true channel labels): {ari_score:.4f}")

# Example prediction for a test tweet
test_tweet = "An abundance of online info can turn us into e-hypochondriacs. Or, worse, lead us to neglect getting the care we need"
Y = vectorizer.transform([test_tweet])
predicted_cluster_idx = model.predict(Y)[0]

print(f"\nPredicted cluster index for test tweet: {predicted_cluster_idx}")
print("Top 10 terms for this cluster:")
order_centroids = model.cluster_centers_.argsort()[:, ::-1]
# Use get_feature_names_out() for scikit-learn 1.0+ (replaces deprecated get_feature_names())
terms = vectorizer.get_feature_names_out()
for ind in order_centroids[predicted_cluster_idx, :10]:
    print(f' • {terms[ind]}')

Key Notes #

• Encoding Handling: If you hit encoding errors, try replacing encoding='utf-8' with encoding='latin-1' to handle non-standard characters in tweets.
• Cluster vs Label Mapping: KMeans clusters are unlabeled—they don't directly map to your channel names. The Adjusted Rand Index (ARI) score tells you how well the clustering matches your true channel groups (score ranges from 0 to 1, higher is better).
• Error Resilience: The code skips empty lines and malformed entries to avoid index errors or crashes.

内容的提问来源于stack exchange，提问作者Mariam Mohie Sleem