基于标准化数据集的输入数据标准化及Pandas代码咨询

Hi there! Let’s break down your standardization code, fill in the gaps, and cover best practices to make your workflow more reliable and aligned with common data science practices.

First: Complete Your Scaling Pipeline #

Your code cuts off at df_scale...—here’s how to finish converting the scaled numpy array back into a Pandas DataFrame (keeping your original column names, which is critical for readability later):

import numpy as np
import pandas as pd
from sklearn.metrics.pairwise import cosine_similarity
from sklearn import preprocessing

# Load your dataset
df = pd.read_csv('DS/RS_DS/final_dataset.csv')

# Drop non-feature columns you don't need for scaling
rec_df = df.drop(['person_id','encounter_id','birthdate','CN','HN','DN','DIAG_DM','DIAG_NONDM','TPN'], axis=1)

# Normalize values to the 0-1 range
df_val = rec_df.values
min_max_scaler = preprocessing.MinMaxScaler()
df_val_scaled = min_max_scaler.fit_transform(df_val)

# Convert scaled array back to a DataFrame with original column names
df_scaled = pd.DataFrame(df_val_scaled, columns=rec_df.columns)

Critical Best Practices to Implement #

1. Handle Missing Values First #

MinMaxScaler (and most sklearn scalers) will throw errors if your data has NaN values. Always check and clean missing data before scaling:

# Check for missing values per column
print(rec_df.isnull().sum())

# Option 1: Drop rows with missing values (use if missing data is minimal)
rec_df_clean = rec_df.dropna()

# Option 2: Impute missing values (better for larger datasets)
from sklearn.impute import SimpleImputer
# Use median for skewed data, mean for normally distributed data
imputer = SimpleImputer(strategy='median')
rec_df_imputed = pd.DataFrame(imputer.fit_transform(rec_df), columns=rec_df.columns)

2. Avoid Data Leakage When Modeling #

If you plan to use this scaled data for machine learning, never fit the scaler on your entire dataset—only fit it on the training set, then transform both training and test sets. This prevents leaking test data information into your scaling process:

from sklearn.model_selection import train_test_split

# Split clean data into train/test sets
X_train, X_test = train_test_split(rec_df_clean, test_size=0.2, random_state=42)

# Fit scaler ONLY on training data
min_max_scaler = preprocessing.MinMaxScaler()
X_train_scaled = min_max_scaler.fit_transform(X_train)
X_test_scaled = min_max_scaler.transform(X_test)

# Convert back to DataFrames for easier manipulation
X_train_scaled_df = pd.DataFrame(X_train_scaled, columns=X_train.columns)
X_test_scaled_df = pd.DataFrame(X_test_scaled, columns=X_test.columns)

3. Choose the Right Scaler for Your Data #

MinMaxScaler is great for 0-1 normalization, but it’s sensitive to outliers. Depending on your data’s distribution, consider alternatives:

• StandardScaler: Use if your features are normally distributed (scales to mean=0, variance=1):

  std_scaler = preprocessing.StandardScaler()
  df_val_scaled_std = std_scaler.fit_transform(rec_df_clean)
  

• RobustScaler: Use if you have extreme outliers (scales based on median and interquartile range, ignoring outliers):

  robust_scaler = preprocessing.RobustScaler()
  df_val_scaled_robust = robust_scaler.fit_transform(rec_df_clean)
  

Integrating with Cosine Similarity #

Since you imported cosine_similarity, here’s how to use your scaled data to compute meaningful similarity scores (e.g., between individuals in your dataset):

# Compute cosine similarity matrix using scaled features
cos_sim_matrix = cosine_similarity(df_scaled)

# Convert to a readable DataFrame using person_id as labels
cos_sim_df = pd.DataFrame(cos_sim_matrix, index=df['person_id'], columns=df['person_id'])

# Example: Get top 5 most similar individuals to person_id 123
top_similar = cos_sim_df[123].sort_values(ascending=False).head(6)  # Includes the person themselves
print(top_similar)

Final Notes #

• Always save your fitted scaler if you need to apply the same scaling to future data (e.g., new patient records):

  import joblib
  joblib.dump(min_max_scaler, 'min_max_scaler.pkl')
  # Load later: scaler = joblib.load('min_max_scaler.pkl')
  

• Double-check that you’re only scaling numerical features—your dropped columns look like non-numerical IDs/categories, which is correct.

内容的提问来源于stack exchange，提问作者Reub