求Pandas DataFrame将列中'?'替换为列均值的高效实现方法

Hey there! Your current code gets the job done, but we can make it more efficient and concise by cutting out intermediate DataFrames and leveraging Pandas' optimized built-in functions. Here are a couple of improved approaches:

Method 1: Convert to Numeric + Fill NaNs (Most Efficient) #

This is the cleanest and fastest way to handle the replacement. We first turn non-numeric values like ? into NaN using pd.to_numeric with errors='coerce', then fill those missing values with the column's mean:

import pandas as pd

df = pd.read_csv("file_name.data", sep="\s+", names=["A","B","Horsepower"])
# Convert Horsepower to numeric, turning '?' into NaN
df['Horsepower'] = pd.to_numeric(df['Horsepower'], errors='coerce')
# Fill NaNs with the mean of valid values
df['Horsepower'] = df['Horsepower'].fillna(df['Horsepower'].mean())

Why this beats your original code:

• No extra DataFrames (df1, df2) cluttering up memory—we modify the column directly (or create a new one if you prefer).
• Pandas' vectorized operations are far faster than manual filtering and apply calls, especially for large datasets.
• The logic is straightforward and easy to maintain.

Method 2: Calculate Mean First, Then Replace #

If you want to stick with a replace-centric approach, you can compute the mean in one line without storing intermediate data:

import pandas as pd

df = pd.read_csv("file_name.data", sep="\s+", names=["A","B","Horsepower"])
# Compute mean of valid Horsepower values (excluding '?')
hp_mean = pd.to_numeric(df[df['Horsepower'] != '?']['Horsepower']).mean()
# Replace '?' with the mean, then convert to numeric type
df['Horsepower'] = pd.to_numeric(df['Horsepower'].replace('?', hp_mean))

Why this is better than your original code:

• We skip creating df1 and df2, which saves memory (critical for large datasets).
• The mean calculation is condensed into a single step, making the code more readable.

Quick Tip: #

After replacement, double-check that Horsepower is a numeric type (float/int). Both methods above ensure this, but Method 1 handles it upfront for cleaner code.

内容的提问来源于stack exchange，提问作者ICantHandleThis