优化Pandas DataFrame转特征与标签数组的方法求助

Hey there! Let's fix that slow nested loop issue you're dealing with. Python-level loops over DataFrames are almost always going to be slow—Pandas is built for vectorized, batch operations that handle this kind of transformation way more efficiently. Here's how to rewrite your code to process those 2700 rows in milliseconds instead of seconds:

The Problem with Your Original Code #

Your nested loops are iterating over every weight and every region column one by one, which means you're doing thousands of individual Python-level operations. Pandas' core strength is handling these bulk transformations under the hood with optimized C-based code, so we can leverage that to cut down runtime drastically.

The Efficient Pandas Solution #

We'll use Pandas' melt() function to reshape your wide DataFrame into a long format—this automatically creates the [weight, region] feature pairs and maps them to their corresponding shipping costs. Then we can extract the arrays you need in one go.

Step 1: Reshape the DataFrame with melt() #

# Convert your wide DataFrame to long format
melted_df = df_ground.melt(
    id_vars='Weight',  # Keep the Weight column as the identifier
    var_name='Region',  # Name for the new column holding region IDs
    value_name='ShippingCost'  # Name for the new column holding shipping costs
)

This takes all your region columns (the ones after Weight) and turns them into rows, pairing each weight with every region and its associated cost in a single, clean table.

Step 2: Extract Features and Labels Arrays #

Now you can pull out your feature and label arrays directly from the reshaped DataFrame:

# Get the feature array in [[weight, region], ...] format
features = melted_df[['Weight', 'Region']].to_numpy()

# Get the label array in [[shipping_cost], ...] format
labels = melted_df[['ShippingCost']].to_numpy()

Example to Illustrate #

Suppose your original DataFrame looks like this:

After melt(), it becomes:

Which gives you exactly the feature and label structure you need, with zero manual looping.

Why This Is Faster #

melt() operates on the entire DataFrame in bulk, avoiding the overhead of Python-level loops. For your 2700-row dataset, this should run in a fraction of a second—no more waiting 1-2 seconds for the transformation to complete.

内容的提问来源于stack exchange，提问作者NiksP