如何在Pandas Series上实现np.logical_and的短路求值优化？

Great question! Since your first condition filters out most rows, implementing short-circuit evaluation is a smart way to cut down on unnecessary computations. Here are two Pythonic approaches to achieve this with pandas:

1. Row-wise apply with Explicit Short-Circuiting #

This method uses apply to check each row individually, stopping early if the hour condition fails:

import operator

def filter_row(row):
    # Check hour condition first - exit early if it fails
    if row['time'].hour not in hours_set:
        return False
    # Only evaluate weekday condition if hour check passes
    return row['time'].weekday_name[:3] in days_set

mask = csv_df.apply(filter_row, axis=1)
csv_df = csv_df.loc[mask, :]

Pros: #

• Super readable: The logic mirrors plain Python short-circuiting (if not A: return False; return B)
• Guaranteed short-circuiting: No unnecessary weekday calculations for rows that fail the hour check

Cons: #

• Row-wise apply can be slower than vectorized operations for extremely large DataFrames. That said, if your hour condition filters out most rows, the total time saved might outweigh this overhead.

2. Vectorized Approach with Targeted Computation #

This method leverages pandas' vectorized operations but only computes the weekday condition for rows that pass the hour check:

import numpy as np
import operator

# First compute the full hour mask
hour_mask = csv_df['time'].map(operator.attrgetter('hour')).isin(hours_set)

# Initialize mask with all False values
mask = np.zeros(len(csv_df), dtype=bool)

# Only calculate the weekday mask for rows where hour_mask is True
weekday_mask = csv_df.loc[hour_mask, 'time'].map(lambda x: x.weekday_name[:3]).isin(days_set)

# Update the main mask with weekday results for relevant rows
mask[hour_mask] = weekday_mask

csv_df = csv_df.loc[mask, :]

Pros: #

• Retains vectorized speed: Avoids the overhead of row-wise apply for large datasets
• Minimizes unnecessary work: Only computes the weekday condition for rows that need it

Cons: #

• Slightly more verbose than the apply method, but still very maintainable

Why Your Original Code Doesn't Short-Circuit #

The np.logical_and function requires both input Series to be fully computed before it can perform the AND operation. This means even if 90% of rows fail the hour check, you're still wasting time computing the weekday condition for every single row. The methods above fix this by only evaluating the second condition when the first one passes.

内容的提问来源于stack exchange，提问作者Mr_and_Mrs_D