如何在Pandas Series的滚动窗口中高效实现按位与运算？

Great question—your current apply() approach works, but it's going to be painfully slow for large windows (100-20000) and thousands of samples, since it's doing Python-level loops over every window element. Let's fix this with a vectorized approach that leverages the nature of bitwise operations.

Why Your Current Method Struggles #

apply() iterates over each window in Python, which is O(n * window) time complexity. For a window size of 20000 and 10,000 samples, that's 200 million Python operations—way too slow. We need to use vectorized operations (implemented in C under the hood) to speed this up.

The Vectorized Solution: Bitwise Independence #

Bitwise AND has a key property: each bit in the result depends only on the corresponding bits in the window values. That means we can split the 32-bit integers into individual bits, compute the rolling "all 1s" check for each bit, then recombine the bits back into integers.

Here's how to implement this efficiently:

Option 1: Using Pandas Rolling (Clean, Readable)

import pandas as pd
import numpy as np

# Sample data
df = pd.Series([0xFFFF, 0xFEAB, 0x0000, 0x1111, 0x5555, 0xAAAA], dtype='uint32')
window_size = 2

# Step 1: Extract each of the 32 bits from the Series
# (bit 0 = least significant bit, bit 31 = most significant)
bit_series = [(df >> i) & 1 for i in range(32)]
bits_df = pd.DataFrame(bit_series).T  # Each row = original number, each column = 1 bit

# Step 2: Rolling min on each bit column (equivalent to "all 1s" in the window)
# Min returns 1 only if all values in the window are 1; 0 otherwise
rolling_bits = bits_df.rolling(window_size).min()

# Step 3: Recombine bits into 32-bit integers
# Multiply each bit by its weight (2^i) and sum across columns
result = (rolling_bits * (1 << np.arange(32))).sum(axis=1)

# Clean up: drop NaNs from the start and cast back to uint32
result = result.dropna().astype('uint32')

Option 2: Using NumPy Sliding Windows (Even Faster for Large Data)

For extremely large windows or datasets, NumPy's sliding_window_view can be even more efficient than Pandas' rolling:

import pandas as pd
import numpy as np

df = pd.Series([0xFFFF, 0xFEAB, 0x0000, 0x1111, 0x5555, 0xAAAA], dtype='uint32')
window_size = 2

# Convert to a 2D array of bits (rows = samples, columns = bits 0-31)
bits_array = np.array([(df >> i) & 1 for i in range(32)]).T

# Create sliding windows over the rows (shape: (n - window + 1, window, 32))
windows = np.lib.stride_tricks.sliding_window_view(bits_array, window_shape=window_size, axis=0)

# Compute min across each window (1 only if all bits in the window are 1)
window_bit_mins = windows.min(axis=1)

# Recombine bits into integers
result_np = (window_bit_mins * (1 << np.arange(32))).sum(axis=1)

# Convert back to Pandas Series with correct index
result = pd.Series(result_np, index=df.index[window_size-1:], dtype='uint32')

Verify the Result #

Both methods will produce the same output as your original apply() code:

1    65195
2        0
3        0
4     4369
5        0
dtype: uint32

Performance Boost #

For a window size of 20000 and 10,000 samples:

• Your original apply() method could take minutes to run.
• The vectorized Pandas method will finish in seconds.
• The NumPy method will be even faster, often sub-second.

This works because we're replacing Python loops with optimized C-level operations, which is the key to handling large datasets in Pandas/NumPy.

内容的提问来源于stack exchange，提问作者Pitt