如何高效将ndarray中的毫秒偏移量转换为np.datetime64

Great question! Let's work through this—you're hitting two common NumPy pitfalls here: implicit type casting and inefficient loops. Here are two solid solutions to fix the overflow and speed up your code:

1. Read offsets directly as int64 (or convert post-read) #

First, you absolutely can read the int32 offset field as int64 to avoid overflow issues. If you know the structure of your binary files, you can define a structured dtype for numpy.fromfile to skip the np.void middleman entirely. Even if you already have a np.void array, converting the offset field to int64 is straightforward.

Example: Read with structured dtype #

# Define the structure of your binary records (adjust fields to match your actual data)
record_dtype = np.dtype([
    ('ms_offset', 'int32'),  # Your millisecond offset from day start
    ('reading', 'float32')   # Example of another data field
])

# Read the file into a structured array (no more np.void!)
data = np.fromfile('daily_data.bin', dtype=record_dtype)

# Convert the offset to int64 to prevent overflow during timestamp calculations
ms_offsets = data['ms_offset'].astype(np.int64)

Example: Convert from existing np.void array #

If you already loaded the data as np.void, extract the offset field and cast it:

# Inspect your void array's dtype to get the field name (run print(data.dtype) first)
ms_offsets = data['ms_offset'].astype(np.int64)

Once you have ms_offsets as int64, you can use them directly as timestamps by adding the Unix epoch timestamp of the day's start:

# Get the base timestamp (milliseconds since 1970-01-01 for your day's start)
base_day = np.datetime64('2024-05-20')  # Replace with your actual date from the file name
base_timestamp = (base_day - np.datetime64('1970-01-01')) // np.timedelta64(1, 'ms')

# Calculate full timestamps as int64
full_timestamps = base_timestamp + ms_offsets

2. Efficiently convert to np.datetime64 (no loops!) #

Looping through NumPy elements is slow and risky for type casting—instead, use NumPy's vectorized operations, which run in C-level code and avoid implicit int32 conversion.

Here's how to batch-convert offsets to datetime64 without loops:

# Using the same ms_offsets and base_day from above
timestamps = base_day + np.timedelta64(1, 'ms') * ms_offsets

This works because multiplying np.timedelta64(1, 'ms') by your int64 offsets creates a timedelta array, and adding that to a datetime64 array is handled entirely in optimized code. datetime64[ms] uses int64 under the hood, so there's no risk of overflow.

Why your loop caused overflow #

When you looped through elements, you were likely assigning results to an array that defaulted to int32 (inherited from the original int32 offset field). The full timestamp (when converted to milliseconds since epoch) is way larger than the int32 limit (~2.1e9 ms, or ~24 days), which triggered the overflow. Vectorized operations with int64 or datetime64 avoid this entirely.

内容的提问来源于stack exchange，提问作者Fabian