如何在Pandas中高效将带时区时间戳转为datetime64[m]？

我通过以下代码创建了一个代表系统数据的DataFrame：

import pandas as pd

data = {
    "date": [
        "2021-03-12 19:50:00-05:00", "2021-03-12 19:51:00-05:00", "2021-03-12 19:52:00-05:00",
        "2021-03-12 19:53:00-05:00", "2021-03-12 19:54:00-05:00", "2021-03-12 19:55:00-05:00",
        "2021-03-12 19:56:00-05:00", "2021-03-12 19:57:00-05:00", "2021-03-12 19:58:00-05:00",
        "2021-03-12 19:59:00-05:00", "2021-03-15 04:00:00-04:00", "2021-03-15 04:01:00-04:00",
        "2021-03-15 04:02:00-04:00", "2021-03-15 04:03:00-04:00", "2021-03-15 04:04:00-04:00",
        "2021-03-15 04:05:00-04:00", "2021-03-15 04:06:00-04:00", "2021-03-15 04:07:00-04:00",
        "2021-03-15 04:08:00-04:00", "2021-03-15 04:09:00-04:00"
    ],
    "open": [81.15, 81.14, 81.15, 81.15, 81.15, 81.17, 81.19, 81.19, 81.20, 81.23, 81.05, 81.05, 81.05, 81.05, 81.05, 81.05, 81.05, 81.05, 81.05, 81.05],
    "high": [81.15, 81.14, 81.15, 81.15, 81.17, 81.17, 81.19, 81.19, 81.20, 81.23, 81.05, 81.05, 81.05, 81.05, 81.05, 81.05, 81.05, 81.05, 81.05, 81.05],
    "low": [81.14, 81.14, 81.14, 81.15, 81.15, 81.17, 81.19, 81.19, 81.20, 81.23, 81.05, 81.05, 81.05, 81.05, 81.05, 81.05, 81.05, 81.05, 81.05, 81.05],
    "close": [81.14, 81.14, 81.15, 81.15, 81.17, 81.17, 81.19, 81.19, 81.20, 81.23, 81.05, 81.05, 81.05, 81.05, 81.05, 81.05, 81.05, 81.05, 81.05, 81.05],
    "volume": [300.0, 100.0, 1684.0, 0.0, 1680.0, 150.0, 448.0, 0.0, 1500.0, 380.0, 162.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
}

df = pd.DataFrame(data)

print(df.info())

输出结果为：

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 20 entries, 0 to 19
Data columns (total 6 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   date    20 non-null     object 
 1   open    20 non-null     float64
 2   high    20 non-null     float64
 3   low     20 non-null     float64
 4   close   20 non-null     float64
 5   volume  20 non-null     float64
dtypes: float64(5), object(1)
memory usage: 1.1+ KB

date列的数据类型为object，存储的是带时区的时间戳。我需要移除时区信息，然后将date列转换为datetime64[m]（分钟精度），但使用以下转换代码后：

df['date'] = df['date'].apply(lambda ts: pd.Timestamp(ts).tz_localize(None).to_numpy().astype('datetime64[m]'))

print(df.info())

输出显示date列的数据类型为datetime64[ns]而非datetime64[m]：

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 20 entries, 0 to 19
Data columns (total 6 columns):
 #   Column  Non-Null Count  Dtype         
---  ------  --------------  -----         
 0   date    20 non-null     datetime64[ns]
 1   open    20 non-null     float64       
 2   high    20 non-null     float64       
 3   low     20 non-null     float64       
 4   close   20 non-null     float64       
 5   volume  20 non-null     float64       
dtypes: datetime64 , float64(5)
memory usage: 1.1 KB

请问如何以最内存高效的方式，正确将带时区信息的date列转换为datetime64[m]类型？

核心原因 #

Pandas的DatetimeArray默认使用datetime64[ns]存储，直接通过astype转换单个元素再赋值会被Pandas自动转回ns精度。要实现datetime64[m]类型，需要直接操作底层数组，避免逐元素处理的额外开销。

内存高效的实现步骤 #

• 一次性解析带时区的时间戳：使用pd.to_datetime批量解析，比apply逐元素处理更高效。
• 移除时区信息：通过tz_localize(None)剥离时区。
• 转换为分钟精度数组：直接将整个Series的底层数组转换为datetime64[m]，再重新赋值回DataFrame。

代码实现：

# 批量解析带时区的时间戳并移除时区
df['date'] = pd.to_datetime(df['date']).dt.tz_localize(None)
# 转换底层数组为datetime64[m]类型
df['date'] = df['date'].values.astype('datetime64[m]')

print(df.info())

执行后输出：

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 20 entries, 0 to 19
Data columns (total 6 columns):
 #   Column  Non-Null Count  Dtype         
---  ------  --------------  -----         
 0   date    20 non-null     datetime64[m]
 1   open    20 non-null     float64       
 2   high    20 non-null     float64       
 3   low     20 non-null     float64       
 4   close   20 non-null     float64       
 5   volume  20 non-null     float64       
dtypes: datetime64[m], float64(5)
memory usage: 1.1 KB

为什么这个方法更高效？ #

• 批量操作替代逐元素处理：pd.to_datetime是向量化操作，比apply循环快得多，内存占用更低。
• 直接操作底层数组：values.astype直接转换整个NumPy数组，避免Pandas自动转回ns精度的问题，同时减少中间对象的创建。

内容的提问来源于stack exchange，提问作者Allan Xu