np.select处理无时区datetime64[ns]数据时触发TypeError问题

问题描述 #

我有一个带时区的datetime类型Pandas Series，需要通过np.select实现以下逻辑：

• 小时数大于12时返回次日时间；
• 小时数小于11时返回当日时间；
• 其他情况返回np.nan。

带时区时代码正常运行，但移除时区后执行np.select抛出TypeError：

TypeError: Choicelists and default value do not have a common dtype: The DType <class 'numpy.dtype[datetime64]'> could not be promoted by <class 'numpy.dtype[float64]'>. This means that no common DType exists for the given inputs. For example they cannot be stored in a single array unless the dtype is `object`. The full list of DTypes is: (<class 'numpy.dtype[datetime64]'>, <class 'numpy.dtype[float64]'>)

完整代码 #

import pandas as pd 
import numpy as np
from datetime import timedelta
import datetime

datetime_series = pd.Series(['2022-09-24 22:00:00+02:00','2022-09-04 11:30:00+02:00', '2022-11-11 02:20:30+02:00',  '2022-11-12 03:20:30+02:00'])
# 转换为datetime类型
datetime_series = pd.to_datetime(datetime_series, errors='coerce')
# 移除时区
datetime_series_no_timezone = datetime_series.dt.tz_localize(None)

print ('datetime_series dtype: ', datetime_series.dtype)
print ('datetime_series_no_timezone dtype: ', datetime_series_no_timezone.dtype)

# 带时区时正常运行
conditions = [
        datetime_series.dt.hour > 12,
        datetime_series.dt.hour < 11]
choices = [
        (datetime_series + datetime.timedelta(days=1)),
        datetime_series ]

print (np.select(conditions, choices, default=np.nan))

# 移除时区后报错
conditions = [
        datetime_series_no_timezone.dt.hour > 12,
        datetime_series_no_timezone.dt.hour < 11]
choices = [
        (datetime_series_no_timezone + datetime.timedelta(days=1)),
        datetime_series_no_timezone ]
print (np.select(conditions, choices, default=np.nan))

执行输出 #

datetime_series dtype:  datetime64[ns, pytz.FixedOffset(120)]
datetime_series_no_timezone dtype:  datetime64[ns]
[Timestamp('2022-09-25 22:00:00+0200', tz='pytz.FixedOffset(120)') nan
 Timestamp('2022-11-11 02:20:30+0200', tz='pytz.FixedOffset(120)')
 Timestamp('2022-11-12 03:20:30+0200', tz='pytz.FixedOffset(120)')]
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-1-63a5d209c9ba> in <module>
     30         (datetime_series_no_timezone + datetime.timedelta(days=1)),
     31         datetime_series_no_timezone ]
---> 32 print (np.select(conditions, choices, default=np.nan))

<__array_function__ internals> in select(*args, **kwargs)

/usr/local/lib/python3.7/dist-packages/numpy/lib/function_base.py in select(condlist, choicelist, default)
    687     except TypeError as e:
    688         msg = f'Choicelists and default value do not have a common dtype: {e}'
---> 689         raise TypeError(msg) from None
    690 
    691     # Convert conditions to arrays and broadcast conditions and choices

原因分析 #

带时区的datetime序列（datetime64[ns, tz]）本质是object dtype的Timestamp对象数组，np.nan可以作为对象混入其中；而移除时区后的序列是原生datetime64[ns] dtype，np.nan是float64类型，两者无法自动转换为共同 dtype，因此np.select报错。

解决方法 #

方法1：用pd.NaT替代np.nan作为默认值 #

pd.NaT是Pandas专为时间序列设计的缺失值标记，与datetime64[ns] dtype完全兼容：

# 修改移除时区后的代码部分
print(np.select(conditions, choices, default=pd.NaT))

方法2：将结果转为object dtype #

如果需要保留np.nan，可以手动指定结果转为object类型：

result = np.select(conditions, choices, default=np.nan).astype(object)
print(result)

方法3：改用Pandas原生操作替代np.select #

用where/mask链式操作更符合Pandas的时间序列处理习惯，从根源避免 dtype 冲突：

result = (datetime_series_no_timezone + pd.Timedelta(days=1)).where(
    datetime_series_no_timezone.dt.hour > 12,
    datetime_series_no_timezone.where(
        datetime_series_no_timezone.dt.hour < 11, 
        pd.NaT
    )
)
print(result)

内容的提问来源于stack exchange，提问作者Leo