Pandas 1.5.3升级至2.2.2后pd.to_datetime()报错的核心变更点

问题场景 #

当使用pd.to_datetime转换带微秒和时区后缀的时间戳字符串时，即使指定的format参数只覆盖到秒级，在Pandas 1.5.3中仍能成功转换；但升级到Pandas 2.2.2后，会直接抛出ValueError提示存在未转换的剩余字符。

测试代码 #

import pandas as pd

data = {
    'timestamp': [
        '2024-05-02 10:00:00.000000+0000',
        '2024-05-02 10:00:01.000000+0000',
        '2024-05-02 10:00:02.000000+0000',
        '2024-05-02 10:00:03.000000+0000',
        '2024-05-02 10:00:04.000000+0000'
    ],
    'value': [False, False, False, False, False]
}

df = pd.DataFrame(data)

STANDARD_DATETIME_FORMAT = '%Y-%m-%d %H:%M:%S'

pd.to_datetime(df['timestamp'], format=STANDARD_DATETIME_FORMAT)

Pandas 2.2.2中的报错信息 #

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[9], line 1
----> 1 pd.to_datetime(df['timestamp'], format=STANDARD_DATETIME_FORMAT)

File ~/tmp/test-pandas/.venv3_12/lib/python3.12/site-packages/pandas/core/tools/datetimes.py:1067, in to_datetime(arg, errors, dayfirst, yearfirst, utc, format, exact, unit, infer_datetime_format, origin, cache)
   1065         result = arg.map(cache_array)
   1066     else:
-> 1067         values = convert_listlike(arg._values, format)
   1068         result = arg._constructor(values, index=arg.index, name=arg.name)
   1069 elif isinstance(arg, (ABCDataFrame, abc.MutableMapping)):

File ~/tmp/test-pandas/.venv3_12/lib/python3.12/site-packages/pandas/core/tools/datetimes.py:433, in _convert_listlike_datetimes(arg, format, name, utc, unit, errors, dayfirst, yearfirst, exact)
    431 # `format` could be inferred, or user didn't ask for mixed-format parsing.
    432 if format is not None and format != "mixed":
-> 433     return _array_strptime_with_fallback(arg, name, utc, format, exact, errors)
    435 result, tz_parsed = objects_to_datetime64(
    436     arg,
    437     dayfirst=dayfirst,
   (...)
    441     allow_object=True,
    442 )
    444 if tz_parsed is not None:
    445     # We can take a shortcut since the datetime64 numpy array
    446     # is in UTC

File ~/tmp/test-pandas/.venv3_12/lib/python3.12/site-packages/pandas/core/tools/datetimes.py:467, in _array_strptime_with_fallback(arg, name, utc, fmt, exact, errors)
    456 def _array_strptime_with_fallback(
    457     arg,
    458     name,
   (...)
    462     errors: str,
    463 ) -> Index:
    464     """
    465     Call array_strptime, with fallback behavior depending on 'errors'.
    466     """
-> 467     result, tz_out = array_strptime(arg, fmt, exact=exact, errors=errors, utc=utc)
    468     if tz_out is not None:
    469         unit = np.datetime_data(result.dtype)[0]

File strptime.pyx:501, in pandas._libs.tslibs.strptime.array_strptime()

File strptime.pyx:451, in pandas._libs.tslibs.strptime.array_strptime()

File strptime.pyx:587, in pandas._libs.tslibs.strptime._parse_with_format()

ValueError: unconverted data remains when parsing with format "%Y-%m-%d %H:%M:%S": ".000000+0000", at position 0. You might want to try:
    - passing `format` if your strings have a consistent format;
    - passing `format='ISO8601'` if your strings are all ISO8601 but not necessarily in exactly the same format;
    - passing `format='mixed'`, and the format will be inferred for each element individually. You might want to use `dayfirst` alongside this.

版本变更细节 #

尽管exact=True（默认参数）在两个版本中未发生变化，但Pandas 2.x版本严格执行了该参数的语义：

• Pandas 1.5.3中，底层解析逻辑会在匹配到format定义的部分后停止，自动忽略字符串末尾的多余字符（如微秒和时区后缀）
• Pandas 2.x版本中，基于Cython重构的array_strptime实现会严格检查整个字符串是否被完全解析，只要存在未被format覆盖的剩余字符，就会抛出ValueError，确保格式匹配的严谨性。

解决方案 #

针对该场景，可采用以下几种方式解决：

1. 修正format参数，匹配完整格式：

   STANDARD_DATETIME_FORMAT = '%Y-%m-%d %H:%M:%S.%f%z'
   pd.to_datetime(df['timestamp'], format=STANDARD_DATETIME_FORMAT)
   

2. 使用ISO8601格式解析：由于输入是标准ISO时间戳，可直接指定format='ISO8601'：

   pd.to_datetime(df['timestamp'], format='ISO8601')
   

3. 宽松匹配（不推荐）：若需临时兼容旧行为，可显式设置exact=False，允许解析部分匹配的字符串：

   pd.to_datetime(df['timestamp'], format=STANDARD_DATETIME_FORMAT, exact=False)
   

内容的提问来源于stack exchange，提问作者Gabriel