Pickle存储的Pandas DataFrame读取时递归深度超限，调优后内核崩溃

问题详情 #

我能成功调用to_pickle()方法存储从Twitter API获取并构建的Pandas DataFrame，但调用read_pickle()读取该文件时，触发递归深度超限错误，报错栈如下：

File ~/.cache/pypoetry/virtualenvs/twitter-gQnmvvjM-py3.11/lib/python3.11/site-packages/pandas/io/pickle.py:208, in read_pickle(filepath_or_buffer, compression, storage_options)
    205     with warnings.catch_warnings(record=True):
    206         # We want to silence any warnings about, e.g. moved modules.
    207         warnings.simplefilter("ignore", Warning)
--> 208         return pickle.load(handles.handle)
    209 except excs_to_catch:
    210     # e.g.
    211     #  "No module named 'pandas.core.sparse.series'"
    212     #  "Can't get attribute '__nat_unpickle' on <module 'pandas._libs.tslib"
    213     return pc.load(handles.handle, encoding=None)

File ~/.cache/pypoetry/virtualenvs/twitter-gQnmvvjM-py3.11/lib/python3.11/site-packages/tweepy/mixins.py:33, in DataMapping.__getattr__(self, name)
     31 def __getattr__(self, name):
     32     try:
--> 33         return self.data[name]
     34     except KeyError:
     35         raise AttributeError from None

File ~/.cache/pypoetry/virtualenvs/twitter-gQnmvvjM-py3.11/lib/python3.11/site-packages/tweepy/mixins.py:33, in DataMapping.__getattr__(self, name)
...
--> 33         return self.data[name]
     34     except KeyError:
     35         raise AttributeError from None

RecursionError: maximum recursion depth exceeded

尝试设置sys.setrecursionlimit(10**5)或更高值后，内核直接崩溃，报错：

Canceled future for execute_request message before replies were done
The Kernel crashed while executing code in the the current cell or a previous cell. Please review the code in the cell(s) to identify a possible cause of the failure. Click here for more info. View Jupyter log for further details.

Jupyter日志内容：

error 12:23:55.212: Raw kernel process exited code: undefined
error 12:23:55.214: Error in waiting for cell to complete Error: Canceled future for execute_request message before replies were done
    at t.KernelShellFutureHandler.dispose (/home/ryan/.vscode-server/extensions/ms-toolsai.jupyter-2023.3.1000892223/out/extension.node.js:2:32419)
    at /home/ryan/.vscode-server/extensions/ms-toolsai.jupyter-2023.3.1000892223/out/extension.node.js:2:51471
    at Map.forEach (<anonymous>)
    at y._clearKernelState (/home/ryan/.vscode-server/extensions/ms-toolsai.jupyter-2023.3.1000892223/out/extension.node.js:2:51456)
    at y.dispose (/home/ryan/.vscode-server/extensions/ms-toolsai.jupyter-2023.3.1000892223/out/extension.node.js:2:44938)
    at /home/ryan/.vscode-server/extensions/ms-toolsai.jupyter-2023.3.1000892223/out/extension.node.js:17:96826
    at ee (/home/ryan/.vscode-server/extensions/ms-toolsai.jupyter-2023.3.1000892223/out/extension.node.js:2:1589492)
    at jh.dispose (/home/ryan/.vscode-server/extensions/ms-toolsai.jupyter-2023.3.1000892223/out/extension.node.js:17:96802)
    at Lh.dispose (/home/ryan/.vscode-server/extensions/ms-toolsai.jupyter-2023.3.1000892223/out/extension.node.js:17:104079)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
warn 12:23:55.215: Cell completed with errors {
  message: 'Canceled future for execute_request message before replies were done'
}

解决方案 #

问题根源是DataFrame中包含Tweepy的DataMapping对象，这类对象反序列化（pickle读取）时会触发无限递归：__getattr__方法访问不存在的属性时会尝试从self.data获取，但如果data属性在反序列化中未正确初始化，就会反复调用__getattr__，最终导致递归超限。

具体解决方法：

1. 存储前转换为原生Python类型：调用to_pickle()前，把DataFrame中所有Tweepy对象转换成字典或其他原生可序列化类型。示例代码：

   # 假设df是目标DataFrame，tweet_data列包含Tweepy对象
   df['tweet_data'] = df['tweet_data'].apply(lambda x: x._json if hasattr(x, '_json') else dict(x))
   

2. 改用其他序列化格式：放弃pickle，改用to_csv()、to_json()或feather、parquet等格式存储，这些格式对原生Python类型兼容性更好，也避免了pickle的对象依赖问题。
3. 避免调高递归深度限制：随意调大递归限制会导致Python进程占用大量内存，最终引发内核崩溃，不要尝试这种方法。

内容的提问来源于stack exchange，提问作者Ryan