使用gcsfs和pandas时仅本地出现FileNotFoundError问题排查

此前一直正常使用pandas和gcsfs读取GCS文件，近期在本地机器调用时出现报错，Python 3.9.16和3.11.1版本均能复现该问题。

测试代码：

import gcsfs
import pandas as pd
x = pd.read_table("gs://MYBUCKET/clinvar/gene_specific_summary.txt")

报错堆栈：

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/seandavis/Documents/git/infra/jupyterlab/venv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1242, in read_table
    return _read(filepath_or_buffer, kwds)
  File "/Users/seandavis/Documents/git/infra/jupyterlab/venv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 577, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/Users/seandavis/Documents/git/infra/jupyterlab/venv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1407, in __init__
    self._engine = self._make_engine(f, self.engine)
  File "/Users/seandavis/Documents/git/infra/jupyterlab/venv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1661, in _make_engine
    self.handles = get_handle(
  File "/Users/seandavis/Documents/git/infra/jupyterlab/venv/lib/python3.9/site-packages/pandas/io/common.py", line 716, in get_handle
    ioargs = _get_filepath_or_buffer(
  File "/Users/seandavis/Documents/git/infra/jupyterlab/venv/lib/python3.9/site-packages/pandas/io/common.py", line 414, in _get_filepath_or_buffer
    file_obj = fsspec.open(
  File "/Users/seandavis/Documents/git/infra/jupyterlab/venv/lib/python3.9/site-packages/fsspec/core.py", line 134, in open
    return self.__enter__()
  File "/Users/seandavis/Documents/git/infra/jupyterlab/venv/lib/python3.9/site-packages/fsspec/core.py", line 102, in __enter__
    f = self.fs.open(self.path, mode=mode)
  File "/Users/seandavis/Documents/git/infra/jupyterlab/venv/lib/python3.9/site-packages/fsspec/spec.py", line 1241, in open
    f = self._open(
  File "/Users/seandavis/Documents/git/infra/jupyterlab/venv/lib/python3.9/site-packages/gcsfs/core.py", line 1343, in _open
    return GCSFile(
  File "/Users/seandavis/Documents/git/infra/jupyterlab/venv/lib/python3.9/site-packages/gcsfs/core.py", line 1502, in __init__
    super().__init__(
  File "/Users/seandavis/Documents/git/infra/jupyterlab/venv/lib/python3.9/site-packages/fsspec/spec.py", line 1597, in __init__
    self.size = self.details["size"]
  File "/Users/seandavis/Documents/git/infra/jupyterlab/venv/lib/python3.9/site-packages/gcsfs/core.py", line 1538, in details
    self._details = self.fs.info(self.path, generation=self.generation)
  File "/Users/seandavis/Documents/git/infra/jupyterlab/venv/lib/python3.9/site-packages/fsspec/asyn.py", line 121, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/Users/seandavis/Documents/git/infra/jupyterlab/venv/lib/python3.9/site-packages/fsspec/asyn.py", line 106, in sync
    raise return_result
  File "/Users/seandavis/Documents/git/infra/jupyterlab/venv/lib/python3.9/site-packages/fsspec/asyn.py", line 61, in _runner
    result[0] = await coro
  File "/Users/seandavis/Documents/git/infra/jupyterlab/venv/lib/python3.9/site-packages/gcsfs/core.py", line 809, in _info
    out = await self._ls(path, **kwargs)
  File "/Users/seandavis/Documents/git/infra/jupyterlab/venv/lib/python3.9/site-packages/gcsfs/core.py", line 846, in _ls
    for entry in await self._list_objects(
  File "/Users/seandavis/Documents/git/infra/jupyterlab/venv/lib/python3.9/site-packages/gcsfs/core.py", line 545, in _list_objects
    items, prefixes = await self._do_list_objects(
  File "/Users/seandavis/Documents/git/infra/jupyterlab/venv/lib/python3.9/site-packages/gcsfs/core.py", line 580, in _do_list_objects
    page = await self._call(
  File "/Users/seandavis/Documents/git/infra/jupyterlab/venv/lib/python3.9/site-packages/gcsfs/core.py", line 430, in _call
    status, headers, info, contents = await self._request(
  File "/Users/seandavis/Documents/git/infra/jupyterlab/venv/lib/python3.9/site-packages/decorator.py", line 221, in fun
    return await caller(func, *(extras + args), **kw)
  File "/Users/seandavis/Documents/git/infra/jupyterlab/venv/lib/python3.9/site-packages/gcsfs/retry.py", line 114, in retry_request
    return await func(*args, **kwargs)
  File "/Users/seandavis/Documents/git/infra/jupyterlab/venv/lib/python3.9/site-packages/gcsfs/core.py", line 423, in _request
    validate_response(status, contents, path, args)
  File "/Users/seandavis/Documents/git/infra/jupyterlab/venv/lib/python3.9/site-packages/gcsfs/retry.py", line 83, in validate_response
    raise FileNotFoundError(path)
FileNotFoundError: b/MYBUCKET/o

从报错信息FileNotFoundError: b/MYBUCKET/o来看，核心是gcsfs无法定位指定的GCS资源，可按以下步骤排查：

• 验证路径与权限

  1. 确认gs://MYBUCKET/clinvar/gene_specific_summary.txt路径完全正确，GCS路径区分大小写，需检查桶名、文件夹层级、文件名的拼写和大小写
  2. 检查本地GCS认证有效性：
     • 运行gcloud auth application-default print-access-token验证是否能获取有效令牌
     • 若使用服务账号密钥，确认GOOGLE_APPLICATION_CREDENTIALS环境变量指向正确的密钥文件，且密钥未过期、拥有该桶的读取权限

• 排查依赖版本冲突

  1. 近期依赖包更新可能导致兼容问题，运行pip list | grep -E "gcsfs|fsspec|pandas"查看当前版本
  2. 尝试回退到之前正常工作的版本，例如执行pip install gcsfs==<历史有效版本> fsspec==<对应版本>

• 测试基础连接能力

  1. 跳过pandas，直接用gcsfs测试资源可达性：

     import gcsfs
     fs = gcsfs.GCSFileSystem()
     # 检查桶是否存在
     print(fs.exists("MYBUCKET"))
     # 检查目标文件是否存在
     print(fs.exists("MYBUCKET/clinvar/gene_specific_summary.txt"))
     

  2. 若上述测试失败，说明问题出在gcsfs的连接或权限层面，与pandas调用逻辑无关

• 检查本地网络环境

  1. 确认本地能正常访问GCS服务，可尝试用浏览器打开https://storage.googleapis.com/MYBUCKET/clinvar/gene_specific_summary.txt（仅文件公开时有效）
  2. 排查是否有代理配置干扰gcsfs请求，可临时关闭代理后重试

内容的提问来源于stack exchange，提问作者seandavi