使用pyiceberg对接Glue Catalog时遭遇403 Forbidden错误求助

问题背景 #

我是Iceberg新手，正在进行POC测试。已在AWS Athena中创建Iceberg表，使用pyiceberg可成功连接Glue Catalog并获取命名空间/表信息，但调用load_table或创建表操作时返回403错误。已配置AWS连接信息环境变量，且通过boto3.client('s3')对元数据文件执行等效HEAD操作正常。

代码示例 #

catalog = catalog.load_catalog('glue', **{
        'type': 'glue'
    })

databases = catalog.list_namespaces()
for d in databases:
    print(d[0])
    tables = catalog.list_tables(d)

table = catalog.load_table('default.iceberg_test')

错误栈 #

FutureWarning: The S3RegionRedirector class has been deprecated for a new internal replacement. A future version of botocore may remove this class.
  warnings.warn(
Traceback (most recent call last):
  File "\/opt\/homebrew\/lib\/python3.10\/site-packages\/s3fs\/core.py", line 112, in _error_wrapper
    return await func(*args, **kwargs)
  File "\/opt\/homebrew\/lib\/python3.10\/site-packages\/aiobotocore\/client.py", line 358, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "\/Users\/dataenginerd\/python\/utilities\/pyiceberg_test.py", line 36, in <module>
    table = catalog.load_table('default.iceberg_test')
  File "\/opt\/homebrew\/lib\/python3.10\/site-packages\/pyiceberg\/catalog\/glue.py", line 278, in load_table
    return self._convert_glue_to_iceberg(load_table_response.get(PROP_GLUE_TABLE, {}))
  File "\/opt\/homebrew\/lib\/python3.10\/site-packages\/pyiceberg\/catalog\/glue.py", line 180, in _convert_glue_to_iceberg
    metadata = FromInputFile.table_metadata(file)
  File "\/opt\/homebrew\/lib\/python3.10\/site-packages\/pyiceberg\/serializers.py", line 56, in table_metadata
    with input_file.open() as input_stream:
  File "\/opt\/homebrew\/lib\/python3.10\/site-packages\/pyiceberg\/io\/fsspec.py", line 166, in open
    return self._fs.open(self.location, "rb")
  File "\/opt\/homebrew\/lib\/python3.10\/site-packages\/fsspec\/spec.py", line 1135, in open
    f = self._open(
  File "\/opt\/homebrew\/lib\/python3.10\/site-packages\/s3fs\/core.py", line 640, in _open
    return S3File(
  File "\/opt\/homebrew\/lib\/python3.10\/site-packages\/s3fs\/core.py", line 1989, in __init__
    super().__init__(
  File "\/opt\/homebrew\/lib\/python3.10\/site-packages\/fsspec\/spec.py", line 1491, in __init__
    self.size = self.details["size"]
  File "\/opt\/homebrew\/lib\/python3.10\/site-packages\/fsspec\/spec.py", line 1504, in details
    self._details = self.fs.info(self.path)
  File "\/opt\/homebrew\/lib\/python3.10\/site-packages\/fsspec\/asyn.py", line 114, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "\/opt\/homebrew\/lib\/python3.10\/site-packages\/fsspec\/asyn.py", line 99, in sync
    raise return_result
  File "\/opt\/homebrew\/lib\/python3.10\/site-packages\/fsspec\/asyn.py", line 54, in _runner
    result[0] = await coro
  File "\/opt\/homebrew\/lib\/python3.10\/site-packages\/s3fs\/core.py", line 1210, in _info
    out = await self._call_s3(
  File "\/opt\/homebrew\/lib\/python3.10\/site-packages\/s3fs\/core.py", line 339, in _call_s3
    return await _error_wrapper(
  File "\/opt\/homebrew\/lib\/python3.10\/site-packages\/s3fs\/core.py", line 139, in _error_wrapper
    raise err
PermissionError: Forbidden

排查与修复思路 #

1. 显式配置S3客户端参数 #

pyiceberg底层依赖s3fs访问S3，可能未自动继承boto3的配置。加载catalog时显式指定S3相关参数：

import os
catalog = catalog.load_catalog('glue', **{
    'type': 'glue',
    's3.access-key-id': os.getenv('AWS_ACCESS_KEY_ID'),
    's3.secret-access-key': os.getenv('AWS_SECRET_ACCESS_KEY'),
    's3.region': os.getenv('AWS_REGION'),
    's3.session-token': os.getenv('AWS_SESSION_TOKEN') # 如果用临时凭证
})

2. 验证S3路径与权限范围 #

• 确认Iceberg表元数据的S3路径与boto3测试的路径完全一致，检查是否存在路径拼写、区域不匹配问题
• 确保IAM权限包含s3:GetObject（HEADObject允许不代表GetObject权限生效）

3. 修复依赖版本兼容性 #

错误栈中的弃用警告可能是s3fs/botocore版本与pyiceberg不兼容导致的，尝试固定版本：

pip install s3fs==2023.6.0 botocore==1.31.16

（版本号可根据pyiceberg官方兼容文档调整）

4. 指定Glue Catalog区域 #

如果Glue Catalog与S3桶区域不一致，可能引发权限拦截，加载catalog时显式指定Glue区域：

catalog = catalog.load_catalog('glue', **{
    'type': 'glue',
    'glue.region': 'us-east-1' # 替换为你的Glue实际区域
})

5. 检查桶策略的用户代理限制 #

部分桶策略会限制特定用户代理的访问，s3fs的用户代理与boto3不同，可在桶策略中添加s3fs的用户代理规则，或者查看日志确认请求的用户代理信息。

6. 启用s3fs调试日志 #

添加日志配置，查看s3fs实际发送的请求细节，对比boto3请求的差异：

import logging
logging.basicConfig(level=logging.DEBUG)

内容的提问来源于stack exchange，提问作者DataEnginerd