使用Polars df.write_delta()对接LocalStack S3存储时遇错误

使用Polars的df.write_delta()方法将Delta表写入AWS LocalStack时出现错误，但相同配置的boto3可正常列出存储桶和对象。

调用代码 #

df.write_delta(
        target=s3_path,
        mode="overwrite",
        overwrite_schema=True,
        storage_options={
            "AWS_REGION": "eu-central-1",
            "AWS_ACCESS_KEY_ID": "test",
            "AWS_SECRET_ACCESS_KEY": "test",
            "endpoint_url": "http://localhost:4566"
        }
    )

错误信息 #

File "/Users/x/Library/Caches/pypoetry/virtualenvs/batch-orZCP5o2-py3.11/lib/python3.11/site-packages/deltalake/table.py", line 415, in __init__
    self._table = RawDeltaTable(
                  ^^^^^^^^^^^^^^
OSError: Generic S3 error: Error after 0 retries in 5.416µs, max_retries:10, retry_timeout:180s, source:builder error for url (http://localhost:4566/my_bucket/my_key/_delta_log/_last_checkpoint)

问题根源在于storage_options中的参数名不匹配。Polars的write_delta()依赖deltalake库，该库要求S3端点的参数名为AWS_ENDPOINT_URL（而非boto3使用的endpoint_url）。修改存储选项参数即可解决问题：

df.write_delta(
        target=s3_path,
        mode="overwrite",
        overwrite_schema=True,
        storage_options={
            "AWS_REGION": "eu-central-1",
            "AWS_ACCESS_KEY_ID": "test",
            "AWS_SECRET_ACCESS_KEY": "test",
            "AWS_ENDPOINT_URL": "http://localhost:4566"  # 修改参数名
        }
    )

若修改后仍存在问题，可额外添加S3_ALLOW_UNSAFE_RENAME: "true"参数，确保LocalStack环境下的文件操作权限：

storage_options={
    "AWS_REGION": "eu-central-1",
    "AWS_ACCESS_KEY_ID": "test",
    "AWS_SECRET_ACCESS_KEY": "test",
    "AWS_ENDPOINT_URL": "http://localhost:4566",
    "S3_ALLOW_UNSAFE_RENAME": "true"
}

deltalake库的存储选项命名规则与boto3不完全一致，所有AWS相关配置参数均以AWS_前缀开头，例如：

• AWS_ACCESS_KEY_ID（对应boto3的aws_access_key_id）
• AWS_SECRET_ACCESS_KEY（对应boto3的aws_secret_access_key）
• AWS_ENDPOINT_URL（对应boto3的endpoint_url）

内容的提问来源于stack exchange，提问作者CodeMonkey74