Zalando Operator克隆PostgreSQL集群失败求助（AWS IAM+S3环境）

环境背景 #

AWS + S3 + IAM Roles环境，使用v1.10.0版本的Zalando Operator克隆PostgreSQL集群失败，原集群备份功能正常。

原集群备份验证 #

在原Pod中执行以下命令可正常查看备份列表：

root@test-postgresql-0:/home/postgres# envdir "/run/etc/wal-e.d/env" wal-g backup-list
name                          modified             wal_segment_backup_start
base_000000010000000000000002 2023-08-17T12:47:14Z 000000010000000000000002
root@test-postgresql-0:/home/postgres# 

克隆配置清单 #

使用以下配置尝试克隆集群：

spec:
  clone:
    uid: "d4355d73-f5d6-4786-a093-7239125a2a12"
    cluster: "test-postgresql"
    timestamp: "2023-08-17T12:47:00+00:00"
    s3_wal_path: "s3://el-postgres-backup-int/spilo/postgresql-test-postgresql/wal/14/"

新Pod手动权限验证 #

在新克隆Pod中手动执行命令，可正常查看备份列表：

root@test-clone-postgresql-0:/home/postgres# envdir "/run/etc/wal-e.d/env-clone-test-postgresql/" wal-g backup-list
name                          modified             wal_segment_backup_start
base_000000010000000000000002 2023-08-17T12:47:14Z 000000010000000000000002
root@test-clone-postgresql-0:/home/postgres# 

克隆失败错误日志 #

集群启动日志显示克隆失败，出现S3目录访问拒绝错误：

2023-08-17 12:59:10,460 INFO: No PostgreSQL configuration items changed, nothing to reload.
2023-08-17 12:59:10,461 INFO: Lock owner: None; I am test-clone-postgresql-0
2023-08-17 12:59:10,541 INFO: trying to bootstrap a new cluster
2023-08-17 12:59:10,541 INFO: Running custom bootstrap script: envdir "/run/etc/wal-e.d/env-clone-test-postgresql" python3 /scripts/clone_with_wale.py --recovery-target-time="2023-08-17T12:47:00+00:00"
2023-08-17 12:59:10,597 INFO: Trying s3://el-postgres-backup-int/spilo/postgresql-test-postgresql/wal/14/ for clone
ERROR: 2023/08/17 12:59:13.824667 failed to list s3 folder: 'spilo/postgresql-test-postgresql/wal/14/basebackups_005/': AccessDenied: Access Denied
    status code: 403, request id: 5TPDP0S5NSN16QFA, host id: 4u4XDWohJwKE+q5R7qMUACDr2tdg//CBGCiyaHW56I8tRj1jeIDrDWpf1RRJeBKN9Y02xmQgXxez7NiihScwOw==
2023-08-17 12:59:13,827 ERROR: Clone failed
Traceback (most recent call last):
  File "/scripts/clone_with_wale.py", line 185, in main
    run_clone_from_s3(options)
  File "/scripts/clone_with_wale.py", line 166, in run_clone_from_s3
    backup_name, update_envdir = find_backup(options.recovery_target_time, env)
  File "/scripts/clone_with_wale.py", line 150, in find_backup
    backup_list = list_backups(env)
  File "/scripts/clone_with_wale.py", line 84, in list_backups
    output = subprocess.check_output(backup_list_cmd, env=env)
  File "/usr/lib/python3.10/subprocess.py", line 420, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['wal-g', 'backup-list']' returned non-zero exit status 1.
2023-08-17 12:59:13,834 INFO: removing initialize key after failed attempt to bootstrap the cluster
Traceback (most recent call last):
  File "/usr/local/bin/patroni", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/patroni/__main__.py", line 144, in main
    return patroni_main()
  File "/usr/local/lib/python3.10/dist-packages/patroni/__main__.py", line 136, in patroni_main
    abstract_main(Patroni, schema)
  File "/usr/local/lib/python3.10/dist-packages/patroni/daemon.py", line 108, in abstract_main
    controller.run()
  File "/usr/local/lib/python3.10/dist-packages/patroni/__main__.py", line 106, in run
    super(Patroni, self).run()
  File "/usr/local/lib/python3.10/dist-packages/patroni/daemon.py", line 65, in run
    self._run_cycle()
  File "/usr/local/lib/python3.10/dist-packages/patroni/__main__.py", line 109, in _run_cycle
    logger.info(self.ha.run_cycle())
  File "/usr/local/lib/python3.10/dist-packages/patroni/ha.py", line 1771, in run_cycle
    info = self._run_cycle()
  File "/usr/local/lib/python3.10/dist-packages/patroni/ha.py", line 1593, in _run_cycle
    return self.post_bootstrap()
  File "/usr/local/lib/python3.10/dist-packages/patroni/ha.py", line 1484, in post_bootstrap
    self.cancel_initialization()
  File "/usr/local/lib/python3.10/dist-packages/patroni/ha.py", line 1477, in cancel_initialization
    raise PatroniFatalException('Failed to bootstrap cluster')
patroni.exceptions.PatroniFatalException: 'Failed to bootstrap cluster'
/etc/runit/runsvdir/default/patroni: finished with code=1 signal=0
/etc/runit/runsvdir/default/patroni: exceeded maximum number of restarts 5
stopping /etc/runit/runsvdir/default/patroni

可能的原因与排查方向 #

1. WAL-G路径解析差异 #

手动执行wal-g backup-list时，WAL-G默认从备份根路径读取备份（对应basebackups_005/目录），但克隆脚本可能未正确设置完整备份根路径：

• 手动执行时，env-clone-test-postgresql中的WALE_S3_PREFIX可能指向s3://el-postgres-backup-int/spilo/postgresql-test-postgresql/（包含basebackups和wal目录）
• 克隆脚本仅使用了s3_wal_path中的路径（wal/14/），导致尝试在wal/14/basebackups_005/下查找备份，而该路径实际不存在

2. IAM角色权限的时间窗口问题 #

虽然手动验证时权限正常，但Patroni/bootstrap脚本执行时，Pod的IAM角色可能尚未完全生效。AWS IAM角色绑定到Pod后需要一定时间完成权限同步，而克隆脚本在Pod启动初期就执行，此时权限可能还未就绪。

3. 克隆脚本的环境变量覆盖问题 #

检查clone_with_wale.py脚本是否在执行时修改了环境变量：

• 脚本可能错误覆盖WALE_S3_PREFIX或其他关键变量，导致WAL-G访问错误的S3路径
• 对比手动执行和脚本执行时的完整环境变量列表，确认差异：

  # 导出手动执行时的环境变量
  envdir "/run/etc/wal-e.d/env-clone-test-postgresql" env > manual_env.txt
  # 导出Patroni进程的环境变量
  ps aux | grep patroni | grep -v grep | awk '{print $2}' | xargs cat /proc/{}/environ | tr '\0' '\n' > script_env.txt
  # 对比两个文件
  diff manual_env.txt script_env.txt
  

4. S3桶的路径权限差异 #

检查S3桶中basebackups_005/目录的权限：

• 确认IAM角色拥有s3:ListBucket权限针对el-postgres-backup-int桶，以及s3:GetObject权限针对spilo/postgresql-test-postgresql/basebackups_005/*路径
• 手动执行时可能使用了不同的路径解析逻辑，而脚本执行时需要明确的目录权限

内容的提问来源于stack exchange，提问作者Oleksandr Khomenko