如何修改Python脚本解决Lambda中pg_dump找不到的问题？

问题背景 #

Python脚本用于生成EC2上Postgres数据库的pg_dump备份，Lambda已配置VPC可连接EC2，但运行时出现文件找不到错误。

原脚本代码 #

def dump():
    ssm = boto3.client('ssm', region_name='us-west-1')
    pg_host = get_parameters(ssm, "pg_host")
    pg_password = get_parameters(ssm, "pg_password")
    pg_database_name = get_parameters(ssm, "pg_database_name")
    pg_user = get_parameters(ssm, "pg_user")
    con = None
    try:

        # connect to the PostgreSQL server
        print('Connecting to the PostgreSQL database...')
        con = psycopg2.connect(
            host=pg_host,
            database=pg_database_name,
            user=pg_user,
            password=pg_password,
            port="5432")
            
        subprocess.call(['pg_dump', '-h', pg_host, '-U', pg_user, '-W', pg_password, '-p', '5432','-F', 't', '-f', '/tmp/pg_dump.sql', '-d', pg_database_name])
  
    finally:
        if con is not None:
            con.close()
            print('Done')

报错信息 #

Response
{
"errorMessage": "[Errno 2] No such file or directory: 'pg_dump'",
"errorType": "FileNotFoundError",
"stackTrace": [
" File "/var/task/lambda_function.py", line 50, in upload\n dump()\n",
" File "/var/task/lambda_function.py", line 42, in dump\n subprocess.call(['pg_dump', '-h', pg_host, '-U', pg_user, '-W', pg_password, '-p', '5432','-F', 't', '-f', '/tmp/pg_dump.sql', '-d', pg_database_name])\n",
]
}

错误原因 #

Lambda默认运行时环境未预装PostgreSQL客户端工具（包括pg_dump），因此subprocess调用外部命令时找不到pg_dump可执行文件。此外，原脚本用-W参数传递密码的方式不安全，且pg_dump不会读取命令行中的密码值。

解决方案 #

方案1：纯Python实现备份（无需依赖系统pg_dump） #

通过psycopg2调用PostgreSQL内置的pg_dump函数生成备份，无需额外配置Lambda层：

def dump():
    import boto3
    import psycopg2
    from psycopg2 import sql

    ssm = boto3.client('ssm', region_name='us-west-1')
    pg_host = get_parameters(ssm, "pg_host")
    pg_password = get_parameters(ssm, "pg_password")
    pg_database_name = get_parameters(ssm, "pg_database_name")
    pg_user = get_parameters(ssm, "pg_user")
    con = None
    try:
        print('Connecting to the PostgreSQL database...')
        con = psycopg2.connect(
            host=pg_host,
            database=pg_database_name,
            user=pg_user,
            password=pg_password,
            port="5432")
        
        # 生成tar格式备份（对应原参数-F t）
        output_path = '/tmp/pg_dump.tar'
        with open(output_path, 'wb') as backup_file:
            cur = con.cursor()
            # 调用PostgreSQL内置pg_dump函数
            cur.execute(
                sql.SQL("SELECT pg_dump(%s, format := 'tar')"),
                [pg_database_name]
            )
            backup_content = cur.fetchone()[0]
            backup_file.write(backup_content)
        
        print(f"备份成功，文件路径：{output_path}")
  
    finally:
        if con is not None:
            con.close()
            print('Connection closed')

注意：该方案要求PostgreSQL版本为12及以上，若使用更低版本，需改用逐表导出或pg_dumpall的替代逻辑。

方案2：将pg_dump打包到Lambda层 #

若必须使用pg_dump命令，需将PostgreSQL客户端工具打包为Lambda层：

1. 准备pg_dump二进制文件：在Amazon Linux 2环境中编译或下载PostgreSQL客户端，获取pg_dump及依赖库（如libpq.so）。
2. 创建Lambda层：将pg_dump放入bin/目录，打包为zip文件，上传至AWS Lambda层并附加到目标函数。
3. 修改脚本：使用环境变量传递密码，并确保pg_dump路径可访问：

def dump():
    import boto3
    import psycopg2
    import subprocess
    import os

    ssm = boto3.client('ssm', region_name='us-west-1')
    pg_host = get_parameters(ssm, "pg_host")
    pg_password = get_parameters(ssm, "pg_password")
    pg_database_name = get_parameters(ssm, "pg_database_name")
    pg_user = get_parameters(ssm, "pg_user")
    con = None
    try:
        print('Connecting to the PostgreSQL database...')
        con = psycopg2.connect(
            host=pg_host,
            database=pg_database_name,
            user=pg_user,
            password=pg_password,
            port="5432")
        
        # 设置环境变量传递密码，避免命令行明文传递
        os.environ['PGPASSWORD'] = pg_password
        # 调用pg_dump（若层的bin目录不在PATH中，需使用绝对路径如/opt/bin/pg_dump）
        subprocess.call([
            'pg_dump', 
            '-h', pg_host, 
            '-U', pg_user, 
            '-p', '5432',
            '-F', 't', 
            '-f', '/tmp/pg_dump.tar', 
            '-d', pg_database_name
        ])
        print("备份成功")
  
    finally:
        if con is not None:
            con.close()
            print('Done')

内容的提问来源于stack exchange，提问作者Igor