Lambda函数中Redshift Connector模块导入失败问题求助

我在AWS Step Functions中使用Lambda函数将S3中的CSV文件加载至Redshift表，该函数需要配置Layer。已尝试通过pip安装redshift_connector库，但仍遭遇模块导入错误。以下是Lambda函数代码及报错信息，求其他解决方法：

Lambda函数代码 #

import redshift_connector 
import pandas as pd
import boto3
import sys


def lambda_handler(event, context):
    print(event)
    # Initialize Redshift Connector
    host ='REDSHIFT ARN'
    port = ******
    database = '*********'
    user = '*******'
    password = '*******'
    
    # Initialize Redshift Connector
    conn = redshift_connector.connect(
        host=host,
        port=port,
        database=database,
        user=user,
        password=password
    )
    print(f"connection success {host}")
    
    # Access the values of the arguments
    bucket_name = event['validation_result']['Payload']['bucket_name']
    print(bucket_name)
    source_folder = event['validation_result']['Payload']['folder_name']
    file_name = event['validation_result']['Payload']['file_name']
    result={}
    result['bucket_name']=bucket_name
    result['file_name']=file_name

    # Load CSV file into a Pandas DataFrame
    csv_file_path = f's3://{bucket_name}/{source_folder}/{file_name}'
    print(csv_file_path)
    df = pd.read_csv(csv_file_path)

    
    # Define the Redshift table name
    table_name = 'club_games'  # Change this to your actual Redshift table name
    
        # Insert data into the Redshift table
    for _, row in df[columns].iterrows():
        values = ', '.join([f"'{value}'" if isinstance(value, str) else str(value) for value in row])
        insert_sql = f"INSERT INTO {table_name} VALUES ({values})"
        cursor.execute(insert_sql)
        conn.commit()
    
    # Archive the processed CSV file
    s3 = boto3.client('s3')

    archive_key='archive' + '/' + file_name 
    object_key= folder_name + '/' + file_name
    # Copy the file to the archive folder
    s3.copy_object(Bucket=bucket_name, CopySource={'Bucket': bucket_name, 'Key': object_key}, Key=archive_key)
    
    # Delete the original file from the input folder
    s3.delete_object(Bucket=bucket_name, Key=object_key)
    
    conn.commit()
    # Close the Redshift connection
    conn.close()
    return result

报错信息 #

{
  "errorMessage": "Unable to import module 'lambda_function': No module named 'redshift_connector'",
  "errorType": "Runtime.ImportModuleError",
  "requestId": "c0aa62a9-e77d-44d3-b92b-acb9e7362ebd",
  "stackTrace": []
}

解决方法 #

1. 修正Layer创建流程 #

Lambda Layer有严格的目录结构和平台要求，按以下步骤重新构建：

• 指定运行时与架构：确保安装命令匹配你的Lambda运行时（比如Python 3.10）和实例架构（x86_64/arm64），执行：

  # 替换X.X为Python版本（如3.10），架构根据实例调整
  pip install redshift_connector -t python/lib/python3.10/site-packages/ --platform manylinux2014_x86_64 --only-binary=:all:
  

• 正确打包：将python目录整体压缩为zip（不要包含外层文件夹），上传到AWS Lambda Layer
• 关联Layer：在Lambda函数的"Layers"配置中添加该Layer，确认版本为最新

2. 替换驱动方案 #

如果redshift_connector的Layer问题无法解决，可改用以下两种替代方案：

• 使用psycopg2-binary：同样需要构建对应运行时的Layer，安装命令类似：

  pip install psycopg2-binary -t python/lib/python3.10/site-packages/ --platform manylinux2014_x86_64 --only-binary=:all:
  

• Redshift Data API：无需额外驱动，直接用boto3调用API执行SQL，示例代码：

  import boto3
  
  def lambda_handler(event, context):
      client = boto3.client('redshift-data')
      bucket_name = event['validation_result']['Payload']['bucket_name']
      source_folder = event['validation_result']['Payload']['folder_name']
      file_name = event['validation_result']['Payload']['file_name']
      
      # 用COPY命令直接从S3加载数据（效率远高于逐行插入）
      copy_sql = f"""
          COPY club_games
          FROM 's3://{bucket_name}/{source_folder}/{file_name}'
          IAM_ROLE 'your-redshift-iam-role-arn'
          CSV
          IGNOREHEADER 1;
      """
      response = client.execute_statement(
          ClusterIdentifier='your-cluster-id',
          Database='your-db-name',
          DbUser='your-db-user',
          Sql=copy_sql
      )
      # 后续归档逻辑不变
      return {"bucket_name": bucket_name, "file_name": file_name}
  

3. 代码优化与问题修复 #

原代码存在两个致命问题，即使解决了导入错误也会失败：

• df[columns]中的columns变量未定义，需替换为实际列名列表（如df[['col1', 'col2']]）
• 逐行插入数据效率极低，Lambda很可能超时，强烈建议改用Redshift的COPY命令直接从S3加载数据（如上述Data API示例）

4. 本地验证Layer #

用Docker模拟Lambda环境验证Layer是否有效：

# 拉取对应运行时的Lambda镜像
docker pull public.ecr.aws/lambda/python:3.10
# 挂载本地Layer目录到容器，测试导入
docker run -v $(pwd)/python:/opt/python public.ecr.aws/lambda/python:3.10 python -c "import redshift_connector; print('Success')"

内容的提问来源于stack exchange，提问作者user22331353