Lambda生成S3日志时如何仅删除文件夹内内容而非文件夹本身

我运行一个Lambda函数，从ALB查询访问日志并将结果发送至S3桶，当前执行两个查询：Daily Logs和Monthly Logs。我在S3桶里创建了DailyLogs、MonthlyLogs两个文件夹，期望Lambda把日志存到对应文件夹中。

原本想实现生成新日志时删除文件夹内旧CSV文件并替换为新日志，但实际执行时，DailyLogs和MonthlyLogs整个文件夹被删除，而且旧文件也没被删掉，恳请解决。

原Lambda代码：

import boto3
import json
import time
from datetime import datetime

# Query string to execute
daily_query = "SELECT * FROM \"DATABASE\".\"TABLE\" WHERE user_agent LIKE '%test%' AND date_parse(time, '%Y-%m-%dT%H:%i:%s.%fZ') >= date_parse(date_format(date_add('day', -1, current_date), '%Y-%m-%d'), '%Y-%m-%d') AND date_parse(time, '%Y-%m-%dT%H:%i:%s.%fZ') < date_parse(date_format(current_date, '%Y-%m-%d'), '%Y-%m-%d') ORDER BY time ASC"
monthly_query = "SELECT * FROM \"DATABASE\".\"TABLE\" WHERE parse_datetime(time,'yyyy-MM-dd''T''HH:mm:ss.SSSSSS''Z') BETWEEN parse_datetime(CAST(date_trunc('month', current_date) AS varchar), 'yyyy-MM-dd') AND parse_datetime(CAST(current_date AS varchar), 'yyyy-MM-dd') AND user_agent LIKE '%test%' ORDER BY time ASC"

# Database to execute the query against
DATABASE = 'DATABASE'

# Output bucket
bucket_name = 'BUCKET_NAME'

# Initialize Boto3 clients
s3_client = boto3.client('s3')
athena_client = boto3.client('athena')

def lambda_handler(event, context):
    try:
        # Get current date
        current_date = datetime.now()

        # Create folder names for daily and monthly logs
        daily_folder = f"DailyLogs/{current_date.strftime('%Y-%m-%d')}/"
        monthly_folder = f"MonthlyLogs/{current_date.strftime('%Y-%m')}/"

        # Delete existing files in the S3 bucket
        delete_daily_files(daily_folder)
        delete_monthly_files(monthly_folder)

        # Start the query executions
        response = athena_client.start_query_execution(
            QueryString=daily_query,
            QueryExecutionContext={'Database': DATABASE},
            ResultConfiguration={'OutputLocation': f's3://{bucket_name}/{daily_folder}'}
        )

        response = athena_client.start_query_execution(
            QueryString=monthly_query,
            QueryExecutionContext={'Database': DATABASE},
            ResultConfiguration={'OutputLocation': f's3://{bucket_name}/{monthly_folder}'}
        )

        return response
    except Exception as e:
        print(f"An error occurred: {str(e)}")
        return {'statusCode': 500, 'body': json.dumps({'error': str(e)})}

def delete_daily_files(folder):
    response = s3_client.list_objects_v2(Bucket=bucket_name, Prefix=folder)
    if 'Contents' in response:
        keys_to_delete = [{'Key': obj['Key']} for obj in response['Contents']]
        if keys_to_delete:
            s3_client.delete_objects(Bucket=bucket_name, Delete={'Objects': keys_to_delete})

def delete_monthly_files(folder):
    response = s3_client.list_objects_v2(Bucket=bucket_name, Prefix=folder)
    if 'Contents' in response:
        keys_to_delete = [{'Key': obj['Key']} for obj in response['Contents']]
        if keys_to_delete:
            s3_client.delete_objects(Bucket=bucket_name, Delete={'Objects': keys_to_delete})

问题原因 #

1. S3无真实文件夹：S3的"文件夹"只是带/后缀的对象，原代码删除逻辑会把这个对象也删掉，导致看起来整个文件夹被删除。
2. 未处理分页：list_objects_v2默认只返回1000个对象，文件夹内文件超过这个数量时，旧文件会残留。
3. 重复冗余代码：两个删除函数逻辑完全一致，维护成本高。

修复后的代码 #

import boto3
import json
from datetime import datetime

# Query string to execute
daily_query = "SELECT * FROM \"DATABASE\".\"TABLE\" WHERE user_agent LIKE '%test%' AND date_parse(time, '%Y-%m-%dT%H:%i:%s.%fZ') >= date_parse(date_format(date_add('day', -1, current_date), '%Y-%m-%d'), '%Y-%m-%d') AND date_parse(time, '%Y-%m-%dT%H:%i:%s.%fZ') < date_parse(date_format(current_date, '%Y-%m-%d'), '%Y-%m-%d') ORDER BY time ASC"
monthly_query = "SELECT * FROM \"DATABASE\".\"TABLE\" WHERE parse_datetime(time,'yyyy-MM-dd''T''HH:mm:ss.SSSSSS''Z') BETWEEN parse_datetime(CAST(date_trunc('month', current_date) AS varchar), 'yyyy-MM-dd') AND parse_datetime(CAST(current_date AS varchar), 'yyyy-MM-dd') AND user_agent LIKE '%test%' ORDER BY time ASC"

# Database to execute the query against
DATABASE = 'DATABASE'

# Output bucket
bucket_name = 'BUCKET_NAME'

# Initialize Boto3 clients
s3_client = boto3.client('s3')
athena_client = boto3.client('athena')

def lambda_handler(event, context):
    try:
        # Get current date
        current_date = datetime.now()

        # Create folder paths (ensure trailing slash)
        daily_folder = f"DailyLogs/{current_date.strftime('%Y-%m-%d')}/"
        monthly_folder = f"MonthlyLogs/{current_date.strftime('%Y-%m')}/"

        # Delete existing files in target folders (exclude folder object itself)
        delete_folder_contents(bucket_name, daily_folder)
        delete_folder_contents(bucket_name, monthly_folder)

        # Start the query executions
        athena_client.start_query_execution(
            QueryString=daily_query,
            QueryExecutionContext={'Database': DATABASE},
            ResultConfiguration={'OutputLocation': f's3://{bucket_name}/{daily_folder}'}
        )

        athena_client.start_query_execution(
            QueryString=monthly_query,
            QueryExecutionContext={'Database': DATABASE},
            ResultConfiguration={'OutputLocation': f's3://{bucket_name}/{monthly_folder}'}
        )

        return {'statusCode': 200, 'body': json.dumps('Queries started successfully')}
    except Exception as e:
        print(f"An error occurred: {str(e)}")
        return {'statusCode': 500, 'body': json.dumps({'error': str(e)})}

def delete_folder_contents(bucket, folder_prefix):
    # Ensure prefix ends with slash to target only contents under this path
    if not folder_prefix.endswith('/'):
        folder_prefix += '/'
    
    paginator = s3_client.get_paginator('list_objects_v2')
    pages = paginator.paginate(Bucket=bucket, Prefix=folder_prefix)
    
    keys_to_delete = []
    for page in pages:
        if 'Contents' in page:
            for obj in page['Contents']:
                # Skip the folder object itself (if it exists)
                if obj['Key'] != folder_prefix:
                    keys_to_delete.append({'Key': obj['Key']})
    
    if keys_to_delete:
        # Delete in batches (max 1000 objects per delete call)
        for i in range(0, len(keys_to_delete), 1000):
            batch = keys_to_delete[i:i+1000]
            s3_client.delete_objects(Bucket=bucket, Delete={'Objects': batch})

关键修复点 #

• 通用删除函数：合并重复逻辑为delete_folder_contents，降低维护成本。
• 分页遍历：使用S3分页器获取所有对象，避免文件数量过多时遗漏删除。
• 保留文件夹对象：通过判断对象Key不等于文件夹前缀，确保只删除内部文件。
• 批量删除优化：按S3批量删除上限（1000个对象）拆分请求，避免请求失败。

内容的提问来源于stack exchange，提问作者Girish Luckhun