使用AWS Lambda定时批量推送CloudWatch日志至S3，求NodeJS可用Lambda库

Absolutely, there are several solid options to handle pushing CloudWatch Logs to S3 on an hourly/daily schedule using Node.js Lambda. Let’s break down the most reliable ones:

You don’t need any third-party libraries here—AWS provides a first-party SDK that’s pre-installed in modern Node.js Lambda runtimes (like Node.js 18.x and above). It gives you full control over the entire workflow: fetching logs from CloudWatch, processing them, and uploading to S3.

To make this work, pair the Lambda with an EventBridge Rule set to a cron schedule (e.g., 0 * * * ? * for hourly runs, 0 0 * * ? * for daily runs) to trigger the function automatically.

Here’s a simplified example of how to implement it:

import { CloudWatchLogsClient, DescribeLogStreamsCommand, GetLogEventsCommand } from "@aws-sdk/client-cloudwatch-logs";
import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";

// Initialize clients with your AWS region
const cwLogsClient = new CloudWatchLogsClient({ region: "us-east-1" });
const s3Client = new S3Client({ region: "us-east-1" });

export const handler = async (event) => {
  const logGroupName = "/your/application/log-group";
  const currentTime = Date.now();
  // Adjust this to 24*3600*1000 for daily runs
  const timeWindowMs = 3600 * 1000; 
  const startTime = currentTime - timeWindowMs;
  const endTime = currentTime;

  // Fetch all log streams in the target group
  const logStreamsResponse = await cwLogsClient.send(new DescribeLogStreamsCommand({
    logGroupName,
    orderBy: "LastEventTime",
    descending: true
  }));

  let allLogEvents = [];

  // Iterate through each log stream to pull events
  for (const stream of logStreamsResponse.logStreams) {
    let nextToken;
    do {
      const logEventsResponse = await cwLogsClient.send(new GetLogEventsCommand({
        logGroupName,
        logStreamName: stream.logStreamName,
        startTime,
        endTime,
        nextToken
      }));
      allLogEvents = [...allLogEvents, ...logEventsResponse.events];
      nextToken = logEventsResponse.nextForwardToken;
    } while (nextToken); // Handle pagination for large log sets
  }

  // Format logs and define S3 path (hourly/daily partitioning)
  const logContent = JSON.stringify(allLogEvents, null, 2);
  const timePrefix = new Date(startTime).toISOString().slice(0, 13); // YYYY-MM-DDTHH for hourly
  // Use slice(0,10) instead for daily partitioning: YYYY-MM-DD
  const s3Key = `cloudwatch-logs/${timePrefix}/logs.json`;

  // Upload to S3
  await s3Client.send(new PutObjectCommand({
    Bucket: "your-target-s3-bucket",
    Key: s3Key,
    Body: logContent,
    ContentType: "application/json"
  }));

  return {
    statusCode: 200,
    body: `Successfully uploaded ${allLogEvents.length} log events to S3: ${s3Key}`
  };
};

If you want to skip writing boilerplate code, there are a few community-maintained libraries that wrap the AWS SDK to simplify the export process. Two worth checking out:

• cloudwatch-logs-exporter: A lightweight library that handles log fetching and S3 uploads with minimal configuration.
• aws-cloudwatch-log-to-s3: Focused specifically on batch exports, with built-in support for scheduling and partitioning.

Note: While these save time, always verify their maintenance status and compatibility with the latest AWS SDK versions before using them in production.

• IAM Permissions: Your Lambda execution role needs permissions to logs:DescribeLogStreams, logs:GetLogEvents (for CloudWatch) and s3:PutObject (for S3). Make sure to scope these permissions to only the log groups and S3 buckets you need.
• Pagination: CloudWatch Logs API limits the number of events returned per request—always handle the nextToken to ensure you fetch all logs in your time window.
• Error Handling: Add retry logic for failed API calls (e.g., using @aws-sdk/util-retry from the AWS SDK) and logging for failures to debug issues quickly.
• Cost Efficiency: Avoid fetching duplicate logs by tracking the last processed timestamp or using CloudWatch Logs filters to narrow down the events you need.

内容的提问来源于stack exchange，提问作者bhb