Hey there! Let's walk through setting up Log4j2 with CloudWatch for your Java Lambda to replace the default logging and fix those duplicate entries when handling large CSVs via Step Functions. You're already off to a good start with the aws-lambda-java-log4j2 dependency, so let's build on that:

1. Finalize Your Maven Dependencies #

First, make sure your pom.xml includes all necessary Log4j2 components and excludes conflicting default Lambda logging libraries to avoid duplication:

<dependencies>
    <!-- AWS Lambda Log4j2 adapter (core integration) -->
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-lambda-java-log4j2</artifactId>
        <version>1.0.0</version>
    </dependency>
    <!-- Log4j2 core and API (required for logging functionality) -->
    <dependency>
        <groupId>org.apache.logging.log4j</groupId>
        <artifactId>log4j-core</artifactId>
        <version>2.20.0</version> <!-- Use the latest compatible version -->
    </dependency>
    <dependency>
        <groupId>org.apache.logging.log4j</groupId>
        <artifactId>log4j-api</artifactId>
        <version>2.20.0</version>
    </dependency>
    <!-- Lambda core dependency, exclude conflicting logging libs -->
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-lambda-java-core</artifactId>
        <version>1.2.21</version>
        <exclusions>
            <exclusion>
                <groupId>commons-logging</groupId>
                <artifactId>commons-logging</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
</dependencies>

2. Create Log4j2 Configuration File #

Create a log4j2.xml file in your src/main/resources directory (this path ensures it's packaged correctly in your Lambda JAR). Use the LambdaAppender provided by AWS to send logs directly to CloudWatch without duplicate entries:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
    <Appenders>
        <!-- Lambda-specific appender: Avoids duplicate logs by sending directly to CloudWatch -->
        <Lambda name="Lambda">
            <PatternLayout pattern="%d{ISO8601} %-5p [%t] %c{1.}: %m%n"/>
        </Lambda>
        <!-- Optional Console appender for local testing (comment out in production) -->
        <Console name="Console" target="SYSTEM_OUT">
            <PatternLayout pattern="%d{ISO8601} %-5p [%t] %c{1.}: %m%n"/>
        </Console>
    </Appenders>
    <Loggers>
        <!-- Root logger: Use Lambda appender for production -->
        <Root level="INFO">
            <AppenderRef ref="Lambda"/>
            <!-- Uncomment below if you need local console logs during testing -->
            <!-- <AppenderRef ref="Console"/> -->
        </Root>
        <!-- Reduce noise from AWS SDK logs -->
        <Logger name="com.amazonaws" level="WARN"/>
        <Logger name="software.amazon.awssdk" level="WARN"/>
    </Loggers>
</Configuration>

3. Update Your Lambda Code to Use Log4j2 #

Replace any Java native logging (like java.util.logging.Logger) or System.out/err calls with Log4j2's logger:

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;

public class CSVSyncHandler implements RequestHandler<Object, String> {
    // Initialize Log4j2 logger
    private static final Logger logger = LogManager.getLogger(CSVSyncHandler.class);

    @Override
    public String handleRequest(Object input, Context context) {
        logger.info("Starting large CSV processing from Step Function");
        
        // Example: Log debug details (enable debug level in config if needed)
        logger.debug("Received CSV input payload snippet: {}", extractPayloadSnippet(input));
        
        try {
            // Your CSV processing logic here
            return "CSV processing completed successfully";
        } catch (Exception e) {
            // Log errors with full stack trace
            logger.error("Failed to process CSV", e);
            throw new RuntimeException("CSV processing failed", e);
        }
    }

    private String extractPayloadSnippet(Object input) {
        // Implement logic to get a short snippet of your CSV payload
        return "Sample CSV content...";
    }
}

4. Fix Duplicate Logs: Disable Default Lambda Logging Capture #

Lambda automatically captures System.out/err and Java Util Logging (JUL) output, which can cause duplicates if you're already using Log4j2. Add this environment variable to your Lambda function to route JUL logs through Log4j2:

• Key: JAVA_TOOL_OPTIONS
• Value: -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

Also, ensure your code doesn't use System.out.println() or System.err.println()—these will still be captured by Lambda and cause duplicates if left in place.

5. Verify Lambda Execution Permissions #

Make sure your Lambda's execution role has the necessary CloudWatch Logs permissions:

• logs:CreateLogGroup
• logs:CreateLogStream
• logs:PutLogEvents

Most default Lambda execution roles include these permissions, but double-check if you've customized your role.

Final Notes #

• When packaging your Lambda, use Maven Shade or Assembly plugin to bundle all dependencies (including Log4j2) into a single JAR.
• For local testing, uncomment the Console appender in your log4j2.xml to see logs in your terminal.

内容的提问来源于stack exchange，提问作者ShaiNe Ram