使用Testcontainers替代Embedded-Kafka进行Spring-Kafka集成测试的问题

Let's break down your two issues and walk through practical solutions for each:

1. Can we use spring-kafka-test with Testcontainers, and how to control @KafkaListener activation? #

Absolutely! spring-kafka-test works seamlessly with Testcontainers—you can still use all its testing utilities (like KafkaTemplate, ConsumerFactory, or helper listeners) while relying on a real Kafka instance from Testcontainers instead of the embedded one.

To restrict @KafkaListener beans to only run in specific tests, here are three reliable approaches:

Option 1: Use Spring Profiles #

• Add @Profile("kafka-test") to any class that contains @KafkaListener methods.
• In your Kafka-related test classes (like PerformSomethingInboundAdapterTest), add @ActiveProfiles("kafka-test") to activate those listener beans.
• For non-Kafka tests, omit the kafka-test profile—Spring won't initialize the listener beans at all, so no unwanted message consumption.

Option 2: Mock Listener Beans in Non-Kafka Tests #

• In tests that don't involve Kafka, use @MockBean to replace your real listener classes with mocks. This prevents the actual listener from starting and consuming messages.
  Example:

  @SpringBootTest
  class NonKafkaTest {
      @MockBean
      private YourKafkaListenerClass listener;
      // ... test logic
  }
  

Option 3: Use Conditional Configuration #

• Add @ConditionalOnProperty(name = "kafka.listeners.enabled", havingValue = "true", matchIfMissing = false) to your listener classes.
• In Kafka test classes, enable the property via @TestPropertySource or a test-specific application.properties:

  @TestPropertySource(properties = "kafka.listeners.enabled=true")
  class KafkaRelatedTest extends IntegrationTest {
      // ... test logic
  }
  

• Non-Kafka tests will leave this property unset, so listeners won't be loaded.

2. Port 9092 conflict when running tests via Maven command line #

The root cause here is multiple JVM processes initializing the Kafka container independently. When running tests via Maven, the Surefire plugin often forks a new JVM for each test class (or batch of classes). Since your kafkaContainer is a static variable, each JVM spins up its own DockerComposeContainer instance—all trying to bind to port 9092 on your host, causing the conflict.

Here's how to fix this:

Solution 1: Use Testcontainers' @Container Annotation #

Testcontainers integrates natively with JUnit 5 via the testcontainers-junit-jupiter dependency. The @Container annotation manages container lifecycle automatically, ensuring only one container instance runs per JVM (and can even share containers across test classes if marked static).

Update your base test class like this:

@SpringBootTest
@Slf4j
public abstract class IntegrationTest {
    // Static @Container ensures the container is shared across all test classes in the same JVM
    @Container
    private static final DockerComposeContainer<?> kafkaContainer = new DockerComposeContainer<>(new File("src/test/resources/kafka-compose.yml"))
            .withExposedService("kafka_1", 9092)
            .withEnv("KAFKA_HOST", "localhost");

    static {
        kafkaContainer.start();
        // Set the correct bootstrap servers for Spring Kafka (replace the embedded Kafka property)
        String bootstrapServers = String.format("PLAINTEXT://%s:%s",
                kafkaContainer.getServiceHost("kafka_1", 9092),
                kafkaContainer.getServicePort("kafka_1", 9092));
        System.setProperty("spring.kafka.bootstrap-servers", bootstrapServers);
        // Note: Avoid using spring.embedded.kafka.brokers—it's exclusively for Embedded-Kafka
    }
}

Solution 2: Use Random Host Ports #

Modify your kafka-compose.yaml to skip hardcoding the host port mapping. Let Testcontainers assign a random port automatically:

kafka:
  image: "wurstmeister/kafka:2.12-2.2.2"
  ports:
    - "9092"  # No host port specified—Testcontainers picks a random available one
  # ... rest of your config

This eliminates port conflicts entirely, even if multiple containers were to run (though @Container should prevent that).

Solution 3: Adjust Maven Surefire Plugin Configuration #

If you want to force tests to run in a single JVM (not recommended for large test suites, as it slows execution), configure the Surefire plugin in your pom.xml:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>3.2.5</version>
    <configuration>
        <forkCount>1</forkCount>
        <reuseForks>true</reuseForks>
    </configuration>
</plugin>

Bonus: Improve Wurstmeister Kafka Configuration #

For better compatibility with Testcontainers, update your kafka-compose.yaml to use KAFKA_ADVERTISED_LISTENERS instead of KAFKA_ADVERTISED_HOST_NAME/KAFKA_ADVERTISED_PORT—this ensures both internal container communication and external host access work correctly:

kafka:
  image: "wurstmeister/kafka:2.12-2.2.2"
  ports:
    - "9092"
  depends_on:
    - "zookeeper"
  environment:
    KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
    KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:${KAFKA_PORT:-9092}
    KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,PLAINTEXT_HOST://0.0.0.0:9092
    KAFKA_CREATE_TOPICS: "recoverer-test:1:1,some-topic"
    KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"

内容的提问来源于stack exchange，提问作者Roman T