Apache Kafka is widely used in Spring Boot applications for building event-driven and streaming systems. While writing integration tests, developers often rely on Embedded Kafka to avoid managing external Kafka infrastructure. However, a common challenge arises when multiple test classes each start their own embedded Kafka broker, leading to slower test execution and unnecessary resource usage. Let us delve into understanding how reusing an embedded Kafka broker in tests helps optimize the testing process.

1. What is Apache Kafka?

Apache Kafka is a distributed event streaming platform designed to handle high-throughput, fault-tolerant, and real-time data streams. It is commonly used for building event-driven architectures, real-time analytics pipelines, log aggregation systems, and data integration platforms. At its core, Kafka works as a publish–subscribe messaging system where:

• Producers publish messages (events) to Kafka topics
• Topics are partitioned and replicated for scalability and fault tolerance
• Consumers subscribe to topics and process messages independently
• Brokers form a Kafka cluster and store message data

In Spring Boot applications, Kafka is typically integrated using Spring for Apache Kafka, which provides abstractions such as KafkaTemplate for producing messages and @KafkaListener for consuming them.

For testing purposes, running a real Kafka cluster is often impractical. To address this, Spring Kafka provides an Embedded Kafka Broker, which starts a lightweight, in-memory Kafka instance within the test JVM. This allows developers to write reliable integration tests without external dependencies.

1.1 Understanding the Problem

While Embedded Kafka greatly simplifies Kafka integration testing, it introduces a performance challenge when used incorrectly. By default, when multiple test classes are annotated with @EmbeddedKafka, Spring creates a new Kafka broker for each test class. This behavior leads to several inefficiencies:

• Significantly increased test startup time due to repeated Kafka initialization
• Higher memory and CPU usage as multiple brokers run sequentially or in parallel
• Slower CI/CD pipelines, especially in large test suites

Consider a typical Kafka-based Spring Boot application test setup:

• Producer integration test validating message publishing
• Consumer integration test validating message consumption
• End-to-end Kafka flow test covering the full message lifecycle

If each of these tests spins up its own embedded Kafka broker, the overall test execution time increases dramatically. In large projects, this can turn a test suite that should run in seconds into one that takes several minutes. Therefore, the key challenge is to reuse a single Embedded Kafka broker across multiple test classes while maintaining test isolation and reliability. Solving this problem results in faster feedback cycles and more efficient automated testing.

1.1.1 Benefits of Reusing an Embedded Kafka Broker

Reusing a single Embedded Kafka broker across multiple test classes provides several tangible benefits, especially for large Spring Boot applications with extensive Kafka-based integration tests:

• Faster test execution by avoiding repeated broker startup and shutdown overhead
• Reduced resource consumption, as only one Kafka instance is initialized during the test lifecycle
• Improved CI/CD pipeline performance, leading to quicker feedback and shorter build times
• More stable test environments with fewer intermittent failures caused by broker startup timing issues
• Better developer productivity, as tests run faster both locally and in automated pipelines

By reusing the Embedded Kafka broker correctly, teams can retain the reliability of integration testing while significantly optimizing execution time and infrastructure usage.

2. Code Example

Reusing an Embedded Kafka broker across multiple test classes is primarily about controlling the Spring test lifecycle and ensuring that Kafka is treated as a shared infrastructure component rather than a per-test resource. When configured correctly, Spring will start Kafka once and reuse it for all tests that share the same application context.

2.1 Adding Dependencies (pom.xml)

<dependency>
 <groupId>org.springframework.kafka</groupId>
 <artifactId>spring-kafka-test</artifactId>
 <scope>test</scope>
</dependency>

This dependency provides the testing utilities required for Kafka integration tests in Spring Boot, including the @EmbeddedKafka annotation, embedded broker support, and helper classes for producing and consuming messages during tests. Declaring it with test scope ensures that Kafka test infrastructure is available only during test execution and is not included in production builds.

2.2 Spring Boot Application

2.2.1 Producer Class

// MessageProducer.java
package com.example.kafka.producer;

import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;

@Service
public class MessageProducer {

 private final KafkaTemplate<String, String> kafkaTemplate;

 public MessageProducer(KafkaTemplate<String, String> kafkaTemplate) {
 this.kafkaTemplate = kafkaTemplate;
 }

 public void send(String topic, String message) {
 kafkaTemplate.send(topic, message);
 }
}

This producer class is a simple Spring-managed service that uses KafkaTemplate to publish messages to Kafka topics. The template is auto-configured by Spring Boot and injected via constructor injection. The send method abstracts Kafka interaction, allowing tests to focus on behavior rather than low-level Kafka APIs.

2.2.2 Consumer Class

// MessageConsumer.java
package com.example.kafka.consumer;

import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;

import java.util.concurrent.CountDownLatch;

@Component
public class MessageConsumer {

 private final CountDownLatch latch = new CountDownLatch(1);
 private String payload;

 @KafkaListener(topics = "test-topic", groupId = "test-group")
 public void receive(String message) {
 this.payload = message;
 latch.countDown();
 }

 public CountDownLatch getLatch() {
 return latch;
 }

 public String getPayload() {
 return payload;
 }
}

This consumer listens to messages from the test-topic using the @KafkaListener annotation. When a message is received, it stores the payload and releases a CountDownLatch, which allows test cases to wait deterministically until consumption is complete. This pattern is commonly used in Kafka integration tests
to avoid timing issues and ensure reliable assertions.

2.3 Creating Test Class(es)

2.3.1 Base Test Class with Embedded Kafka

This base test class is responsible for initializing the Embedded Kafka broker only once and sharing it across all Kafka-related test classes. By centralizing the Kafka setup here, Spring is able to cache and reuse the same application context, preventing redundant broker startup.

// BaseKafkaTest.java
package com.example.kafka;

import org.junit.jupiter.api.TestInstance;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.kafka.test.context.EmbeddedKafka;
import org.springframework.test.context.ActiveProfiles;

@SpringBootTest
@ActiveProfiles("test")
@EmbeddedKafka(
 partitions = 1,
 topics = { "test-topic" },
 brokerProperties = {
 "listeners=PLAINTEXT://localhost:9092",
 "port=9092"
 }
)
@TestInstance(TestInstance.Lifecycle.PER_CLASS)
public abstract class BaseKafkaTest {
}

Here, @SpringBootTest loads the full Spring application context, @ActiveProfiles("test") ensures test-specific configuration is used, and @EmbeddedKafka starts a single Kafka broker with a predefined topic. Because all test classes extend this abstract base class and share identical annotations, Spring’s test context caching mechanism reuses the same context and broker. The PER_CLASS test instance lifecycle further reduces unnecessary reinitialization within a test class.

2.3.2 Producer Test Class

// ProducerIntegrationTest.java
package com.example.kafka;

import com.example.kafka.producer.MessageProducer;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;

class ProducerIntegrationTest extends BaseKafkaTest {

 @Autowired
 private MessageProducer producer;

 @Test
 void shouldSendMessageToKafka() {
 producer.send("test-topic", "hello kafka");
 }
}

This test class verifies that the Kafka producer can successfully send a message. By extending BaseKafkaTest, it automatically uses the shared Embedded Kafka broker.The MessageProducer bean is injected from the Spring context, and the test simply
publishes a message to the topic without worrying about broker setup or teardown.

2.3.3 Consumer Test Class

// ConsumerIntegrationTest.java
package com.example.kafka;

import com.example.kafka.consumer.MessageConsumer;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;

import java.util.concurrent.TimeUnit;

import static org.assertj.core.api.Assertions.assertThat;

class ConsumerIntegrationTest extends BaseKafkaTest {

 @Autowired
 private MessageConsumer consumer;

 @Test
 void shouldConsumeMessageFromKafka() throws Exception {
 boolean messageConsumed =
 consumer.getLatch().await(10, TimeUnit.SECONDS);

 assertThat(messageConsumed).isTrue();
 assertThat(consumer.getPayload()).isEqualTo("hello kafka");
 }
}

This consumer test also extends the shared base test class, ensuring it runs against the same Embedded Kafka broker used by the producer test. The test waits on a CountDownLatch until a message is consumed, then asserts both successful consumption and payload correctness. This demonstrates how multiple test classes can reliably interact with a single embedded broker while remaining isolated at the test logic level.

2.3.4 Code Run and Output

Once all the components and test classes are in place, running the Kafka integration tests is straightforward because the Embedded Kafka broker is managed entirely by Spring’s test framework.

No external Kafka installation or Docker setup is required. The broker starts automatically when the Spring test context is initialized and shuts down when the test suite completes.

mvn clean test

This command compiles the application and test classes, starts a single Embedded Kafka broker, executes all test classes extending BaseKafkaTest, and reuses the same Kafka broker across both producer and consumer integration tests.

Started Embedded Kafka broker on PLAINTEXT://localhost:9092
Sending message: hello kafka
Message received from topic test-topic: hello kafka

BUILD SUCCESS

This output shows that the Embedded Kafka broker starts successfully and is available at the configured address. The producer test publishes the message hello kafka to the topic, and the consumer test receives the same message from test-topic, confirming end-to-end message flow. The presence of a single broker startup message indicates that the Kafka broker was reused across test classes, and the BUILD SUCCESS status confirms that all integration tests completed successfully.

3. Conclusion

Reusing an Embedded Kafka broker across multiple test classes significantly improves test performance and stability in Spring Boot applications by moving @EmbeddedKafka to a shared base test class and leveraging Spring’s test context caching, which helps reduce overall test execution time, avoids unnecessary Kafka restarts, and enables faster and more reliable CI pipelines. This approach is especially useful for large microservice projects with extensive Kafka-based integration tests.