How to Subscribe to the Topic in Apache Kafka from the Java application?

Subscribing to a Kafka topic from a Java application requires setting up a Kafka consumer that reads messages from a specific topic. This is a key part of many microservice architectures where services must process messages asynchronously. Apache Kafka provides a robust and scalable platform for building such message-driven applications.

Kafka consumers are responsible for reading messages from Kafka topics. The customer subscribes to the topic and always polls for new messages. The key components involved in setting up a Kafka consumer include:

• Consumer Configuration: Define properties such as the Kafka server address, group ID, key deserializer, and value deserializer.
• Kafka Listener: An annotation provided by Spring Kafka to mark a method as a listener for a specific topic.
• Consumer Group: A group of consumers that work together to consume messages from a topic. Each message is delivered to only one consumer in the group.

Implementation to Subscribe to the Topic in Apache Kafka from the Java application

Below are the implementation steps to subscribe to the topic in Apache Kafka from the Java application.

Step 1: Setup the Apache Kafka

Refer to this link to confirm that Kafka is installed and running on your local system.

Step 2: Create the Spring Boot Project

Create the Spring Boot project using the Spring Initializr and add the below required dependencies.

• Spring Web
• Spring For Apache Kafka
• Lombok
• Spring DevTools

After completing this step, the project structure will be like below:

👁 Project Folder Structure

Step 3: Configure the application properties

Open the application.properties and add the below properties for configuration of the server port.

spring.application.name=kafka-demo
server.port=8080

Step 4: Consumer Kafka Configuration

Now, we will create the KafkaConsumerConfig class that can configure the configuration of the Consumer service.

Go to src > main > java > org.example.kafkademo > KafkaConsumerConfig and put the below code.

Step 5: Main Class

Open the main class to subscribe the topic of the Kafka in the Spring application.

Go to src > main > java > org.example.kafkademo > KafkaDemoApplication and put the below code.

pom.xml:

Step 6: Run the Application

After completing all the above steps, now run the application and it will start at port 8080.

👁 Application Running

Step 7: Sending Messages to the Topic

Now, we will use the Kafka producer to send the messages to the topic for testing. We can use the below Kafka command line tool.

.\bin\windows\kafka-topics.bat --create --topic kafka-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

Output:

👁 Topic Created

Type the message:

bin/kafka-console-producer.bat --broker-list localhost:9092 --topic kafka-topic
>hello kafka message

Output:

👁 Kafka Message

Spring Boot Console Output:

👁 Final Output

By the following these steps, we can set up the Kafka consumer in the Spring Boot application and it allows microservices to process messages asynchronously and efficiently.