Spring Boot | How to consume JSON messages using Apache Kafka

Apache Kafka is widely used for real-time data streaming, and Spring Boot makes it easy to integrate Kafka into Java applications. This article demonstrates how to consume JSON messages in a Spring Boot application and process them efficiently using Kafka consumers.

• Configure Kafka consumer properties and JSON deserialization in the application.
• Use @KafkaListener to receive and process incoming JSON messages.
• Map JSON data to Java objects (POJOs) for structured handling.

Step by step Implementation process To consume Jason Massage 

Follow these steps to implement JSON message consumption in Apache Kafka using a Spring Boot application.

Step 1: Create a Spring Boot Project

Create a Spring Boot project using Spring Initializr and

• Add the dependency: Spring for Apache Kafka
• Provides Kafka producer and consumer support.
• Enables Kafka listener annotations.
• Simplifies Kafka configuration in Spring Boot

Step 2: Create the Student Model Class

Since the application will consume student details in JSON format, create a Student model class.

• Represents the JSON structure.
• Stores incoming Kafka message data.

Step 3: Configure Kafka Consumer

Create a configuration class named Config and annotate it with @Configuration and @EnableKafka.

• Establishes communication between Spring Boot and Kafka.
• Defines consumer properties.
• Configures JSON deserialization.

Step 4: Create Kafka Consumer Service 

Create a service class that listens for incoming messages from Kafka.

• Receives messages from Kafka topics.
• Converts JSON data into Student objects.

Step 5: Start ZooKeeper and Kafka Server

Before running the Spring Boot application, start ZooKeeper and the Kafka broker.

• ZooKeeper manages Kafka cluster metadata.
• Kafka broker handles message storage and delivery.
• Consumers cannot receive messages unless the Kafka server is running.

Step 6: Create a Kafka Topic

Create a topic named JsonTopic.

For macOS/Linux

bin/kafka-topics.sh --create \

--zookeeper localhost:2181 \

--replication-factor 1 \

--partitions 1 \

--topic JsonTopic

For Windows

.\bin\windows\kafka-topics.bat --create ^

--zookeeper localhost:2181 ^

--replication-factor 1 ^

--partitions 1 ^

--topic JsonTopic

Step 7: Start Kafka Producer Console

Open a new terminal and run the Kafka producer console.

For macOS/Linux

bin/kafka-console-producer.sh \

--broker-list localhost:9092 \

--topic JsonTopic

For Windows

.\bin\windows\kafka-console-producer.bat ^

--broker-list localhost:9092 ^

--topic JsonTopic

Step 8: Run the Spring Boot Application

Start the Spring Boot application.

Now enter JSON data in the Kafka producer console:

{
"id": 101,
"firstName": "John",
"lastName": "Doe"
}

Press Enter to send the message.

Expected Output

New Entry: Student{
id=101,
firstName='John',
lastName='Doe'
}

The JSON message is automatically deserialized into a Student object and displayed on the console.