Implementing Request Response in Java Apache Kafka

Apache Kafka is a very powerful distributed event streaming platform that can be used for building real-time pipelines and streaming applications. It is highly scalable, fault-tolerant, and provides high throughput. One of the common patterns used in the Kafka application is the request-response pattern which is slightly unconventional as Kafka is designed for asynchronous messaging. We can implement the request-response communication between the services using Kafka.

In the request-response pattern, the Consumer sends the request message to the Producer, and the producer can process the request and send it back to the response message. In Kafka, It can achieved using separate topics for the requests and responses along with the correlation IDs to match the responses to corresponding requests.

Prerequisites:

• Basic understanding of the Java and Apache Kafka.
• JDK and IntelliJ Idea installed in your local system.
• Maven for building dependency management.
• Required the Kafka Client Library.

Implementing Request Response in Java Apache Kafka

Step 1: Create a Maven Project

Create a new Java maven project using IntelliJ Idea, after that add the following dependencies into the project dependency.

Note: To Install and setup the Apache Kafka refer this link

Required Dependencies:

 <!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients -->
 <dependency>
 <groupId>org.apache.kafka</groupId>
 <artifactId>kafka-clients</artifactId>
 <version>3.7.0</version>
 </dependency>

 <!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-api -->
 <dependency>
 <groupId>org.slf4j</groupId>
 <artifactId>slf4j-api</artifactId>
 <version>2.0.13</version>
 </dependency>

 <!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-simple -->
 <dependency>
 <groupId>org.slf4j</groupId>
 <artifactId>slf4j-simple</artifactId>
 <version>2.0.13</version>
 <scope>test</scope>
 </dependency>

After creating the project, the folder structure in the IDE will look like the below image:

Step 2: Add the Dependencies into the pom.xml file

Open the pom.xml add the required dependencies into the project.

Step 3: Create the KafkaProducerService Class

The KafkaProducerService class manages a Kafka Producer instance configured with bootstrap servers and serializers for keys and values. It sends messages asynchronously to specified topics, handles delivery callbacks, and provides a method to close the producer cleanly.

Step 4: Create the KafkaConsumerService Class

• KafkaConsumerService initializes a Kafka consumer with specified configuration (bootstrap servers, group ID, topic).
• It continuously polls messages from the subscribed Kafka topic using a loop.
• The class provides a method to close the Kafka consumer instance when no longer needed.

Step 5: Create the RequestResponseService Class

• It manages Kafka producer and consumer services for implementing request-response pattern.
• Sends requests with unique correlation IDs to a specified topic and prints the ID.
• Receives and prints responses from a designated Kafka topic indefinitely.

Step 6: Main Class

Step 7: Run the application

Output:

This example project demonstrates the basic setup and implementation of the request and response pattern in Apache Kafka.