Kafka MirrorMaker Management via CLI

Kafka MirrorMaker complies with strict certifications, allowing you to keep a replica of your Kafka cluster in a second data center. It accomplishes this by leveraging the existing consumer and producer APIs. Kafka MirrorMaker allows you to copy data across two Kafka clusters. This service allows you to focus on developing your apps rather than worrying about their configuration, monitoring, backup, redundancy, or updates.

What is Kafka MirrorMaker?

Kafka MirrorMaker is a tool for mirroring data across Apache Kafka clusters and it mostly uses the Kafka Connect technology to replicate data, hence increasing resiliency. However, SaaS applications have an extensive deployment across numerous worldwide data centers as this resulted in the development and release of Kafka MirrorMaker 2, which includes additional features. MirrorMaker 2 is used by developers for a variety of applications, such as a communication channel for transferring messages between Kafka clusters in different data centers.

How Does Kafka MirrorMaker Work?

Connectors

• MirrorSinkConnector: Consumes primary cluster topics and replicates them to a single target cluster.
• MirrorHeartBeatConnector: Checks for cluster connectivity.
• MirrorCheckpointConnector: Generates consumer offset checkpoints.
• MirrorSourceConnector: Replicates topics, topic ACLs, and configurations from a single source cluster and generates offset syncs.

Running Modes

• MirrorMaker in a Connect cluster: This option is appropriate for organizations that already have a previously installed Connect cluster.
• Dedicated MirrorMaker cluster: This mode is also known as driver mode and it is the simplest production-ready mode to build because it assumes no prior knowledge of the Kafka Connect infrastructure.
• Standalone MirrorMaker connector: This mode is appropriate for testing a single Connect worker.

Features of Kafka MirrorMaker

• MirrorMaker 2 improves on MirrorMaker 1 (MM1) in a variety of ways. MirrorMaker 2's built-in high-level driver simplifies initial configuration and connector management in a dedicated cluster.
• The auto-discovery feature also automatically detects new topics and partitions. It takes some time, but ultimately the new topic is copied.
• MirrorMaker 2 supports a wide range of topologies and data flows, including "active/active" and "active/passive" cluster pairings, cross-datacenter replication, and aggregation.
• MirrorMaker 2 even automatically synchronizes topic setup, including ACLs, across clusters. This is a departure from MM1, where topics simply selected the default cluster configuration options.

Implementation of Kafka MirrorMaker Management via CLI

Step 1: Setup and Configuration

First, you need to ensure files the consumer.properties and producer.properties files are well maintained in Kafka.

consumer.properties:

bootstrap.servers=source_kafka_broker1:9092,source_kafka_broker2:9092
group.id=mirrormaker-consumer-group
key.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer
value.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer

producer.properties:

bootstrap.servers=target_kafka_broker1:9092,target_kafka_broker2:9092
key.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer

Step 2: Create MirrorMaker for Kafka

Next, You have to use the below command to create MirrorMaker:

bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config consumer.properties --producer.config producer.properties --whitelist topic1,topic2 --num.streams 4

Output:

Step 3: Monitor MirrorMaker

You can review the logs to keep an eye on the status:

tail -f /path/to/kafka/logs/mirrormaker.log

Output:

Step 4: Stop MirrorMaker

At last, the process needs to stop now you have to find the process and stop it:

ps -ef | grep MirrorMaker

Output:

Best Practices of Kafka MirrorMaker Management via CLI

Before you begin Mirror Maker, you should establish any topics that will be mirrored on the destination cluster. Mirror Maker can generate subjects automatically, although they may not have the same configuration as the originals.

If you have many instances, it is advised that they all use the same group in the consumer properties.

Conclusion

In this article, we have learned about Kafka MirrorMaker Management via CLI. Kafka MirrorMaker mirrors data between Apache Kafka clusters. It employs Kafka Connect technology to repeat data, enhancing resilience. However, current SaaS applications require massive deployments across several global data centers.