 |
VOOZH | about |
|
VOOZH | about |
We’re so glad you’re here. You can expect all the best TNS content to arrive Monday through Friday to keep you on top of the news and at the top of your game.
Check your inbox for a confirmation email where you can adjust your preferences and even join additional groups.
Follow TNS on your favorite social media networks.
Become a TNS follower on LinkedIn.
Check out the latest featured and trending stories while you wait for your first TNS newsletter.
Kafka is popular, there’s no question. Due to its high performance optimized for throughput, it can also scale to large amounts. As Kafka scales, cost is a concern, so the price/performance value is key to most organizations running Kafka. Price/performance is always a critical measure when we look at the infrastructure costs and the desired throughput and performance of the system.
With the increased deployment and usage of ARM architectures, and most especially Amazon Web Services‘ Graviton2, Graviton3 and Ampere, there is a need to understand the price/performance. In this analysis, we are focused on benchmarking these in various conditions.
There have been some great articles on the migration benefits of moving Kafka onto Graviton processors. Most of them have been specific to certain types of workloads, versus generic benchmarking. This is the most in-depth article: AWS Graviton2 and gp3 Support for Apache Kafka – DZone. There are also some good benchmarks published by Honeycomb on their migration Tuning Apache Kafka and Confluent Platform for Graviton2 using Amazon Corretto | AWS Developer Tools Blog. None of the blogs are looking at Graviton3, nor are any of them doing controlled testing, so we figured we would run some experiments.
The setup we selected includes an Apache Kafka cluster running 3 nodes with the version 3.3. Every node will run the same kernel version (being 6.0.12). It will host 1 topic with 2000 partitions. The retention is set to 10GB for each partition and the retention time is 10 minutes. Every broker will host the same number of partitions.
In front of this cluster, we will run a scenario of 8 producers and 8 consumers sending an unlimited number of 500-byte messages.
Every benchmark runs for 5 minutes of warm-up, then 20 minutes of maintained load. Note that these benchmarks are measured at peak performance; however, all instances chosen here are throttleable instances. This means that peak performances can only be maintained for less than 1h.
Every producer will be an instance of kafka-producer-perf-test.sh; respectively, every consumer will be an instance of kafka-consumer-perf-test.sh. Each of these instances will run m6a.2xlarge nodes.
The nodes picked to compare the three different architectures will be:
The disk setup was made up of 3 EBS volumes with enough extra I/O to not be throttled and allow for maximum throughput.
One of the most interesting things we discovered was the overall stability of different architectures under heavy load. x86 architectures (Intel and AMD) increasingly saw IOWait times increase until the cluster became unresponsive as the CPU was trying to handle IO and not serving requests properly, hence some metrics for this scenario appear much lower than others.
This is a direct result of the EBS limits of these instances, as EBS quotas were reached, the IO throttling had a very high impact on the capacity of the machine to handle IO requests, thus increasing IOWait times. On the other hand, we’ve realized that AMD instances with similar EBS capacity as Intel instances (m6a vs. m5) were less resilient to the IO throttling. As a result, they were more likely to cause instability to the cluster once EBS quotas were fully consumed.
Note that we ran the benchmark 3 times for each architecture, and the failures on AMD architecture instances were consistent.
Too many of our Kafka customers focus on OS-level metrics, which are good to keep an eye on but not indicative of the actual performance of the application. We will get to those soon, but first, the most important metric for benchmarking is requests per second, and the number of messages being ingested from producers. We will also track the induced latency on the client side from when the message is sent.
One should expect the same number of requests on all tests, observing a lower number of requests per second highlights an induced latency perceived on producers’ and consumers’ side.
Some clarification on the metrics measured below. “Messages in” is the number of incoming message rates per topic. This does not consider messages which are being consumed from Kafka, but only messages received by Kafka.
RequestsPerSec, on the other hand, considers requests for messages consumed and produced. In the graph of Network Metrics for data going out of the machine (Network Tx), m7g sends way more data than m6i on the network.
In the end, this reflects that — a bit more data — was produced against m7g, but a lot more data was consumed from it.
Requests Per Second
Although the requests per second showed measurable differences in the highest performing instances, the data below indicates that there was throttling and queuing occurring as the performance of messages per second was significantly closer between the two.
The data here is interesting, although Graviton3 does very well performance-wise, the Intel Ice Lake architecture is the best performing under load. Graviton3 is close.
Messages In
Every five seconds, we captured the MessagesInPerSec counter and aggregated it over the three machines. This measures the messages coming into Kafka per second. Intel outperformed or essentially matched Graviton processors. AMD did well but didn’t match the throughput.
We’ve observed less stability from AMD instances as compared to the other instance types. Those were the two types of machines not able to continuously handle the pressure put on the cluster, thus leading to these spiky graphs compared to m7g, m6g, m6i and m5. However, this isn’t the whole story as you start to look at the other metrics.
Looking at the latency is critical when measuring how the broker is performing. Here we have three charts breaking down the latency by 99th, 95th and 50th percentiles.
The first set of charts looks at the producer latency:
There is not a lot of variance between the architectures of the following processors. All are showing quite low latency under load. This is not that interesting, however, the story soon changes.
The second set of charts shows the consumer latency:
These graphs show there are issues with consumer latency, which strongly suggests that there was a bottleneck on M6a/m6g/m5a as they had much higher latency (here this was EBS bandwidth). Let’s dig into the metrics to determine which subsystems were showing the issue.
CPU consumption has been measured separately in 4 metrics:
Under higher load (8 producers/consumers), ARM machines have shown more stability and less CPU consumption than both Intel and AMD architecture. As mentioned above, Intel and AMD crashed multiple times during the last phase of the benchmark, leading to these numbers.
The IOWait times are interesting as we increase loads, both the m6i and the m7g instances have lower IOWait times because of the higher network bandwidth, hence they are able to process more work than the other instances. Additionally, both of these architectures have higher memory bandwidth, reducing IOWait times. Furthermore, the disk subsystems in all instances were fully saturated based on the total EBS volume bandwidth, which we will show below.
Total CPU time shows how busy the processor is, including the slices that we’ve reviewed above. This is an overview of the CPU usage including system, IRQ, user and IOWait times.
We see that due to the IOWait times, the processors which had the most system time available had the best performance. Below you can see the overall CPU usage is mostly driven by the higher IOWait times due to the IO subsystems being saturated.
Interestingly, m6i was the least overloaded processor, most were running near maximum percentages
Here you can find the disk details from the tests showing the throughput of the disk subsystem. We were using the attached EBS volumes which were striped to allow for higher throughput.
Each of the instances disk subsystems were fully saturated. The following EBS volume throughput is the maximum for EBS in total at burst capacity (can only be reached for under 1h), which is highlighted below. For reference, you may see the sustained performance here: Amazon EBS–optimized instances – Amazon Elastic Compute Cloud.
Note: m7g and m6i and m6a were able to handle approximately the same throughput, whereas m6g, m5a and m5 were showing other bottlenecks.
Note: m7g had the highest outbound traffic, which is due to the higher bandwidth available in this instance. Just for details:
Finally, wrapping up the benchmarks, we can see there is merit to the performance and stability of the ARM architecture chips versus x86 on Kafka. When we consider the price differences between these instances:
Another way to slice this data is by bringing in the cost per message and variance from the price/performance leader of the pack, which was the Graviton3 based m7g instances. (For data, see: Benchmarking Kafka : Price/Performance.)
The Graviton3 is clearly the price-performance winner of the competition and seems like the right choice for Kafka workloads based on these benchmarks. However, there are several things to note.
In summary, as far as performance goes, Intel Ice Lake is the best performer; however, older Intel architectures are behind Graviton3. AMD architectures fall short but have a similar performance profile to the older Graviton2 CPU. Graviton CPUs are more stable than Intel or AMD architectures when running under heavy load.
We hope to do more benchmarking on other platforms, if you want to see OpenSearch, PostgreSQL, or something else, please let us know Tweet @jkowall or @mweaeng! Or book a call with us!
