Valkey 8.1’s Performance Gains Disrupt In-Memory Databases

NAPA, Calif — A year ago, Redis announced that it was dumping the open source BSD 3-clause license for its Redis in-memory key-value database and was moving it to a “source-available” Redis Source Available License (RSALv2) and Server Side Public License (SSPLv1).

That move went over like a lead brick with many Redis developers and users. So the disgruntled developers forked a new project, Valkey, as “an open source alternative to the Redis in-memory NoSQL data store.” Now it’s become clear that this has become a remarkably successful fork. 

How successful? According to a Percona research paper, “75% of surveyed Redis users are considering migration due to recent licensing changes. … Of those considering migration, more than 75% are testing, considering, or have adopted Valkey.” Perhaps a more telling point is that third-party Redis developer companies, such as Redisson, are supporting both Redis and Valkey.

Multithreading and Scalability

It’s not just the licensing changes that make Valkey attractive, though. At the Linux Foundation Member Summit,  Madelyn Olson, a principal software engineer at Amazon Web Services (AWS) and Valkey project maintainer said in her keynote speech that Valkey is far faster thanks to incorporating enhanced multithreading and scalability features. 

That, Olson added, was not the original plan. “We wanted to keep the open source spirit of the Redis project alive, but we also wanted the value to be more than just a fork. We organized a contributor summit in Seattle where we got together developers and users to try to figure out what this new project should look like. At the time, I was really expecting us to just focus on caching, the main workload that Redis open source was serving. What we heard from our users is that they wanted so much more. They wanted Valkey to be a high-performance database for all sorts of distributed workloads. And so although that would add a lot of complexity to the project, the new core team sort of took on that mantle, and we tried to build that for our community.”

They were successful. By August of 2024, Dirk Hohndel, a Linux kernel developer and long-time open source leader, said the Valkey 8.0 redesign of Redis’s single-threaded event loop threading model with a more sophisticated multithreaded approach to I/O operations had given him “roughly a threefold improvement in performance, and I stream a lot of data, 60 million data points a day.” In addition, with Valkey 8, he saw about a “20% reduction in the size of separate cache tables. When you’re talking about terabytes and more on Amazon Web Services, that’s a real savings in size and cash.”

Shifting back to the current day, Olson added, “Over the last couple of months, we’ve been dramatically improving the core engine by adding Rust into the core to add memory safety. We’ve been changing the internal algorithm for how the cluster mode works to improve reliability and improve the failover times. We’re also dramatically changing how the internal data structures work since they were based on 10-year-old pieces of software, so they can better take advantage of modern hardware.”

In addition, the developer team has rebuilt the key-value store from scratch to take better advantage of modern hardware based on the work done at Google on the so-called Swiss Tables. Olson continued, “In just a few short weeks, we’ll release these improvements as part of Valkey 8.1 just one year after the project’s anniversary.” This new release includes up to 20% memory efficiency improvements, the most common bottleneck within caching systems, and state-of-the-art data structures.

Looking ahead, Valkey plans to introduce more multithreaded performance improvements, a highly scalable clustering system, and new core changes to data types. Does that sound good to you? The project remains open to new contributors and invites interested parties to join via GitHub.