Valkey: What’s New and What’s Next?

RALEIGH, N.C. — Valkey, the open source fork of Redis created this past March, released its Version 8.0 into general availability in September. Amazon Web Services is among the enterprises already using the in-memory storage engine, according to Kyle Davis, senior developer advocate for open source Valkey at AWS.

“AWS has a service called ElastiCache, and they just launched a few weeks ago their Valkey service,” said Davis on this episode of The New Stack Makers. “It’s for users who want to deploy their own in-memory data store, and they can now spin that up as a new engine in that service.”

Valkey was created after Redis moved its previously open source code base to more restrictive licensing; in his Makers conversation with Alex Williams, founder and publisher of The New Stack, recorded at the All Things Open conference in October, the AWS dev advocate spoke about how the project is being used, what’s new in Valkey 8.0 (released in September) and offered a peek at what’s coming up with Valkey 9.0.

The Valkey project, Davis said, has drawn contributors from not only AWS but also Alibaba, Ericsson, Google, Huawei, Oracle, and Percona.

“We had people that kind of came out of the woodwork,” he told the Makers audience. For a while, the top contributor was someone who wasn’t previously on any of the participating companies’ radar.

”Come to find out it was a guy really interested in the project, working out of Vietnam, and he was just the biggest contributor for a couple of weeks,” Davis said, adding, “We had people that just kind of drove by, found the project and contributed a lot of code to it.”

Data Structure Changes Equal More Efficiency

Previous versions of Valkey, Davis said, mimicked the architecture of Redis, but Valkey 8.0 introduced multithreading as a capability.

Redis, created in 2009, “was built in a way that mirrored the hardware that was available at the time,” Davis said. “Architecturally, it’s based on what’s called an event loop, and the software would just do a little bit and then go to the next thing and run the next thing. Consequently, it was single-threaded, and that matched what was available at the time.”

But now, he added, “we work on these multithreaded CPUs that have so many options to utilize it.”

Today’s more advanced processors, Davis said, have made it more possible to introduce multithreading into the I/O.

“The multithreading was one of the first things that we looked at doing,” he said. “So that came together really quickly because it was just kind of a pent-up demand. We had a lot of people who focused on that. And we knew that early, it’s gonna be a differentiating feature.”

Valkey is a key value database with keys stored in RAM. Its project maintainers have found that changing Valkey’s data structure can help increase its memory efficiency, Davis said.

“Sometimes you have very large keys and actually a relatively small amount of data; you can get huge benefits by making tweaks to that data structure,” he said. “So we change the data structure a little bit for that key dictionary, got a few bytes per key, and that can end up being up to 20% more memory efficient.”

The maintainers found overhead that slowed down memory, Davis said. For instance, “every piece of data has to have metadata, like when it was stored, how it was stored, and then the data structure itself has metadata as well .. all those things take up precious bytes

He added, “Just refactoring that, changing the way we organized that, how the metadata was packed into it, made a big difference in the amount of storage that was required for the key set.”

What Comes After Valkey 8.0?

Going forward, the Valkey team intends to release two new versions per year, “a major and a minor,” Davis said. The next version won’t come out until 2025, he told the Makers audience.

But new enhancements are emerging on the horizon. Project maintainers are looking at ways to extend the database through modules, Davis said. “One of those is a JSON module. So you could take JSON in its natural form, then go ahead and put that into Valkey, and then you can operate on that JSON without having to deserialize or serialize it again. So if you want to make a deep change into it, you can make that deep change, retrieve things back.”

Another potential module would be a bloom filter. “Bloom filters are a probabilistic data structure for testing presence,” he said. “We’re implementing a Bloom filter module for 8.1.”

Further into the future, he said, possibly a vector similarity search module.

And as for Valkey 9.0? “It’s pretty far into the future,” Davis said. “I think we’re gonna see more fundamental changes to the system. What I hear a lot of people talk about, is changes to how the cluster is going to work.

“Again, we have a cluster that was made in the early teens, and now moving that cluster to use things more efficiently, maybe change some of the properties around the cluster itself, will be kind of that big change. “

Check out the full episode for more about Valkey and how it’s being used.