What Is Data Management in the Kubernetes Age?

Data management poses challenges for those organizations seeking to make the shift to Kubernetes and cloud native environments.  In this episode of The New Stack Analysts podcast, TNS founder and publisher Alex Williams virtually shared pancakes and syrup with guests to discuss how Apache Cassandra, gRPC and other tools and platforms play a role in managing data on Kubernetes.

The podcast featured Mya Pitzeruse, software engineer and OSS contributor from microservices service provider effx; Sam Ramji, chief strategy officer at Datastax; and Tom Offermann, lead software engineer at observability service provider New Relic as guests offering deep perspectives about the evolution of data management on Kubernetes and the work that remains to be done.

What is Data Management in the Kubernetes Age?

There are “a lot of lessons on the stateless side of the world that are starting to make their way down into the stateful side,” said Pitzeruse. “As a result, we’re seeing some really healthy growth, [such as] running these really complicated stateful systems inside of these more ephemeral compute platforms like Kubernetes or Nomad — or even the managed ones in the cloud,” said Pitzeruse.

Indeed, the original idea “was, as we increase the appetite for compute, people will use more and more cloudy stateful services, but you don’t have to stuff that inside Kubernetes,” said Ramji, who was vice president of product management at Google Cloud Platform overseeing Kubernetes and cloud DevOps when Kubernetes was in development.

“We thought ‘we’ll just take care of the compute and the data will take care of itself, but instead … that has pulled in a lot more orientation into ‘hey, why shouldn’t data be as easy to use as the compute layer?’” said Ramji. “So, bringing statefulness to the Kubernetes environment is kind of the big challenge and opportunity right now.”

However, while managing stateful data on Kubernetes remains a key challenge — as mentioned above — work had already begun when  Kubernetes was first being developed. “The early activity in Kubernetes really was around stateless services,” said Offermann. “And now I think we’re kind of at the point where we’re ready to tackle how do you manage stateful service? How do you manage databases? How do you manage datastores on Kubernetes? And that’s pretty exciting.”

The new tools and platforms that support data management largely support how data is shared and transferred on Kubernetes. “A lot of it gets into how you efficiently move data around since a lot of the traditional systems have been bound to single machines, and with this kind of new era of compute, we need to start working on taking our data workloads and spreading them across them as well,” said Pitzeruse.

For this talk, Williams also offered a “Kubernetes pancake” as a bonus, and,  for one lucky podcast listener among those who Tweet their questions, TNS extended its pancake-making vibes by offering a Star Wars spatula set.