Q&A: Kubernetes Storage SIG Chair on the State of State in K8s

👁 Image
Just how difficult is it to run stateful workloads in Kubernetes? As we found when talking about the developer experience in Kubernetes, the biggest challenge isn’t necessarily a lack of technical solutions but rather the learning curve for everyone involved, from application developers to cluster administrators and traditional storage administrators. There are, of course, still some tooling gaps and places where the ecosystem is immature and not yet battle-tested.

We talked with Saad Ali, software engineer at Google and the chair of the Kubernetes Storage Special Interest Group at the Cloud Native Computing Foundation, about how Kubernetes has made running stateful workloads easier, the challenges that remain and what to look for in the future.

Can you gives us some background on managing state in Kubernetes?

Ali: There’s been a lot of innovation at the lower level of storage and a lot of innovation at the higher, application level, but not a lot in the middle. You know, making storage available dynamically as your workloads move around hasn’t really been tackled.

Before Kubernetes existed, if you had workloads deployed on a virtual machine you had to create some volumes, carve them out in your storage array or your cloud provider, make sure that they’re attached to the right VM for the workload and make sure it’s formatted property. You had to either do that manually or with some custom scripts your DevOps team wrote ahead of time. If you needed to move that workload to a different VM, that’s a very manual process.

Kubernetes simplified this process, introducing the concept of dynamic volume provisioning, where you don’t need to provision volumes ahead of time, you give Kubernetes credentials to your storage system and let it take care of provisioning automatically. Now the Container Storage Interface (CSI) allows Kubernetes to interact directly with an arbitrary storage system.

What are some of the key challenges related to handling state in Kubernetes?

Ali: I think for most people understanding the whole ecosystem is a challenge. There are a lot of moving parts involved. You have to understand what storage is going to be available to my cluster, is that going to satisfy the users? How do I integrate that storage with my Kubernetes cluster? Do I use a CSI driver? Are there other options? How do I set up the storage classes in a way that will make sense for my end users?

We’ve focused on end-users as application developers and have made their lives easy. But if you’re the cluster administrator, I think there are still some challenges in setting up and getting everything running. There are more than 50 CSI drivers out there, but they are all pretty immature. Storage vendors are still testing the waters and trying to figure out if the CSI driver will, for example, be able to handle it if the cluster scales up. It’s all headed in the right direction but there needs to be a lot of education.

I do think most of the primitives are already there, which would allow you to run the vast majority of stateful workloads on Kubernetes successfully. There’s definitely a problem with education. I also think there’s an immense amount of complexity. How do you even begin to port an application that has traditionally run on virtual machines to Kubernetes? That is a massive endeavor and it’s daunting.

There is definitely room for improvement in the overall Kubernetes ecosystem. Low-level disk discovery for the software-defined storage system and local storage support within Kubernetes as areas that could be made easier. In general, though, as long as they are containerized the primitives exist to handle the vast majority of stateful workloads in Kubernetes.

What are the most common ways that you see people handling stateful workloads in Kubernetes?

Ali: First of all, there are a lot of aspects of storage, and we generally stay away from application-layer storage and focus on the block and file. Applications can depend on some sort of managed data service, which means you’ve offloaded this work to somebody else and it’s not your problem — you just have an SDK. But in cases where folks want to run their own databases or some sort of stateful service that will need to write to file and block storage, we see three basic patterns. They are a single writer block storage, a multiwriter file storage and some kind of really fast local storage. If you choose to manage your database yourself and deploy it on Kubernetes, then you use the primitives that we provide for block and file storage interfaces in Kubernetes.

What projects are the Kubernetes Storage SIG working on now?

Ali: The next frontier for us is making this enterprise-ready, production-ready and looking at day two, day three issues. We’re looking at data protection and how we do volume snapshots and volume backups. We’ve started a new working group focused on data protection that’s a collaboration between the Kubernetes Storage SIG and the Kubernetes Application SIG, because there’s a lot of intersection when you start talking about backing up workloads. It’s not just storage, it’s not just application.

What does the future look like for stateful workloads in Kubernetes?

Ali: I’m really excited about some of the big projects that we’re going to be releasing as part of Kubernetes 1.18. We’ve been working on adding support for Windows for a while, and the Windows CSI is going to be hitting Alpha in this next release. Raw Block Volumes is another project that will finally be going to general availability after being in the works for years. The vast majority of workloads that run on Kubernetes use storage as a file system that ends up as some sort of mounted directory inside the container. But some prefer to access the raw disk, and Raw Block Volumes makes that possible.

We’re also looking at better interfaces for CSI volume health checks. At this point, once Kubernetes has made a volume available to a workload, it’s pretty hands-off and you don’t have much information. We’re building mechanisms to allow the storage system to surface health information about the volume continuously to Kubernetes. Initially, it would just be to alert users, but eventually, we want it so that Kubernetes could programmatically respond to events that the storage system is surfacing, or end-users could write their own tools to respond to the information.

Answers have been edited and condensed for clarity.

The Cloud Native Computing Foundation is a sponsor of The New Stack.

Feature image by Jen Theodore on Unsplash.