Configure Storage Volumes for Kubeflow Notebook Servers

In the last installment of this series, we have created custom Docker images for provisioning Jupyter Notebook Servers targeting the data preparation, training, and inference stages of a machine learning project.

Before we launch the fully customized environments for data scientists and ML developers, we need to make sure that the storage is configured to facilitate collaboration among teams.

This tutorial focuses on provisioning the storage backend for Jupyter Notebook Servers running in the Kubeflow Kubernetes machine learning operations (MLops) platform.

There are two prerequisites for configuring storage for Kubeflow:

1. 1. A storage class that supports dynamic provisioning
   2. A storage backend with support for shared volumes

While the first prerequisite is met by most of the overlay storage choices available for Kubernetes, only a few support the second.

As discussed in the earlier parts, Kubeflow has a unique requirement of shared volumes and dedicated volumes to run MLOps pipelines. Shared volumes make it possible for multiple pods to read and write simultaneously. Dedicated volumes are traditional Kubernetes persistent volume claims (PVCs) mounted in a single pod.

This tutorial will show how Portworx and NFS provisioner can be used to configure the storage volumes. For detailed instructions on deploying and configuring Kubeflow storage, refer to the DeepOps guide for NFS and Portworx. You can use one of these storage providers as the preferred overlay storage backend for Notebook Servers.

Configure NFS-Based Shared Volumes for Kubeflow

When you install the NFS client provisioner through NVIDIA DeepOps, you get a storage class by name nfs-client which supports shared volumes.

👁 Image

Let’s create two shared PVCs for sharing the dataset and models.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: datasets
 namespace: kubeflow-user-example-com
spec:
 accessModes:
 - ReadWriteMany
 storageClassName: nfs-client
 resources:
 requests:
 storage: 10Gi

This PVC will be used for performing the ETL operations and data pre-processing.

The next one that we create would be used for storing the model artifacts.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: models
 namespace: kubeflow-user-example-com
spec:
 accessModes:
 - ReadWriteMany
 storageClassName: nfs-client
 resources:
 requests:
 storage: 10Gi

Configure Portworx-Based PVCs for Kubeflow

Portworx supports shared volumes through the SharedV4 storage class. Let’s start by creating the storage class.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
 name: shared-sc
provisioner: kubernetes.io/portworx-volume
parameters:
 repl: "2"
 allow_all_ips: "true"
 sharedv4: "true"

Now, we can define the PVCs based on the above storage class.

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
 name: datasets
 namespace: kubeflow-user-example-com
 annotations:
 volume.beta.kubernetes.io/storage-class: shared-sc
spec:
 accessModes:
 - ReadWriteMany
 resources:
 requests:
 storage: 10Gi

The below YAML specification creates the PVC for storing the model artifacts:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
 name: models
 namespace: kubeflow-user-example-com
 annotations:
 volume.beta.kubernetes.io/storage-class: shared-sc
spec:
 accessModes:
 - ReadWriteMany
 resources:
 requests:
 storage: 10Gi

Use the Kubeflow 1.3 Volumes UI to Configure PVCs

If the storage class is already created, we can also configure the PVCs through the new volumes UI introduced in Kubeflow 1.3.

👁 Image

Notice how the dropdown list picks up the storage classes and gives us a choice of access modes.

Apart from the shared storage volumes, we also need a dedicated volume (RWO) that will be mounted as the home directory of the Notebook Server. When the Notebook Servers are launched, the volume dashboard shows all the bound PVCs.

👁 Image

With the custom Docker container images and storage volumes in place, we are all set to launch the Notebook Servers for data preparation, training, and inference.

In the next part of this series, we will configure the first Notebook Server that performs the ETL and data pre-processing job. Stay tuned!