A Practical Guide to Kubernetes Stateful Backup and Recovery

Kubernetes is a very powerful and robust platform for orchestrating containerized applications, excelling in managing both stateful and stateless applications. Managing stateful applications can be challenging because of the need to maintain data consistency, integrity and availability.

Proper, well-documented and tested backup and recovery strategies are essential so that when there is a disaster, you can easily restore service without any data loss. There are different methods for achieving backup and restore in a Kubernetes environment, so you must ensure the strategy you use aligns with your use case.

I will walk you through the essential strategies and tools that can be adapted to perform backup and recovery in the Kubernetes environment for business continuity.

Why Is Backup and Recovery Essential for Stateful Applications?

Losing data in a stateful application can be catastrophic. Unlike stateless applications, which do not require persistent storage and can be easily scaled and replaced anytime, it’s essential to have a tested and reliable backup and recovery strategy for your stateful applications. Examples of stateful applications include:

• Databases: MySQL, MongoDB and PostgreSQL
• Memory caches: Redis, RabbitMQ
• Storage systems: Elasticsearch, Cassandra

Understanding Kubernetes Stateful Workloads

A StatefulSet in Kubernetes is the workload API object used to manage stateful applications. A StatefulSet provides the capability for the pod to maintain a sticky identity, unique network identity, persistent volumes (PVs) and persistent volume claims (PVCs). A StatefulSet makes it easier to get each pod’s identity, which in turn makes it easier to perform database backup and restore.

Backup Strategies for Kubernetes Stateful Applications

1. Volume Snapshots

Kubernetes provides a standardized way of copying a volume’s contents at a specific time without creating an entirely new volume. Volume snapshots are handy and powerful when database administrators want to quickly restore a previous state. This can also be useful when a maintenance activity needs to be performed on the Kubernetes cluster. The backup will be performed before the activity, and the administrator will perform the restore after the activity.

How to use volume snapshots: Kubernetes has built-in support for managing volume snapshots through the Container Storage Interface (CSI) Snapshot API, which integrates seamlessly with storage in cloud environments.

• In AWS, use AWS EBS.
• In Azure, use Azure Managed Disks snapshots.
• In Google Cloud Platform (GCP), use persistent disk snapshots.

Volume snapshot tools to consider:

• Velero is a popular open source tool used to perform backup, restore and migration of Kubernetes resources such as PVCs and PVs. It also performs scheduled backups and integrates with major cloud providers.
• You can also set up a Kubernetes cron job. Create a Kubernetes cron job resource that schedules regular tasks to execute rsync commands. The rsync tool will synchronize or back up data from a PV to a backup location, such as external storage, cloud storage or another PV.

2. Application-Level Backups

Stateful applications (particularly databases) require consistent routine backups. Simply copying the data may lead to data corruption, so it is better to utilize the database’s built-in tool to perform a backup.

Database backup tools to consider:

• PostgreSQL: Use pg_dump.
• MySQL: Use mysqldump.
• Use Velero for regular backups.

3. Incremental and Differential Backups

In cases where the database is very large, performing incremental and differential backups will come in handy. Incremental and differential backups will back up only data that’s changed, saving time, bandwidth and storage.

• Incremental backups: Capture changes since the last backup.
• Differential backups: Capture changes since the last full backup.

Incremental and differential backup tools to consider:

• Restic supports efficient and encrypted incremental backups.
• BorgBackup can be used to back up Kubernetes volumes on a node.

4. Offsite and Multiregion Backups

To prevent a single point of failure at the database location and protect against database location failures, store backups offsite or in multiple regions. To store backups in the cloud, you can try:

• Amazon S3: S3 is a reliable and scalable object storage service that can replicate data to other regions.
• Google Cloud Storage: GCP Cloud Storage integrates with GCP services, stores any amount of data and retrieves it as often as you like.
• Azure Blob Storage: It integrates with Microsoft Azure services and provides scalable, cost-efficient object storage in the cloud.

Recovery Strategies for Kubernetes Stateful Applications

There are so many strategies that can be adapted to perform data restoration for a stateful application.

1. Restore From Volume Snapshots

The following methods can be used to restore volume snapshots from a Kubernetes environment. You also need to validate the integrity of the volume snapshot before attempting a restore.

• Use Velero to perform volume restore.
• Use Kubernetes resources to restore VolumeSnapshot into a new PV.

2. Application-Level Restore

Use the built-in tool provided by the database to perform a database restore. You can only use these tools to restore backups created with the same tool (e.g., if mysqldump was used to perform a MySQL backup).

Database-specific restore tools:

• PostgreSQL: Use pg_dump.
• MySQL: Use mysqldump.
• Use Velero for regular backups.

3. Full Kubernetes Restore With Velero

Velero can both back up and restore Kubernetes resources. You can use Velero to restore Kubernetes resources such as StatefulSets, ConfigMaps, Kubernetes secrets, PVs and PVCs.

Once all the resources have been successfully restored, you can reattach the PV.

Recommended Best Practices for Backup and Recovery

When establishing a backup and recovery strategy, make sure it includes the following best practices:

• Perform regular, scheduled backups and retention policies.
• Organize periodic testing (backup restores) of the backups to validate their integrity and authenticity. This can be automated and generate reports for analysis.
• Monitor and send alerts for backup failures. You can use tools like Nagios or Datadog to perform backup monitoring.
• Document your recovery procedures.
• Use encryption on backups for security.

Disaster Recovery Automation Tools

The following tools can be used for automating backup and restore in a Kubernetes environment.

• Velero is an open source tool for backing up and restoring Kubernetes workloads. It also has support for cloud storage and snapshots.
• Stash is a native Kubernetes disaster recovery solution for backing up and restoring volumes and databases in Kubernetes.
• Ark is an open source tool created by Heptio for backing up and restoring Kubernetes clusters and PVs. Ark allows you to back up all or part of a resource in your Kubernetes cluster, including PVs, deployments, tags and more.

Conclusion

It is very important to perform regular disaster recovery (DR) drills to ensure business continuity in a disaster situation. You can also do regular activities such as chaos engineering, which will simulate failures and validate your infrastructure’s recovery process, on your Kubernetes cluster.

By implementing a strategy for the backup and recovery process that aligns with your use case, leveraging StatefulSets, PV snapshots and PVCs; using backup solutions such as Velero; and maintaining a backup and restore policy, you can ensure that your stateful applications remain resilient to data loss or corruption.

An architected backup and recovery strategy not only mitigates the risk associated with data loss but also enhances the overall reliability and trustworthiness of your Kubernetes-managed applications. It is a future investment in your infrastructure setup, ensuring that operations can continue running smoothly even when faced with unexpected disruptions.

Want to dive further into the possibilities of Kubernetes? Check out Andela’s guide, Make a Scalable CI/CD Pipeline for Kubernetes With GitHub and Argo CD, and continue your Kubernetes journey.