Kubernetes Secrets Management: 3 Approaches, 9 Best Practices

Secrets, such as usernames, passwords, API tokens and TLS certificates, contain confidential data that can be used to authenticate and authorize users, groups or entities. As the name implies, secrets are not meant to be known or seen by others. So how do we keep them safe?

The key to keeping secrets safe lies within the way you manage them. When migrating an application or microservice to Kubernetes, developers must make early design choices about where to store secrets, how to retrieve them and how to make them available in an application. Part of this design choice is to ensure secrets can become available without compromising the application’s security posture.

In this article, I will provide approaches and recommended best practices for managing secrets in Kubernetes.

How to Approach Secrets Management in Kubernetes

Let’s start with some approaches. Below are three approaches I recommend for Kubernetes secrets management.

etcd

etcd is a supported datastore in Kubernetes, and a lot of developers opt to store secrets in a Base64-encoded format in etcd as a key-value pair. Secrets stored in etcd can be made available from within Kubernetes deployment specs as an environment variable, which is stored in memory and makes the secrets harder to extract. Secrets can also be made available from inside the container as a volume mount, but since volume mounts store secrets on a container filesystem, it is easier to extract secrets from it, compared to environment variables. If you’re worried about flexibility and security, then you’ll be happy to hear that etcd access is controlled through Kubernetes role-based access control (RBAC).

Through strong concurrency primitives, APIs and linearizable reads, secrets can be managed at scale with etcd. However, there are drawbacks: secrets are stored, retrieved and sent in plaintext (Base64 encoding). I strongly advise against storing secrets in plaintext because this means anyone or any machine can easily read your confidential data without you knowing. This issue can be solved by configuring etcd to encrypt data in transit or at rest using TLS.

Note that access to etcd is not audited, meaning anyone who has access to it can access all secrets. Secrets stored in etcd are also not versioned nor recoverable. For those who require large-scale organizational secrets management, etcd may not be the right choice, since it is a Kubernetes datastore.

Secrets Management Service

Most public cloud providers offer secrets management services. Some of the most popular secrets management services include Google Secret Manager, Azure Key Vault and AWS Secrets Manager.

The HashiCorp Vault is another option, as it provides key management, encryption, rotation, public key infrastructure (PKI) and other essential services. As a centralized secrets manager, Vault can be used together with a cloud provider’s secrets management service. With Vault, operators can avoid handling plaintext keys, as when the Vault is initialized, initial keys can be encrypted and stored in a cloud key management service (KMS).

Container Storage Interface Driver

The container storage interface (CSI) driver is a secret store that’s been available since

version 1.13 of Kubernetes. The CSI driver combines external secret stores, such as Azure, AWS, GCP and HashiCorp’s Vault, into Kubernetes via CSI.

By authenticating your secret store service through volume attributes and mounting necessary secrets into the pod, the CSI driver approach steers clear of using the Kubernetes etcd datastore. This allows you to manage your secrets effectively and scale if needed.

Best Practices for Kubernetes Secrets Management

Avoid Secrets Sprawl

Secrets sprawl happens when application secrets are stored in different places, such as in configuration filers, git repositories and YAML. Secrets sprawl means no secrets management workflow is in place.

Mitigate the risk of secrets sprawl by implementing centralized secrets management. Set up a place where credentials can be securely stored and retrieved from a single point, and where they can be used by the entire organization through proper authorization, monitoring and logging mechanisms.

Encrypt Data in Transit and at Rest

Kubernetes does not safely store or transmit secrets by default. Hence, it is crucial to have a structure that encrypts secrets in transit with end-to-end TLS encryption, and a solution that stores secrets in an encrypted form.

Use Automated Secret Rotation

Organizations often have varying rotation schedules for different secrets to control exposure in case of attacks. Cloud secrets management services and external solutions are usually able to provide automated secrets rotation, so your secrets can be rotated as often as hourly, depending on data sensitivity.

Have an Audit Log

It’s a good idea to know your secrets’ activity. In the event of a breach, an audit log provides visibility and helps you assess the incident, including whether the compromise was intentional, the affected area and other investigative steps.

Implement Anti-Affinity Rules

Your secrets management solution should be a single process on a limited number of designated VMs or hosts. If run as a microservice on Kubernetes, this process will be running in a dedicated pod. To specify which nodes the pods should run on, you can implement an anti-affinity rule to control the distribution of pods on the nodes that are set to run a secrets management solution.

Leverage Dynamic Secrets

Dynamic secrets are ephemeral secrets generated on demand with a limited lifespan. Even if an attacker accesses a secret through means such as leaked application codes or debug logs, the secret would have been changed within a short period of time, thereby limiting exposure and protecting the application. Dynamic secrets can also make it easier to pin down the time frame a secret was discovered by an attacker and accelerate investigations.

Use a Custom Certificate Authority

A custom certificate authority (CA) can be used to implement end-to-end TLS as part of a defense-in-depth approach. Organizations can opt to sign their certificates with their own CA, meaning the organization-signed certificate must be presented in order to access a service.

Store Secrets in Container Memory

Storing secrets in memory will make it harder for attackers to find the secrets during a breach. Whenever a containerized application receives a secret, instead of storing the secret on a disk or in the volumeMount available in the host, it should be in memory.

Protect Secret Zero

Envelope encryption is often adopted by many secrets management solutions, where the data encryption keys (DEKs) are protected by a key encryption key (KEK). Here, the KEK is considered secret zero, the ultimate master key. If the KEK becomes compromised, attackers can decrypt the DEK, and consequently, the data encrypted by the DEK.

To protect secret zero, use your chosen cloud service provider’s identity and access management framework (IAM) in conjunction with key management servers (KMS). Note that both the IAM and KMS would have their own secret zero in the trust chain, which should also be considered sensitive data.

Summary

When it comes to protecting sensitive data, the most effective method is to adhere to a zero trust model, assume compromise, and have measures in place to mitigate risks.

The approaches and best practices I have recommended in this article lay a foundation for secrets management. For those interested in having a more thorough understanding, I recommend reading chapter three of “Kubernetes security and observability: A holistic approach to securing containers and cloud native applications,” an ebook I co-authored.